mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 23:03:09 +00:00
fix(studio,core): unify the audio id space, emit group elements, gate both canaries
Six of the fifteen findings from the max-effort review of the audio
groups / mute-solo / pitch-shift stack. Nothing in the stack is merged;
this sits on top of wa-24-timeline-fx.
The id space (findings 1-3). Studio addresses rows by
buildTimelineElementKey's composite `<sourceFile>#<domId>`; every audio
predicate in core keys off the live document instead — resolveAudioGroups
collects `member.id`, isAudibleUnderSolo compares `el.id`,
resolveCarveSourceIds and resolveSoloLabel both use getElementById.
Nobody checked the boundary, so:
* solo put a composite key in the set the runtime matches against
`el.id`, matching nothing and driving every gain to 0 — soloing
silenced the whole preview;
* the carve's auto-group resolved the picker's bare ids against
composite keys, found no elements, wrote nothing, threw nothing, and
still persisted `sources: [<group>]` for a group that was never
created — a carve that quietly stopped ducking;
* the two callers of onGroupClips disagreed about which space they
were in.
Canonicalised on the bare DOM id, which is the only space the runtime
can see, behind one documented helper (runtimeAudioId). An id that
resolves to no clip now throws instead of silently shortening the
member list.
The group element (finding 4). Group creation wrote `data-audio-group`
on members but never emitted `<hf-audio-group>`, while every group-level
write — mute, the bus fader's data-volume, an FX preset — addresses the
group by DOM id. Groups the product created were exactly the groups
nothing could edit. Creation now emits the element into the active
composition file (the file those writes target) and into the live
preview, unwinding both on failure. Group ids are validated before being
interpolated into markup.
The canary leaks (findings 5-6). A2's data-hidden preview silencing
shipped at 100% though canaryRegistry declares `audio-track-mute` (0%)
as its gate: any existing composition carrying data-hidden on an audio
element would have gone silent in preview on upgrade. Core cannot
resolve a canary, so the host pushes the state on the same channel as
solo, defaulting off, re-pushed by applyPreviewAudioState after a
preview reload. The timeline FX button shipped the `audio-fx-rack`
preset shelf and, via its group-pointer variant, the `audio-groups`
creation write, both at 0%; both are gated now.
Tests. Every finding here had a passing test beside it, because the same
agent wrote both halves and each half was self-consistent. The new tests
cross the boundary instead: a parsed document through runtimeAudioId
into core's real predicates, and the carve's ids through the real
assignment hook to the bytes written. Each was mutation-checked against
the pre-fix code.
Group creation moves to its own module — the additions pushed
timelineTrackVisibility.ts past the 600-line ceiling. Also swaps two raw
NUL bytes in useFxCarve.ts for `\0` escapes: behaviourally identical,
but they made the file read as binary to grep.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
1d5405e3e0
commit
5850a0c978
@@ -1349,6 +1349,10 @@ describe("initSandboxRuntimeModular", () => {
|
||||
|
||||
window.__timelines = { main: createMockTimeline(10) };
|
||||
initSandboxRuntimeModular();
|
||||
// Behind the `audio-track-mute` canary — off until the host pushes it, so a
|
||||
// composition that already carries data-hidden on an audio element keeps
|
||||
// playing in preview for anyone not enrolled.
|
||||
window.__hf?.setAudioMuteHidden?.(true);
|
||||
|
||||
const decodeSpy = vi
|
||||
.spyOn(WebAudioTransport.prototype, "decodeAudioElement")
|
||||
@@ -1362,6 +1366,39 @@ describe("initSandboxRuntimeModular", () => {
|
||||
expect(decodeSpy.mock.calls[0]?.[0]).toBe(audibleAudio);
|
||||
});
|
||||
|
||||
it("still schedules a data-hidden audio clip when the host has not opted in", () => {
|
||||
const root = document.createElement("div");
|
||||
root.setAttribute("data-composition-id", "main");
|
||||
root.setAttribute("data-root", "true");
|
||||
root.setAttribute("data-start", "0");
|
||||
root.setAttribute("data-duration", "10");
|
||||
root.setAttribute("data-width", "1920");
|
||||
root.setAttribute("data-height", "1080");
|
||||
document.body.appendChild(root);
|
||||
|
||||
const hiddenAudio = document.createElement("audio");
|
||||
hiddenAudio.setAttribute("data-start", "0");
|
||||
hiddenAudio.setAttribute("data-duration", "10");
|
||||
hiddenAudio.setAttribute("data-hidden", "");
|
||||
hiddenAudio.load = () => {};
|
||||
hiddenAudio.play = vi.fn(() => Promise.resolve());
|
||||
root.appendChild(hiddenAudio);
|
||||
|
||||
window.__timelines = { main: createMockTimeline(10) };
|
||||
initSandboxRuntimeModular();
|
||||
|
||||
const decodeSpy = vi
|
||||
.spyOn(WebAudioTransport.prototype, "decodeAudioElement")
|
||||
.mockResolvedValue(null);
|
||||
|
||||
const player = window.__player;
|
||||
player?.play();
|
||||
player?.seek(0);
|
||||
|
||||
expect(decodeSpy).toHaveBeenCalledTimes(1);
|
||||
expect(decodeSpy.mock.calls[0]?.[0]).toBe(hiddenAudio);
|
||||
});
|
||||
|
||||
it("batches a mid-playback data-hidden toggle into exactly one Web Audio reschedule", () => {
|
||||
const root = document.createElement("div");
|
||||
root.setAttribute("data-composition-id", "main");
|
||||
|
||||
@@ -192,6 +192,22 @@ export function initSandboxRuntimeModular(): void {
|
||||
soloedIds = new Set(ids);
|
||||
webAudio.setSolo(soloedIds);
|
||||
};
|
||||
// A2's preview/export parity fix — silencing `data-hidden` audio the way the
|
||||
// render already does — behind the `audio-track-mute` canary, which is what
|
||||
// that canary was declared for. Core cannot read the registry (canaries are
|
||||
// resolved from the studio's install id), so the host pushes the resolved
|
||||
// state on the same channel as solo. Default OFF = the shipped behaviour: a
|
||||
// composition carrying `data-hidden` on an audio element keeps playing in
|
||||
// preview until its author is enrolled. Non-studio hosts (CLI preview, the
|
||||
// bare player) never push, so they stay on the old behaviour too.
|
||||
let silenceHiddenAudio = false;
|
||||
window.__hf.setAudioMuteHidden = (enabled) => {
|
||||
if (silenceHiddenAudio === enabled) return;
|
||||
silenceHiddenAudio = enabled;
|
||||
// The active-clip set is built with this predicate baked in, so a flip
|
||||
// mid-session has to rebuild it — same reason a `data-hidden` toggle does.
|
||||
if (clock.isPlaying()) scheduleWebAudioForActiveClips();
|
||||
};
|
||||
// `_auto` is a Studio-internal keyframe marker (an auto-tracked endpoint the
|
||||
// parser reads back), NOT an animatable property. Register it as a no-op GSAP
|
||||
// plugin so GSAP doesn't log "Invalid property _auto" on every tween build —
|
||||
@@ -2080,6 +2096,7 @@ export function initSandboxRuntimeModular(): void {
|
||||
isWebAudioOwned: (el) => webAudio.ownsElement(el),
|
||||
isWebAudioRouted: (el) => webAudio.routesElement(el),
|
||||
isAudibleUnderSolo: (el) => isAudibleUnderSolo(soloedIds, el.id, audioGroupOf(el)),
|
||||
silenceHiddenAudio,
|
||||
onAutoplayBlocked: () => {
|
||||
if (state.mediaAutoplayBlockedPosted) return;
|
||||
state.mediaAutoplayBlockedPosted = true;
|
||||
@@ -2982,7 +2999,7 @@ export function initSandboxRuntimeModular(): void {
|
||||
let foundActive = false;
|
||||
for (const rawEl of audioEls) {
|
||||
if (!(rawEl instanceof HTMLMediaElement) || !rawEl.isConnected) continue;
|
||||
if (rawEl.closest("[data-hidden]")) continue;
|
||||
if (silenceHiddenAudio && rawEl.closest("[data-hidden]")) continue;
|
||||
const start = Number.parseFloat(rawEl.dataset.start ?? "");
|
||||
const durAttr = parseStrictFiniteTimingNumber(rawEl.dataset.duration);
|
||||
const end = durAttr != null && durAttr > 0 ? start + durAttr : Infinity;
|
||||
@@ -3090,7 +3107,7 @@ export function initSandboxRuntimeModular(): void {
|
||||
const audioEls = document.querySelectorAll("audio[data-start]");
|
||||
for (const rawEl of audioEls) {
|
||||
if (!(rawEl instanceof HTMLMediaElement) || !rawEl.isConnected) continue;
|
||||
if (rawEl.closest("[data-hidden]")) continue;
|
||||
if (silenceHiddenAudio && rawEl.closest("[data-hidden]")) continue;
|
||||
const compStart = Number.parseFloat(rawEl.dataset.start ?? "");
|
||||
if (!Number.isFinite(compStart)) continue;
|
||||
const mediaStart = readElementPlaybackStart(rawEl);
|
||||
|
||||
@@ -574,38 +574,57 @@ describe("syncRuntimeMedia", () => {
|
||||
});
|
||||
|
||||
describe("data-hidden silences preview volume", () => {
|
||||
it("zeroes effective volume for a clip under a data-hidden ancestor", () => {
|
||||
const hiddenClip = () => {
|
||||
const clip = createMockClip({ start: 0, end: 10, volume: 0.8 });
|
||||
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
|
||||
const hiddenAncestor = document.createElement("div");
|
||||
hiddenAncestor.setAttribute("data-hidden", "");
|
||||
document.body.appendChild(hiddenAncestor);
|
||||
hiddenAncestor.appendChild(clip.el);
|
||||
|
||||
return clip;
|
||||
};
|
||||
const volumeSeen = (clip: ReturnType<typeof hiddenClip>, silenceHiddenAudio?: boolean) => {
|
||||
let seen = -1;
|
||||
syncRuntimeMedia({
|
||||
clips: [clip],
|
||||
timeSeconds: 1,
|
||||
playing: true,
|
||||
playbackRate: 1,
|
||||
...(silenceHiddenAudio === undefined ? {} : { silenceHiddenAudio }),
|
||||
onElementVolume: (_el, v) => {
|
||||
seen = v;
|
||||
},
|
||||
});
|
||||
return seen;
|
||||
};
|
||||
|
||||
expect(seen).toBe(0);
|
||||
it("zeroes effective volume for a clip under a data-hidden ancestor", () => {
|
||||
expect(volumeSeen(hiddenClip(), true)).toBe(0);
|
||||
});
|
||||
|
||||
// The `audio-track-mute` canary sits at 0%: an existing composition that
|
||||
// carries data-hidden on an audio element must keep playing in preview
|
||||
// until its author is enrolled, or the upgrade silences them with no way
|
||||
// back short of a revert.
|
||||
it("leaves a hidden clip audible when the host has not opted in", () => {
|
||||
expect(volumeSeen(hiddenClip(), false)).toBe(0.8);
|
||||
});
|
||||
|
||||
it("defaults to audible when the flag is absent entirely", () => {
|
||||
expect(volumeSeen(hiddenClip())).toBe(0.8);
|
||||
});
|
||||
|
||||
it("does not touch el.muted when silencing a hidden clip (RULES trap: transport owns el.muted)", () => {
|
||||
const clip = createMockClip({ start: 0, end: 10, volume: 0.8 });
|
||||
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
|
||||
const hiddenAncestor = document.createElement("div");
|
||||
hiddenAncestor.setAttribute("data-hidden", "");
|
||||
document.body.appendChild(hiddenAncestor);
|
||||
hiddenAncestor.appendChild(clip.el);
|
||||
const clip = hiddenClip();
|
||||
clip.el.muted = false;
|
||||
|
||||
syncRuntimeMedia({ clips: [clip], timeSeconds: 1, playing: true, playbackRate: 1 });
|
||||
syncRuntimeMedia({
|
||||
clips: [clip],
|
||||
timeSeconds: 1,
|
||||
playing: true,
|
||||
playbackRate: 1,
|
||||
silenceHiddenAudio: true,
|
||||
});
|
||||
|
||||
expect(clip.el.muted).toBe(false);
|
||||
});
|
||||
|
||||
@@ -222,6 +222,11 @@ export function syncRuntimeMedia(params: {
|
||||
* solo gain instead — see `WebAudioTransport.setSolo`). Absent when solo
|
||||
* isn't wired up at all, which reads as "always audible". */
|
||||
isAudibleUnderSolo?: (el: HTMLMediaElement) => boolean;
|
||||
/** Silence media under a `data-hidden` ancestor, matching the render. Opt-in:
|
||||
* the host pushes it via `__hf.setAudioMuteHidden` when the `audio-track-mute`
|
||||
* canary is on. Absent/false = the shipped behaviour (hidden audio still
|
||||
* plays in preview). */
|
||||
silenceHiddenAudio?: boolean;
|
||||
forceSync?: boolean;
|
||||
}): void {
|
||||
const forceMuteAll = !!(params.outputMuted || params.userMuted);
|
||||
@@ -321,13 +326,19 @@ export function syncRuntimeMedia(params: {
|
||||
}
|
||||
|
||||
// A data-hidden ancestor is silent in the export (audioMixer.ts drops
|
||||
// it); preview must match. Folded into the per-tick volume, not
|
||||
// el.muted (RULES trap: el.muted is the transport's ownership flag).
|
||||
// Solo rides the same fold for the same reason — never el.muted, and
|
||||
// never touching any attribute (it is session-only, unlike hidden).
|
||||
// it); preview matches once the host opts in (`silenceHiddenAudio`, the
|
||||
// `audio-track-mute` canary — see init.ts). Folded into the per-tick
|
||||
// volume, not el.muted (RULES trap: el.muted is the transport's ownership
|
||||
// flag). Solo rides the same fold for the same reason — never el.muted,
|
||||
// and never touching any attribute (it is session-only, unlike hidden) —
|
||||
// but is NOT gated: it is a session control with no shipped behaviour to
|
||||
// preserve.
|
||||
const silencedByHidden = params.silenceHiddenAudio
|
||||
? el.closest("[data-hidden]") !== null
|
||||
: false;
|
||||
const silencedBySolo = params.isAudibleUnderSolo ? !params.isAudibleUnderSolo(el) : false;
|
||||
const effectiveVolume =
|
||||
el.closest("[data-hidden]") || silencedBySolo ? 0 : clampVolume(authorVolume * userVol);
|
||||
silencedByHidden || silencedBySolo ? 0 : clampVolume(authorVolume * userVol);
|
||||
el.volume = effectiveVolume;
|
||||
lastRuntimeAppliedVolume.set(el, effectiveVolume);
|
||||
params.onElementVolume?.(el, effectiveVolume, authorVolume);
|
||||
|
||||
+6
@@ -43,6 +43,12 @@ declare global {
|
||||
* read from or written to any document attribute.
|
||||
*/
|
||||
setAudioSolo?: (ids: readonly string[]) => void;
|
||||
/**
|
||||
* Studio's `audio-track-mute` canary state: silence audio under a
|
||||
* `data-hidden` ancestor in preview, the way the render already does.
|
||||
* Off until pushed — core cannot resolve a canary itself.
|
||||
*/
|
||||
setAudioMuteHidden?: (enabled: boolean) => void;
|
||||
};
|
||||
__playerReady?: boolean;
|
||||
__renderReady?: boolean;
|
||||
|
||||
Reference in New Issue
Block a user