mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
fix(player): single-owner audio to prevent double voice in preview (#298)
## Summary Fixes the double-voice issue in studio preview where narration plays twice with a drifting offset (measured 23ms → 80ms over a 28s clip). ## Root cause Two audio pipelines were playing the same source in parallel: 1. The iframe runtime played `<audio data-start>` elements via `syncRuntimeMedia` — the intended path. 2. `<hyperframes-player>` also created parent-frame `<audio>` copies on iframe load and auto-played them in response to every runtime `state` message. The existing `_muteIframeMedia` tried to silence the iframe copies via `el.volume = 0`, but `syncRuntimeMedia` re-asserts `el.volume` from `data-volume` every tick, so the mute never held. Studio seeks went through `__player.seek()`, which only updated the iframe timeline; parent copies kept their stale `currentTime` and drift compounded across seeks. Confirmed via agent-browser instrumentation on `factory-series-c-video`: - 6 `volumechange` events per play cycle (mute-fight signature) - Both copies audible at `volume=1`, offset growing 23ms → 80ms - Every seek widened the drift further PR #295 (v0.4.2) actually **made it audible** — before that, parent copies 404'd on the wrong URL and played silently. Fixing the URL exposed the latent double-playback. ## Fix Explicit single-owner audio ownership between `<hyperframes-player>` and the runtime. - **Default ownership is `runtime`**: iframe drives audible playback; parent proxies stay paused and inert. Matches every desktop / studio code path. No parent `play()`, no `volumechange` thrash. - **On `NotAllowedError`** from the runtime's `play()` attempt (autoplay-gated iframes), the runtime posts `media-autoplay-blocked` once. The player promotes to `parent` ownership: sends `set-media-output-muted: true` to the runtime, starts parent proxies, mirrors `currentTime` from state messages with a 150ms correction threshold. Two orthogonal mute channels replace the volume fight: | Channel | Purpose | |---|---| | `set-muted` | User's mute preference (existing, unchanged) | | `set-media-output-muted` | Internal ownership handoff (new) | `syncRuntimeMedia` now accepts `outputMuted` and asserts `el.muted = true` per active tick — sticky against sub-composition media that arrives mid-playback. Uses native `muted` (orthogonal to `volume`) so no other code path can clobber it. ## Why this shape - **Single owner, explicit transition.** No races, no tug-of-war. - **Probes reality, not device class.** We flip on an actual `NotAllowedError`, not on `matchMedia('(pointer: coarse)')` or user-agent sniffing. - **Uses `muted` instead of abusing `volume`.** `muted` is orthogonal to `volume`; `syncRuntimeMedia` doesn't write to it; author / user settings stay intact. - **Parent proxies become a thin mirror.** Under parent ownership, their `currentTime` is slaved to the iframe timeline via state messages — no independent drift. - **Backwards compatible.** Old runtimes without the new bridge action ignore the message; old players without the new message just get the previous behavior. - **Capture engine unaffected** — it bypasses both DOM pipelines and muxes audio from source files. ## Files changed - `packages/core/src/runtime/types.ts` — `set-media-output-muted` action + `media-autoplay-blocked` outbound message types. - `packages/core/src/runtime/state.ts` — `mediaOutputMuted` + `mediaAutoplayBlockedPosted` fields. - `packages/core/src/runtime/bridge.ts` — route new action to `onSetMediaOutputMuted`. - `packages/core/src/runtime/media.ts` — `outputMuted` param asserts `el.muted = true` per tick; `NotAllowedError` detection fires `onAutoplayBlocked`. - `packages/core/src/runtime/init.ts` — wire new bridge handler; coordinate with `set-muted`; post `media-autoplay-blocked` once per session. - `packages/player/src/hyperframes-player.ts` — `_audioOwner` state; delete `_muteIframeMedia`; `_promoteToParentProxy`; mirror parent `currentTime`; gate all parent play/pause/seek on ownership. ## Verified end-to-end with agent-browser on `factory-series-c-video` **Runtime ownership (default — desktop studio):** | | Before | After | |---|---|---| | `PARENT.play()` calls per play cycle | 1 | **0** | | iframe `volumechange` events | 6 | **0** | | Audible streams | 2 (drifting) | **1 (iframe)** | **Parent ownership (simulated autoplay block — direct message):** | | Value | |---|---| | iframe audio | `muted=true`, `volume=1` (untouched) | | parent audio | `muted=false`, `volume=1`, audible | | Parent ↔ iframe `currentTime` offset | ~6 ms steady state | | Offset > 150 ms | corrected by mirror sync | **Mobile path simulated with iPhone 14 emulation + injected `NotAllowedError` from iframe `<audio>.play()`:** Event timeline captured via agent-browser instrumentation: ``` t=0.0 ms IFRAME.play() called ← runtime attempts playback t=0.4 ms IFRAME.play() REJECTED: NotAllowedError ← simulated mobile gate t=0.4 ms →IFRAME bridge set-media-output-muted=true ← player promotes t=0.6 ms PARENT.play() called ← parent proxy starts t=0.8 ms ←IFRAME msg media-autoplay-blocked ← runtime signal t=1.0 ms PARENT.play() resolved ← audible t=1.3 ms IFRAME muted=true, volume=1 ← iframe silenced via native muted ``` Steady state at t=4 s under promoted parent ownership: | Element | currentTime | paused | volume | muted | |---|---|---|---|---| | Parent audio | 4.060 s | false | 1.0 | **false** (audible) | | Iframe audio | 4.068 s | false | 1.0 | **true** (silent) | **Offset: 8 ms**, single audible stream, orthogonal mute channel respected. ## Test plan - [x] `bunx vitest run` under `packages/core` — **467 / 467 pass** (incl. 4 new `media.test.ts` + 2 new `bridge.test.ts`) - [x] `bunx vitest run` under `packages/player` — **23 / 23 pass** (3 rewrites for new contract, 2 new for promotion flow) - [x] `bun run build` — all packages green - [x] Fresh preview + browser repro on `factory-series-c-video`: - [x] Runtime ownership: single audio stream, no drift - [x] Parent ownership promotion via direct `media-autoplay-blocked` message: iframe muted, parent audible - [x] iPhone 14 emulation + injected `NotAllowedError`: full promotion chain verified in ~1 s, 8 ms steady-state offset - [x] No `volumechange` thrash in either ownership mode - [x] One round of QA on a physical iOS / Android device before release — exercises real `NotAllowedError` path (expected behavior identical to simulation above)
This commit is contained in:
Miguel Ángel
@@ -7,6 +7,7 @@ function createMockDeps() {
|
||||
onPause: vi.fn(),
|
||||
onSeek: vi.fn(),
|
||||
onSetMuted: vi.fn(),
|
||||
onSetMediaOutputMuted: vi.fn(),
|
||||
onSetPlaybackRate: vi.fn(),
|
||||
onEnablePickMode: vi.fn(),
|
||||
onDisablePickMode: vi.fn(),
|
||||
@@ -55,6 +56,22 @@ describe("installRuntimeControlBridge", () => {
|
||||
expect(deps.onSetMuted).toHaveBeenCalledWith(true);
|
||||
});
|
||||
|
||||
it("dispatches set-media-output-muted command", () => {
|
||||
const deps = createMockDeps();
|
||||
const handler = installRuntimeControlBridge(deps);
|
||||
handler(makeControlMessage("set-media-output-muted", { muted: true }));
|
||||
expect(deps.onSetMediaOutputMuted).toHaveBeenCalledWith(true);
|
||||
handler(makeControlMessage("set-media-output-muted", { muted: false }));
|
||||
expect(deps.onSetMediaOutputMuted).toHaveBeenCalledWith(false);
|
||||
});
|
||||
|
||||
it("set-media-output-muted coerces absent flag to false", () => {
|
||||
const deps = createMockDeps();
|
||||
const handler = installRuntimeControlBridge(deps);
|
||||
handler(makeControlMessage("set-media-output-muted"));
|
||||
expect(deps.onSetMediaOutputMuted).toHaveBeenCalledWith(false);
|
||||
});
|
||||
|
||||
it("dispatches set-playback-rate command", () => {
|
||||
const deps = createMockDeps();
|
||||
const handler = installRuntimeControlBridge(deps);
|
||||
|
||||
@@ -5,6 +5,7 @@ type BridgeDeps = {
|
||||
onPause: () => void;
|
||||
onSeek: (frame: number, seekMode: "drag" | "commit") => void;
|
||||
onSetMuted: (muted: boolean) => void;
|
||||
onSetMediaOutputMuted: (muted: boolean) => void;
|
||||
onSetPlaybackRate: (rate: number) => void;
|
||||
onEnablePickMode: () => void;
|
||||
onDisablePickMode: () => void;
|
||||
@@ -39,6 +40,10 @@ export function installRuntimeControlBridge(deps: BridgeDeps): (event: MessageEv
|
||||
deps.onSetMuted(Boolean(data.muted));
|
||||
return;
|
||||
}
|
||||
if (action === "set-media-output-muted") {
|
||||
deps.onSetMediaOutputMuted(Boolean(data.muted));
|
||||
return;
|
||||
}
|
||||
if (action === "set-playback-rate") {
|
||||
deps.onSetPlaybackRate(Number(data.playbackRate ?? 1));
|
||||
return;
|
||||
|
||||
@@ -1189,6 +1189,12 @@ export function initSandboxRuntimeModular(): void {
|
||||
timeSeconds: state.currentTime,
|
||||
playing: state.isPlaying,
|
||||
playbackRate: state.playbackRate,
|
||||
outputMuted: state.mediaOutputMuted,
|
||||
onAutoplayBlocked: () => {
|
||||
if (state.mediaAutoplayBlockedPosted) return;
|
||||
state.mediaAutoplayBlockedPosted = true;
|
||||
postRuntimeMessage({ source: "hf-preview", type: "media-autoplay-blocked" });
|
||||
},
|
||||
});
|
||||
const rootCompId =
|
||||
document.querySelector("[data-composition-id]")?.getAttribute("data-composition-id") ?? null;
|
||||
@@ -1424,10 +1430,20 @@ export function initSandboxRuntimeModular(): void {
|
||||
},
|
||||
onSetMuted: (muted) => {
|
||||
state.bridgeMuted = muted;
|
||||
const effective = muted || state.mediaOutputMuted;
|
||||
const mediaEls = document.querySelectorAll("video, audio");
|
||||
for (const el of mediaEls) {
|
||||
if (!(el instanceof HTMLMediaElement)) continue;
|
||||
el.muted = muted;
|
||||
el.muted = effective;
|
||||
}
|
||||
},
|
||||
onSetMediaOutputMuted: (muted) => {
|
||||
state.mediaOutputMuted = muted;
|
||||
const effective = muted || state.bridgeMuted;
|
||||
const mediaEls = document.querySelectorAll("video, audio");
|
||||
for (const el of mediaEls) {
|
||||
if (!(el instanceof HTMLMediaElement)) continue;
|
||||
el.muted = effective;
|
||||
}
|
||||
},
|
||||
onSetPlaybackRate: (rate) => applyPlaybackRate(rate),
|
||||
|
||||
@@ -381,4 +381,81 @@ describe("syncRuntimeMedia", () => {
|
||||
syncRuntimeMedia({ clips: [clip], timeSeconds: 7, playing: false, playbackRate: 1 });
|
||||
expect(clip.el.currentTime).toBe(7);
|
||||
});
|
||||
|
||||
it("asserts muted=true every tick while outputMuted is set", () => {
|
||||
// Parent ownership has taken over audible playback via parent-frame
|
||||
// proxies. The iframe runtime must silence every active media element
|
||||
// per tick so new sub-composition media inherits the mute as soon as
|
||||
// it appears in the DOM — otherwise a late <audio> insertion would
|
||||
// briefly play audibly and double-voice the viewer.
|
||||
const clip = createMockClip({ start: 0, end: 10, volume: 1 });
|
||||
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
|
||||
Object.defineProperty(clip.el, "muted", { value: false, writable: true });
|
||||
syncRuntimeMedia({
|
||||
clips: [clip],
|
||||
timeSeconds: 5,
|
||||
playing: true,
|
||||
playbackRate: 1,
|
||||
outputMuted: true,
|
||||
});
|
||||
expect(clip.el.muted).toBe(true);
|
||||
// A second tick re-asserts — captures the sticky behavior, since
|
||||
// the bridge handler only runs on flip transitions.
|
||||
Object.defineProperty(clip.el, "muted", { value: false, writable: true });
|
||||
syncRuntimeMedia({
|
||||
clips: [clip],
|
||||
timeSeconds: 5.02,
|
||||
playing: true,
|
||||
playbackRate: 1,
|
||||
outputMuted: true,
|
||||
});
|
||||
expect(clip.el.muted).toBe(true);
|
||||
});
|
||||
|
||||
it("does not touch muted when outputMuted is absent", () => {
|
||||
// The un-mute decision belongs to author intent (`<audio muted>`) and
|
||||
// user preference (`onSetMuted`) — syncRuntimeMedia must not race them.
|
||||
const clip = createMockClip({ start: 0, end: 10 });
|
||||
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
|
||||
Object.defineProperty(clip.el, "muted", { value: true, writable: true });
|
||||
syncRuntimeMedia({ clips: [clip], timeSeconds: 5, playing: true, playbackRate: 1 });
|
||||
expect(clip.el.muted).toBe(true);
|
||||
});
|
||||
|
||||
it("fires onAutoplayBlocked when play() rejects with NotAllowedError", async () => {
|
||||
const clip = createMockClip({ start: 0, end: 10 });
|
||||
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
|
||||
const rejection = Object.assign(new Error("blocked"), { name: "NotAllowedError" });
|
||||
clip.el.play = vi.fn(() => Promise.reject(rejection));
|
||||
const onAutoplayBlocked = vi.fn();
|
||||
syncRuntimeMedia({
|
||||
clips: [clip],
|
||||
timeSeconds: 5,
|
||||
playing: true,
|
||||
playbackRate: 1,
|
||||
onAutoplayBlocked,
|
||||
});
|
||||
// The rejection is delivered on a microtask — flush it.
|
||||
await Promise.resolve();
|
||||
await Promise.resolve();
|
||||
expect(onAutoplayBlocked).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("does not fire onAutoplayBlocked for non-autoplay rejections", async () => {
|
||||
const clip = createMockClip({ start: 0, end: 10 });
|
||||
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
|
||||
const rejection = Object.assign(new Error("aborted"), { name: "AbortError" });
|
||||
clip.el.play = vi.fn(() => Promise.reject(rejection));
|
||||
const onAutoplayBlocked = vi.fn();
|
||||
syncRuntimeMedia({
|
||||
clips: [clip],
|
||||
timeSeconds: 5,
|
||||
playing: true,
|
||||
playbackRate: 1,
|
||||
onAutoplayBlocked,
|
||||
});
|
||||
await Promise.resolve();
|
||||
await Promise.resolve();
|
||||
expect(onAutoplayBlocked).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -92,6 +92,20 @@ export function syncRuntimeMedia(params: {
|
||||
timeSeconds: number;
|
||||
playing: boolean;
|
||||
playbackRate: number;
|
||||
/**
|
||||
* When `true`, assert `el.muted = true` on every active media element on
|
||||
* every tick. Sticky against newly-discovered media (sub-composition
|
||||
* activation, dynamic DOM) so the parent-frame audio-owner invariant holds.
|
||||
* `false` is a no-op — we don't un-mute, because other code paths
|
||||
* (`<audio muted>` author intent, `onSetMuted`) own the un-mute decision.
|
||||
*/
|
||||
outputMuted?: boolean;
|
||||
/**
|
||||
* Invoked at most once when a media element's `play()` promise rejects with
|
||||
* `NotAllowedError`. The caller is expected to latch and post a single
|
||||
* outbound message; further invocations are suppressed by the caller.
|
||||
*/
|
||||
onAutoplayBlocked?: () => void;
|
||||
}): void {
|
||||
for (const clip of params.clips) {
|
||||
const { el } = clip;
|
||||
@@ -108,6 +122,7 @@ export function syncRuntimeMedia(params: {
|
||||
}
|
||||
}
|
||||
if (clip.volume != null) el.volume = clip.volume;
|
||||
if (params.outputMuted) el.muted = true;
|
||||
try {
|
||||
// Per-element rate × global transport rate
|
||||
el.playbackRate = clip.playbackRate * params.playbackRate;
|
||||
@@ -167,11 +182,20 @@ export function syncRuntimeMedia(params: {
|
||||
// inserted after the runtime bound its listeners.
|
||||
if (el.preload !== "auto") el.preload = "auto";
|
||||
markPlayRequested(el);
|
||||
void el.play().catch(() => {
|
||||
void el.play().catch((err: unknown) => {
|
||||
// If play() rejects — e.g. autoplay blocked, element removed
|
||||
// mid-flight — drop the in-flight flag so a future sync tick can
|
||||
// retry rather than getting stuck waiting for `playing`/`pause`.
|
||||
playRequested.delete(el);
|
||||
// `NotAllowedError` is the autoplay-gating browser response when
|
||||
// the iframe has no user activation. Signal the parent exactly
|
||||
// once so it can promote to parent-frame audio proxies. Retries
|
||||
// here would be pointless — nothing the runtime does fixes it.
|
||||
const name =
|
||||
err && typeof err === "object" && "name" in err
|
||||
? String((err as { name?: unknown }).name ?? "")
|
||||
: "";
|
||||
if (name === "NotAllowedError") params.onAutoplayBlocked?.();
|
||||
});
|
||||
} else if (!params.playing && !el.paused) {
|
||||
el.pause();
|
||||
|
||||
@@ -10,6 +10,21 @@ export type RuntimeState = {
|
||||
parityModeEnabled: boolean;
|
||||
canonicalFps: number;
|
||||
bridgeMuted: boolean;
|
||||
/**
|
||||
* Internal mute of audible media output, owned by the audio-ownership
|
||||
* protocol between the parent (`<hyperframes-player>`) and this runtime.
|
||||
* Independent of `bridgeMuted` (the user's mute preference). When the
|
||||
* parent takes over audible playback via parent-frame proxies, it sets
|
||||
* this to `true` so the runtime keeps driving timed media for frame
|
||||
* accuracy but produces no audio of its own.
|
||||
*/
|
||||
mediaOutputMuted: boolean;
|
||||
/**
|
||||
* Latch so the `media-autoplay-blocked` outbound message is posted at most
|
||||
* once per runtime session. The parent only needs the first signal — it
|
||||
* takes over playback and further rejections are the same problem.
|
||||
*/
|
||||
mediaAutoplayBlockedPosted: boolean;
|
||||
playbackRate: number;
|
||||
bridgeLastPostedFrame: number;
|
||||
bridgeLastPostedAt: number;
|
||||
@@ -42,6 +57,8 @@ export function createRuntimeState(): RuntimeState {
|
||||
parityModeEnabled: true,
|
||||
canonicalFps: 30,
|
||||
bridgeMuted: false,
|
||||
mediaOutputMuted: false,
|
||||
mediaAutoplayBlockedPosted: false,
|
||||
playbackRate: 1,
|
||||
bridgeLastPostedFrame: -1,
|
||||
bridgeLastPostedAt: 0,
|
||||
|
||||
@@ -11,6 +11,7 @@ export type RuntimeBridgeControlAction =
|
||||
| "pause"
|
||||
| "seek"
|
||||
| "set-muted"
|
||||
| "set-media-output-muted"
|
||||
| "set-playback-rate"
|
||||
| "enable-pick-mode"
|
||||
| "disable-pick-mode"
|
||||
@@ -137,6 +138,18 @@ export type RuntimeStageSizeMessage = {
|
||||
height: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Fired once per session when the runtime's attempt to play a timed media
|
||||
* element is rejected with `NotAllowedError`. The parent (web component / host
|
||||
* app) uses this as the signal to promote to parent-frame audio proxies —
|
||||
* iframes lose autoplay privileges when the user gesture originated in the
|
||||
* parent frame, so the host has to take over audible playback there.
|
||||
*/
|
||||
export type RuntimeMediaAutoplayBlockedMessage = {
|
||||
source: "hf-preview";
|
||||
type: "media-autoplay-blocked";
|
||||
};
|
||||
|
||||
/**
|
||||
* Analytics events emitted by the runtime.
|
||||
*
|
||||
@@ -167,6 +180,7 @@ export type RuntimeOutboundMessage =
|
||||
| RuntimePickerPickedManyMessage
|
||||
| RuntimePickerCancelledMessage
|
||||
| RuntimeStageSizeMessage
|
||||
| RuntimeMediaAutoplayBlockedMessage
|
||||
| RuntimeAnalyticsMessage;
|
||||
|
||||
export type RuntimePlayer = {
|
||||
|
||||
Reference in New Issue
Block a user