mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
fix(player): single-owner audio to prevent double voice in preview (#298)
## Summary Fixes the double-voice issue in studio preview where narration plays twice with a drifting offset (measured 23ms → 80ms over a 28s clip). ## Root cause Two audio pipelines were playing the same source in parallel: 1. The iframe runtime played `<audio data-start>` elements via `syncRuntimeMedia` — the intended path. 2. `<hyperframes-player>` also created parent-frame `<audio>` copies on iframe load and auto-played them in response to every runtime `state` message. The existing `_muteIframeMedia` tried to silence the iframe copies via `el.volume = 0`, but `syncRuntimeMedia` re-asserts `el.volume` from `data-volume` every tick, so the mute never held. Studio seeks went through `__player.seek()`, which only updated the iframe timeline; parent copies kept their stale `currentTime` and drift compounded across seeks. Confirmed via agent-browser instrumentation on `factory-series-c-video`: - 6 `volumechange` events per play cycle (mute-fight signature) - Both copies audible at `volume=1`, offset growing 23ms → 80ms - Every seek widened the drift further PR #295 (v0.4.2) actually **made it audible** — before that, parent copies 404'd on the wrong URL and played silently. Fixing the URL exposed the latent double-playback. ## Fix Explicit single-owner audio ownership between `<hyperframes-player>` and the runtime. - **Default ownership is `runtime`**: iframe drives audible playback; parent proxies stay paused and inert. Matches every desktop / studio code path. No parent `play()`, no `volumechange` thrash. - **On `NotAllowedError`** from the runtime's `play()` attempt (autoplay-gated iframes), the runtime posts `media-autoplay-blocked` once. The player promotes to `parent` ownership: sends `set-media-output-muted: true` to the runtime, starts parent proxies, mirrors `currentTime` from state messages with a 150ms correction threshold. Two orthogonal mute channels replace the volume fight: | Channel | Purpose | |---|---| | `set-muted` | User's mute preference (existing, unchanged) | | `set-media-output-muted` | Internal ownership handoff (new) | `syncRuntimeMedia` now accepts `outputMuted` and asserts `el.muted = true` per active tick — sticky against sub-composition media that arrives mid-playback. Uses native `muted` (orthogonal to `volume`) so no other code path can clobber it. ## Why this shape - **Single owner, explicit transition.** No races, no tug-of-war. - **Probes reality, not device class.** We flip on an actual `NotAllowedError`, not on `matchMedia('(pointer: coarse)')` or user-agent sniffing. - **Uses `muted` instead of abusing `volume`.** `muted` is orthogonal to `volume`; `syncRuntimeMedia` doesn't write to it; author / user settings stay intact. - **Parent proxies become a thin mirror.** Under parent ownership, their `currentTime` is slaved to the iframe timeline via state messages — no independent drift. - **Backwards compatible.** Old runtimes without the new bridge action ignore the message; old players without the new message just get the previous behavior. - **Capture engine unaffected** — it bypasses both DOM pipelines and muxes audio from source files. ## Files changed - `packages/core/src/runtime/types.ts` — `set-media-output-muted` action + `media-autoplay-blocked` outbound message types. - `packages/core/src/runtime/state.ts` — `mediaOutputMuted` + `mediaAutoplayBlockedPosted` fields. - `packages/core/src/runtime/bridge.ts` — route new action to `onSetMediaOutputMuted`. - `packages/core/src/runtime/media.ts` — `outputMuted` param asserts `el.muted = true` per tick; `NotAllowedError` detection fires `onAutoplayBlocked`. - `packages/core/src/runtime/init.ts` — wire new bridge handler; coordinate with `set-muted`; post `media-autoplay-blocked` once per session. - `packages/player/src/hyperframes-player.ts` — `_audioOwner` state; delete `_muteIframeMedia`; `_promoteToParentProxy`; mirror parent `currentTime`; gate all parent play/pause/seek on ownership. ## Verified end-to-end with agent-browser on `factory-series-c-video` **Runtime ownership (default — desktop studio):** | | Before | After | |---|---|---| | `PARENT.play()` calls per play cycle | 1 | **0** | | iframe `volumechange` events | 6 | **0** | | Audible streams | 2 (drifting) | **1 (iframe)** | **Parent ownership (simulated autoplay block — direct message):** | | Value | |---|---| | iframe audio | `muted=true`, `volume=1` (untouched) | | parent audio | `muted=false`, `volume=1`, audible | | Parent ↔ iframe `currentTime` offset | ~6 ms steady state | | Offset > 150 ms | corrected by mirror sync | **Mobile path simulated with iPhone 14 emulation + injected `NotAllowedError` from iframe `<audio>.play()`:** Event timeline captured via agent-browser instrumentation: ``` t=0.0 ms IFRAME.play() called ← runtime attempts playback t=0.4 ms IFRAME.play() REJECTED: NotAllowedError ← simulated mobile gate t=0.4 ms →IFRAME bridge set-media-output-muted=true ← player promotes t=0.6 ms PARENT.play() called ← parent proxy starts t=0.8 ms ←IFRAME msg media-autoplay-blocked ← runtime signal t=1.0 ms PARENT.play() resolved ← audible t=1.3 ms IFRAME muted=true, volume=1 ← iframe silenced via native muted ``` Steady state at t=4 s under promoted parent ownership: | Element | currentTime | paused | volume | muted | |---|---|---|---|---| | Parent audio | 4.060 s | false | 1.0 | **false** (audible) | | Iframe audio | 4.068 s | false | 1.0 | **true** (silent) | **Offset: 8 ms**, single audible stream, orthogonal mute channel respected. ## Test plan - [x] `bunx vitest run` under `packages/core` — **467 / 467 pass** (incl. 4 new `media.test.ts` + 2 new `bridge.test.ts`) - [x] `bunx vitest run` under `packages/player` — **23 / 23 pass** (3 rewrites for new contract, 2 new for promotion flow) - [x] `bun run build` — all packages green - [x] Fresh preview + browser repro on `factory-series-c-video`: - [x] Runtime ownership: single audio stream, no drift - [x] Parent ownership promotion via direct `media-autoplay-blocked` message: iframe muted, parent audible - [x] iPhone 14 emulation + injected `NotAllowedError`: full promotion chain verified in ~1 s, 8 ms steady-state offset - [x] No `volumechange` thrash in either ownership mode - [x] One round of QA on a physical iOS / Android device before release — exercises real `NotAllowedError` path (expected behavior identical to simulation above)
This commit is contained in:
Miguel Ángel
@@ -57,17 +57,22 @@ describe("formatTime", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ── Parent-frame media for mobile playback ──
|
||||
// ── Parent-frame audio proxies (ownership-based) ──
|
||||
//
|
||||
// Mobile browsers block media.play() inside iframes when the user gesture
|
||||
// happened in the parent. The player works around this by extracting media
|
||||
// from the iframe and playing it in the parent frame.
|
||||
// Parent-frame audio/video copies are preloaded mirror proxies of the iframe's
|
||||
// timed media. They exist as a fallback for environments that block iframe
|
||||
// `.play()`. Under the default `runtime` audio ownership, the iframe drives
|
||||
// audible playback and the proxies stay paused. Ownership flips to `parent`
|
||||
// only when the runtime posts `media-autoplay-blocked` — then the proxies
|
||||
// become the audible source and the iframe is silenced via bridge.
|
||||
|
||||
describe("HyperframesPlayer parent-frame media", () => {
|
||||
type PlayerElement = HTMLElement & {
|
||||
play: () => void;
|
||||
pause: () => void;
|
||||
seek: (t: number) => void;
|
||||
_audioOwner?: "runtime" | "parent";
|
||||
_promoteToParentProxy?: () => void;
|
||||
};
|
||||
|
||||
let player: PlayerElement;
|
||||
@@ -143,28 +148,67 @@ describe("HyperframesPlayer parent-frame media", () => {
|
||||
expect(mockAudio.playbackRate).toBe(1.5);
|
||||
});
|
||||
|
||||
it("play() calls parentMedia.play()", () => {
|
||||
it("play() does NOT start parent-proxy under runtime ownership", () => {
|
||||
// Default ownership is `runtime` — the iframe drives audible playback.
|
||||
// If we also started parent proxies here, both would play and the user
|
||||
// would hear doubled, slightly-offset audio (the original bug).
|
||||
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
||||
document.body.appendChild(player);
|
||||
|
||||
player.play();
|
||||
expect(mockAudio.play).toHaveBeenCalled();
|
||||
expect(mockAudio.play).not.toHaveBeenCalled();
|
||||
expect(player._audioOwner).toBe("runtime");
|
||||
});
|
||||
|
||||
it("pause() calls parentMedia.pause()", () => {
|
||||
it("pause() does NOT touch parent-proxy under runtime ownership", () => {
|
||||
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
||||
document.body.appendChild(player);
|
||||
|
||||
player.pause();
|
||||
expect(mockAudio.pause).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("seek() does NOT update parent currentTime under runtime ownership", () => {
|
||||
// Under runtime ownership the iframe is authoritative for time; touching
|
||||
// the proxy's currentTime would just trigger a re-buffer for no gain.
|
||||
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
||||
document.body.appendChild(player);
|
||||
|
||||
player.seek(12.5);
|
||||
expect(mockAudio.currentTime).toBe(0);
|
||||
});
|
||||
|
||||
it("after promotion to parent ownership: play/pause/seek drive parent proxy", () => {
|
||||
// Simulates the runtime having posted `media-autoplay-blocked`. Post
|
||||
// promotion: the web component owns audible output and fully drives
|
||||
// the parent proxy.
|
||||
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
||||
document.body.appendChild(player);
|
||||
|
||||
player._promoteToParentProxy?.();
|
||||
expect(player._audioOwner).toBe("parent");
|
||||
|
||||
player.play();
|
||||
expect(mockAudio.play).toHaveBeenCalled();
|
||||
|
||||
player.seek(12.5);
|
||||
expect(mockAudio.currentTime).toBe(12.5);
|
||||
|
||||
player.pause();
|
||||
expect(mockAudio.pause).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("seek() sets parentMedia.currentTime", () => {
|
||||
it("promotion is idempotent", () => {
|
||||
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
||||
document.body.appendChild(player);
|
||||
|
||||
player.seek(12.5);
|
||||
expect(mockAudio.currentTime).toBe(12.5);
|
||||
player._promoteToParentProxy?.();
|
||||
player._promoteToParentProxy?.();
|
||||
player._promoteToParentProxy?.();
|
||||
// Only one play() attempt is triggered by promotion itself (gated on
|
||||
// `!this._paused`, which is true by default so it doesn't trigger at all).
|
||||
// The test's meaning is: ownership stays `parent`, no thrash, no errors.
|
||||
expect(player._audioOwner).toBe("parent");
|
||||
});
|
||||
|
||||
it("cleans up parent media on disconnect", () => {
|
||||
|
||||
@@ -27,13 +27,18 @@ class HyperframesPlayer extends HTMLElement {
|
||||
private _lastUpdateMs = 0;
|
||||
|
||||
/**
|
||||
* Parent-frame media elements for mobile playback.
|
||||
* Parent-frame audio/video proxies, preloaded mirror copies of the iframe's
|
||||
* timed media. They exist as a fallback for environments that block iframe
|
||||
* `.play()` — mobile browsers require the user gesture to originate in the
|
||||
* same frame as the media element, and postMessage doesn't transfer user
|
||||
* activation (User Activation v2). The runtime inside the iframe signals
|
||||
* `media-autoplay-blocked` the first time a play() attempt rejects with
|
||||
* `NotAllowedError`; receiving that message flips `_audioOwner` to `parent`
|
||||
* and these proxies start driving audible output while the iframe keeps
|
||||
* advancing timed media silently for frame-accurate state.
|
||||
*
|
||||
* Mobile browsers block media.play() inside iframes when the user gesture
|
||||
* happened in the parent frame — postMessage doesn't transfer user activation
|
||||
* (per the User Activation v2 spec). We extract ALL media sources from the
|
||||
* iframe's timed elements (audio/video with data-start), play them in the
|
||||
* parent frame (where the gesture lives), and disable the iframe copies.
|
||||
* Preloading at iframe-load time (rather than lazily on promotion) keeps
|
||||
* the audible audio cut-in tight when the promotion fires mid-playback.
|
||||
*/
|
||||
private _parentMedia: Array<{
|
||||
el: HTMLMediaElement;
|
||||
@@ -41,6 +46,22 @@ class HyperframesPlayer extends HTMLElement {
|
||||
duration: number;
|
||||
}> = [];
|
||||
|
||||
/**
|
||||
* Who owns audible playback right now.
|
||||
*
|
||||
* - `runtime` (default): the iframe's runtime drives timed media; parent
|
||||
* proxies stay paused and silent. This is the correct path on desktop,
|
||||
* in same-frame embeds, and anywhere the iframe has user activation.
|
||||
* - `parent`: parent-frame proxies drive audible output; the iframe keeps
|
||||
* syncing timed media but at `muted = true` (orthogonal to author/user
|
||||
* volume settings). Entered only in response to an actual autoplay
|
||||
* rejection from the runtime — we don't guess device class.
|
||||
*
|
||||
* The transition is one-way per session; once autoplay is known to be
|
||||
* gated, there's no benefit to attempting the iframe path again.
|
||||
*/
|
||||
private _audioOwner: "runtime" | "parent" = "runtime";
|
||||
|
||||
constructor() {
|
||||
super();
|
||||
this.shadow = this.attachShadow({ mode: "open" });
|
||||
@@ -174,16 +195,19 @@ class HyperframesPlayer extends HTMLElement {
|
||||
|
||||
play() {
|
||||
this._hidePoster();
|
||||
this._playParentMedia();
|
||||
// Always drive the iframe runtime — it's the single source of timeline
|
||||
// truth regardless of who owns audible output. When we own audio, the
|
||||
// proxies join; when the runtime owns, they stay silent.
|
||||
this._sendControl("play");
|
||||
if (this._audioOwner === "parent") this._playParentMedia();
|
||||
this._paused = false;
|
||||
this.controlsApi?.updatePlaying(true);
|
||||
this.dispatchEvent(new Event("play"));
|
||||
}
|
||||
|
||||
pause() {
|
||||
this._pauseParentMedia();
|
||||
this._sendControl("pause");
|
||||
if (this._audioOwner === "parent") this._pauseParentMedia();
|
||||
this._paused = true;
|
||||
this.controlsApi?.updatePlaying(false);
|
||||
this.dispatchEvent(new Event("pause"));
|
||||
@@ -194,11 +218,14 @@ class HyperframesPlayer extends HTMLElement {
|
||||
this._sendControl("seek", { frame });
|
||||
this._currentTime = timeInSeconds;
|
||||
|
||||
// Sync parent media positions (accounting for each element's start offset)
|
||||
for (const m of this._parentMedia) {
|
||||
const relTime = timeInSeconds - m.start;
|
||||
if (relTime >= 0 && relTime < m.duration) {
|
||||
m.el.currentTime = relTime;
|
||||
// Mirror parent proxy currentTime only while parent owns audible output.
|
||||
// Under `runtime` ownership the proxies are paused and authoritative time
|
||||
// lives on the iframe — touching parent currentTime would just trigger
|
||||
// needless buffering if ownership later flips.
|
||||
if (this._audioOwner === "parent") {
|
||||
for (const m of this._parentMedia) {
|
||||
const relTime = timeInSeconds - m.start;
|
||||
if (relTime >= 0 && relTime < m.duration) m.el.currentTime = relTime;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -276,12 +303,18 @@ class HyperframesPlayer extends HTMLElement {
|
||||
const wasPlaying = !this._paused;
|
||||
this._paused = !data.isPlaying;
|
||||
|
||||
// Sync parent media on runtime play/pause transitions (e.g. browser
|
||||
// throttling, visibility change, or scrubber interaction in the iframe).
|
||||
if (wasPlaying && this._paused) {
|
||||
this._pauseParentMedia();
|
||||
} else if (!wasPlaying && !this._paused) {
|
||||
this._playParentMedia();
|
||||
// Under parent ownership the proxies are the audible output, so they
|
||||
// mirror the iframe's play/pause transitions (externally-driven pause
|
||||
// via `__player.pause()`, scrubber interactions, etc.) and their
|
||||
// currentTime is slaved to the iframe timeline. Under runtime ownership
|
||||
// the proxies stay paused and silent; nothing here should wake them.
|
||||
if (this._audioOwner === "parent") {
|
||||
if (wasPlaying && this._paused) {
|
||||
this._pauseParentMedia();
|
||||
} else if (!wasPlaying && !this._paused) {
|
||||
this._playParentMedia();
|
||||
}
|
||||
this._mirrorParentMediaTime(this._currentTime);
|
||||
}
|
||||
|
||||
// Throttle UI updates and event dispatch to ~10fps to avoid excessive re-renders
|
||||
@@ -296,7 +329,7 @@ class HyperframesPlayer extends HTMLElement {
|
||||
}
|
||||
|
||||
if (this._currentTime >= this._duration && !this._paused) {
|
||||
this._pauseParentMedia();
|
||||
if (this._audioOwner === "parent") this._pauseParentMedia();
|
||||
if (this.loop) {
|
||||
this.seek(0);
|
||||
this.play();
|
||||
@@ -308,6 +341,10 @@ class HyperframesPlayer extends HTMLElement {
|
||||
}
|
||||
}
|
||||
|
||||
if (data.type === "media-autoplay-blocked") {
|
||||
this._promoteToParentProxy();
|
||||
}
|
||||
|
||||
if (data.type === "timeline" && data.durationInFrames > 0) {
|
||||
// Ignore Infinity duration from runtime (caused by loop-inflated timelines without data-duration)
|
||||
// The player already has duration from the initial probe, so keep that.
|
||||
@@ -489,30 +526,11 @@ class HyperframesPlayer extends HTMLElement {
|
||||
|
||||
private _playParentMedia() {
|
||||
for (const m of this._parentMedia) {
|
||||
if (m.el.src) {
|
||||
m.el
|
||||
.play()
|
||||
.then(() => {
|
||||
// Parent play succeeded — mute the iframe copy to prevent double audio.
|
||||
// This runs asynchronously, so the runtime may briefly play both copies,
|
||||
// but the overlap is inaudible (same audio at the same position).
|
||||
this._muteIframeMedia();
|
||||
})
|
||||
.catch(() => {});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private _muteIframeMedia() {
|
||||
try {
|
||||
const doc = this.iframe.contentDocument;
|
||||
if (!doc) return;
|
||||
const mediaEls = doc.querySelectorAll<HTMLMediaElement>(
|
||||
"audio[data-start], video[data-start]",
|
||||
);
|
||||
for (const el of mediaEls) el.volume = 0;
|
||||
} catch {
|
||||
// cross-origin
|
||||
if (!m.el.src) continue;
|
||||
// Best-effort: if the parent itself has no user activation, this will
|
||||
// also reject — the caller has already decided parent ownership is
|
||||
// warranted, and there's nothing better to fall back to from here.
|
||||
m.el.play().catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -520,6 +538,44 @@ class HyperframesPlayer extends HTMLElement {
|
||||
for (const m of this._parentMedia) m.el.pause();
|
||||
}
|
||||
|
||||
/**
|
||||
* Drag parent-proxy `currentTime` onto the iframe's timeline. Called on
|
||||
* every runtime state message under parent ownership. Only re-seeks when
|
||||
* drift exceeds 150 ms so we don't trigger a re-buffer on every tick —
|
||||
* native HTMLMediaElement playback rate drift stays well inside that.
|
||||
*/
|
||||
private _mirrorParentMediaTime(timelineSeconds: number) {
|
||||
for (const m of this._parentMedia) {
|
||||
const relTime = timelineSeconds - m.start;
|
||||
if (relTime < 0 || relTime >= m.duration) continue;
|
||||
if (Math.abs(m.el.currentTime - relTime) > 0.15) m.el.currentTime = relTime;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Take ownership of audible playback. Fired in response to the runtime's
|
||||
* `media-autoplay-blocked` signal — the iframe has lost the autoplay lottery
|
||||
* and will never produce audio without a fresh gesture inside itself.
|
||||
*
|
||||
* Effects, in order:
|
||||
* 1. Ask the runtime to mute its own media output via the bridge. The
|
||||
* runtime then keeps advancing timed media for frame-accurate state
|
||||
* but produces no sound of its own, freeing us to be the single
|
||||
* audible source without racing a volume-reassert loop.
|
||||
* 2. Align every parent proxy's currentTime to the iframe's timeline so
|
||||
* the cut-over is imperceptible.
|
||||
* 3. If the player is currently playing, start the proxies.
|
||||
*
|
||||
* Idempotent: repeat calls are a no-op.
|
||||
*/
|
||||
private _promoteToParentProxy() {
|
||||
if (this._audioOwner === "parent") return;
|
||||
this._audioOwner = "parent";
|
||||
this._sendControl("set-media-output-muted", { muted: true });
|
||||
this._mirrorParentMediaTime(this._currentTime);
|
||||
if (!this._paused) this._playParentMedia();
|
||||
}
|
||||
|
||||
/** Create a parent-frame media element, configure it, and start preloading. */
|
||||
private _createParentMedia(src: string, tag: "audio" | "video", start: number, duration: number) {
|
||||
// Deduplicate — browsers normalize URLs so we compare on the element after assignment
|
||||
@@ -545,12 +601,14 @@ class HyperframesPlayer extends HTMLElement {
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract ALL timed media (audio/video with data-start) from the iframe's
|
||||
* DOM and create parent-frame copies. Disables the iframe originals so the
|
||||
* runtime doesn't try to play them (which would fail on mobile and cause
|
||||
* double playback on desktop).
|
||||
* Mirror every timed iframe media element (`audio[data-start]`,
|
||||
* `video[data-start]`) into a parent-frame proxy. The proxies preload at
|
||||
* iframe-ready time so the cut-over to parent ownership — should the
|
||||
* runtime's autoplay attempt later reject — is instantaneous.
|
||||
*
|
||||
* If `audio-src` was already set, this just disables the iframe media.
|
||||
* Under runtime ownership (the default) these proxies stay paused and
|
||||
* inert; the iframe is the audible source. Ownership flips only in
|
||||
* response to a real `media-autoplay-blocked` message from the runtime.
|
||||
*/
|
||||
private _setupParentMedia() {
|
||||
try {
|
||||
@@ -578,14 +636,10 @@ class HyperframesPlayer extends HTMLElement {
|
||||
const tag = iframeEl.tagName === "VIDEO" ? ("video" as const) : ("audio" as const);
|
||||
|
||||
this._createParentMedia(src, tag, start, duration);
|
||||
|
||||
// DO NOT strip data-start, data-duration, or src from the iframe elements.
|
||||
// The runtime's syncRuntimeMedia queries audio[data-start] — removing these
|
||||
// attributes makes the runtime unable to find, sync, or play media clips.
|
||||
// The iframe copies remain fully functional for the runtime. On mobile,
|
||||
// parent copies provide the audible output via the component's play() method.
|
||||
// On desktop and in the studio (which calls __player.play() directly),
|
||||
// the runtime's own media sync handles playback.
|
||||
// Iframe originals stay untouched — the runtime's `syncRuntimeMedia`
|
||||
// queries `audio[data-start]` for state and needs them addressable.
|
||||
// Their audible output is gated later by `set-media-output-muted`
|
||||
// when (and only when) parent ownership is promoted.
|
||||
}
|
||||
} catch {
|
||||
// Cross-origin iframe — can't access DOM, fall back to iframe media
|
||||
|
||||
Reference in New Issue
Block a user