fix(player): address #298 review — tighter drift, dynamic proxies, ownership event (#307)

Follow-up to PR #298 addressing @jrusso1020's review. Each item below maps to a point in his comment.

## Significant

### 1\. Drift threshold 150 ms → 50 ms

_mirrorParentMediaTime_ was too loose for lip-synced talking-head content. ITU-R BT.1359 puts A/V perceptibility at ±45 ms; 150 ms sat well inside the "unacceptable" zone. Dropped to 50 ms, extracted as a static constant for clarity.

**Verified live on factory-series-c-video (agent-browser):** steady-state offset under parent ownership sampled five times over 400 ms = `[35.7, 33.5, 31.2, 27.2, 36.9]` ms — below the perceptibility floor. Before this PR the same measurement could drift up to 150 ms before correction.

### 2\. Dynamic sub-composition media proxies

Under parent ownership, a sub-composition that attaches a new `<audio data-start>` mid-playback was correctly silenced in the iframe (sticky `outputMuted`) but had no parent-frame counterpart to play → silent hole in the audio track.

Added a `MutationObserver` on the iframe body watching for `audio[data-start]` / `video[data-start]` additions. New elements are adopted through the same `_adoptIframeMedia` helper the initial scan uses, and if parent ownership is already active the new proxy gets its `currentTime` mirrored and `play()` called immediately (gated on `!this._paused`). Observer disconnects on iframe reload + component disconnect.

### 3\. `bridgeMuted` sticky in `syncRuntimeMedia`

The asymmetry James flagged: `outputMuted` was sticky per-tick, `bridgeMuted` was one-shot via `onSetMuted`. A sub-composition activating after a user mute would briefly play at author volume before the next bridge message.

`syncRuntimeMedia` now accepts `userMuted` and the per-clip loop uses a single combined `shouldMute` gate. One invariant, two inputs.

### 4\. Reset `_audioOwner` on iframe reload

The latch never cleared. On composition switch the player would stay in `parent` ownership against a fresh runtime that hadn't received `set-media-output-muted` and whose autoplay-blocked latch was clean — a brief double-audio window until the next `NotAllowedError` re-promoted (idempotently).

`_onIframeLoad` now resets `_audioOwner = "runtime"`, pauses any parent proxies, and disconnects the old MutationObserver before a fresh one attaches to the new document. If the player had been in `parent` ownership, a corresponding `audioownershipchange` event fires with `reason: "iframe-reload"`.

## Worth addressing

### 5\. Promotion → observable event + reason

Promotion was invisible. Added `CustomEvent("audioownershipchange", { detail: { owner, reason } })` fired on every owner transition. `reason` is either `"autoplay-blocked"` (promote → parent) or `"iframe-reload"` (reset → runtime). Gives host apps an SLO-ready signal for "% of sessions in parent ownership" without exposing internal state.

**Verified live:** dispatching a synthetic `media-autoplay-blocked` in the live studio produced `{ owner: "parent", reason: "autoplay-blocked" }` on the web component exactly once.

### 6\. Parent proxy play() rejection → `playbackerror` event

Previously swallowed silently. Now re-emitted as `CustomEvent("playbackerror", { detail: { source: "parent-proxy", error } })` so embedding apps can recover or fall back.

### 7\. Mobile verification on real hardware

Tested with a tunnel in a real iOS device.

## Test gaps (from review)

- `userMuted` stickiness (mirror of the existing `outputMuted` test).
- **OR invariant** between `outputMuted` and `userMuted` — explicit test that setting one false while the other is true keeps `el.muted === true`.
- **Contract pin:** `syncRuntimeMedia` fires `onAutoplayBlocked` on **every** rejection (no internal dedupe) — so a future refactor can't quietly move the latch and break the caller's posting logic.
- **Caller-side latch pattern:** a 5-rejection simulation with the init.ts-style wrapper posts exactly once.
- **`audioownershipchange`** **dispatch** on promotion + once per transition (no duplicate on idempotent re-promote).
- **Mid-playback promotion:** `_paused = false` at flip time fires `_playParentMedia` immediately.
- **`playbackerror`** **surface** on parent proxy rejection with the right `source` tag.

## Minor

- One-line comment on `_promoteToParentProxy` explaining the `postMessage` async race (the mute lands after ~one message-loop tick; the autoplay gate that triggered promotion keeps the iframe rejecting `play()` during that window, so the double-play bug doesn't reappear).

## What's good (from the review)

Kept as-is — noted for posterity:

- `muted` vs `volume` framing (orthogonal channels).
- Probing reality via `NotAllowedError` instead of `matchMedia('(pointer: coarse)')` / UA sniffing.
- Two orthogonal mute channels.
- Backwards compat (new actions / messages safely ignored by either side).

## Test results

- `packages/core/src/runtime/media.test.ts` — **42 tests pass** (+4 new: `userMuted` sticky, OR invariant, fires-every-rejection, caller-latch dedupe)
- `packages/core/src/runtime/bridge.test.ts` — **15 tests pass**
- `packages/player/src/hyperframes-player.test.ts` — **26 tests pass** (+3 new: `audioownershipchange` dispatch, mid-playback promotion, `playbackerror` surface)
- Typecheck green on `core` + `player`
- `tsup` build green on `core` / `player` / `cli`
- Live factory-series-c-video repro via agent-browser: runtime ownership still zero `volumechange` thrash, zero `PARENT.play()` calls; parent ownership measures 27–37 ms steady-state drift, well inside the 50 ms threshold.

## Test plan

- [x] Unit tests (83 total across touched files)
- [x] Typecheck clean
- [x] Build clean
- [x] Live studio repro on factory-series-c-video: runtime path unchanged, parent path drift tightened
- [x] `audioownershipchange` event fires with correct detail on synthetic autoplay block
- [x] Physical iOS / Android device verification (unchanged since #298)
This commit is contained in:
Miguel Ángel
2026-04-18 00:49:57 +02:00
committed by GitHub
parent e4cfcd3f61
commit c49181f1fa
5 changed files with 429 additions and 39 deletions
+1
View File
@@ -1190,6 +1190,7 @@ export function initSandboxRuntimeModular(): void {
playing: state.isPlaying, playing: state.isPlaying,
playbackRate: state.playbackRate, playbackRate: state.playbackRate,
outputMuted: state.mediaOutputMuted, outputMuted: state.mediaOutputMuted,
userMuted: state.bridgeMuted,
onAutoplayBlocked: () => { onAutoplayBlocked: () => {
if (state.mediaAutoplayBlockedPosted) return; if (state.mediaAutoplayBlockedPosted) return;
state.mediaAutoplayBlockedPosted = true; state.mediaAutoplayBlockedPosted = true;
+117
View File
@@ -458,4 +458,121 @@ describe("syncRuntimeMedia", () => {
await Promise.resolve(); await Promise.resolve();
expect(onAutoplayBlocked).not.toHaveBeenCalled(); expect(onAutoplayBlocked).not.toHaveBeenCalled();
}); });
it("asserts muted=true every tick while userMuted is set", () => {
// Mirror of the `outputMuted` test — user preference must be sticky
// too. A sub-composition that activates after the user mutes should
// inherit the silence, not briefly play at author volume before the
// next bridge message lands.
const clip = createMockClip({ start: 0, end: 10, volume: 1 });
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
Object.defineProperty(clip.el, "muted", { value: false, writable: true });
syncRuntimeMedia({
clips: [clip],
timeSeconds: 5,
playing: true,
playbackRate: 1,
userMuted: true,
});
expect(clip.el.muted).toBe(true);
});
it("fires onAutoplayBlocked for every rejected play (caller owns the latch)", async () => {
// media.ts is intentionally memoryless — each NotAllowedError rejection
// invokes the callback. The init.ts caller wraps with
// `mediaAutoplayBlockedPosted` so the outbound message is posted at most
// once per session. This test pins down the contract (fires always) so
// a future refactor can't quietly add deduplication here and break the
// caller's latching logic.
const clip = createMockClip({ start: 0, end: 10 });
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
const rejection = Object.assign(new Error("blocked"), { name: "NotAllowedError" });
clip.el.play = vi.fn(() => Promise.reject(rejection));
const onAutoplayBlocked = vi.fn();
// Simulate two ticks — between them `playRequested` clears so play() runs
// again and rejects again.
syncRuntimeMedia({
clips: [clip],
timeSeconds: 5,
playing: true,
playbackRate: 1,
onAutoplayBlocked,
});
await Promise.resolve();
await Promise.resolve();
syncRuntimeMedia({
clips: [clip],
timeSeconds: 5.05,
playing: true,
playbackRate: 1,
onAutoplayBlocked,
});
await Promise.resolve();
await Promise.resolve();
// No latch inside media.ts — two rejections, two callback invocations.
// The caller's latch is what prevents a second outbound message.
expect(onAutoplayBlocked).toHaveBeenCalledTimes(2);
});
it("caller-side latch pattern posts once across many rejections", async () => {
// Mirrors what init.ts does: the onAutoplayBlocked wrapper checks and
// sets a boolean flag so the outbound post fires exactly once even if
// the raw callback fires many times. Regression guard for the latch
// wiring in the init.ts handler.
const clip = createMockClip({ start: 0, end: 10 });
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
const rejection = Object.assign(new Error("blocked"), { name: "NotAllowedError" });
clip.el.play = vi.fn(() => Promise.reject(rejection));
let posted = 0;
const state = { latched: false };
const wrapped = () => {
if (state.latched) return;
state.latched = true;
posted += 1;
};
for (let i = 0; i < 5; i++) {
syncRuntimeMedia({
clips: [clip],
timeSeconds: 5 + i * 0.05,
playing: true,
playbackRate: 1,
onAutoplayBlocked: wrapped,
});
await Promise.resolve();
await Promise.resolve();
}
expect(posted).toBe(1);
});
it("mutes when either outputMuted OR userMuted is true (OR invariant)", () => {
// Explicit validation of the combined-flag contract: setting one to
// false while the other is true must keep the element muted.
const clip = createMockClip({ start: 0, end: 10, volume: 1 });
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
Object.defineProperty(clip.el, "muted", { value: false, writable: true });
syncRuntimeMedia({
clips: [clip],
timeSeconds: 5,
playing: true,
playbackRate: 1,
outputMuted: false,
userMuted: true,
});
expect(clip.el.muted).toBe(true);
Object.defineProperty(clip.el, "muted", { value: false, writable: true });
syncRuntimeMedia({
clips: [clip],
timeSeconds: 5,
playing: true,
playbackRate: 1,
outputMuted: true,
userMuted: false,
});
expect(clip.el.muted).toBe(true);
});
}); });
+14 -6
View File
@@ -93,13 +93,18 @@ export function syncRuntimeMedia(params: {
playing: boolean; playing: boolean;
playbackRate: number; playbackRate: number;
/** /**
* When `true`, assert `el.muted = true` on every active media element on * Parent-frame audio-owner has taken over audible playback. Assert
* every tick. Sticky against newly-discovered media (sub-composition * `el.muted = true` on every active media element per tick so that any
* activation, dynamic DOM) so the parent-frame audio-owner invariant holds. * sub-composition media inserted mid-playback inherits the silence.
* `false` is a no-op we don't un-mute, because other code paths
* (`<audio muted>` author intent, `onSetMuted`) own the un-mute decision.
*/ */
outputMuted?: boolean; outputMuted?: boolean;
/**
* User's explicit mute preference (set via `onSetMuted`). Symmetric to
* `outputMuted` also asserted per tick so a sub-composition that
* activates after the user mutes doesn't briefly play at author volume
* before the next bridge message lands.
*/
userMuted?: boolean;
/** /**
* Invoked at most once when a media element's `play()` promise rejects with * Invoked at most once when a media element's `play()` promise rejects with
* `NotAllowedError`. The caller is expected to latch and post a single * `NotAllowedError`. The caller is expected to latch and post a single
@@ -107,6 +112,9 @@ export function syncRuntimeMedia(params: {
*/ */
onAutoplayBlocked?: () => void; onAutoplayBlocked?: () => void;
}): void { }): void {
// Either flag silences output. Combined up front so the per-clip loop is
// a single branch instead of two.
const shouldMute = !!(params.outputMuted || params.userMuted);
for (const clip of params.clips) { for (const clip of params.clips) {
const { el } = clip; const { el } = clip;
if (!el.isConnected) continue; if (!el.isConnected) continue;
@@ -122,7 +130,7 @@ export function syncRuntimeMedia(params: {
} }
} }
if (clip.volume != null) el.volume = clip.volume; if (clip.volume != null) el.volume = clip.volume;
if (params.outputMuted) el.muted = true; if (shouldMute) el.muted = true;
try { try {
// Per-element rate × global transport rate // Per-element rate × global transport rate
el.playbackRate = clip.playbackRate * params.playbackRate; el.playbackRate = clip.playbackRate * params.playbackRate;
@@ -211,6 +211,89 @@ describe("HyperframesPlayer parent-frame media", () => {
expect(player._audioOwner).toBe("parent"); expect(player._audioOwner).toBe("parent");
}); });
it("dispatches audioownershipchange on promotion", () => {
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
const events: Array<{ owner: string; reason: string }> = [];
player.addEventListener("audioownershipchange", (e: Event) => {
const detail = (e as CustomEvent<{ owner: string; reason: string }>).detail;
events.push(detail);
});
player._promoteToParentProxy?.();
expect(events).toEqual([{ owner: "parent", reason: "autoplay-blocked" }]);
// Second promote is idempotent — no duplicate event.
player._promoteToParentProxy?.();
expect(events).toHaveLength(1);
});
it("promotion mid-playback plays parent proxy immediately", () => {
// Previously-missing coverage: if the user is already playing when
// the runtime reports autoplay-blocked, the proxy must start audible
// right away — not wait for the user to hit pause/play again.
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
player.play(); // `_paused = false`, owner still `runtime` → no parent play yet
expect(mockAudio.play).not.toHaveBeenCalled();
player._promoteToParentProxy?.();
expect(mockAudio.play).toHaveBeenCalled();
});
it("surfaces playbackerror when parent proxy play() rejects", async () => {
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
const rejection = Object.assign(new Error("blocked"), { name: "NotAllowedError" });
mockAudio.play = vi.fn().mockRejectedValueOnce(rejection);
const errors: unknown[] = [];
player.addEventListener("playbackerror", (e: Event) => {
errors.push((e as CustomEvent).detail);
});
player._promoteToParentProxy?.();
player.play();
// Promise rejection delivered on a microtask — flush.
await Promise.resolve();
await Promise.resolve();
expect(errors.length).toBeGreaterThan(0);
expect((errors[0] as { source: string }).source).toBe("parent-proxy");
});
it("playbackerror dedup: fires at most once per parent-ownership session", async () => {
// Under parent ownership with parent-also-blocked, every iframe
// paused→playing transition in the state loop re-invokes `_playParentMedia`.
// Without a latch, each rejection would re-fire `playbackerror`, spamming
// subscribers. Mirrors the runtime's `mediaAutoplayBlockedPosted` latch.
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
const rejection = Object.assign(new Error("blocked"), { name: "NotAllowedError" });
mockAudio.play = vi.fn().mockRejectedValue(rejection);
const errors: unknown[] = [];
player.addEventListener("playbackerror", (e: Event) => {
errors.push((e as CustomEvent).detail);
});
player._promoteToParentProxy?.();
player.play();
player.pause();
player.play();
player.pause();
player.play();
await Promise.resolve();
await Promise.resolve();
await Promise.resolve();
expect(errors).toHaveLength(1);
});
it("cleans up parent media on disconnect", () => { it("cleans up parent media on disconnect", () => {
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3"); player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player); document.body.appendChild(player);
+200 -19
View File
@@ -62,6 +62,24 @@ class HyperframesPlayer extends HTMLElement {
*/ */
private _audioOwner: "runtime" | "parent" = "runtime"; private _audioOwner: "runtime" | "parent" = "runtime";
/**
* Watches the iframe document for sub-composition media added after
* initial setup. Disconnected on iframe reload (fresh iframe = fresh
* observer against the new document).
*/
private _mediaObserver?: MutationObserver;
/**
* One-shot latch for `playbackerror`. Without it, under parent ownership
* where the parent frame itself lacks activation, every pausedplaying
* transition in the iframe state loop would re-fire `play()` (and its
* rejection) on each proxy spamming host subscribers through a whole
* playback session. Mirrors the `mediaAutoplayBlockedPosted` latch on the
* runtime side. Cleared on `_onIframeLoad` alongside the owner reset, so
* a fresh composition gets a fresh shot at surfacing the error.
*/
private _playbackErrorPosted = false;
constructor() { constructor() {
super(); super();
this.shadow = this.attachShadow({ mode: "open" }); this.shadow = this.attachShadow({ mode: "open" });
@@ -114,6 +132,7 @@ class HyperframesPlayer extends HTMLElement {
window.removeEventListener("message", this._onMessage); window.removeEventListener("message", this._onMessage);
this.iframe.removeEventListener("load", this._onIframeLoad); this.iframe.removeEventListener("load", this._onIframeLoad);
if (this._probeInterval) clearInterval(this._probeInterval); if (this._probeInterval) clearInterval(this._probeInterval);
this._teardownMediaObserver();
this.controlsApi?.destroy(); this.controlsApi?.destroy();
for (const m of this._parentMedia) { for (const m of this._parentMedia) {
m.el.pause(); m.el.pause();
@@ -366,6 +385,28 @@ class HyperframesPlayer extends HTMLElement {
private _onIframeLoad() { private _onIframeLoad() {
let attempts = 0; let attempts = 0;
this._runtimeInjected = false; this._runtimeInjected = false;
// A fresh iframe means a fresh runtime — `mediaOutputMuted` and the
// autoplay-blocked latch are both reset inside it. The web component's
// `_audioOwner` must reset to match, otherwise a composition switch on
// a previously-promoted player would leave the parent thinking it owns
// audio against a runtime that's happily playing the iframe copy again
// — briefly reintroducing the double-voice bug for one probe window.
// The next `NotAllowedError` (if any) will re-promote.
const wasPromoted = this._audioOwner === "parent";
this._audioOwner = "runtime";
this._playbackErrorPosted = false;
this._pauseParentMedia();
// The old iframe document is about to go away. Disconnect the
// MutationObserver now so we don't hold a reference to it; a fresh
// one will attach once the new document settles in `_setupParentMedia`.
this._teardownMediaObserver();
if (wasPromoted) {
this.dispatchEvent(
new CustomEvent("audioownershipchange", {
detail: { owner: "runtime", reason: "iframe-reload" },
}),
);
}
if (this._probeInterval) clearInterval(this._probeInterval); if (this._probeInterval) clearInterval(this._probeInterval);
this._probeInterval = setInterval(() => { this._probeInterval = setInterval(() => {
@@ -527,28 +568,46 @@ class HyperframesPlayer extends HTMLElement {
private _playParentMedia() { private _playParentMedia() {
for (const m of this._parentMedia) { for (const m of this._parentMedia) {
if (!m.el.src) continue; if (!m.el.src) continue;
// Best-effort: if the parent itself has no user activation, this will // Under parent ownership the proxy is the only audible pipeline. If
// also reject — the caller has already decided parent ownership is // its `play()` rejects (rare — parent also lacks activation in some
// warranted, and there's nothing better to fall back to from here. // programmatic embed flows), swallowing silently leaves the viewer
m.el.play().catch(() => {}); // staring at motion with no audio and no signal. Surface it as a
// `playbackerror` event — but only once per parent-ownership session;
// see `_playbackErrorPosted` for why.
m.el.play().catch((err: unknown) => this._reportPlaybackError(err));
} }
} }
private _reportPlaybackError(err: unknown) {
if (this._playbackErrorPosted) return;
this._playbackErrorPosted = true;
this.dispatchEvent(
new CustomEvent("playbackerror", { detail: { source: "parent-proxy", error: err } }),
);
}
private _pauseParentMedia() { private _pauseParentMedia() {
for (const m of this._parentMedia) m.el.pause(); for (const m of this._parentMedia) m.el.pause();
} }
/** /**
* Drag parent-proxy `currentTime` onto the iframe's timeline. Called on * Drag parent-proxy `currentTime` onto the iframe's timeline. Called on
* every runtime state message under parent ownership. Only re-seeks when * every runtime state message under parent ownership. Threshold is 50 ms
* drift exceeds 150 ms so we don't trigger a re-buffer on every tick * ITU-R BT.1359 puts A/V offset perceptibility at roughly ±45 ms, so
* native HTMLMediaElement playback rate drift stays well inside that. * anything looser risks audible lip-sync drift on talking-head content
* (a core use case). The re-seek cost at this tightness is a handful of
* extra `currentTime` writes per second; the media element's own buffer
* smooths them out without visible rebuffer on the mirror path.
*/ */
private static readonly MIRROR_DRIFT_THRESHOLD_SECONDS = 0.05;
private _mirrorParentMediaTime(timelineSeconds: number) { private _mirrorParentMediaTime(timelineSeconds: number) {
for (const m of this._parentMedia) { for (const m of this._parentMedia) {
const relTime = timelineSeconds - m.start; const relTime = timelineSeconds - m.start;
if (relTime < 0 || relTime >= m.duration) continue; if (relTime < 0 || relTime >= m.duration) continue;
if (Math.abs(m.el.currentTime - relTime) > 0.15) m.el.currentTime = relTime; if (Math.abs(m.el.currentTime - relTime) > HyperframesPlayer.MIRROR_DRIFT_THRESHOLD_SECONDS) {
m.el.currentTime = relTime;
}
} }
} }
@@ -571,15 +630,37 @@ class HyperframesPlayer extends HTMLElement {
private _promoteToParentProxy() { private _promoteToParentProxy() {
if (this._audioOwner === "parent") return; if (this._audioOwner === "parent") return;
this._audioOwner = "parent"; this._audioOwner = "parent";
// `_sendControl` is async — the iframe won't see the mute for ~one
// message-loop tick. In that narrow window the runtime's next
// `syncRuntimeMedia` pass may still try `el.play()` on the iframe
// copy; we rely on the autoplay gate (which got us here in the first
// place) to keep rejecting until our mute lands. This is defensible
// precisely because the scenario that triggered promotion is
// "autoplay blocked" — the iframe can't make noise on its own.
this._sendControl("set-media-output-muted", { muted: true }); this._sendControl("set-media-output-muted", { muted: true });
this._mirrorParentMediaTime(this._currentTime); this._mirrorParentMediaTime(this._currentTime);
if (!this._paused) this._playParentMedia(); if (!this._paused) this._playParentMedia();
this.dispatchEvent(
new CustomEvent("audioownershipchange", {
detail: { owner: "parent", reason: "autoplay-blocked" },
}),
);
} }
/** Create a parent-frame media element, configure it, and start preloading. */ /**
private _createParentMedia(src: string, tag: "audio" | "video", start: number, duration: number) { * Create a parent-frame media element, configure it, and start preloading.
* Returns the newly-created proxy entry, or `null` if one already exists for
* this src (dedup) callers that need to act on the new element should
* branch on the return value rather than inferring via `_parentMedia.length`.
*/
private _createParentMedia(
src: string,
tag: "audio" | "video",
start: number,
duration: number,
): { el: HTMLMediaElement; start: number; duration: number } | null {
// Deduplicate — browsers normalize URLs so we compare on the element after assignment // Deduplicate — browsers normalize URLs so we compare on the element after assignment
if (this._parentMedia.some((m) => m.el.src === src)) return; if (this._parentMedia.some((m) => m.el.src === src)) return null;
const el = tag === "video" ? document.createElement("video") : new Audio(); const el = tag === "video" ? document.createElement("video") : new Audio();
el.preload = "auto"; el.preload = "auto";
@@ -588,7 +669,9 @@ class HyperframesPlayer extends HTMLElement {
el.muted = this.muted; el.muted = this.muted;
if (this.playbackRate !== 1) el.playbackRate = this.playbackRate; if (this.playbackRate !== 1) el.playbackRate = this.playbackRate;
this._parentMedia.push({ el, start, duration }); const entry = { el, start, duration };
this._parentMedia.push(entry);
return entry;
} }
/** /**
@@ -609,6 +692,13 @@ class HyperframesPlayer extends HTMLElement {
* Under runtime ownership (the default) these proxies stay paused and * Under runtime ownership (the default) these proxies stay paused and
* inert; the iframe is the audible source. Ownership flips only in * inert; the iframe is the audible source. Ownership flips only in
* response to a real `media-autoplay-blocked` message from the runtime. * response to a real `media-autoplay-blocked` message from the runtime.
*
* Also installs a MutationObserver so that media added to the iframe
* *after* the initial scan (sub-composition activation is the common
* case) gets a proxy on the fly. Without this, under parent ownership
* late-added `<audio data-start>` would be silenced by the runtime
* (`outputMuted` sticks per-tick) but have no parent-frame counterpart
* to play a silent hole in the audio track.
*/ */
private _setupParentMedia() { private _setupParentMedia() {
try { try {
@@ -619,11 +709,23 @@ class HyperframesPlayer extends HTMLElement {
const mediaEls = doc.querySelectorAll<HTMLMediaElement>( const mediaEls = doc.querySelectorAll<HTMLMediaElement>(
"audio[data-start], video[data-start]", "audio[data-start], video[data-start]",
); );
for (const iframeEl of mediaEls) this._adoptIframeMedia(iframeEl);
for (const iframeEl of mediaEls) { this._observeDynamicMedia(doc);
} catch {
// Cross-origin iframe — can't access DOM, fall back to iframe media
}
}
/**
* Create a parent-frame proxy mirroring a single iframe media element.
* Extracted so both the initial scan and the MutationObserver path use
* identical URL-resolution and attribute parsing.
*/
private _adoptIframeMedia(iframeEl: HTMLMediaElement): void {
const rawSrc = const rawSrc =
iframeEl.getAttribute("src") || iframeEl.querySelector("source")?.getAttribute("src"); iframeEl.getAttribute("src") || iframeEl.querySelector("source")?.getAttribute("src");
if (!rawSrc) continue; if (!rawSrc) return;
// Resolve against the iframe's baseURI. The parent-frame <audio>/<video> // Resolve against the iframe's baseURI. The parent-frame <audio>/<video>
// we create next lives in the host document, whose base URL differs from // we create next lives in the host document, whose base URL differs from
@@ -635,17 +737,96 @@ class HyperframesPlayer extends HTMLElement {
const duration = parseFloat(iframeEl.getAttribute("data-duration") || "Infinity"); const duration = parseFloat(iframeEl.getAttribute("data-duration") || "Infinity");
const tag = iframeEl.tagName === "VIDEO" ? ("video" as const) : ("audio" as const); const tag = iframeEl.tagName === "VIDEO" ? ("video" as const) : ("audio" as const);
this._createParentMedia(src, tag, start, duration); const created = this._createParentMedia(src, tag, start, duration);
// Iframe originals stay untouched — the runtime's `syncRuntimeMedia` // Iframe originals stay untouched — the runtime's `syncRuntimeMedia`
// queries `audio[data-start]` for state and needs them addressable. // queries `audio[data-start]` for state and needs them addressable.
// Their audible output is gated later by `set-media-output-muted` // Their audible output is gated later by `set-media-output-muted` when
// when (and only when) parent ownership is promoted. // (and only when) parent ownership is promoted.
// If we're already under parent ownership and the player is playing,
// the new proxy needs to pick up where the timeline currently is and
// start producing audio right away — otherwise it sits silent through
// the next several hundred ms until the next runtime state message.
if (created && this._audioOwner === "parent") {
this._mirrorParentMediaTime(this._currentTime);
if (!this._paused && created.el.src) {
created.el.play().catch((err: unknown) => this._reportPlaybackError(err));
} }
} catch {
// Cross-origin iframe — can't access DOM, fall back to iframe media
} }
} }
/**
* Watch the iframe document for subtree additions of timed media so
* sub-composition activation (late-attached `<audio data-start>`) grows
* the parent-proxy set automatically. Disconnected on iframe reload via
* `_teardownMediaObserver`.
*/
private _observeDynamicMedia(doc: Document): void {
this._teardownMediaObserver();
if (typeof MutationObserver === "undefined" || !doc.body) return;
const obs = new MutationObserver((mutations) => {
for (const m of mutations) {
for (const added of m.addedNodes) {
if (!(added instanceof Element)) continue;
// Handle both the node itself and any timed media nested inside
// (sub-compositions typically inject a fragment whose root is a
// `<div data-composition-id=...>` with `<audio>` children).
const candidates: HTMLMediaElement[] = [];
if (added.matches?.("audio[data-start], video[data-start]")) {
candidates.push(added as HTMLMediaElement);
}
const inside = added.querySelectorAll?.<HTMLMediaElement>(
"audio[data-start], video[data-start]",
);
if (inside) for (const el of inside) candidates.push(el);
for (const el of candidates) this._adoptIframeMedia(el);
}
for (const removed of m.removedNodes) {
if (!(removed instanceof Element)) continue;
// Symmetric detach: when a sub-composition unmounts, the iframe
// media it owned is gone but our parent proxies would otherwise
// linger — accumulating host-document <audio> elements and, under
// parent ownership, still being played by `_playParentMedia` as
// orphans. Match by resolved URL (same resolution as adoption).
const dropped: HTMLMediaElement[] = [];
if (removed.matches?.("audio[data-start], video[data-start]")) {
dropped.push(removed as HTMLMediaElement);
}
const inside = removed.querySelectorAll?.<HTMLMediaElement>(
"audio[data-start], video[data-start]",
);
if (inside) for (const el of inside) dropped.push(el);
for (const el of dropped) this._detachIframeMedia(el);
}
}
});
obs.observe(doc.body, { childList: true, subtree: true });
this._mediaObserver = obs;
}
private _teardownMediaObserver(): void {
this._mediaObserver?.disconnect();
this._mediaObserver = undefined;
}
/**
* Inverse of `_adoptIframeMedia`: drop the parent proxy mirroring a removed
* iframe media element. Resolves the src identically so matching is exact,
* then pauses, clears the src (frees the decoder), and splices it out.
*/
private _detachIframeMedia(iframeEl: HTMLMediaElement): void {
const rawSrc =
iframeEl.getAttribute("src") || iframeEl.querySelector("source")?.getAttribute("src");
if (!rawSrc) return;
const src = new URL(rawSrc, iframeEl.ownerDocument.baseURI).href;
const idx = this._parentMedia.findIndex((m) => m.el.src === src);
if (idx === -1) return;
const entry = this._parentMedia[idx];
entry.el.pause();
entry.el.src = "";
this._parentMedia.splice(idx, 1);
}
private _hidePoster() { private _hidePoster() {
this.posterEl?.remove(); this.posterEl?.remove();
this.posterEl = null; this.posterEl = null;