fix: prevent nested composition videos from autoplaying on seek (#477)

## Problem

Studio seek could still wake nested composition media even when the transport itself stayed paused.

In the real repro from `apple-presentation`, scrubbing to `0:29` without pressing play lands on the `slide-translation` composition. That composition contains `Multilingual_Journey.mp4` inside the composition host. On the broken path:

- the main Studio transport remained paused
- the nested video advanced and stayed playing anyway
- the user saw autoplay-like behavior even though the only action was a seek

That was especially confusing because the seek was otherwise correct: the timeline moved to the right point, but the nested media stopped obeying the paused transport state.

## What this fixes

### Nested media now participates in runtime media sync

- the runtime media cache no longer assumes only `video[data-start]` / `audio[data-start]` are relevant
- nested media inside a composition host can now be included in the same timed-media sync pass even when the inner media element does not carry its own authored `data-start`

### Nested media timing is resolved in the host composition window

- nested media start time is resolved against the enclosing composition host instead of falling back to scene-local `0`
- nested media duration is clamped to the enclosing composition window so it stays aligned with the authored host clip timing

### Paused seeks land on the right frame and stay paused

- after seeking into a nested composition, the inner media is now seeked to the correct frame relative to the host timeline
- because it is now part of the managed media set, the runtime also keeps it paused when the transport is paused instead of letting it continue playing on its own

### Regression coverage

- adds a runtime regression test that covers a nested composition video with no local `data-start`
- the test verifies that `player.seek(29)` leaves the nested video paused while landing it at the expected `currentTime`

## Root cause

The bug came from a mismatch between deterministic timeline seeking and media ownership.

### 1. The runtime only managed media with direct timing attrs

`refreshRuntimeMediaCache()` only collected `video[data-start]` and `audio[data-start]`. That works for root-level timed media, but not for media embedded inside a composition host where timing is inherited from the host composition rather than duplicated onto the inner media node.

### 2. Nested composition seek could still advance inner media

The runtime intentionally rearms sibling timelines during deterministic seek so nested timelines land on the right local offsets. That part is necessary and correct.

But because the nested video was not part of the managed media cache, it could advance during that seek path without being brought back under the paused transport state afterward.

### 3. The runtime had no way to reconcile the two

So the system had an inconsistent split:

- timeline seek knew about the nested composition timeline
- media sync did not know about the nested media inside it

The fix closes that split by resolving nested media start/duration from the enclosing composition context and running it through the same sync logic as other managed media.

## Verification

### Local checks

- `bun run --filter @hyperframes/core typecheck`
- `bun run test -- src/runtime/init.test.ts src/runtime/media.test.ts src/runtime/player.test.ts` in `packages/core`
- `bunx oxlint packages/core/src/runtime/media.ts packages/core/src/runtime/init.ts packages/core/src/runtime/init.test.ts`
- `bunx oxfmt --check packages/core/src/runtime/media.ts packages/core/src/runtime/init.ts packages/core/src/runtime/init.test.ts`

### Browser verification

Verified against a repo-backed local Studio preview of `apple-presentation`:

- opened `http://127.0.0.1:3014/#project/apple-presentation`
- seeked to `0:29` without pressing play
- confirmed the visible composition switched to `slide-translation`
- confirmed `Multilingual_Journey.mp4` landed at a non-zero `currentTime` (`3.067` in the verified run)
- confirmed the nested video stayed `paused` and its `currentTime` remained stable across a follow-up check instead of autoplaying

## Notes

- the local browser proof artifacts under `qa-artifacts/autoplay-seek/` are verification-only and are not part of this PR
- this PR is intentionally scoped to nested media ownership during paused seek; it does not broaden into unrelated runtime media refactors beyond bringing inherited nested media under the existing sync contract
This commit is contained in:
Miguel Ángel
2026-04-24 22:10:11 +02:00
committed by GitHub
parent 6b21ead737
commit e8c43f0889
3 changed files with 146 additions and 7 deletions
+92
View File
@@ -123,4 +123,96 @@ describe("initSandboxRuntimeModular", () => {
expect(child.style.visibility).toBe("hidden");
});
it("pauses nested media that is outside the timed-media cache after a seek", () => {
const root = document.createElement("div");
root.setAttribute("data-composition-id", "main");
root.setAttribute("data-root", "true");
root.setAttribute("data-width", "1920");
root.setAttribute("data-height", "1080");
document.body.appendChild(root);
const child = document.createElement("div");
child.setAttribute("data-composition-id", "slide-translation");
child.setAttribute("data-start", "20");
child.setAttribute("data-duration", "16");
root.appendChild(child);
const video = document.createElement("video");
child.appendChild(video);
Object.defineProperty(video, "duration", { value: 20, writable: true, configurable: true });
Object.defineProperty(video, "paused", { value: false, writable: true, configurable: true });
Object.defineProperty(video, "readyState", { value: 4, writable: true, configurable: true });
Object.defineProperty(video, "currentTime", { value: 0, writable: true, configurable: true });
const pause = () => {
Object.defineProperty(video, "paused", { value: true, writable: true, configurable: true });
};
video.load = () => {};
video.pause = pause;
(window as Window & { __timelines?: Record<string, RuntimeTimelineLike> }).__timelines = {
main: createMockTimeline(40),
"slide-translation": createMockTimeline(16),
};
initSandboxRuntimeModular();
const player = (
window as Window & {
__player?: { seek: (timeSeconds: number) => void };
}
).__player;
expect(player).toBeDefined();
player?.seek(29);
expect(video.paused).toBe(true);
expect(video.currentTime).toBe(9);
});
it("clamps nested media to the authored host window on seek", () => {
const root = document.createElement("div");
root.setAttribute("data-composition-id", "main");
root.setAttribute("data-root", "true");
root.setAttribute("data-width", "1920");
root.setAttribute("data-height", "1080");
document.body.appendChild(root);
const child = document.createElement("div");
child.setAttribute("data-composition-id", "slide-translation");
child.setAttribute("data-start", "20");
child.setAttribute("data-duration", "16");
root.appendChild(child);
const video = document.createElement("video");
child.appendChild(video);
Object.defineProperty(video, "duration", { value: 20, writable: true, configurable: true });
Object.defineProperty(video, "paused", { value: false, writable: true, configurable: true });
Object.defineProperty(video, "readyState", { value: 4, writable: true, configurable: true });
Object.defineProperty(video, "currentTime", { value: 0, writable: true, configurable: true });
const pause = () => {
Object.defineProperty(video, "paused", { value: true, writable: true, configurable: true });
};
video.load = () => {};
video.pause = pause;
(window as Window & { __timelines?: Record<string, RuntimeTimelineLike> }).__timelines = {
main: createMockTimeline(40),
"slide-translation": createMockTimeline(16),
};
initSandboxRuntimeModular();
const player = (
window as Window & {
__player?: { seek: (timeSeconds: number) => void };
}
).__player;
expect(player).toBeDefined();
player?.seek(37);
expect(video.paused).toBe(true);
expect(video.currentTime).toBe(0);
});
});
+42 -1
View File
@@ -1199,8 +1199,49 @@ export function initSandboxRuntimeModular(): void {
};
const syncMediaForCurrentState = () => {
const resolveMediaCompositionContext = (element: HTMLVideoElement | HTMLAudioElement) => {
const compositionRoot = element.closest("[data-composition-id]");
const inheritedStart = compositionRoot ? resolveStartForElement(compositionRoot, 0) : null;
// Media sync intentionally uses the authored host window here instead of
// the live child timeline duration. Visibility prefers live truth so a
// shrinking child composition hides early, but nested media needs a
// stable authored window so seeks clamp against the host clip timing.
const inheritedDuration = compositionRoot
? resolveDurationForElement(compositionRoot, { includeAuthoredTimingAttrs: true })
: null;
return { compositionRoot, inheritedStart, inheritedDuration };
};
const cache = refreshRuntimeMediaCache({
resolveStartSeconds: (element) => resolveStartForElement(element, 0),
shouldIncludeElement: (element) =>
element.hasAttribute("data-start") ||
Boolean(resolveMediaCompositionContext(element).compositionRoot),
resolveStartSeconds: (element) => {
const context = resolveMediaCompositionContext(
element as HTMLVideoElement | HTMLAudioElement,
);
return resolveStartForElement(element, context.inheritedStart ?? 0);
},
resolveDurationSeconds: (element) => {
const context = resolveMediaCompositionContext(element);
const start = resolveStartForElement(element, context.inheritedStart ?? 0);
const mediaStart =
Number.parseFloat(element.dataset.playbackStart ?? element.dataset.mediaStart ?? "0") ||
0;
const hostRemaining =
context.inheritedStart != null &&
context.inheritedDuration != null &&
context.inheritedDuration > 0
? Math.max(0, context.inheritedStart + context.inheritedDuration - start)
: null;
const sourceDuration =
Number.isFinite(element.duration) && element.duration > mediaStart
? Math.max(0, element.duration - mediaStart)
: null;
if (sourceDuration != null && hostRemaining != null) {
return Math.min(sourceDuration, hostRemaining);
}
return sourceDuration ?? hostRemaining;
},
});
syncRuntimeMedia({
clips: cache.mediaClips,
+12 -6
View File
@@ -13,19 +13,24 @@ export type RuntimeMediaClip = {
export function refreshRuntimeMediaCache(params?: {
resolveStartSeconds?: (element: Element) => number;
resolveDurationSeconds?: (element: HTMLVideoElement | HTMLAudioElement) => number | null;
shouldIncludeElement?: (element: HTMLVideoElement | HTMLAudioElement) => boolean;
}): {
timedMediaEls: Array<HTMLVideoElement | HTMLAudioElement>;
mediaClips: RuntimeMediaClip[];
videoClips: RuntimeMediaClip[];
maxMediaEnd: number;
} {
const mediaEls = Array.from(
document.querySelectorAll("video[data-start], audio[data-start]"),
) as Array<HTMLVideoElement | HTMLAudioElement>;
const mediaEls = Array.from(document.querySelectorAll("video, audio")) as Array<
HTMLVideoElement | HTMLAudioElement
>;
const timedMediaEls = params?.shouldIncludeElement
? mediaEls.filter((el) => params.shouldIncludeElement?.(el))
: mediaEls.filter((el) => el.hasAttribute("data-start"));
const mediaClips: RuntimeMediaClip[] = [];
const videoClips: RuntimeMediaClip[] = [];
let maxMediaEnd = 0;
for (const el of mediaEls) {
for (const el of timedMediaEls) {
const start = params?.resolveStartSeconds
? params.resolveStartSeconds(el)
: Number.parseFloat(el.dataset.start ?? "0");
@@ -39,7 +44,8 @@ export function refreshRuntimeMediaCache(params?: {
Number.isFinite(rawRate) && rawRate > 0 ? Math.max(0.1, Math.min(5, rawRate)) : 1;
const loop = el.loop;
const sourceDuration = Number.isFinite(el.duration) && el.duration > 0 ? el.duration : null;
let duration = Number.parseFloat(el.dataset.duration ?? "");
let duration =
params?.resolveDurationSeconds?.(el) ?? Number.parseFloat(el.dataset.duration ?? "");
if ((!Number.isFinite(duration) || duration <= 0) && sourceDuration != null) {
// Effective duration accounts for playback rate:
// at 0.5x, a 10s source plays for 20s on the timeline
@@ -63,7 +69,7 @@ export function refreshRuntimeMediaCache(params?: {
if (el.tagName === "VIDEO") videoClips.push(clip);
if (Number.isFinite(end)) maxMediaEnd = Math.max(maxMediaEnd, end);
}
return { timedMediaEls: mediaEls, mediaClips, videoClips, maxMediaEnd };
return { timedMediaEls, mediaClips, videoClips, maxMediaEnd };
}
// Per-element timeline→media offset from the previous tick. Used to tell a