fix(runtime): silent-first-play + loading overlay for preview (#293)

## Summary

Fixes three audio-sync defects in the studio preview plus a small UX improvement. All four land in one commit so the PR stays aligned with one bug fix per commit.

### 1\. Silent / very-late first play on slow-loading audio (`packages/core/src/runtime/media.ts`)

`syncRuntimeMedia`'s old flow — when it hit `readyState < HAVE_FUTURE_DATA` — called `el.load()` and attached a `canplay` listener to retry `play()`. Two real problems, neither of which is "lost user activation" (the sync runs from a 50 ms `setInterval`, well outside any gesture window):

- `bindMediaMetadataListeners` already sets `preload="auto"` and calls `el.load()` at runtime init. The sync's duplicate `el.load()` aborts that in-flight fetch and restarts from zero — on slow networks this delayed playback by seconds, which users perceived as "silent until a second click."
- The `canplay` listener was racy: the event can fire between `load()` and `addEventListener`, leaving the element wedged.

`HTMLMediaElement.play()` is already spec'd to queue playback until data arrives, so we can unconditionally call it. Drop the `readyState` gate, the redundant `load()`, and the `canplay` listener. Also dedup in-flight `play()` calls with a `WeakSet` (cleared on `playing`/`pause`/`error`) — without it the 50 ms poll fires 20–40 spurious calls per element during buffer, each silencing real `AbortError`/`NotAllowedError` diagnostics in the `.catch`.

### 2\. Audible stutter on rapid pause/play (`packages/core/src/runtime/media.ts`)

The 0.3 s drift-seek threshold fired on nearly every toggle because pause/play ordering between timeline and media produces 0.1–0.4 s of transient drift. Each forced `el.currentTime = relTime` drops `readyState` and surfaces as a `waiting` event the user hears as a stutter. Threshold raised to 0.5 s.

### 3\. Skipped words on cold first play (`packages/core/src/runtime/media.ts`)

Even with 0.5 s, drift grew past 0.5 s during initial buffering while the audio element was stuck at `currentTime = 0`. The old logic would then force-seek audio forward and the user missed the opening of the narration.

Fix distinguishes drift that grows _gradually_ (buffer catch-up, ~16 ms/tick) from drift that _jumps_ in one tick (a scrub). Only jumps, first-tick clip activation, or catastrophic drift (>3 s) trigger a resync. Inline tradeoff note in code: strictly lip-synced dialogue would want a tighter threshold (~0.15 s) outside a 500 ms toggle window — deferred to a future PR.

### 4\. "Loading assets…" overlay in the studio preview (`packages/studio/src/player/components/Player.tsx`)

Spinner while every timed `<audio>`/`<video>` has enough buffered data and every Lottie animation is loaded. Preserves the previous overlay state on cross-origin / transient-DOM catches so a brief access failure doesn't flicker, and logs `console.debug` when the 10 s safety cap trips so a stuck asset is diagnosable. Lottie readiness handles both `lottie-web` (`isLoaded`) and `@dotlottie/player-component` (`totalFrames > 0`), with an inline `@see` pointing to `packages/core/src/runtime/adapters/lottie.ts` so the two sites stay in sync.

## Verification

- 456 core tests pass; 34 in `media.test.ts` cover synchronous play, preload nudge, play-request dedup, offset-jump vs gradual drift, first-tick hard-sync, catastrophic-drift safety valve, and inactive-clip baseline reset.
- Full monorepo build green (`bun run build`), typecheck clean, lint/format clean.
- End-to-end with agent-browser against a composition that uses a 50 s voiceover plus multiple sub-composition video clips. Four scenarios, all pass:

| Scenario | Metric | Result |
| --- | --- | --- |
| Normal first play | Audio plays from click, smooth progression |  |
| Cold play (forced unbuffered) | First `play` event fires at `ct: 0` — no word-skip |  |
| Rapid pause/play (12 toggles) | `waiting` events: 1 (was 40+ bursts) |  |
| Scrub mid-playback | Lands exactly at target frame |  |

## Files changed

- `packages/core/src/runtime/media.ts` — unconditional synchronous `play()`; play-request dedup WeakSet; offset-jump-only drift correction; 0.5 s threshold; first-tick hard-sync; catastrophic-drift safety valve.
- `packages/core/src/runtime/media.test.ts` — coverage for the above plus the gradual-drift cold-play case, scrub offset-jump, in-flight dedup, and inactive-clip baseline reset.
- `packages/core/src/runtime/adapters/lottie.ts` — exported `isLottieAnimationLoaded` helper documenting the two supported player shapes.
- `packages/studio/src/player/components/Player.tsx` — loading-assets overlay with cached-return catch, timeout debug log, and the Lottie readiness check.

## Follow-ups (deferred)

- Tight-threshold short-window drift correction for lip-synced dialogue.
- A perf-regression test that fails on `waiting`\-event resurgence.

## Test plan

- [x] `hyperframes preview` a composition with audio, `Cmd+Shift+R`, click play immediately — audio starts from the very beginning, no skipped words.
- [x] Rapidly pause/play the preview — audio stays smooth (no stutter, no `waiting` events).
- [x] Cold-load a composition — "Loading assets…" overlay appears and disappears once media buffers.
- [ ] Scrub the timeline mid-playback — audio follows the scrub, lands on frame.
This commit is contained in:
Miguel Ángel
2026-04-16 21:51:06 +02:00
committed by GitHub
parent 87f4c77e2f
commit f1a37400b4
4 changed files with 308 additions and 53 deletions
+76 -22
View File
@@ -66,9 +66,26 @@ export function refreshRuntimeMediaCache(params?: {
return { timedMediaEls: mediaEls, mediaClips, videoClips, maxMediaEnd };
}
// Elements with a pending deferred play — prevents re-calling load()/addEventListener
// on every tick while the media is still buffering.
const pendingPlay = new WeakSet<HTMLMediaElement>();
// Per-element timeline→media offset from the previous tick. Used to tell a
// gradual drift (initial buffer catch-up, where offset grows ~16ms/tick) from
// a scrub (where offset jumps in one tick). Cleared when a clip becomes
// inactive so the next activation gets a hard resync on its first tick.
const lastOffset = new WeakMap<HTMLMediaElement, number>();
// Elements whose play() is in flight. The sync runs on a 50 ms poll and with
// a 12 s buffer that would fire 2040 spurious play() calls per element —
// noise in devtools and, worse, each `.catch(() => {})` would swallow a real
// AbortError / NotAllowedError that should surface. Cleared on the `playing`
// event (actual playback started) or on `pause`/`error` (state ended).
const playRequested = new WeakSet<HTMLMediaElement>();
function markPlayRequested(el: HTMLMediaElement): void {
if (playRequested.has(el)) return;
playRequested.add(el);
const clear = () => playRequested.delete(el);
el.addEventListener("playing", clear, { once: true });
el.addEventListener("pause", clear, { once: true });
el.addEventListener("error", clear, { once: true });
}
export function syncRuntimeMedia(params: {
clips: RuntimeMediaClip[];
@@ -97,36 +114,73 @@ export function syncRuntimeMedia(params: {
} catch {
// ignore unsupported playbackRate
}
if (Math.abs((el.currentTime || 0) - relTime) > 0.3) {
// Drift correction. Forcing `el.currentTime = relTime` every frame
// causes an audible seek+rebuffer hiccup (readyState drops briefly).
//
// We only want to correct drift that came from an *event* — an explicit
// user seek, a sub-composition activation, or a timeline jump — not
// drift that grew naturally from initial-buffer latency. Telling them
// apart by timing: scrubs move the timeline-to-media offset by seconds
// in a single tick; buffer catch-up grows the offset by ~one frame
// (<20ms) per tick.
//
// The first tick a clip is active we don't have a previous offset to
// compare against — treat that as a hard resync so sub-compositions
// with non-zero `mediaStart` land on the right frame.
//
// Tradeoff: the 3 s catastrophic-drift valve means an unnoticed
// steady-state drift can accumulate up to ~3 s before we correct.
// For music / motion graphics this is inaudible; for lip-synced
// dialogue it is not. If that becomes a target use case, switch to
// a short-window tight threshold (e.g. tighten to 0.15 s when the
// last play/pause transition was >500 ms ago).
const currentElTime = el.currentTime || 0;
const drift = Math.abs(currentElTime - relTime);
const offset = relTime - currentElTime;
const prevOffset = lastOffset.get(el);
lastOffset.set(el, offset);
const firstTickOfClip = prevOffset === undefined;
const offsetJumped = !firstTickOfClip && Math.abs(offset - prevOffset!) > 0.5;
const catastrophicDrift = drift > 3;
if (drift > 0.5 && (firstTickOfClip || offsetJumped || catastrophicDrift)) {
try {
el.currentTime = relTime;
} catch {
// ignore browser seek restrictions
}
}
if (params.playing && el.paused && !pendingPlay.has(el)) {
if (el.readyState >= HTMLMediaElement.HAVE_FUTURE_DATA) {
void el.play().catch(() => {});
} else {
pendingPlay.add(el);
if (el.preload !== "auto") el.preload = "auto";
el.addEventListener(
"canplay",
() => {
pendingPlay.delete(el);
if (!el.paused) return;
void el.play().catch(() => {});
},
{ once: true },
);
el.addEventListener("error", () => pendingPlay.delete(el), { once: true });
el.load();
}
if (params.playing && el.paused && !playRequested.has(el)) {
// `HTMLMediaElement.play()` is spec'd to queue playback and resolve
// once enough data is buffered, so we can unconditionally call it —
// no need to gate on `readyState` or defer to a `canplay` listener.
//
// The old `readyState < HAVE_FUTURE_DATA` branch called `el.load()`
// inside the listener, which *aborts* the in-flight fetch that
// `bindMediaMetadataListeners` already started at init time and
// restarts from zero. On slow networks this delayed playback by
// seconds. The canplay listener was also racey — the event could
// fire between `load()` and `addEventListener` attachment, wedging
// the element waiting for a callback that never came.
//
// preload="auto" is already set at bind time in init.ts; the
// re-assignment here is defensive for media elements that were
// inserted after the runtime bound its listeners.
if (el.preload !== "auto") el.preload = "auto";
markPlayRequested(el);
void el.play().catch(() => {
// If play() rejects — e.g. autoplay blocked, element removed
// mid-flight — drop the in-flight flag so a future sync tick can
// retry rather than getting stuck waiting for `playing`/`pause`.
playRequested.delete(el);
});
} else if (!params.playing && !el.paused) {
el.pause();
}
continue;
}
// Clip left its active window — drop the offset baseline so the next
// activation (e.g. re-entering a sub-composition) gets a hard resync.
lastOffset.delete(el);
if (!el.paused) el.pause();
}
}