mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-10 22:20:14 +00:00
fix(runtime): silent-first-play + loading overlay for preview (#293)
## Summary Fixes three audio-sync defects in the studio preview plus a small UX improvement. All four land in one commit so the PR stays aligned with one bug fix per commit. ### 1\. Silent / very-late first play on slow-loading audio (`packages/core/src/runtime/media.ts`) `syncRuntimeMedia`'s old flow — when it hit `readyState < HAVE_FUTURE_DATA` — called `el.load()` and attached a `canplay` listener to retry `play()`. Two real problems, neither of which is "lost user activation" (the sync runs from a 50 ms `setInterval`, well outside any gesture window): - `bindMediaMetadataListeners` already sets `preload="auto"` and calls `el.load()` at runtime init. The sync's duplicate `el.load()` aborts that in-flight fetch and restarts from zero — on slow networks this delayed playback by seconds, which users perceived as "silent until a second click." - The `canplay` listener was racy: the event can fire between `load()` and `addEventListener`, leaving the element wedged. `HTMLMediaElement.play()` is already spec'd to queue playback until data arrives, so we can unconditionally call it. Drop the `readyState` gate, the redundant `load()`, and the `canplay` listener. Also dedup in-flight `play()` calls with a `WeakSet` (cleared on `playing`/`pause`/`error`) — without it the 50 ms poll fires 20–40 spurious calls per element during buffer, each silencing real `AbortError`/`NotAllowedError` diagnostics in the `.catch`. ### 2\. Audible stutter on rapid pause/play (`packages/core/src/runtime/media.ts`) The 0.3 s drift-seek threshold fired on nearly every toggle because pause/play ordering between timeline and media produces 0.1–0.4 s of transient drift. Each forced `el.currentTime = relTime` drops `readyState` and surfaces as a `waiting` event the user hears as a stutter. Threshold raised to 0.5 s. ### 3\. Skipped words on cold first play (`packages/core/src/runtime/media.ts`) Even with 0.5 s, drift grew past 0.5 s during initial buffering while the audio element was stuck at `currentTime = 0`. The old logic would then force-seek audio forward and the user missed the opening of the narration. Fix distinguishes drift that grows _gradually_ (buffer catch-up, ~16 ms/tick) from drift that _jumps_ in one tick (a scrub). Only jumps, first-tick clip activation, or catastrophic drift (>3 s) trigger a resync. Inline tradeoff note in code: strictly lip-synced dialogue would want a tighter threshold (~0.15 s) outside a 500 ms toggle window — deferred to a future PR. ### 4\. "Loading assets…" overlay in the studio preview (`packages/studio/src/player/components/Player.tsx`) Spinner while every timed `<audio>`/`<video>` has enough buffered data and every Lottie animation is loaded. Preserves the previous overlay state on cross-origin / transient-DOM catches so a brief access failure doesn't flicker, and logs `console.debug` when the 10 s safety cap trips so a stuck asset is diagnosable. Lottie readiness handles both `lottie-web` (`isLoaded`) and `@dotlottie/player-component` (`totalFrames > 0`), with an inline `@see` pointing to `packages/core/src/runtime/adapters/lottie.ts` so the two sites stay in sync. ## Verification - 456 core tests pass; 34 in `media.test.ts` cover synchronous play, preload nudge, play-request dedup, offset-jump vs gradual drift, first-tick hard-sync, catastrophic-drift safety valve, and inactive-clip baseline reset. - Full monorepo build green (`bun run build`), typecheck clean, lint/format clean. - End-to-end with agent-browser against a composition that uses a 50 s voiceover plus multiple sub-composition video clips. Four scenarios, all pass: | Scenario | Metric | Result | | --- | --- | --- | | Normal first play | Audio plays from click, smooth progression | ✅ | | Cold play (forced unbuffered) | First `play` event fires at `ct: 0` — no word-skip | ✅ | | Rapid pause/play (12 toggles) | `waiting` events: 1 (was 40+ bursts) | ✅ | | Scrub mid-playback | Lands exactly at target frame | ✅ | ## Files changed - `packages/core/src/runtime/media.ts` — unconditional synchronous `play()`; play-request dedup WeakSet; offset-jump-only drift correction; 0.5 s threshold; first-tick hard-sync; catastrophic-drift safety valve. - `packages/core/src/runtime/media.test.ts` — coverage for the above plus the gradual-drift cold-play case, scrub offset-jump, in-flight dedup, and inactive-clip baseline reset. - `packages/core/src/runtime/adapters/lottie.ts` — exported `isLottieAnimationLoaded` helper documenting the two supported player shapes. - `packages/studio/src/player/components/Player.tsx` — loading-assets overlay with cached-return catch, timeout debug log, and the Lottie readiness check. ## Follow-ups (deferred) - Tight-threshold short-window drift correction for lip-synced dialogue. - A perf-regression test that fails on `waiting`\-event resurgence. ## Test plan - [x] `hyperframes preview` a composition with audio, `Cmd+Shift+R`, click play immediately — audio starts from the very beginning, no skipped words. - [x] Rapidly pause/play the preview — audio stays smooth (no stutter, no `waiting` events). - [x] Cold-load a composition — "Loading assets…" overlay appears and disappears once media buffers. - [ ] Scrub the timeline mid-playback — audio follows the scrub, lands on frame.
This commit is contained in:
@@ -66,9 +66,26 @@ export function refreshRuntimeMediaCache(params?: {
|
||||
return { timedMediaEls: mediaEls, mediaClips, videoClips, maxMediaEnd };
|
||||
}
|
||||
|
||||
// Elements with a pending deferred play — prevents re-calling load()/addEventListener
|
||||
// on every tick while the media is still buffering.
|
||||
const pendingPlay = new WeakSet<HTMLMediaElement>();
|
||||
// Per-element timeline→media offset from the previous tick. Used to tell a
|
||||
// gradual drift (initial buffer catch-up, where offset grows ~16ms/tick) from
|
||||
// a scrub (where offset jumps in one tick). Cleared when a clip becomes
|
||||
// inactive so the next activation gets a hard resync on its first tick.
|
||||
const lastOffset = new WeakMap<HTMLMediaElement, number>();
|
||||
|
||||
// Elements whose play() is in flight. The sync runs on a 50 ms poll and with
|
||||
// a 1–2 s buffer that would fire 20–40 spurious play() calls per element —
|
||||
// noise in devtools and, worse, each `.catch(() => {})` would swallow a real
|
||||
// AbortError / NotAllowedError that should surface. Cleared on the `playing`
|
||||
// event (actual playback started) or on `pause`/`error` (state ended).
|
||||
const playRequested = new WeakSet<HTMLMediaElement>();
|
||||
function markPlayRequested(el: HTMLMediaElement): void {
|
||||
if (playRequested.has(el)) return;
|
||||
playRequested.add(el);
|
||||
const clear = () => playRequested.delete(el);
|
||||
el.addEventListener("playing", clear, { once: true });
|
||||
el.addEventListener("pause", clear, { once: true });
|
||||
el.addEventListener("error", clear, { once: true });
|
||||
}
|
||||
|
||||
export function syncRuntimeMedia(params: {
|
||||
clips: RuntimeMediaClip[];
|
||||
@@ -97,36 +114,73 @@ export function syncRuntimeMedia(params: {
|
||||
} catch {
|
||||
// ignore unsupported playbackRate
|
||||
}
|
||||
if (Math.abs((el.currentTime || 0) - relTime) > 0.3) {
|
||||
// Drift correction. Forcing `el.currentTime = relTime` every frame
|
||||
// causes an audible seek+rebuffer hiccup (readyState drops briefly).
|
||||
//
|
||||
// We only want to correct drift that came from an *event* — an explicit
|
||||
// user seek, a sub-composition activation, or a timeline jump — not
|
||||
// drift that grew naturally from initial-buffer latency. Telling them
|
||||
// apart by timing: scrubs move the timeline-to-media offset by seconds
|
||||
// in a single tick; buffer catch-up grows the offset by ~one frame
|
||||
// (<20ms) per tick.
|
||||
//
|
||||
// The first tick a clip is active we don't have a previous offset to
|
||||
// compare against — treat that as a hard resync so sub-compositions
|
||||
// with non-zero `mediaStart` land on the right frame.
|
||||
//
|
||||
// Tradeoff: the 3 s catastrophic-drift valve means an unnoticed
|
||||
// steady-state drift can accumulate up to ~3 s before we correct.
|
||||
// For music / motion graphics this is inaudible; for lip-synced
|
||||
// dialogue it is not. If that becomes a target use case, switch to
|
||||
// a short-window tight threshold (e.g. tighten to 0.15 s when the
|
||||
// last play/pause transition was >500 ms ago).
|
||||
const currentElTime = el.currentTime || 0;
|
||||
const drift = Math.abs(currentElTime - relTime);
|
||||
const offset = relTime - currentElTime;
|
||||
const prevOffset = lastOffset.get(el);
|
||||
lastOffset.set(el, offset);
|
||||
const firstTickOfClip = prevOffset === undefined;
|
||||
const offsetJumped = !firstTickOfClip && Math.abs(offset - prevOffset!) > 0.5;
|
||||
const catastrophicDrift = drift > 3;
|
||||
if (drift > 0.5 && (firstTickOfClip || offsetJumped || catastrophicDrift)) {
|
||||
try {
|
||||
el.currentTime = relTime;
|
||||
} catch {
|
||||
// ignore browser seek restrictions
|
||||
}
|
||||
}
|
||||
if (params.playing && el.paused && !pendingPlay.has(el)) {
|
||||
if (el.readyState >= HTMLMediaElement.HAVE_FUTURE_DATA) {
|
||||
void el.play().catch(() => {});
|
||||
} else {
|
||||
pendingPlay.add(el);
|
||||
if (el.preload !== "auto") el.preload = "auto";
|
||||
el.addEventListener(
|
||||
"canplay",
|
||||
() => {
|
||||
pendingPlay.delete(el);
|
||||
if (!el.paused) return;
|
||||
void el.play().catch(() => {});
|
||||
},
|
||||
{ once: true },
|
||||
);
|
||||
el.addEventListener("error", () => pendingPlay.delete(el), { once: true });
|
||||
el.load();
|
||||
}
|
||||
if (params.playing && el.paused && !playRequested.has(el)) {
|
||||
// `HTMLMediaElement.play()` is spec'd to queue playback and resolve
|
||||
// once enough data is buffered, so we can unconditionally call it —
|
||||
// no need to gate on `readyState` or defer to a `canplay` listener.
|
||||
//
|
||||
// The old `readyState < HAVE_FUTURE_DATA` branch called `el.load()`
|
||||
// inside the listener, which *aborts* the in-flight fetch that
|
||||
// `bindMediaMetadataListeners` already started at init time and
|
||||
// restarts from zero. On slow networks this delayed playback by
|
||||
// seconds. The canplay listener was also racey — the event could
|
||||
// fire between `load()` and `addEventListener` attachment, wedging
|
||||
// the element waiting for a callback that never came.
|
||||
//
|
||||
// preload="auto" is already set at bind time in init.ts; the
|
||||
// re-assignment here is defensive for media elements that were
|
||||
// inserted after the runtime bound its listeners.
|
||||
if (el.preload !== "auto") el.preload = "auto";
|
||||
markPlayRequested(el);
|
||||
void el.play().catch(() => {
|
||||
// If play() rejects — e.g. autoplay blocked, element removed
|
||||
// mid-flight — drop the in-flight flag so a future sync tick can
|
||||
// retry rather than getting stuck waiting for `playing`/`pause`.
|
||||
playRequested.delete(el);
|
||||
});
|
||||
} else if (!params.playing && !el.paused) {
|
||||
el.pause();
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// Clip left its active window — drop the offset baseline so the next
|
||||
// activation (e.g. re-entering a sub-composition) gets a hard resync.
|
||||
lastOffset.delete(el);
|
||||
if (!el.paused) el.pause();
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user