mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
Miguel Ángel
f1a37400b4
## Summary Fixes three audio-sync defects in the studio preview plus a small UX improvement. All four land in one commit so the PR stays aligned with one bug fix per commit. ### 1\. Silent / very-late first play on slow-loading audio (`packages/core/src/runtime/media.ts`) `syncRuntimeMedia`'s old flow — when it hit `readyState < HAVE_FUTURE_DATA` — called `el.load()` and attached a `canplay` listener to retry `play()`. Two real problems, neither of which is "lost user activation" (the sync runs from a 50 ms `setInterval`, well outside any gesture window): - `bindMediaMetadataListeners` already sets `preload="auto"` and calls `el.load()` at runtime init. The sync's duplicate `el.load()` aborts that in-flight fetch and restarts from zero — on slow networks this delayed playback by seconds, which users perceived as "silent until a second click." - The `canplay` listener was racy: the event can fire between `load()` and `addEventListener`, leaving the element wedged. `HTMLMediaElement.play()` is already spec'd to queue playback until data arrives, so we can unconditionally call it. Drop the `readyState` gate, the redundant `load()`, and the `canplay` listener. Also dedup in-flight `play()` calls with a `WeakSet` (cleared on `playing`/`pause`/`error`) — without it the 50 ms poll fires 20–40 spurious calls per element during buffer, each silencing real `AbortError`/`NotAllowedError` diagnostics in the `.catch`. ### 2\. Audible stutter on rapid pause/play (`packages/core/src/runtime/media.ts`) The 0.3 s drift-seek threshold fired on nearly every toggle because pause/play ordering between timeline and media produces 0.1–0.4 s of transient drift. Each forced `el.currentTime = relTime` drops `readyState` and surfaces as a `waiting` event the user hears as a stutter. Threshold raised to 0.5 s. ### 3\. Skipped words on cold first play (`packages/core/src/runtime/media.ts`) Even with 0.5 s, drift grew past 0.5 s during initial buffering while the audio element was stuck at `currentTime = 0`. The old logic would then force-seek audio forward and the user missed the opening of the narration. Fix distinguishes drift that grows _gradually_ (buffer catch-up, ~16 ms/tick) from drift that _jumps_ in one tick (a scrub). Only jumps, first-tick clip activation, or catastrophic drift (>3 s) trigger a resync. Inline tradeoff note in code: strictly lip-synced dialogue would want a tighter threshold (~0.15 s) outside a 500 ms toggle window — deferred to a future PR. ### 4\. "Loading assets…" overlay in the studio preview (`packages/studio/src/player/components/Player.tsx`) Spinner while every timed `<audio>`/`<video>` has enough buffered data and every Lottie animation is loaded. Preserves the previous overlay state on cross-origin / transient-DOM catches so a brief access failure doesn't flicker, and logs `console.debug` when the 10 s safety cap trips so a stuck asset is diagnosable. Lottie readiness handles both `lottie-web` (`isLoaded`) and `@dotlottie/player-component` (`totalFrames > 0`), with an inline `@see` pointing to `packages/core/src/runtime/adapters/lottie.ts` so the two sites stay in sync. ## Verification - 456 core tests pass; 34 in `media.test.ts` cover synchronous play, preload nudge, play-request dedup, offset-jump vs gradual drift, first-tick hard-sync, catastrophic-drift safety valve, and inactive-clip baseline reset. - Full monorepo build green (`bun run build`), typecheck clean, lint/format clean. - End-to-end with agent-browser against a composition that uses a 50 s voiceover plus multiple sub-composition video clips. Four scenarios, all pass: | Scenario | Metric | Result | | --- | --- | --- | | Normal first play | Audio plays from click, smooth progression | ✅ | | Cold play (forced unbuffered) | First `play` event fires at `ct: 0` — no word-skip | ✅ | | Rapid pause/play (12 toggles) | `waiting` events: 1 (was 40+ bursts) | ✅ | | Scrub mid-playback | Lands exactly at target frame | ✅ | ## Files changed - `packages/core/src/runtime/media.ts` — unconditional synchronous `play()`; play-request dedup WeakSet; offset-jump-only drift correction; 0.5 s threshold; first-tick hard-sync; catastrophic-drift safety valve. - `packages/core/src/runtime/media.test.ts` — coverage for the above plus the gradual-drift cold-play case, scrub offset-jump, in-flight dedup, and inactive-clip baseline reset. - `packages/core/src/runtime/adapters/lottie.ts` — exported `isLottieAnimationLoaded` helper documenting the two supported player shapes. - `packages/studio/src/player/components/Player.tsx` — loading-assets overlay with cached-return catch, timeout debug log, and the Lottie readiness check. ## Follow-ups (deferred) - Tight-threshold short-window drift correction for lip-synced dialogue. - A perf-regression test that fails on `waiting`\-event resurgence. ## Test plan - [x] `hyperframes preview` a composition with audio, `Cmd+Shift+R`, click play immediately — audio starts from the very beginning, no skipped words. - [x] Rapidly pause/play the preview — audio stays smooth (no stutter, no `waiting` events). - [x] Cold-load a composition — "Loading assets…" overlay appears and disappears once media buffers. - [ ] Scrub the timeline mid-playback — audio follows the scrub, lands on frame.