mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
fix(runtime): silent-first-play + loading overlay for preview (#293)
## Summary Fixes three audio-sync defects in the studio preview plus a small UX improvement. All four land in one commit so the PR stays aligned with one bug fix per commit. ### 1\. Silent / very-late first play on slow-loading audio (`packages/core/src/runtime/media.ts`) `syncRuntimeMedia`'s old flow — when it hit `readyState < HAVE_FUTURE_DATA` — called `el.load()` and attached a `canplay` listener to retry `play()`. Two real problems, neither of which is "lost user activation" (the sync runs from a 50 ms `setInterval`, well outside any gesture window): - `bindMediaMetadataListeners` already sets `preload="auto"` and calls `el.load()` at runtime init. The sync's duplicate `el.load()` aborts that in-flight fetch and restarts from zero — on slow networks this delayed playback by seconds, which users perceived as "silent until a second click." - The `canplay` listener was racy: the event can fire between `load()` and `addEventListener`, leaving the element wedged. `HTMLMediaElement.play()` is already spec'd to queue playback until data arrives, so we can unconditionally call it. Drop the `readyState` gate, the redundant `load()`, and the `canplay` listener. Also dedup in-flight `play()` calls with a `WeakSet` (cleared on `playing`/`pause`/`error`) — without it the 50 ms poll fires 20–40 spurious calls per element during buffer, each silencing real `AbortError`/`NotAllowedError` diagnostics in the `.catch`. ### 2\. Audible stutter on rapid pause/play (`packages/core/src/runtime/media.ts`) The 0.3 s drift-seek threshold fired on nearly every toggle because pause/play ordering between timeline and media produces 0.1–0.4 s of transient drift. Each forced `el.currentTime = relTime` drops `readyState` and surfaces as a `waiting` event the user hears as a stutter. Threshold raised to 0.5 s. ### 3\. Skipped words on cold first play (`packages/core/src/runtime/media.ts`) Even with 0.5 s, drift grew past 0.5 s during initial buffering while the audio element was stuck at `currentTime = 0`. The old logic would then force-seek audio forward and the user missed the opening of the narration. Fix distinguishes drift that grows _gradually_ (buffer catch-up, ~16 ms/tick) from drift that _jumps_ in one tick (a scrub). Only jumps, first-tick clip activation, or catastrophic drift (>3 s) trigger a resync. Inline tradeoff note in code: strictly lip-synced dialogue would want a tighter threshold (~0.15 s) outside a 500 ms toggle window — deferred to a future PR. ### 4\. "Loading assets…" overlay in the studio preview (`packages/studio/src/player/components/Player.tsx`) Spinner while every timed `<audio>`/`<video>` has enough buffered data and every Lottie animation is loaded. Preserves the previous overlay state on cross-origin / transient-DOM catches so a brief access failure doesn't flicker, and logs `console.debug` when the 10 s safety cap trips so a stuck asset is diagnosable. Lottie readiness handles both `lottie-web` (`isLoaded`) and `@dotlottie/player-component` (`totalFrames > 0`), with an inline `@see` pointing to `packages/core/src/runtime/adapters/lottie.ts` so the two sites stay in sync. ## Verification - 456 core tests pass; 34 in `media.test.ts` cover synchronous play, preload nudge, play-request dedup, offset-jump vs gradual drift, first-tick hard-sync, catastrophic-drift safety valve, and inactive-clip baseline reset. - Full monorepo build green (`bun run build`), typecheck clean, lint/format clean. - End-to-end with agent-browser against a composition that uses a 50 s voiceover plus multiple sub-composition video clips. Four scenarios, all pass: | Scenario | Metric | Result | | --- | --- | --- | | Normal first play | Audio plays from click, smooth progression | ✅ | | Cold play (forced unbuffered) | First `play` event fires at `ct: 0` — no word-skip | ✅ | | Rapid pause/play (12 toggles) | `waiting` events: 1 (was 40+ bursts) | ✅ | | Scrub mid-playback | Lands exactly at target frame | ✅ | ## Files changed - `packages/core/src/runtime/media.ts` — unconditional synchronous `play()`; play-request dedup WeakSet; offset-jump-only drift correction; 0.5 s threshold; first-tick hard-sync; catastrophic-drift safety valve. - `packages/core/src/runtime/media.test.ts` — coverage for the above plus the gradual-drift cold-play case, scrub offset-jump, in-flight dedup, and inactive-clip baseline reset. - `packages/core/src/runtime/adapters/lottie.ts` — exported `isLottieAnimationLoaded` helper documenting the two supported player shapes. - `packages/studio/src/player/components/Player.tsx` — loading-assets overlay with cached-return catch, timeout debug log, and the Lottie readiness check. ## Follow-ups (deferred) - Tight-threshold short-window drift correction for lip-synced dialogue. - A perf-regression test that fails on `waiting`\-event resurgence. ## Test plan - [x] `hyperframes preview` a composition with audio, `Cmd+Shift+R`, click play immediately — audio starts from the very beginning, no skipped words. - [x] Rapidly pause/play the preview — audio stays smooth (no stutter, no `waiting` events). - [x] Cold-load a composition — "Loading assets…" overlay appears and disappears once media buffers. - [ ] Scrub the timeline mid-playback — audio follows the scrub, lands on frame.
This commit is contained in:
Miguel Ángel
@@ -1,4 +1,4 @@
|
||||
import { forwardRef, useRef } from "react";
|
||||
import { forwardRef, useRef, useState } from "react";
|
||||
import { useMountEffect } from "../../hooks/useMountEffect";
|
||||
import type { HyperframesPlayer } from "@hyperframes/player";
|
||||
// NOTE: importing "@hyperframes/player" registers a class extending HTMLElement
|
||||
@@ -12,6 +12,63 @@ interface PlayerProps {
|
||||
portrait?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Readiness check for a Lottie animation instance. Duck-types both supported
|
||||
* player shapes:
|
||||
*
|
||||
* - `lottie-web` exposes a boolean `isLoaded` on `AnimationItem`.
|
||||
* - `@dotlottie/player-component` doesn't; we infer readiness from
|
||||
* `totalFrames > 0` since that value is only populated once the animation
|
||||
* JSON has been parsed.
|
||||
*
|
||||
* Kept in sync with the runtime adapter's own checks in
|
||||
* `@hyperframes/core/runtime/adapters/lottie.ts` — that module would be a
|
||||
* more canonical home for the helper, but importing from the core package's
|
||||
* root index pulls Node-only submodules (path, url) into this browser bundle
|
||||
* and breaks Vite. If the helper grows, split a browser-safe submodule
|
||||
* export in core and switch this to import it.
|
||||
*/
|
||||
function isLottieAnimationReady(anim: unknown): boolean {
|
||||
if (typeof anim !== "object" || anim === null) return true;
|
||||
const maybe = anim as { isLoaded?: boolean; totalFrames?: number };
|
||||
if (maybe.isLoaded === true) return true;
|
||||
if (typeof maybe.totalFrames === "number" && maybe.totalFrames > 0) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
// Assets are considered ready when every `<video>`/`<audio>` has enough data
|
||||
// to play through without buffering, and every registered Lottie animation has
|
||||
// finished loading.
|
||||
//
|
||||
// Returns whichever value was returned last on cross-origin / transient DOM
|
||||
// races so a brief access failure (e.g. an iframe that just swapped src)
|
||||
// doesn't flicker the overlay state — we keep showing whatever was most
|
||||
// recently true.
|
||||
function hasUnloadedAssets(iframe: HTMLIFrameElement, lastResult: boolean): boolean {
|
||||
try {
|
||||
const win = iframe.contentWindow as unknown as (Window & { __hfLottie?: unknown[] }) | null;
|
||||
const doc = iframe.contentDocument;
|
||||
if (!win || !doc) return lastResult;
|
||||
|
||||
for (const el of doc.querySelectorAll("video, audio")) {
|
||||
if (el instanceof HTMLMediaElement && el.readyState < HTMLMediaElement.HAVE_FUTURE_DATA) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
const lotties = win.__hfLottie;
|
||||
if (lotties?.length) {
|
||||
for (const anim of lotties) {
|
||||
if (!isLottieAnimationReady(anim)) return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
} catch {
|
||||
return lastResult;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Renders a composition preview using the <hyperframes-player> web component.
|
||||
*
|
||||
@@ -24,6 +81,8 @@ export const Player = forwardRef<HTMLIFrameElement, PlayerProps>(
|
||||
({ projectId, directUrl, onLoad, portrait }, ref) => {
|
||||
const containerRef = useRef<HTMLDivElement>(null);
|
||||
const loadCountRef = useRef(0);
|
||||
const assetPollRef = useRef<ReturnType<typeof setInterval> | null>(null);
|
||||
const [assetsLoading, setAssetsLoading] = useState(false);
|
||||
|
||||
useMountEffect(() => {
|
||||
const container = containerRef.current;
|
||||
@@ -72,12 +131,48 @@ export const Player = forwardRef<HTMLIFrameElement, PlayerProps>(
|
||||
container.addEventListener("animationend", onEnd, { once: true });
|
||||
}
|
||||
onLoad();
|
||||
|
||||
// Show a loading overlay until every `<video>`/`<audio>` and Lottie
|
||||
// asset is ready. Without this users can click play before audio has
|
||||
// buffered — the runtime is resilient (queued play() resolves once
|
||||
// data arrives), but the overlay communicates why the first frame
|
||||
// or first audio beat may lag.
|
||||
//
|
||||
// Poll with a 10 s safety cap (100 ticks × 100 ms). If the cap
|
||||
// trips we hide the overlay so the UI doesn't appear stuck forever,
|
||||
// but we log a debug warning so the case is diagnosable — a long
|
||||
// cold video or a broken asset can legitimately exceed 10 s on a
|
||||
// slow network.
|
||||
if (assetPollRef.current) clearInterval(assetPollRef.current);
|
||||
let lastUnloaded = hasUnloadedAssets(iframe, false);
|
||||
if (lastUnloaded) {
|
||||
setAssetsLoading(true);
|
||||
let attempts = 0;
|
||||
assetPollRef.current = setInterval(() => {
|
||||
attempts += 1;
|
||||
lastUnloaded = hasUnloadedAssets(iframe, lastUnloaded);
|
||||
if (!lastUnloaded || attempts > 100) {
|
||||
if (assetPollRef.current) clearInterval(assetPollRef.current);
|
||||
assetPollRef.current = null;
|
||||
setAssetsLoading(false);
|
||||
if (lastUnloaded) {
|
||||
console.debug(
|
||||
"[Player] Asset-loading overlay timed out after 10s; hiding anyway. Check network or asset integrity.",
|
||||
);
|
||||
}
|
||||
}
|
||||
}, 100);
|
||||
} else {
|
||||
setAssetsLoading(false);
|
||||
}
|
||||
};
|
||||
iframe.addEventListener("load", handleLoad);
|
||||
|
||||
cleanup = () => {
|
||||
iframe.removeEventListener("load", handleLoad);
|
||||
player.removeEventListener("click", preventToggle, { capture: true });
|
||||
if (assetPollRef.current) clearInterval(assetPollRef.current);
|
||||
assetPollRef.current = null;
|
||||
container.removeChild(player);
|
||||
// Clear the forwarded ref
|
||||
if (typeof ref === "function") {
|
||||
@@ -95,10 +190,15 @@ export const Player = forwardRef<HTMLIFrameElement, PlayerProps>(
|
||||
});
|
||||
|
||||
return (
|
||||
<div
|
||||
ref={containerRef}
|
||||
className="w-full h-full max-w-full max-h-full overflow-hidden bg-black flex items-center justify-center"
|
||||
/>
|
||||
<div className="relative w-full h-full max-w-full max-h-full overflow-hidden bg-black flex items-center justify-center">
|
||||
<div ref={containerRef} className="w-full h-full" />
|
||||
{assetsLoading && (
|
||||
<div className="absolute inset-0 bg-black/80 flex flex-col items-center justify-center z-20 pointer-events-none">
|
||||
<div className="w-8 h-8 border-2 border-white/20 border-t-white rounded-full animate-spin" />
|
||||
<span className="text-white/60 text-xs mt-3">Loading assets…</span>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
Reference in New Issue
Block a user