## Summary Fixes the double-voice issue in studio preview where narration plays twice with a drifting offset (measured 23ms → 80ms over a 28s clip). ## Root cause Two audio pipelines were playing the same source in parallel: 1. The iframe runtime played `<audio data-start>` elements via `syncRuntimeMedia` — the intended path. 2. `<hyperframes-player>` also created parent-frame `<audio>` copies on iframe load and auto-played them in response to every runtime `state` message. The existing `_muteIframeMedia` tried to silence the iframe copies via `el.volume = 0`, but `syncRuntimeMedia` re-asserts `el.volume` from `data-volume` every tick, so the mute never held. Studio seeks went through `__player.seek()`, which only updated the iframe timeline; parent copies kept their stale `currentTime` and drift compounded across seeks. Confirmed via agent-browser instrumentation on `factory-series-c-video`: - 6 `volumechange` events per play cycle (mute-fight signature) - Both copies audible at `volume=1`, offset growing 23ms → 80ms - Every seek widened the drift further PR #295 (v0.4.2) actually **made it audible** — before that, parent copies 404'd on the wrong URL and played silently. Fixing the URL exposed the latent double-playback. ## Fix Explicit single-owner audio ownership between `<hyperframes-player>` and the runtime. - **Default ownership is `runtime`**: iframe drives audible playback; parent proxies stay paused and inert. Matches every desktop / studio code path. No parent `play()`, no `volumechange` thrash. - **On `NotAllowedError`** from the runtime's `play()` attempt (autoplay-gated iframes), the runtime posts `media-autoplay-blocked` once. The player promotes to `parent` ownership: sends `set-media-output-muted: true` to the runtime, starts parent proxies, mirrors `currentTime` from state messages with a 150ms correction threshold. Two orthogonal mute channels replace the volume fight: | Channel | Purpose | |---|---| | `set-muted` | User's mute preference (existing, unchanged) | | `set-media-output-muted` | Internal ownership handoff (new) | `syncRuntimeMedia` now accepts `outputMuted` and asserts `el.muted = true` per active tick — sticky against sub-composition media that arrives mid-playback. Uses native `muted` (orthogonal to `volume`) so no other code path can clobber it. ## Why this shape - **Single owner, explicit transition.** No races, no tug-of-war. - **Probes reality, not device class.** We flip on an actual `NotAllowedError`, not on `matchMedia('(pointer: coarse)')` or user-agent sniffing. - **Uses `muted` instead of abusing `volume`.** `muted` is orthogonal to `volume`; `syncRuntimeMedia` doesn't write to it; author / user settings stay intact. - **Parent proxies become a thin mirror.** Under parent ownership, their `currentTime` is slaved to the iframe timeline via state messages — no independent drift. - **Backwards compatible.** Old runtimes without the new bridge action ignore the message; old players without the new message just get the previous behavior. - **Capture engine unaffected** — it bypasses both DOM pipelines and muxes audio from source files. ## Files changed - `packages/core/src/runtime/types.ts` — `set-media-output-muted` action + `media-autoplay-blocked` outbound message types. - `packages/core/src/runtime/state.ts` — `mediaOutputMuted` + `mediaAutoplayBlockedPosted` fields. - `packages/core/src/runtime/bridge.ts` — route new action to `onSetMediaOutputMuted`. - `packages/core/src/runtime/media.ts` — `outputMuted` param asserts `el.muted = true` per tick; `NotAllowedError` detection fires `onAutoplayBlocked`. - `packages/core/src/runtime/init.ts` — wire new bridge handler; coordinate with `set-muted`; post `media-autoplay-blocked` once per session. - `packages/player/src/hyperframes-player.ts` — `_audioOwner` state; delete `_muteIframeMedia`; `_promoteToParentProxy`; mirror parent `currentTime`; gate all parent play/pause/seek on ownership. ## Verified end-to-end with agent-browser on `factory-series-c-video` **Runtime ownership (default — desktop studio):** | | Before | After | |---|---|---| | `PARENT.play()` calls per play cycle | 1 | **0** | | iframe `volumechange` events | 6 | **0** | | Audible streams | 2 (drifting) | **1 (iframe)** | **Parent ownership (simulated autoplay block — direct message):** | | Value | |---|---| | iframe audio | `muted=true`, `volume=1` (untouched) | | parent audio | `muted=false`, `volume=1`, audible | | Parent ↔ iframe `currentTime` offset | ~6 ms steady state | | Offset > 150 ms | corrected by mirror sync | **Mobile path simulated with iPhone 14 emulation + injected `NotAllowedError` from iframe `<audio>.play()`:** Event timeline captured via agent-browser instrumentation: ``` t=0.0 ms IFRAME.play() called ← runtime attempts playback t=0.4 ms IFRAME.play() REJECTED: NotAllowedError ← simulated mobile gate t=0.4 ms →IFRAME bridge set-media-output-muted=true ← player promotes t=0.6 ms PARENT.play() called ← parent proxy starts t=0.8 ms ←IFRAME msg media-autoplay-blocked ← runtime signal t=1.0 ms PARENT.play() resolved ← audible t=1.3 ms IFRAME muted=true, volume=1 ← iframe silenced via native muted ``` Steady state at t=4 s under promoted parent ownership: | Element | currentTime | paused | volume | muted | |---|---|---|---|---| | Parent audio | 4.060 s | false | 1.0 | **false** (audible) | | Iframe audio | 4.068 s | false | 1.0 | **true** (silent) | **Offset: 8 ms**, single audible stream, orthogonal mute channel respected. ## Test plan - [x] `bunx vitest run` under `packages/core` — **467 / 467 pass** (incl. 4 new `media.test.ts` + 2 new `bridge.test.ts`) - [x] `bunx vitest run` under `packages/player` — **23 / 23 pass** (3 rewrites for new contract, 2 new for promotion flow) - [x] `bun run build` — all packages green - [x] Fresh preview + browser repro on `factory-series-c-video`: - [x] Runtime ownership: single audio stream, no drift - [x] Parent ownership promotion via direct `media-autoplay-blocked` message: iframe muted, parent audible - [x] iPhone 14 emulation + injected `NotAllowedError`: full promotion chain verified in ~1 s, 8 ms steady-state offset - [x] No `volumechange` thrash in either ownership mode - [x] One round of QA on a physical iOS / Android device before release — exercises real `NotAllowedError` path (expected behavior identical to simulation above)
Write HTML. Render video. Built for agents.
Hyperframes is an open-source video rendering framework that lets you create, preview, and render HTML-based video compositions — with first-class support for AI agents.
Quick Start
Option 1: With an AI coding agent (recommended)
Install the HyperFrames skills, then describe the video you want:
npx skills add heygen-com/hyperframes
This teaches your agent (Claude Code, Cursor, Gemini CLI, Codex) how to write correct compositions and GSAP animations. In Claude Code, the skills register as slash commands — invoke /hyperframes to author compositions, /hyperframes-cli for CLI commands, and /gsap for animation help.
Try it: example prompts
Copy any of these into your agent to get started. The /hyperframes prefix loads the skill context explicitly so you get correct output the first time.
Cold start — describe what you want:
Using
/hyperframes, create a 10-second product intro with a fade-in title, a background video, and background music.
Warm start — turn existing context into a video:
Take a look at this GitHub repo https://github.com/heygen-com/hyperframes and explain its uses and architecture to me using
/hyperframes.
Summarize the attached PDF into a 45-second pitch video using
/hyperframes.
Turn this CSV into an animated bar chart race using
/hyperframes.
Format-specific:
Make a 9:16 TikTok-style hook video about [topic] using
/hyperframes, with bouncy captions synced to a TTS narration.
Iterate — talk to the agent like a video editor:
Make the title 2x bigger, swap to dark mode, and add a fade-out at the end.
Add a lower third at 0:03 with my name and title.
The agent handles scaffolding, animation, and rendering. See the prompting guide for more patterns.
Option 2: Start a project manually
npx hyperframes init my-video
cd my-video
npx hyperframes preview # preview in browser (live reload)
npx hyperframes render # render to MP4
hyperframes init installs skills automatically, so you can hand off to your AI agent at any point.
Requirements: Node.js >= 22, FFmpeg
Why Hyperframes?
- HTML-native — compositions are HTML files with data attributes. No React, no proprietary DSL.
- AI-first — agents already speak HTML. The CLI is non-interactive by default, designed for agent-driven workflows.
- Deterministic rendering — same input = identical output. Built for automated pipelines.
- Frame Adapter pattern — bring your own animation runtime (GSAP, Lottie, CSS, Three.js).
How It Works
Define your video as HTML with data attributes:
<div id="stage" data-composition-id="my-video" data-start="0" data-width="1920" data-height="1080">
<video
id="clip-1"
data-start="0"
data-duration="5"
data-track-index="0"
src="intro.mp4"
muted
playsinline
></video>
<img
id="overlay"
class="clip"
data-start="2"
data-duration="3"
data-track-index="1"
src="logo.png"
/>
<audio
id="bg-music"
data-start="0"
data-duration="9"
data-track-index="2"
data-volume="0.5"
src="music.wav"
></audio>
</div>
Preview instantly in the browser. Render to MP4 locally or in Docker.
Catalog
50+ ready-to-use blocks and components — social overlays, shader transitions, data visualizations, and cinematic effects:
npx hyperframes add flash-through-white # shader transition
npx hyperframes add instagram-follow # social overlay
npx hyperframes add data-chart # animated chart
Browse the full catalog at hyperframes.heygen.com/catalog.
Documentation
Full documentation at hyperframes.heygen.com/introduction — Quickstart | Guides | API Reference | Catalog
Packages
| Package | Description |
|---|---|
hyperframes |
CLI — create, preview, lint, and render compositions |
@hyperframes/core |
Types, parsers, generators, linter, runtime, frame adapters |
@hyperframes/engine |
Seekable page-to-video capture engine (Puppeteer + FFmpeg) |
@hyperframes/producer |
Full rendering pipeline (capture + encode + audio mix) |
@hyperframes/studio |
Browser-based composition editor UI |
@hyperframes/player |
Embeddable <hyperframes-player> web component |
@hyperframes/shader-transitions |
WebGL shader transitions for compositions |
Skills
HyperFrames ships skills that teach AI agents framework-specific patterns that generic docs don't cover.
npx skills add heygen-com/hyperframes
| Skill | What it teaches |
|---|---|
hyperframes |
HTML composition authoring, captions, TTS, audio-reactive animation, transitions |
hyperframes-cli |
CLI commands: init, lint, preview, render, transcribe, tts, doctor |
hyperframes-registry |
Block and component installation via hyperframes add |
gsap |
GSAP animation API, timelines, easing, ScrollTrigger, plugins, React/Vue/Svelte, performance |
Contributing
See CONTRIBUTING.md for guidelines.
