Miguel Ángel c49181f1fa fix(player): address #298 review — tighter drift, dynamic proxies, ownership event (#307)
Follow-up to PR #298 addressing @jrusso1020's review. Each item below maps to a point in his comment.

## Significant

### 1\. Drift threshold 150 ms → 50 ms

_mirrorParentMediaTime_ was too loose for lip-synced talking-head content. ITU-R BT.1359 puts A/V perceptibility at ±45 ms; 150 ms sat well inside the "unacceptable" zone. Dropped to 50 ms, extracted as a static constant for clarity.

**Verified live on factory-series-c-video (agent-browser):** steady-state offset under parent ownership sampled five times over 400 ms = `[35.7, 33.5, 31.2, 27.2, 36.9]` ms — below the perceptibility floor. Before this PR the same measurement could drift up to 150 ms before correction.

### 2\. Dynamic sub-composition media proxies

Under parent ownership, a sub-composition that attaches a new `<audio data-start>` mid-playback was correctly silenced in the iframe (sticky `outputMuted`) but had no parent-frame counterpart to play → silent hole in the audio track.

Added a `MutationObserver` on the iframe body watching for `audio[data-start]` / `video[data-start]` additions. New elements are adopted through the same `_adoptIframeMedia` helper the initial scan uses, and if parent ownership is already active the new proxy gets its `currentTime` mirrored and `play()` called immediately (gated on `!this._paused`). Observer disconnects on iframe reload + component disconnect.

### 3\. `bridgeMuted` sticky in `syncRuntimeMedia`

The asymmetry James flagged: `outputMuted` was sticky per-tick, `bridgeMuted` was one-shot via `onSetMuted`. A sub-composition activating after a user mute would briefly play at author volume before the next bridge message.

`syncRuntimeMedia` now accepts `userMuted` and the per-clip loop uses a single combined `shouldMute` gate. One invariant, two inputs.

### 4\. Reset `_audioOwner` on iframe reload

The latch never cleared. On composition switch the player would stay in `parent` ownership against a fresh runtime that hadn't received `set-media-output-muted` and whose autoplay-blocked latch was clean — a brief double-audio window until the next `NotAllowedError` re-promoted (idempotently).

`_onIframeLoad` now resets `_audioOwner = "runtime"`, pauses any parent proxies, and disconnects the old MutationObserver before a fresh one attaches to the new document. If the player had been in `parent` ownership, a corresponding `audioownershipchange` event fires with `reason: "iframe-reload"`.

## Worth addressing

### 5\. Promotion → observable event + reason

Promotion was invisible. Added `CustomEvent("audioownershipchange", { detail: { owner, reason } })` fired on every owner transition. `reason` is either `"autoplay-blocked"` (promote → parent) or `"iframe-reload"` (reset → runtime). Gives host apps an SLO-ready signal for "% of sessions in parent ownership" without exposing internal state.

**Verified live:** dispatching a synthetic `media-autoplay-blocked` in the live studio produced `{ owner: "parent", reason: "autoplay-blocked" }` on the web component exactly once.

### 6\. Parent proxy play() rejection → `playbackerror` event

Previously swallowed silently. Now re-emitted as `CustomEvent("playbackerror", { detail: { source: "parent-proxy", error } })` so embedding apps can recover or fall back.

### 7\. Mobile verification on real hardware

Tested with a tunnel in a real iOS device.

## Test gaps (from review)

- `userMuted` stickiness (mirror of the existing `outputMuted` test).
- **OR invariant** between `outputMuted` and `userMuted` — explicit test that setting one false while the other is true keeps `el.muted === true`.
- **Contract pin:** `syncRuntimeMedia` fires `onAutoplayBlocked` on **every** rejection (no internal dedupe) — so a future refactor can't quietly move the latch and break the caller's posting logic.
- **Caller-side latch pattern:** a 5-rejection simulation with the init.ts-style wrapper posts exactly once.
- **`audioownershipchange`** **dispatch** on promotion + once per transition (no duplicate on idempotent re-promote).
- **Mid-playback promotion:** `_paused = false` at flip time fires `_playParentMedia` immediately.
- **`playbackerror`** **surface** on parent proxy rejection with the right `source` tag.

## Minor

- One-line comment on `_promoteToParentProxy` explaining the `postMessage` async race (the mute lands after ~one message-loop tick; the autoplay gate that triggered promotion keeps the iframe rejecting `play()` during that window, so the double-play bug doesn't reappear).

## What's good (from the review)

Kept as-is — noted for posterity:

- `muted` vs `volume` framing (orthogonal channels).
- Probing reality via `NotAllowedError` instead of `matchMedia('(pointer: coarse)')` / UA sniffing.
- Two orthogonal mute channels.
- Backwards compat (new actions / messages safely ignored by either side).

## Test results

- `packages/core/src/runtime/media.test.ts` — **42 tests pass** (+4 new: `userMuted` sticky, OR invariant, fires-every-rejection, caller-latch dedupe)
- `packages/core/src/runtime/bridge.test.ts` — **15 tests pass**
- `packages/player/src/hyperframes-player.test.ts` — **26 tests pass** (+3 new: `audioownershipchange` dispatch, mid-playback promotion, `playbackerror` surface)
- Typecheck green on `core` + `player`
- `tsup` build green on `core` / `player` / `cli`
- Live factory-series-c-video repro via agent-browser: runtime ownership still zero `volumechange` thrash, zero `PARENT.play()` calls; parent ownership measures 27–37 ms steady-state drift, well inside the 50 ms threshold.

## Test plan

- [x] Unit tests (83 total across touched files)
- [x] Typecheck clean
- [x] Build clean
- [x] Live studio repro on factory-series-c-video: runtime path unchanged, parent path drift tightened
- [x] `audioownershipchange` event fires with correct detail on synthetic autoplay block
- [x] Physical iOS / Android device verification (unchanged since #298)
2026-04-18 00:49:57 +02:00
2026-03-21 22:43:56 -07:00
2026-03-21 22:43:56 -07:00

HyperFrames

npm version npm downloads License Node.js

Write HTML. Render video. Built for agents.

HyperFrames demo — HTML code on the left transforms into a rendered video on the right

Hyperframes is an open-source video rendering framework that lets you create, preview, and render HTML-based video compositions — with first-class support for AI agents.

Quick Start

Install the HyperFrames skills, then describe the video you want:

npx skills add heygen-com/hyperframes

This teaches your agent (Claude Code, Cursor, Gemini CLI, Codex) how to write correct compositions and GSAP animations. In Claude Code, the skills register as slash commands — invoke /hyperframes to author compositions, /hyperframes-cli for CLI commands, and /gsap for animation help.

Try it: example prompts

Copy any of these into your agent to get started. The /hyperframes prefix loads the skill context explicitly so you get correct output the first time.

Cold start — describe what you want:

Using /hyperframes, create a 10-second product intro with a fade-in title, a background video, and background music.

Warm start — turn existing context into a video:

Take a look at this GitHub repo https://github.com/heygen-com/hyperframes and explain its uses and architecture to me using /hyperframes.

Summarize the attached PDF into a 45-second pitch video using /hyperframes.

Turn this CSV into an animated bar chart race using /hyperframes.

Format-specific:

Make a 9:16 TikTok-style hook video about [topic] using /hyperframes, with bouncy captions synced to a TTS narration.

Iterate — talk to the agent like a video editor:

Make the title 2x bigger, swap to dark mode, and add a fade-out at the end.

Add a lower third at 0:03 with my name and title.

The agent handles scaffolding, animation, and rendering. See the prompting guide for more patterns.

Option 2: Start a project manually

npx hyperframes init my-video
cd my-video
npx hyperframes preview      # preview in browser (live reload)
npx hyperframes render       # render to MP4

hyperframes init installs skills automatically, so you can hand off to your AI agent at any point.

Requirements: Node.js >= 22, FFmpeg

Why Hyperframes?

  • HTML-native — compositions are HTML files with data attributes. No React, no proprietary DSL.
  • AI-first — agents already speak HTML. The CLI is non-interactive by default, designed for agent-driven workflows.
  • Deterministic rendering — same input = identical output. Built for automated pipelines.
  • Frame Adapter pattern — bring your own animation runtime (GSAP, Lottie, CSS, Three.js).

How It Works

Define your video as HTML with data attributes:

<div id="stage" data-composition-id="my-video" data-start="0" data-width="1920" data-height="1080">
  <video
    id="clip-1"
    data-start="0"
    data-duration="5"
    data-track-index="0"
    src="intro.mp4"
    muted
    playsinline
  ></video>
  <img
    id="overlay"
    class="clip"
    data-start="2"
    data-duration="3"
    data-track-index="1"
    src="logo.png"
  />
  <audio
    id="bg-music"
    data-start="0"
    data-duration="9"
    data-track-index="2"
    data-volume="0.5"
    src="music.wav"
  ></audio>
</div>

Preview instantly in the browser. Render to MP4 locally or in Docker.

Catalog

50+ ready-to-use blocks and components — social overlays, shader transitions, data visualizations, and cinematic effects:

npx hyperframes add flash-through-white   # shader transition
npx hyperframes add instagram-follow      # social overlay
npx hyperframes add data-chart            # animated chart

Browse the full catalog at hyperframes.heygen.com/catalog.

Documentation

Full documentation at hyperframes.heygen.com/introduction — Quickstart | Guides | API Reference | Catalog

Packages

Package Description
hyperframes CLI — create, preview, lint, and render compositions
@hyperframes/core Types, parsers, generators, linter, runtime, frame adapters
@hyperframes/engine Seekable page-to-video capture engine (Puppeteer + FFmpeg)
@hyperframes/producer Full rendering pipeline (capture + encode + audio mix)
@hyperframes/studio Browser-based composition editor UI
@hyperframes/player Embeddable <hyperframes-player> web component
@hyperframes/shader-transitions WebGL shader transitions for compositions

Skills

HyperFrames ships skills that teach AI agents framework-specific patterns that generic docs don't cover.

npx skills add heygen-com/hyperframes
Skill What it teaches
hyperframes HTML composition authoring, captions, TTS, audio-reactive animation, transitions
hyperframes-cli CLI commands: init, lint, preview, render, transcribe, tts, doctor
hyperframes-registry Block and component installation via hyperframes add
gsap GSAP animation API, timelines, easing, ScrollTrigger, plugins, React/Vue/Svelte, performance

Contributing

See CONTRIBUTING.md for guidelines.

License

Apache 2.0

S
Description
Write HTML. Render video. Built for agents.
Readme
745 MiB
Languages
JavaScript 56.2%
TypeScript 43.6%