## Summary Adds **scenario 06: live-playback parity** — the third and final tranche of the P0-1 perf-test buildout (`p0-1a` infra → `p0-1b` fps/scrub/drift → this). The scenario plays the `gsap-heavy` fixture, freezes it mid-animation, screenshots the live frame, then synchronously seeks the same player back to that exact timestamp and screenshots the reference. The two PNGs are diffed with `ffmpeg -lavfi ssim` and the resulting average SSIM is emitted as `parity_ssim_min`. Baseline gate: **SSIM ≥ 0.95**. This pins the player's two frame-production paths (the runtime's animation loop vs. `_trySyncSeek`) to each other visually, so any future drift between scrub and playback fails CI instead of silently shipping. ## Motivation `<hyperframes-player>` produces frames two different ways: 1. **Live playback** — the runtime's animation loop advances the GSAP timeline frame-by-frame. 2. **Synchronous seek** (`_trySyncSeek`, landed in #397) — for same-origin embeds, the player calls into the iframe runtime's `seek()` directly and asks for a specific time. These paths must agree. If they don't — different rounding, different sub-frame sampling, different state ordering — scrubbing a paused composition shows different pixels than a paused-during-playback frame at the same time. That's a class of bug that only surfaces visually, never in unit tests, and only at specific timestamps where many things are mid-flight. `gsap-heavy` is a 10s composition with 60 tiles each running a staggered 4s out-and-back tween. At t=5.0s a large fraction of those tiles are mid-flight, so the rendered frame has many distinct, position-sensitive pixels — the worst-case input for any sub-frame disagreement. If the two paths produce identical pixels here, they'll produce identical pixels everywhere that matters. ## What changed - **`packages/player/tests/perf/scenarios/06-parity.ts`** — new scenario (~340 lines). Owns capture, seek, screenshot, SSIM, artifact persistence, and aggregation. - **`packages/player/tests/perf/index.ts`** — register `parity` as a scenario id, default-runs = 3, dispatch to `runParity`, include in the default scenario list. - **`packages/player/tests/perf/perf-gate.ts`** — extend `PerfBaseline` with `paritySsimMin`. - **`packages/player/tests/perf/baseline.json`** — `paritySsimMin: 0.95`. - **`.github/workflows/player-perf.yml`** — add a `parity` shard (3 runs) to the matrix alongside `load` / `fps` / `scrub` / `drift`. ## How the scenario works The hard part is making the two captures land on the *exact same timestamp* without trusting `postMessage` round-trips or arbitrary `setTimeout` settling. 1. **Install an iframe-side rAF watcher** before issuing `play()`. The watcher polls `__player.getTime()` every animation frame and, the first time `getTime() >= 5.0`, calls `__player.pause()` *from inside the same rAF tick*. `pause()` is synchronous (it calls `timeline.pause()`), so the timeline freezes at exactly that `getTime()` value with no postMessage round-trip. The watcher's Promise resolves with that frozen value as the canonical `T_actual` for the run. 2. **Confirm `isPlaying() === true`** via `frame.waitForFunction` before awaiting the watcher. Without this, the test can hang if `play()` hasn't kicked the timeline yet. 3. **Wait for paint** — two `requestAnimationFrame` ticks on the host page. The first flushes pending style/layout, the second guarantees a painted compositor commit. Same paint-settlement pattern as `packages/producer/src/parity-harness.ts`. 4. **Screenshot the live frame** — `page.screenshot({ type: "png" })`. 5. **Synchronously seek to `T_actual`** — call `el.seek(capturedTime)` on the host page. The player's public `seek()` calls `_trySyncSeek` which (same-origin) calls `__player.seek()` synchronously, so no postMessage await is needed. The runtime's deterministic `seek()` rebuilds frame state at exactly the requested time. 6. **Wait for paint** again, screenshot the reference frame. 7. **Diff with ffmpeg** — `ffmpeg -hide_banner -i reference.png -i actual.png -lavfi ssim -f null -`. ffmpeg writes per-channel + overall SSIM to stderr; we parse the `All:` value, clamp at 1.0 (ffmpeg occasionally reports 1.000001 on identical inputs), and treat it as the run's score. 8. **Persist artifacts** under `tests/perf/results/parity/run-N/` (`actual.png`, `reference.png`, `captured-time.txt`) so CI can upload them and so a failed run is locally reproducible. Directory is already gitignored via the existing `packages/player/tests/perf/results/` rule. ### Aggregation `min()` across runs, **not** mean. We want the *worst observed* parity to pass the gate so a single bad run can't get masked by averaging. Both per-run scores and the aggregate are logged. ### Output metric | name | direction | baseline | |-------------------|------------------|----------------------| | `parity_ssim_min` | higher-is-better | `paritySsimMin: 0.95` | With deterministic rendering enabled in the runner, identical pixels produce SSIM very close to 1.0; the 0.95 threshold leaves headroom for legitimate fixture-level noise (font hinting, GPU compositor variance) while still catching any real disagreement between the two paths. ## Test plan - `bun run player:perf -- --scenarios=parity --runs=3` locally on `gsap-heavy` — passes with SSIM ≈ 0.999 across all 3 runs. - Inspected `results/parity/run-1/actual.png` and `reference.png` side-by-side — visually identical. - Inspected `captured-time.txt` to confirm `T_actual` lands just past 5.0s (within one frame). - Sanity test: temporarily forced a 1-frame offset between live and reference capture; SSIM dropped well below 0.95 as expected, confirming the threshold catches real drift. - CI: `parity` shard added alongside the existing `load` / `fps` / `scrub` / `drift` shards; same `measure`-mode / artifact-upload / aggregation flow. - `bunx oxlint` and `bunx oxfmt --check` clean on the new scenario. ## Stack This is the top of the perf stack: 1. #393 `perf/x-1-emit-performance-metric` — performance.measure() emission 2. #394 `perf/p1-1-share-player-styles-via-adopted-stylesheets` — adopted stylesheets 3. #395 `perf/p1-2-scope-media-mutation-observer` — scoped MutationObserver 4. #396 `perf/p1-4-coalesce-mirror-parent-media-time` — coalesce currentTime writes 5. #397 `perf/p3-1-sync-seek-same-origin` — synchronous seek path (the path this PR pins) 6. #398 `perf/p3-2-srcdoc-composition-switching` — srcdoc switching 7. #399 `perf/p0-1a-perf-test-infra` — server, runner, perf-gate, CI 8. #400 `perf/p0-1b-perf-tests-for-fps-scrub-drift` — fps / scrub / drift scenarios 9. **#401 `perf/p0-1c-live-playback-parity-test` ← you are here** With this PR landed the perf harness covers all five proposal scenarios: `load`, `fps`, `scrub`, `drift`, `parity`.
Write HTML. Render video. Built for agents.
Hyperframes is an open-source video rendering framework that lets you create, preview, and render HTML-based video compositions — with first-class support for AI agents.
Quick Start
Option 1: With an AI coding agent (recommended)
Install the HyperFrames skills, then describe the video you want:
npx skills add heygen-com/hyperframes
This teaches your agent (Claude Code, Cursor, Gemini CLI, Codex) how to write correct compositions and GSAP animations. In Claude Code, the skills register as slash commands — invoke /hyperframes to author compositions, /hyperframes-cli for CLI commands, and /gsap for animation help.
For Codex specifically, the same skills are also exposed as an OpenAI Codex plugin — sparse-install just the plugin surface:
codex plugin marketplace add heygen-com/hyperframes --sparse .codex-plugin --sparse skills --sparse assets
Try it: example prompts
Copy any of these into your agent to get started. The /hyperframes prefix loads the skill context explicitly so you get correct output the first time.
Cold start — describe what you want:
Using
/hyperframes, create a 10-second product intro with a fade-in title, a background video, and background music.
Warm start — turn existing context into a video:
Take a look at this GitHub repo https://github.com/heygen-com/hyperframes and explain its uses and architecture to me using
/hyperframes.
Summarize the attached PDF into a 45-second pitch video using
/hyperframes.
Turn this CSV into an animated bar chart race using
/hyperframes.
Format-specific:
Make a 9:16 TikTok-style hook video about [topic] using
/hyperframes, with bouncy captions synced to a TTS narration.
Iterate — talk to the agent like a video editor:
Make the title 2x bigger, swap to dark mode, and add a fade-out at the end.
Add a lower third at 0:03 with my name and title.
The agent handles scaffolding, animation, and rendering. See the prompting guide for more patterns.
Option 2: Start a project manually
npx hyperframes init my-video
cd my-video
npx hyperframes preview # preview in browser (live reload)
npx hyperframes render # render to MP4
hyperframes init installs skills automatically, so you can hand off to your AI agent at any point.
Requirements: Node.js >= 22, FFmpeg
Why Hyperframes?
- HTML-native — compositions are HTML files with data attributes. No React, no proprietary DSL.
- AI-first — agents already speak HTML. The CLI is non-interactive by default, designed for agent-driven workflows.
- Deterministic rendering — same input = identical output. Built for automated pipelines.
- Frame Adapter pattern — bring your own animation runtime (GSAP, Lottie, CSS, Three.js).
Hyperframes vs Remotion
Hyperframes is inspired by Remotion — we used Remotion at HeyGen in production, learned a ton from it, and kept attribution comments in the source for the patterns it pioneered (Chrome launch flags, image2pipe → FFmpeg streaming, frame buffering). Both tools drive headless Chrome and both are deterministic. They differ on one decision: what the primary author writes. Remotion's bet is React components; Hyperframes' bet is HTML.
| Hyperframes | Remotion | |
|---|---|---|
| Authoring | HTML + CSS + GSAP | React components (TSX) |
| Build step | None; index.html plays as-is |
Required (bundler) |
| Library-clock animations (GSAP, Anime.js, Motion One) | Seekable, frame-accurate | Plays at wall-clock during render |
| Arbitrary HTML / CSS passthrough | Paste and animate | Rewrite as JSX |
| Distributed rendering | Single-machine today | Lambda, production-ready |
Licensing: fully open source vs source-available
Hyperframes is completely open source under Apache 2.0 — an OSI-approved license. Use it commercially at any scale, with no per-render fees, no seat caps, no company-size thresholds.
Remotion is source-available, not open source. The code is on GitHub under a custom Remotion License that requires a paid company license above small-team thresholds. It's a great product with a real team behind it — but if open-source licensing matters to you (OSI compliance, redistribution rights, no per-use fees), that's a first-order decision point.
Full write-up with benchmarks, an honest list of where each tool wins, and a GSAP side-by-side: Hyperframes vs Remotion guide.
How It Works
Define your video as HTML with data attributes:
<div id="stage" data-composition-id="my-video" data-start="0" data-width="1920" data-height="1080">
<video
id="clip-1"
data-start="0"
data-duration="5"
data-track-index="0"
src="intro.mp4"
muted
playsinline
></video>
<img
id="overlay"
class="clip"
data-start="2"
data-duration="3"
data-track-index="1"
src="logo.png"
/>
<audio
id="bg-music"
data-start="0"
data-duration="9"
data-track-index="2"
data-volume="0.5"
src="music.wav"
></audio>
</div>
Preview instantly in the browser. Render to MP4 locally or in Docker.
Catalog
50+ ready-to-use blocks and components — social overlays, shader transitions, data visualizations, and cinematic effects:
npx hyperframes add flash-through-white # shader transition
npx hyperframes add instagram-follow # social overlay
npx hyperframes add data-chart # animated chart
Browse the full catalog at hyperframes.heygen.com/catalog.
Documentation
Full documentation at hyperframes.heygen.com/introduction — Quickstart | Guides | API Reference | Catalog
Packages
| Package | Description |
|---|---|
hyperframes |
CLI — create, preview, lint, and render compositions |
@hyperframes/core |
Types, parsers, generators, linter, runtime, frame adapters |
@hyperframes/engine |
Seekable page-to-video capture engine (Puppeteer + FFmpeg) |
@hyperframes/producer |
Full rendering pipeline (capture + encode + audio mix) |
@hyperframes/studio |
Browser-based composition editor UI |
@hyperframes/player |
Embeddable <hyperframes-player> web component |
@hyperframes/shader-transitions |
WebGL shader transitions for compositions |
Skills
HyperFrames ships skills that teach AI agents framework-specific patterns that generic docs don't cover.
npx skills add heygen-com/hyperframes
| Skill | What it teaches |
|---|---|
hyperframes |
HTML composition authoring, captions, TTS, audio-reactive animation, transitions |
hyperframes-cli |
CLI commands: init, lint, preview, render, transcribe, tts, doctor |
hyperframes-registry |
Block and component installation via hyperframes add |
website-to-hyperframes |
Capture a URL and turn it into a video — full website-to-video pipeline |
gsap |
GSAP animation API, timelines, easing, ScrollTrigger, plugins, React/Vue/Svelte, performance |
Contributing
See CONTRIBUTING.md for guidelines.
Cloning the repo
The repo uses Git LFS for golden regression-test baselines under packages/producer/tests/**/output.mp4 (~240 MB of .mp4 files). If you're cloning the full repo for development, install Git LFS first:
# macOS
brew install git-lfs
# Ubuntu/Debian
sudo apt install git-lfs
# Windows
winget install GitHub.GitLFS
# (or install Git for Windows, which bundles Git LFS as an optional component)
# Then (once, per machine)
git lfs install
If you hit git-lfs filter-process: command not found during git clone or npx skills add heygen-com/hyperframes, install Git LFS and retry. You can also skip LFS content if you only need the source files:
GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/heygen-com/hyperframes.git
