Miguel Ángel 6a59ef6106 fix: skip metadata waits for injected video frames (#575)
## Problem

Closes #574.

On Windows with cached headless-shell Chrome, a composition that reuses the same video file in three timeline clips can fail before frame capture starts:

```html
<video id="video1" src="1.mp4" data-start="0" muted data-duration="4" data-track-index="0" data-media-start="0"></video>
<video id="video2" src="1.mp4" data-start="4" muted data-duration="4" data-track-index="0" data-media-start="4"></video>
<video id="video3" src="1.mp4" data-start="8" muted data-duration="4" data-track-index="0" data-media-start="8"></video>
```

The reported render reaches video frame extraction, then dies at frame-capture initialization with:

```text
[FrameCapture] video metadata not ready after 45000ms. Video elements must load metadata before capture starts.
```

The important detail is that by this stage HyperFrames has already extracted video pixels through FFmpeg. Native Chromium video metadata is only being waited on for DOM layout stability, not because Chromium is the source of rendered pixels.

## Root Cause

The render pipeline has two separate media responsibilities:

- FFmpeg extracts video frames and audio from declared media.
- Chromium owns DOM layout and capture, while injected FFmpeg frames supply the video pixels before each captured frame.

Before this PR, every capture session still waited for every DOM `<video>` to reach `readyState >= 1` unless the element was a native HDR exception. That made native browser media metadata a hard render prerequisite even when the browser would not decode or provide the final video pixels.

That is why the issue fails at `25% Starting frame capture`: FFmpeg extraction has already succeeded, but capture initialization blocks on repeated native `<video src="1.mp4">` metadata loading in cached Windows headless-shell Chrome.

There was a second constraint: the readiness wait also prevents first-frame layout bugs. If a skipped `<video>` has no native metadata, Chromium can use the default `300x150` intrinsic video size, which breaks layouts such as `width: 100%; height: auto` before the first injected frame. The fix therefore must not simply skip all video readiness waits; it must provide dimensions for any skipped videos.

## What This Fixes

- Treats videos with successfully extracted FFmpeg frames and usable dimensions as out-of-band rendered video sources.
- Skips native browser metadata readiness waits for those extracted videos because Chromium is not responsible for their pixels.
- Passes FFmpeg-probed dimensions into capture as `videoMetadataHints`.
- Applies those hints before the readiness wait in both screenshot and BeginFrame initialization paths.
- Sets missing `width` / `height` attributes and an explicit `aspect-ratio` only when the element does not already provide one, preserving author styles where present.
- Keeps native HDR video IDs in the skip list, preserving the existing HEVC/HDR behavior where Chrome may not decode the source but FFmpeg/native HDR compositing can still render it.
- Uses one `buildCaptureOptions()` helper so calibration, HDR DOM capture, streaming capture, parallel capture, and sequential capture receive the same skip IDs and metadata hints.
- Adds tests for the skip-list and metadata-hint contract.
- Adds a Windows CI regression that reproduces the issue shape after the canary render warms the cached-browser path.

## Reviewer Map

Primary files:

- `packages/producer/src/services/renderOrchestrator.ts`
  - `collectVideoReadinessSkipIds()` includes native HDR IDs plus extracted videos that have finite positive FFmpeg dimensions.
  - `collectVideoMetadataHints()` converts extracted FFmpeg metadata into capture hints.
  - `buildCaptureOptions()` threads `skipReadinessVideoIds` and `videoMetadataHints` into every capture path.
- `packages/engine/src/services/frameCapture.ts`
  - `applyVideoMetadataHints()` runs in the page before video readiness polling.
  - Both screenshot and BeginFrame initialization call it before checking non-skipped videos for `readyState >= 1`.
- `packages/engine/src/types.ts`
  - Adds `CaptureVideoMetadataHint` and documents that readiness skips should be paired with metadata hints when layout may depend on intrinsic dimensions.
- `packages/producer/src/services/renderOrchestrator.test.ts`
  - Covers that extracted videos with dimensions are skipped, invalid dimensions are not, native HDR IDs are preserved, and hints are stable/sorted.
- `.github/workflows/windows-render.yml`
  - Adds the issue #574 Windows regression with the exact three-clip markup and a generated deterministic `1.mp4`.

## Why This Is Safe

The skip is intentionally gated:

- A standard video is skipped only after `extractAllVideoFrames()` succeeded for that video and returned usable dimensions.
- Videos with invalid dimensions are not skipped, so the old browser readiness guard still applies.
- DOM videos are still present for layout and element bounds; only the native metadata wait is skipped for sources whose pixels come from FFmpeg injection.
- Metadata hints are applied conservatively: existing `width`, `height`, and explicit `aspect-ratio` are not overwritten.
- Non-extracted videos, images, fonts, page readiness, and `window.__hf` readiness keep the existing waits.
- The fix is not limited to the sequential path from the issue; it is threaded through calibration, HDR DOM capture, streaming encode, parallel capture, and sequential capture.

A first local revision skipped readiness too broadly and caused `overlay-montage-prod` first-frame layout shrinkage. The current version fixes that by pairing skips with FFmpeg metadata hints; `overlay-montage-prod` now passes and is listed in verification below.

## Verification

### Root-Cause Reproduction Before Fix

The reporter did not attach the actual `1.mp4`, so the regression uses the exact issue markup and a deterministic generated 12s H.264 file named `1.mp4`.

I reproduced the failure in GitHub Actions by running this branch's new Windows workflow against unpatched `main`:

```bash
gh workflow run windows-render.yml --repo heygen-com/hyperframes --ref fix/reused-video-metadata -f ref=main
```

That means the workflow contains the new issue #574 regression, but the code under test is `main` without this fix.

Baseline failure:

- Run: https://github.com/heygen-com/hyperframes/actions/runs/25174603730
- Failed job: https://github.com/heygen-com/hyperframes/actions/runs/25174603730/job/73803179086
- Checkout proof: `ref: main`, `origin/main`, commit `8662598a3ac64018a2999d189ffb369e6d46b53a`.
- Failure proof: `Browser: cache`, `staticDuration:12`, `videoCount:3`, then `25% Starting frame capture` -> `[FrameCapture] video metadata not ready after 15000ms`.

This is the same failure class as the issue, on Windows, in cached-browser mode, before the fix.

### Fixed Windows Regression

The same regression passes on this PR branch:

- Run: https://github.com/heygen-com/hyperframes/actions/runs/25175048215
- Passing job: https://github.com/heygen-com/hyperframes/actions/runs/25175048215/job/73804781017
- Checkout proof: PR merge contains `79d6b41c9f2ba137cbfb9301678e0815b16c4f5a` merged into `8662598a3ac64018a2999d189ffb369e6d46b53a`.
- Passing proof: `Browser: cache`, `staticDuration:12`, `videoCount:3`, `25% Starting frame capture`, captures `360/360` frames, renders `issue-574.mp4`, and `ffprobe` verifies `1920x1080 @ 30/1, 12s`.

### Local Checks

- `bun run build:hyperframes-runtime`
- `bunx vitest run packages/producer/src/services/renderOrchestrator.test.ts`
- `bun run --filter @hyperframes/producer typecheck`
- `bun run --filter @hyperframes/engine typecheck`
- `bunx oxlint packages/engine/src/services/frameCapture.ts packages/engine/src/types.ts packages/engine/src/index.ts packages/producer/src/services/renderOrchestrator.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `bunx oxfmt --check .github/workflows/windows-render.yml packages/engine/src/services/frameCapture.ts packages/engine/src/types.ts packages/engine/src/index.ts packages/producer/src/services/renderOrchestrator.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `git diff --check`
- Lefthook pre-commit: lint, format, typecheck where applicable
- Lefthook commit-msg: commitlint

### Local Render Checks

- Created `/tmp/hf-issue-574-repro` with the issue shape: three clips using the same `1.mp4`, `data-media-start=0/4/8`, 12s total.
- `PRODUCER_PLAYER_READY_TIMEOUT_MS=5000 bun packages/cli/src/cli.ts render /tmp/hf-issue-574-repro --workers 1 --quality draft --fps 30 --output /tmp/hf-issue-574-h264-fixed-v2.mp4` -> completed.
- Created `/tmp/hf-issue-574-prores` with the same three-clip shape using one FFmpeg-readable ProRes `.mov`, which exercises the browser-metadata failure class because Chromium should not be needed to decode the source.
- `PRODUCER_PLAYER_READY_TIMEOUT_MS=3000 bun packages/cli/src/cli.ts render /tmp/hf-issue-574-prores --workers 1 --quality draft --fps 30 --output /tmp/hf-issue-574-prores-fixed-v2.mp4` -> completed.
- `bun run --filter @hyperframes/producer test --sequential --keep-temp overlay-montage-prod` -> passed; this guards against skipped metadata shrinking `height:auto` video layout before the first injected frame.
- `ffmpeg -v error -i /tmp/hf-issue-574-prores-fixed-v2.mp4 -f null -`
- `ffmpeg -v error -i /tmp/hf-issue-574-h264-fixed-v2.mp4 -f null -`
- `ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of json /tmp/hf-issue-574-h264-fixed-v2.mp4` -> H.264, 320x180, 30fps, 12.0s.

### Current PR Checks

- Windows render verification: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048215.
- Windows tests: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048215.
- Main CI build/lint/typecheck/test/smoke jobs: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048175.
- Regression shards observed passing include HDR, render-compat, styles A-G, and `overlay-montage-prod`. At the time this body was updated, the `fast` regression shard was still in progress in run https://github.com/heygen-com/hyperframes/actions/runs/25174515546.

### Browser Verification

- Used `agent-browser` to open `file:///tmp/hf-issue-574-h264-fixed-v2.mp4` and verify the rendered output displays in Chromium.
- Screenshot: `.debug/issue-574/h264-output-page.png`
- Agent-browser recording: `.debug/issue-574/h264-output-playback.webm`

## Notes / Caveats

- The reporter's exact `1.mp4` was not attached to #574. The committed Windows regression uses a generated deterministic H.264 file with the same filename and exact markup from the issue.
- The exact H.264 issue shape did not reproduce the timeout on this macOS/system-Chrome machine before the fix; it rendered successfully locally. The GitHub Actions baseline above reproduces it on Windows/cache without the fix.
- The Windows fixture intentionally runs after the existing canary render so the browser path is `Browser: cache`, matching the reporter's environment.
- The generated fixture emits sparse-keyframe warnings. Those warnings are expected and are not the failure being fixed; the baseline failure occurs before any frame capture because native browser video metadata never becomes ready.
- Browser proof artifacts are local-only under `.debug/issue-574/` and intentionally not committed.
2026-04-30 18:50:57 +02:00
2026-04-27 18:16:09 -04:00
2026-03-21 22:43:56 -07:00

HyperFrames

npm version npm downloads License Node.js Discord

Write HTML. Render video. Built for agents.

HyperFrames demo — HTML code on the left transforms into a rendered video on the right

Hyperframes is an open-source video rendering framework that lets you create, preview, and render HTML-based video compositions — with first-class support for AI agents.

Quick Start

Install the HyperFrames skills, then describe the video you want:

npx skills add heygen-com/hyperframes

This teaches your agent (Claude Code, Cursor, Gemini CLI, Codex) how to write correct compositions, GSAP timelines, and first-party adapter animations. In Claude Code, the skills register as slash commands — invoke /hyperframes to author compositions, /hyperframes-cli for CLI commands, /gsap for timeline animation help, or the adapter skills (/animejs, /css-animations, /lottie, /three, /waapi) when a composition uses those runtimes.

For Claude Design, open docs/guides/claude-design-hyperframes.md on GitHub and click the download button (↓) to save it, then attach the file to your Claude Design chat. It produces a valid first draft; refine in any AI coding agent. See the Claude Design guide.

For Codex specifically, the same skills are also exposed as an OpenAI Codex plugin — sparse-install just the plugin surface:

codex plugin marketplace add heygen-com/hyperframes --sparse .codex-plugin --sparse skills --sparse assets

For Claude Code, the repo also ships a Claude Code plugin manifest: test it locally with claude --plugin-dir .. The manifest intentionally omits skills because Claude Code auto-discovers the root skills/ directory by convention, and for marketplace submission use the title HyperFrames by HeyGen plus the black/white icon assets at assets/claude-code-icon-dark.svg and assets/claude-code-icon-light.svg for the two theme slots. For Cursor, the same skills are packaged as a Cursor plugin — install from the Cursor Marketplace, or sideload by cloning this repo and pointing Settings → Plugins → Load unpacked at the repo root.

Try it: example prompts

Copy any of these into your agent to get started. The /hyperframes prefix loads the skill context explicitly so you get correct output the first time.

Cold start — describe what you want:

Using /hyperframes, create a 10-second product intro with a fade-in title, a background video, and background music.

Warm start — turn existing context into a video:

Take a look at this GitHub repo https://github.com/heygen-com/hyperframes and explain its uses and architecture to me using /hyperframes.

Summarize the attached PDF into a 45-second pitch video using /hyperframes.

Turn this CSV into an animated bar chart race using /hyperframes.

Format-specific:

Make a 9:16 TikTok-style hook video about [topic] using /hyperframes, with bouncy captions synced to a TTS narration.

Iterate — talk to the agent like a video editor:

Make the title 2x bigger, swap to dark mode, and add a fade-out at the end.

Add a lower third at 0:03 with my name and title.

The agent handles scaffolding, animation, and rendering. See the prompting guide for more patterns.

Option 2: Start a project manually

npx hyperframes init my-video
cd my-video
npx hyperframes preview      # preview in browser (live reload)
npx hyperframes render       # render to MP4

hyperframes init installs skills automatically, so you can hand off to your AI agent at any point.

Requirements: Node.js >= 22, FFmpeg

Why Hyperframes?

  • HTML-native — compositions are HTML files with data attributes. No React, no proprietary DSL.
  • AI-first — agents already speak HTML. The CLI is non-interactive by default, designed for agent-driven workflows.
  • Deterministic rendering — same input = identical output. Built for automated pipelines.
  • Frame Adapter pattern — bring your own animation runtime (GSAP, Lottie, CSS, Three.js).

Hyperframes vs Remotion

Hyperframes is inspired by Remotion — we used Remotion at HeyGen in production, learned a ton from it, and kept attribution comments in the source for the patterns it pioneered (Chrome launch flags, image2pipe → FFmpeg streaming, frame buffering). Both tools drive headless Chrome and both are deterministic. They differ on one decision: what the primary author writes. Remotion's bet is React components; Hyperframes' bet is HTML.

Hyperframes Remotion
Authoring HTML + CSS + GSAP React components (TSX)
Build step None; index.html plays as-is Required (bundler)
Library-clock animations (GSAP, Anime.js, Motion One) Seekable, frame-accurate Plays at wall-clock during render
Arbitrary HTML / CSS passthrough Paste and animate Rewrite as JSX
Distributed rendering Single-machine today Lambda, production-ready

Licensing: fully open source vs source-available

Hyperframes is completely open source under Apache 2.0 — an OSI-approved license. Use it commercially at any scale, with no per-render fees, no seat caps, no company-size thresholds.

Remotion is source-available, not open source. The code is on GitHub under a custom Remotion License that requires a paid company license above small-team thresholds. It's a great product with a real team behind it — but if open-source licensing matters to you (OSI compliance, redistribution rights, no per-use fees), that's a first-order decision point.

Full write-up with benchmarks, an honest list of where each tool wins, and a GSAP side-by-side: Hyperframes vs Remotion guide.

How It Works

Define your video as HTML with data attributes:

<div id="stage" data-composition-id="my-video" data-start="0" data-width="1920" data-height="1080">
  <video
    id="clip-1"
    data-start="0"
    data-duration="5"
    data-track-index="0"
    src="intro.mp4"
    muted
    playsinline
  ></video>
  <img
    id="overlay"
    class="clip"
    data-start="2"
    data-duration="3"
    data-track-index="1"
    src="logo.png"
  />
  <audio
    id="bg-music"
    data-start="0"
    data-duration="9"
    data-track-index="2"
    data-volume="0.5"
    src="music.wav"
  ></audio>
</div>

Preview instantly in the browser. Render to MP4 locally or in Docker.

Catalog

50+ ready-to-use blocks and components — social overlays, shader transitions, data visualizations, and cinematic effects:

npx hyperframes add flash-through-white   # shader transition
npx hyperframes add instagram-follow      # social overlay
npx hyperframes add data-chart            # animated chart

Browse the full catalog at hyperframes.heygen.com/catalog.

Documentation

Full documentation at hyperframes.heygen.com/introduction — Quickstart | Guides | API Reference | Catalog

Packages

Package Description
hyperframes CLI — create, preview, lint, and render compositions
@hyperframes/core Types, parsers, generators, linter, runtime, frame adapters
@hyperframes/engine Seekable page-to-video capture engine (Puppeteer + FFmpeg)
@hyperframes/producer Full rendering pipeline (capture + encode + audio mix)
@hyperframes/studio Browser-based composition editor UI
@hyperframes/player Embeddable <hyperframes-player> web component
@hyperframes/shader-transitions WebGL shader transitions for compositions

Skills

HyperFrames ships skills that teach AI agents framework-specific patterns that generic docs don't cover.

npx skills add heygen-com/hyperframes
Skill What it teaches
hyperframes HTML composition authoring, captions, TTS, audio-reactive animation, transitions
hyperframes-cli CLI commands: init, lint, preview, render, transcribe, tts, doctor
hyperframes-registry Block and component installation via hyperframes add
website-to-hyperframes Capture a URL and turn it into a video — full website-to-video pipeline
remotion-to-hyperframes Translate a Remotion (React) composition into a HyperFrames HTML composition
gsap GSAP timelines for HyperFrames: paused registration, deterministic seeking, easing, sequencing, performance
animejs Anime.js animations and timelines registered on window.__hfAnime for deterministic HyperFrames seeking
css-animations CSS keyframe animation patterns that HyperFrames can discover, pause, and seek
lottie lottie-web and dotLottie players registered on window.__hfLottie with local assets and paused playback
three Three.js scenes that render from HyperFrames hf-seek events and window.__hfThreeTime instead of wall-clock time
waapi Web Animations API element.animate() patterns seeked through document.getAnimations()

Contributing

See CONTRIBUTING.md for guidelines.

Cloning the repo

The repo uses Git LFS for golden regression-test baselines under packages/producer/tests/**/output.mp4 (~240 MB of .mp4 files). If you're cloning the full repo for development, install Git LFS first:

# macOS
brew install git-lfs

# Ubuntu/Debian
sudo apt install git-lfs

# Windows
winget install GitHub.GitLFS
# (or install Git for Windows, which bundles Git LFS as an optional component)

# Then (once, per machine)
git lfs install

If you hit git-lfs filter-process: command not found during git clone or npx skills add heygen-com/hyperframes, install Git LFS and retry. You can also skip LFS content if you only need the source files:

GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/heygen-com/hyperframes.git

License

Apache 2.0

S
Description
Write HTML. Render video. Built for agents.
Readme
756 MiB
Languages
JavaScript 55.6%
TypeScript 44.2%