Miguel Ángel e8c43f0889 fix: prevent nested composition videos from autoplaying on seek (#477)
## Problem

Studio seek could still wake nested composition media even when the transport itself stayed paused.

In the real repro from `apple-presentation`, scrubbing to `0:29` without pressing play lands on the `slide-translation` composition. That composition contains `Multilingual_Journey.mp4` inside the composition host. On the broken path:

- the main Studio transport remained paused
- the nested video advanced and stayed playing anyway
- the user saw autoplay-like behavior even though the only action was a seek

That was especially confusing because the seek was otherwise correct: the timeline moved to the right point, but the nested media stopped obeying the paused transport state.

## What this fixes

### Nested media now participates in runtime media sync

- the runtime media cache no longer assumes only `video[data-start]` / `audio[data-start]` are relevant
- nested media inside a composition host can now be included in the same timed-media sync pass even when the inner media element does not carry its own authored `data-start`

### Nested media timing is resolved in the host composition window

- nested media start time is resolved against the enclosing composition host instead of falling back to scene-local `0`
- nested media duration is clamped to the enclosing composition window so it stays aligned with the authored host clip timing

### Paused seeks land on the right frame and stay paused

- after seeking into a nested composition, the inner media is now seeked to the correct frame relative to the host timeline
- because it is now part of the managed media set, the runtime also keeps it paused when the transport is paused instead of letting it continue playing on its own

### Regression coverage

- adds a runtime regression test that covers a nested composition video with no local `data-start`
- the test verifies that `player.seek(29)` leaves the nested video paused while landing it at the expected `currentTime`

## Root cause

The bug came from a mismatch between deterministic timeline seeking and media ownership.

### 1. The runtime only managed media with direct timing attrs

`refreshRuntimeMediaCache()` only collected `video[data-start]` and `audio[data-start]`. That works for root-level timed media, but not for media embedded inside a composition host where timing is inherited from the host composition rather than duplicated onto the inner media node.

### 2. Nested composition seek could still advance inner media

The runtime intentionally rearms sibling timelines during deterministic seek so nested timelines land on the right local offsets. That part is necessary and correct.

But because the nested video was not part of the managed media cache, it could advance during that seek path without being brought back under the paused transport state afterward.

### 3. The runtime had no way to reconcile the two

So the system had an inconsistent split:

- timeline seek knew about the nested composition timeline
- media sync did not know about the nested media inside it

The fix closes that split by resolving nested media start/duration from the enclosing composition context and running it through the same sync logic as other managed media.

## Verification

### Local checks

- `bun run --filter @hyperframes/core typecheck`
- `bun run test -- src/runtime/init.test.ts src/runtime/media.test.ts src/runtime/player.test.ts` in `packages/core`
- `bunx oxlint packages/core/src/runtime/media.ts packages/core/src/runtime/init.ts packages/core/src/runtime/init.test.ts`
- `bunx oxfmt --check packages/core/src/runtime/media.ts packages/core/src/runtime/init.ts packages/core/src/runtime/init.test.ts`

### Browser verification

Verified against a repo-backed local Studio preview of `apple-presentation`:

- opened `http://127.0.0.1:3014/#project/apple-presentation`
- seeked to `0:29` without pressing play
- confirmed the visible composition switched to `slide-translation`
- confirmed `Multilingual_Journey.mp4` landed at a non-zero `currentTime` (`3.067` in the verified run)
- confirmed the nested video stayed `paused` and its `currentTime` remained stable across a follow-up check instead of autoplaying

## Notes

- the local browser proof artifacts under `qa-artifacts/autoplay-seek/` are verification-only and are not part of this PR
- this PR is intentionally scoped to nested media ownership during paused seek; it does not broaden into unrelated runtime media refactors beyond bringing inherited nested media under the existing sync contract
2026-04-24 22:10:11 +02:00
2026-03-21 22:43:56 -07:00

HyperFrames

npm version npm downloads License Node.js

Write HTML. Render video. Built for agents.

HyperFrames demo — HTML code on the left transforms into a rendered video on the right

Hyperframes is an open-source video rendering framework that lets you create, preview, and render HTML-based video compositions — with first-class support for AI agents.

Quick Start

Install the HyperFrames skills, then describe the video you want:

npx skills add heygen-com/hyperframes

This teaches your agent (Claude Code, Cursor, Gemini CLI, Codex) how to write correct compositions and GSAP animations. In Claude Code, the skills register as slash commands — invoke /hyperframes to author compositions, /hyperframes-cli for CLI commands, and /gsap for animation help.

For Claude Design, open skills/claude-design-hyperframes/SKILL.md on GitHub and click the download button (↓) to save it, then attach the file to your Claude Design chat. It produces a valid first draft; refine in any AI coding agent. See the Claude Design guide.

For Codex specifically, the same skills are also exposed as an OpenAI Codex plugin — sparse-install just the plugin surface:

codex plugin marketplace add heygen-com/hyperframes --sparse .codex-plugin --sparse skills --sparse assets

For Claude Code, the repo also ships a Claude Code plugin manifest: test it locally with claude --plugin-dir .. The manifest intentionally omits skills because Claude Code auto-discovers the root skills/ directory by convention, and for marketplace submission use the title HyperFrames by HeyGen plus the black/white icon assets at assets/claude-code-icon-dark.svg and assets/claude-code-icon-light.svg for the two theme slots. For Cursor, the same skills are packaged as a Cursor plugin — install from the Cursor Marketplace, or sideload by cloning this repo and pointing Settings → Plugins → Load unpacked at the repo root.

Try it: example prompts

Copy any of these into your agent to get started. The /hyperframes prefix loads the skill context explicitly so you get correct output the first time.

Cold start — describe what you want:

Using /hyperframes, create a 10-second product intro with a fade-in title, a background video, and background music.

Warm start — turn existing context into a video:

Take a look at this GitHub repo https://github.com/heygen-com/hyperframes and explain its uses and architecture to me using /hyperframes.

Summarize the attached PDF into a 45-second pitch video using /hyperframes.

Turn this CSV into an animated bar chart race using /hyperframes.

Format-specific:

Make a 9:16 TikTok-style hook video about [topic] using /hyperframes, with bouncy captions synced to a TTS narration.

Iterate — talk to the agent like a video editor:

Make the title 2x bigger, swap to dark mode, and add a fade-out at the end.

Add a lower third at 0:03 with my name and title.

The agent handles scaffolding, animation, and rendering. See the prompting guide for more patterns.

Option 2: Start a project manually

npx hyperframes init my-video
cd my-video
npx hyperframes preview      # preview in browser (live reload)
npx hyperframes render       # render to MP4

hyperframes init installs skills automatically, so you can hand off to your AI agent at any point.

Requirements: Node.js >= 22, FFmpeg

Why Hyperframes?

  • HTML-native — compositions are HTML files with data attributes. No React, no proprietary DSL.
  • AI-first — agents already speak HTML. The CLI is non-interactive by default, designed for agent-driven workflows.
  • Deterministic rendering — same input = identical output. Built for automated pipelines.
  • Frame Adapter pattern — bring your own animation runtime (GSAP, Lottie, CSS, Three.js).

Hyperframes vs Remotion

Hyperframes is inspired by Remotion — we used Remotion at HeyGen in production, learned a ton from it, and kept attribution comments in the source for the patterns it pioneered (Chrome launch flags, image2pipe → FFmpeg streaming, frame buffering). Both tools drive headless Chrome and both are deterministic. They differ on one decision: what the primary author writes. Remotion's bet is React components; Hyperframes' bet is HTML.

Hyperframes Remotion
Authoring HTML + CSS + GSAP React components (TSX)
Build step None; index.html plays as-is Required (bundler)
Library-clock animations (GSAP, Anime.js, Motion One) Seekable, frame-accurate Plays at wall-clock during render
Arbitrary HTML / CSS passthrough Paste and animate Rewrite as JSX
Distributed rendering Single-machine today Lambda, production-ready

Licensing: fully open source vs source-available

Hyperframes is completely open source under Apache 2.0 — an OSI-approved license. Use it commercially at any scale, with no per-render fees, no seat caps, no company-size thresholds.

Remotion is source-available, not open source. The code is on GitHub under a custom Remotion License that requires a paid company license above small-team thresholds. It's a great product with a real team behind it — but if open-source licensing matters to you (OSI compliance, redistribution rights, no per-use fees), that's a first-order decision point.

Full write-up with benchmarks, an honest list of where each tool wins, and a GSAP side-by-side: Hyperframes vs Remotion guide.

How It Works

Define your video as HTML with data attributes:

<div id="stage" data-composition-id="my-video" data-start="0" data-width="1920" data-height="1080">
  <video
    id="clip-1"
    data-start="0"
    data-duration="5"
    data-track-index="0"
    src="intro.mp4"
    muted
    playsinline
  ></video>
  <img
    id="overlay"
    class="clip"
    data-start="2"
    data-duration="3"
    data-track-index="1"
    src="logo.png"
  />
  <audio
    id="bg-music"
    data-start="0"
    data-duration="9"
    data-track-index="2"
    data-volume="0.5"
    src="music.wav"
  ></audio>
</div>

Preview instantly in the browser. Render to MP4 locally or in Docker.

Catalog

50+ ready-to-use blocks and components — social overlays, shader transitions, data visualizations, and cinematic effects:

npx hyperframes add flash-through-white   # shader transition
npx hyperframes add instagram-follow      # social overlay
npx hyperframes add data-chart            # animated chart

Browse the full catalog at hyperframes.heygen.com/catalog.

Documentation

Full documentation at hyperframes.heygen.com/introductionQuickstart | Guides | API Reference | Catalog

Packages

Package Description
hyperframes CLI — create, preview, lint, and render compositions
@hyperframes/core Types, parsers, generators, linter, runtime, frame adapters
@hyperframes/engine Seekable page-to-video capture engine (Puppeteer + FFmpeg)
@hyperframes/producer Full rendering pipeline (capture + encode + audio mix)
@hyperframes/studio Browser-based composition editor UI
@hyperframes/player Embeddable <hyperframes-player> web component
@hyperframes/shader-transitions WebGL shader transitions for compositions

Skills

HyperFrames ships skills that teach AI agents framework-specific patterns that generic docs don't cover.

npx skills add heygen-com/hyperframes
Skill What it teaches
claude-design-hyperframes Template-first Claude Design skill — pre-valid skeletons, produces video drafts for refinement in any coding agent
hyperframes HTML composition authoring, captions, TTS, audio-reactive animation, transitions
hyperframes-cli CLI commands: init, lint, preview, render, transcribe, tts, doctor
hyperframes-registry Block and component installation via hyperframes add
website-to-hyperframes Capture a URL and turn it into a video — full website-to-video pipeline
gsap GSAP animation API, timelines, easing, ScrollTrigger, plugins, React/Vue/Svelte, performance

Contributing

See CONTRIBUTING.md for guidelines.

Cloning the repo

The repo uses Git LFS for golden regression-test baselines under packages/producer/tests/**/output.mp4 (~240 MB of .mp4 files). If you're cloning the full repo for development, install Git LFS first:

# macOS
brew install git-lfs

# Ubuntu/Debian
sudo apt install git-lfs

# Windows
winget install GitHub.GitLFS
# (or install Git for Windows, which bundles Git LFS as an optional component)

# Then (once, per machine)
git lfs install

If you hit git-lfs filter-process: command not found during git clone or npx skills add heygen-com/hyperframes, install Git LFS and retry. You can also skip LFS content if you only need the source files:

GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/heygen-com/hyperframes.git

License

Apache 2.0

S
Description
Write HTML. Render video. Built for agents.
Readme
580 MiB
Languages
TypeScript 86%
JavaScript 9.3%
CSS 4.1%
Shell 0.3%
Python 0.2%