mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
Vance Ingalls
37404f23da
## Summary **Whisper improvements:** - Auto-detect language and switch from `.en` to multilingual model when needed - Detect speech onset in WAV to strip hallucinated words before speech begins - Merge whisper-cpp token fragments: contractions (`didn` + `'t` → `didn't`), split capitals (`C` + `aught` → `Caught`), dropped-g (`shin` + `in'` → `shinin'`) - Interpolate zero-duration word clusters for reliable karaoke timing **Captions skill updates (folded from #176):** - Rewrite script-to-style mapping as an energy detection table (high → low) with mandatory animation requirements: karaoke baseline, 2+ highlight techniques, kinetic exits - Replace `tl.call()` per-frame audio-reactive pattern with group-level GSAP tweens — read peak bass/treble for each group's time range and modulate entrance intensity at build time - Add transcript quality check with automatic retry rules (>20% music tokens = retry with larger model) - Add caption word structure lint rule (`.caption-group` + `<span>`) for studio editor compatibility **Multilingual defaults (folded from #186):** - Default whisper model changed from `small.en` to `small` to prevent silent translation of non-English audio - Added non-negotiable language rule to captions skill ## Test plan - [ ] `pnpm test` passes (contraction merging, fragment merging, zero-duration interpolation, speech onset) - [ ] Transcribe non-English audio — verify it transcribes in original language, not translates - [ ] Skill files render correctly, cross-references resolve - [ ] `dynamic-techniques.md` audio-reactive section uses `tl.to()`/`tl.set()` only, no `tl.call()` loops 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Hyperframes
Write HTML. Render video. Built for agents.
Hyperframes is an open-source video rendering framework that lets you create, preview, and render HTML-based video compositions — with first-class support for AI agents via MCP.
Why Hyperframes?
- HTML-native — AI agents already speak HTML. No React required.
- Frame Adapter pattern — bring your own animation runtime (GSAP, Lottie, CSS, Three.js).
- Deterministic rendering — same input = identical output. Built for automated pipelines.
- AI-first design — not a bolted-on afterthought.
Quick Start
npx hyperframes init my-video
cd my-video
Then open the project with your AI coding agent (Claude Code, Cursor, etc.) — it has HyperFrames skills installed and knows how to create and edit compositions.
npx hyperframes preview # preview in browser (live reload)
npx hyperframes render # render to MP4
Requirements: Node.js >= 22, FFmpeg
Documentation
Full documentation at hyperframes.heygen.com — start with the Quickstart, then explore guides, concepts, API reference, and package docs.
How It Works
Define your video as HTML with data attributes:
<div id="stage" data-composition-id="my-video" data-start="0" data-width="1920" data-height="1080">
<video
id="clip-1"
data-start="0"
data-duration="5"
data-track="0"
src="intro.mp4"
muted
playsinline
></video>
<img id="overlay" data-start="2" data-duration="3" data-track="1" src="logo.png" />
<audio
id="bg-music"
data-start="0"
data-duration="9"
data-track="2"
data-volume="0.5"
src="music.wav"
></audio>
</div>
Preview instantly in the browser. Render to MP4 locally. Let AI agents compose videos using tools they already understand.
Packages
| Package | Description |
|---|---|
hyperframes |
CLI — create, preview, lint, and render compositions |
@hyperframes/core |
Types, parsers, generators, linter, runtime, frame adapters |
@hyperframes/engine |
Seekable page-to-video capture engine (Puppeteer + FFmpeg) |
@hyperframes/producer |
Full rendering pipeline (capture + encode + audio mix) |
@hyperframes/studio |
Browser-based composition editor UI |
AI Agent Skills
HyperFrames ships skills that teach AI coding agents (Claude Code, Gemini CLI, Codex, Cursor) how to write correct compositions and GSAP animations. Use these instead of writing from scratch — they encode framework-specific patterns that generic docs don't cover.
Install via CLI (recommended)
# Install all skills (HyperFrames + GSAP) — runs automatically during `hyperframes init`
npx hyperframes skills
# Or install to a specific agent
npx hyperframes skills --claude
npx hyperframes skills --cursor
Or via npx skills add
# HyperFrames skills (hyperframes-compose, hyperframes-captions)
npx skills add heygen-com/hyperframes
# GSAP skills (gsap-core, gsap-timeline, gsap-scrolltrigger, gsap-plugins, gsap-performance, gsap-utils, gsap-react, gsap-frameworks)
npx skills add greensock/gsap-skills
Installed Skills
| Source | Skills | What they teach |
|---|---|---|
| HyperFrames | hyperframes-compose, hyperframes-captions |
HTML composition structure, class="clip" rules, data-* attributes, timeline registration, rendering constraints |
| GSAP | gsap-core, gsap-timeline, gsap-performance, gsap-plugins, gsap-scrolltrigger, gsap-utils, gsap-react, gsap-frameworks |
Core API, timeline sequencing, ScrollTrigger, plugin usage, performance best practices |
In Claude Code, invoke with /hyperframes-compose, /hyperframes-captions, /gsap-core, etc.
Contributing
See CONTRIBUTING.md for guidelines on how to contribute.
License
See LICENSE for details.