mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
James RussoandClaude Opus 4.6
2f99e33bbe
* feat(cli,core): add standalone transcribe command, transcript normalization, and caption lint rules
- Add `hyperframes transcribe` command for transcribing audio/video and importing
existing transcripts (SRT, VTT, OpenAI Whisper API JSON, whisper.cpp JSON)
- Add transcript format normalizer (normalize.ts) with auto-detection and
conversion to standard [{text, start, end}] word arrays
- Upgrade default whisper model from base.en to small.en for better accuracy
- Add --model and --language flags to both `transcribe` and `init` commands
- Extract shared patchCaptionHtml() to eliminate duplication between init.ts
and transcribe.ts (init.ts reduced by ~55 lines)
- Add 3 caption lint rules: caption_exit_missing_hard_kill,
caption_text_overflow_risk, caption_container_relative_position
- Update captions skill with model guide, format docs, music guidance,
text overflow prevention, caption exit guarantee pattern
- Expand captions skill trigger to cover lyrics, karaoke, lyric videos
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(cli): add transcribe command and --model/--language flags to CLI docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): fix blank template lint issues
- blank/index.html: remove data-start from video (was nested in timed parent),
add class="clip" for initial hidden state
- blank/captions.html: add max-width + overflow:hidden to prevent text clipping,
add tl.set hard kill after exit tween to prevent stuck captions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: add lint-after-edit rule to repo and project CLAUDE.md
Agents must run `npx hyperframes lint` after editing compositions.
Also expand captions skill description in project template.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style: format _shared/CLAUDE.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
119 lines
6.4 KiB
Markdown
119 lines
6.4 KiB
Markdown
# Hyperframes
|
|
|
|
## Skills — USE THESE FIRST
|
|
|
|
This repo ships skills that are installed globally via `npx hyperframes skills` (runs automatically during `hyperframes init`). **Always use the appropriate skill instead of writing code from scratch or fetching external docs.**
|
|
|
|
### HyperFrames Skills (from this repo)
|
|
|
|
| Skill | Invoke with | When to use |
|
|
| ------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| **hyperframes-compose** | `/hyperframes-compose` | Creating ANY HTML composition — videos, animations, title cards, overlays. Contains required HTML structure, `class="clip"` rules, GSAP timeline patterns, and rendering constraints. |
|
|
| **hyperframes-captions** | `/hyperframes-captions` | Any task involving text synced to audio: captions, subtitles, lyrics, lyric videos, karaoke. Also covers transcription strategy (whisper model selection, transcript format). |
|
|
|
|
### GSAP Skills (from [greensock/gsap-skills](https://github.com/greensock/gsap-skills))
|
|
|
|
| Skill | Invoke with | When to use |
|
|
| ---------------------- | --------------------- | -------------------------------------------------------------------------------- |
|
|
| **gsap-core** | `/gsap-core` | `gsap.to()`, `from()`, `fromTo()`, easing, duration, stagger, defaults |
|
|
| **gsap-timeline** | `/gsap-timeline` | Timeline sequencing, position parameter, labels, nesting, playback |
|
|
| **gsap-performance** | `/gsap-performance` | Performance best practices — transforms over layout props, will-change, batching |
|
|
| **gsap-plugins** | `/gsap-plugins` | ScrollTrigger, Flip, Draggable, SplitText, and other GSAP plugins |
|
|
| **gsap-scrolltrigger** | `/gsap-scrolltrigger` | Scroll-linked animations, pinning, scrub, triggers |
|
|
| **gsap-utils** | `/gsap-utils` | `gsap.utils` helpers — clamp, mapRange, snap, toArray, wrap, pipe |
|
|
|
|
### Why this matters
|
|
|
|
The skills encode HyperFrames-specific patterns (e.g., required `class="clip"` on all timed elements, GSAP timeline registration via `window.__GSAP_TIMELINE`, `data-*` attribute semantics) that are NOT in generic web docs. Skipping the skills and writing from scratch will produce broken compositions.
|
|
|
|
### Rules
|
|
|
|
- When creating or modifying HTML compositions → invoke `/hyperframes-compose` BEFORE writing any code
|
|
- When adding captions, subtitles, lyrics, or any text synced to audio → invoke `/hyperframes-captions` BEFORE writing any code
|
|
- When transcribing audio or choosing a whisper model → invoke `/hyperframes-captions` BEFORE running any transcription tool
|
|
- When creating a video from audio (music video, lyric video, audio visualizer with text) → invoke BOTH `/hyperframes-compose` AND `/hyperframes-captions`
|
|
- When writing GSAP animations → invoke `/gsap-core` and `/gsap-timeline` BEFORE writing any code
|
|
- When optimizing animation performance → invoke `/gsap-performance` BEFORE making changes
|
|
- After creating or editing any `.html` composition → run `npx hyperframes lint` and fix all errors before considering the task complete
|
|
|
|
### Installing skills
|
|
|
|
```bash
|
|
npx hyperframes skills # install all to Claude, Gemini, Codex
|
|
npx hyperframes skills --claude # Claude Code only
|
|
npx skills add greensock/gsap-skills # alternative: via skills CLI
|
|
```
|
|
|
|
## Project Overview
|
|
|
|
Open-source video rendering framework: write HTML, render video.
|
|
|
|
```
|
|
packages/
|
|
cli/ → hyperframes CLI (create, preview, lint, render)
|
|
core/ → Types, parsers, generators, linter, runtime, frame adapters
|
|
engine/ → Seekable page-to-video capture engine (Puppeteer + FFmpeg)
|
|
producer/ → Full rendering pipeline (capture + encode + audio mix)
|
|
studio/ → Browser-based composition editor UI
|
|
```
|
|
|
|
## Development
|
|
|
|
```bash
|
|
pnpm install # Install dependencies
|
|
pnpm build # Build all packages
|
|
pnpm test # Run tests
|
|
```
|
|
|
|
## Key Concepts
|
|
|
|
- **Compositions** are HTML files with `data-*` attributes defining timeline, tracks, and media
|
|
- **Frame Adapters** bridge animation runtimes (GSAP, Lottie, CSS) to the capture engine
|
|
- **Producer** orchestrates capture → encode → audio mix into final MP4
|
|
- **BeginFrame rendering** uses `HeadlessExperimental.beginFrame` for deterministic frame capture
|
|
|
|
## Transcription
|
|
|
|
HyperFrames uses word-level timestamps for captions. The `hyperframes transcribe` command handles both transcription and format conversion.
|
|
|
|
### Quick reference
|
|
|
|
```bash
|
|
# Transcribe audio/video (local whisper.cpp, no API key)
|
|
npx hyperframes transcribe audio.mp3
|
|
npx hyperframes transcribe video.mp4 --model medium.en --language en
|
|
|
|
# Import existing transcript from another tool
|
|
npx hyperframes transcribe subtitles.srt
|
|
npx hyperframes transcribe subtitles.vtt
|
|
npx hyperframes transcribe openai-response.json
|
|
```
|
|
|
|
### Whisper models
|
|
|
|
Default is `small.en`. Upgrade for better accuracy:
|
|
|
|
| Model | Size | Use case |
|
|
| ----------- | ------ | -------------------------------- |
|
|
| `tiny.en` | 75 MB | Quick testing |
|
|
| `base.en` | 142 MB | Short clips, clear audio |
|
|
| `small.en` | 466 MB | **Default** — most content |
|
|
| `medium.en` | 1.5 GB | Important content, noisy audio |
|
|
| `large-v3` | 3.1 GB | Multilingual, production quality |
|
|
|
|
Use `.en` suffix for English-only (more accurate). Drop it for multilingual content.
|
|
|
|
### Supported transcript formats
|
|
|
|
The CLI auto-detects and normalizes: whisper.cpp JSON, OpenAI Whisper API JSON, SRT, VTT, and pre-normalized `[{text, start, end}]` arrays.
|
|
|
|
### Improving transcription quality
|
|
|
|
If captions are inaccurate (wrong words, bad timing):
|
|
|
|
1. **Upgrade the model**: `--model medium.en` or `--model large-v3`
|
|
2. **Set language**: `--language en` to filter non-target speech
|
|
3. **Use an external API**: Transcribe via OpenAI or Groq Whisper API, then import the JSON with `hyperframes transcribe response.json`
|
|
|
|
See the `/hyperframes-captions` skill for full details on model selection and API usage.
|