ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-03 04:38:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #619 from heygen-com/split-cli-media-skills

Browse Source 
refactor(skills): split asset preprocessing out of hyperframes-cli
This commit is contained in:
James Russo
2026-05-04 20:27:21 -07:00
committed by GitHub
parent 20895eecdd 94dc6c895e
commit 06f5422d34
11 changed files with 152 additions and 161 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.codex-plugin/plugin.json
+1 -1
View File
@@ -31,7 +31,7 @@
"interface": {
"displayName": "HyperFrames by HeyGen",
"shortDescription": "Write HTML, render video",
"longDescription": "Build videos from HTML with HyperFrames. Author compositions with HTML, CSS, Tailwind v4 browser-runtime styles, GSAP, Anime.js, Lottie, Three.js, and WAAPI adapter patterns, use the CLI for init/preview/render/transcribe/tts, install reusable registry blocks and components, and turn any website into a video with the 7-step capture-to-video pipeline.",
"longDescription": "Build videos from HTML with HyperFrames. Author compositions with HTML, CSS, Tailwind v4 browser-runtime styles, GSAP, Anime.js, Lottie, Three.js, and WAAPI adapter patterns, use the CLI for the dev loop (init/preview/render), preprocess assets (tts/transcribe/remove-background) for compositions, install reusable registry blocks and components, and turn any website into a video with the 7-step capture-to-video pipeline.",
"developerName": "HeyGen",
"category": "Design",
"capabilities": ["Read", "Write"],
CLAUDE.md
+1 -1
View File
@@ -67,4 +67,4 @@ will not match CI. Use it only for local-only experimentation.
## Skills
Composition authoring (not repo development) is guided by skills installed via `npx skills add heygen-com/hyperframes`. See `skills/` for source. Invoke `/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/tailwind`, or `/gsap` when authoring compositions. Use `/tailwind` for projects created with `hyperframes init --tailwind` so agents follow the pinned Tailwind v4 browser-runtime contract instead of Studio's Tailwind v3 setup. Use `/animejs`, `/css-animations`, `/lottie`, `/three`, or `/waapi` when a composition uses those first-party runtime adapters. When a user provides a website URL and wants a video, invoke `/website-to-hyperframes` — it runs the full 7-step capture-to-video pipeline.
Composition authoring (not repo development) is guided by skills installed via `npx skills add heygen-com/hyperframes`. See `skills/` for source. Invoke `/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/tailwind`, or `/gsap` when authoring compositions. Use `/tailwind` for projects created with `hyperframes init --tailwind` so agents follow the pinned Tailwind v4 browser-runtime contract instead of Studio's Tailwind v3 setup. Use `/animejs`, `/css-animations`, `/lottie`, `/three`, or `/waapi` when a composition uses those first-party runtime adapters. Invoke `/hyperframes-media` for asset preprocessing (TTS narration, audio/video transcription, background removal for transparent overlays) — these commands have their own skill so the CLI skill stays focused on the dev loop. When a user provides a website URL and wants a video, invoke `/website-to-hyperframes` — it runs the full 7-step capture-to-video pipeline.
README.md
+3 -2
View File
@@ -32,7 +32,7 @@ Install the HyperFrames skills, then describe the video you want:
npx skills add heygen-com/hyperframes
```
This teaches your agent (Claude Code, Cursor, Gemini CLI, Codex) how to write correct compositions, GSAP timelines, Tailwind v4 browser-runtime styles, and first-party adapter animations. In Claude Code, the skills register as slash commands — invoke `/hyperframes` to author compositions, `/hyperframes-cli` for CLI commands, `/tailwind` for `init --tailwind` projects, `/gsap` for timeline animation help, or the adapter skills (`/animejs`, `/css-animations`, `/lottie`, `/three`, `/waapi`) when a composition uses those runtimes.
This teaches your agent (Claude Code, Cursor, Gemini CLI, Codex) how to write correct compositions, GSAP timelines, Tailwind v4 browser-runtime styles, and first-party adapter animations. In Claude Code, the skills register as slash commands — invoke `/hyperframes` to author compositions, `/hyperframes-cli` for the dev-loop commands (init, lint, preview, render), `/hyperframes-media` for asset preprocessing (TTS, transcription, background removal), `/tailwind` for `init --tailwind` projects, `/gsap` for timeline animation help, or the adapter skills (`/animejs`, `/css-animations`, `/lottie`, `/three`, `/waapi`) when a composition uses those runtimes.
For Claude Design, open [`docs/guides/claude-design-hyperframes.md`](https://github.com/heygen-com/hyperframes/blob/main/docs/guides/claude-design-hyperframes.md) on GitHub and click the download button (↓) to save it, then attach the file to your Claude Design chat. It produces a valid first draft; refine in any AI coding agent. See the [Claude Design guide](https://hyperframes.heygen.com/guides/claude-design).
@@ -188,7 +188,8 @@ npx skills add heygen-com/hyperframes
| Skill | What it teaches |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `hyperframes` | HTML composition authoring, captions, TTS, audio-reactive animation, transitions |
| `hyperframes-cli` | CLI commands: init, lint, preview, render, transcribe, tts, doctor |
| `hyperframes-cli` | Dev-loop CLI: init, lint, inspect, preview, render, doctor |
| `hyperframes-media` | Asset preprocessing: tts (Kokoro), transcribe (Whisper), remove-background (u2net) — voice/model/codec selection |
| `hyperframes-registry` | Block and component installation via `hyperframes add` |
| `website-to-hyperframes` | Capture a URL and turn it into a video — full website-to-video pipeline |
| `remotion-to-hyperframes` | Translate a Remotion (React) composition into a HyperFrames HTML composition |
docs/guides/prompting.mdx
+2 -1
View File
@@ -18,7 +18,8 @@ In Claude Code, restart the session after installing. Skills register as **slash
| Slash command | What it loads |
| -------------------------- | -------------------------------------------------------------------------- |
| `/hyperframes` | Composition authoring — HTML structure, timing, captions, TTS, transitions |
| `/hyperframes-cli` | CLI commands — `init`, `lint`, `preview`, `render`, `transcribe`, `tts` |
| `/hyperframes-cli` | Dev-loop CLI — `init`, `lint`, `inspect`, `preview`, `render`, `doctor` |
| `/hyperframes-media` | Asset preprocessing — `tts`, `transcribe`, `remove-background` |
| `/hyperframes-registry` | Block and component installation via `hyperframes add` |
| `/website-to-hyperframes` | Capture a URL and turn it into a video — full website-to-video pipeline |
| `/gsap` | GSAP animation API — timelines, easing, ScrollTrigger, plugins |
docs/quickstart.mdx
+1 -1
View File
@@ -13,7 +13,7 @@ Install the HyperFrames skills, then describe the video you want:
npx skills add heygen-com/hyperframes
```
This teaches your agent (Claude Code, Cursor, Gemini CLI, Codex) how to write correct compositions, GSAP timelines, Tailwind v4 browser-runtime styles, and first-party adapter animations. In Claude Code the skills register as slash commands — `/hyperframes` for composition authoring, `/hyperframes-cli` for CLI commands, `/tailwind` for `init --tailwind` projects, `/gsap` for timeline animation help, and `/animejs`, `/css-animations`, `/lottie`, `/three`, or `/waapi` when a composition uses those runtimes. Invoking the slash command loads the skill context explicitly, which produces correct output the first time.
This teaches your agent (Claude Code, Cursor, Gemini CLI, Codex) how to write correct compositions, GSAP timelines, Tailwind v4 browser-runtime styles, and first-party adapter animations. In Claude Code the skills register as slash commands — `/hyperframes` for composition authoring, `/hyperframes-cli` for the dev-loop commands (init, lint, preview, render), `/hyperframes-media` for asset preprocessing (TTS, transcription, background removal), `/tailwind` for `init --tailwind` projects, `/gsap` for timeline animation help, and `/animejs`, `/css-animations`, `/lottie`, `/three`, or `/waapi` when a composition uses those runtimes. Invoking the slash command loads the skill context explicitly, which produces correct output the first time.
<Note>
Claude Design uses a different entry path. Open [`docs/guides/claude-design-hyperframes.md`](https://github.com/heygen-com/hyperframes/blob/main/docs/guides/claude-design-hyperframes.md) on GitHub, click the download button (↓) to save it, then attach to your Claude Design chat. It produces a valid first draft you can refine in any AI coding agent. See the [Claude Design guide](/guides/claude-design).
packages/cli/src/templates/_shared/CLAUDE.md
+2 -1
View File
@@ -7,7 +7,8 @@
| Skill | Command | When to use |
| -------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------- |
| **hyperframes** | `/hyperframes` | Creating or editing HTML compositions, captions, TTS, audio-reactive animation, marker highlights |
| **hyperframes-cli** | `/hyperframes-cli` | CLI commands: init, lint, preview, render, transcribe, tts |
| **hyperframes-cli** | `/hyperframes-cli` | Dev-loop CLI: init, lint, inspect, preview, render, doctor |
| **hyperframes-media** | `/hyperframes-media` | Asset preprocessing: tts (Kokoro), transcribe (Whisper), remove-background (u2net) |
| **hyperframes-registry** | `/hyperframes-registry` | Installing blocks and components via `hyperframes add` |
| **website-to-hyperframes** | `/website-to-hyperframes` | Capturing a URL and turning it into a video — full website-to-video pipeline |
| **tailwind** | `/tailwind` | Tailwind v4 browser-runtime styles for projects created with `hyperframes init --tailwind` |
skills/hyperframes-cli/SKILL.md
+3 -31
View File
@@ -1,6 +1,6 @@
---
name: hyperframes-cli
description: HyperFrames CLI tool — hyperframes init, lint, inspect, preview, render, transcribe, tts, remove-background, doctor, browser, info, upgrade, compositions, docs, benchmark. Use when scaffolding a project, linting, validating, inspecting visual layout in compositions, previewing in the studio, rendering to video, transcribing audio, generating TTS, removing the background from an avatar video for transparent overlays, or troubleshooting the HyperFrames environment.
description: HyperFrames CLI dev loop — `npx hyperframes` for scaffolding (init), validation (lint, inspect), preview, render, and environment troubleshooting (doctor, browser, info, upgrade). Use when running any of these commands or troubleshooting the HyperFrames build/render environment. For asset preprocessing commands (`tts`, `transcribe`, `remove-background`), invoke the `hyperframes-media` skill instead.
---
# HyperFrames CLI
@@ -120,37 +120,9 @@ npx hyperframes render --docker # byte-identical
**Parametrized renders:** the composition declares its variables on the `<html>` root with **`data-composition-variables`** — a JSON **array of declarations** (`{id, type, label, default}` per entry) that defines the schema. Scripts inside read the resolved values via `window.__hyperframes.getVariables()`. The CLI **`--variables '{"title":"Q4 Report"}'`** is a JSON **object keyed by id** that overrides those declared defaults for one render; missing keys fall through, so the same composition runs unchanged in dev preview and in production. (Sub-comp hosts can also override per-instance with **`data-variable-values`** — same object shape, scoped to one mount of the sub-composition. See the `hyperframes` skill for the full pattern.)
## Transcription
## Asset Preprocessing
```bash
npx hyperframes transcribe audio.mp3
npx hyperframes transcribe video.mp4 --model medium.en --language en
npx hyperframes transcribe subtitles.srt # import existing
npx hyperframes transcribe subtitles.vtt
npx hyperframes transcribe openai-response.json
```
## Text-to-Speech
```bash
npx hyperframes tts "Text here" --voice af_nova --output narration.wav
npx hyperframes tts script.txt --voice bf_emma
npx hyperframes tts --list # show all voices
```
## Background Removal (transparent video)
Remove the background from a video or image so it can be used as a transparent overlay in a composition (e.g. an avatar floating on a background).
```bash
npx hyperframes remove-background avatar.mp4 -o transparent.webm # default: VP9 alpha WebM
npx hyperframes remove-background avatar.mp4 -o transparent.mov # ProRes 4444 for editing
npx hyperframes remove-background portrait.jpg -o cutout.png # single-image cutout
npx hyperframes remove-background avatar.mp4 -o transparent.webm --device cpu
npx hyperframes remove-background --info # detected providers
```
Uses `u2net_human_seg` (MIT). First run downloads ~168 MB of weights to `~/.cache/hyperframes/background-removal/models/` and reuses them after. Drop the resulting `.webm` into a composition with `<video src="transparent.webm" autoplay muted loop>` — Chrome decodes the alpha natively.
`npx hyperframes tts`, `transcribe`, and `remove-background` produce assets (narration audio, word-level transcripts, transparent video) that get dropped into a composition. Each downloads its own model on first run. For voice selection, whisper model rules (the `.en`-translates-non-English gotcha), output format choice (VP9 alpha WebM vs ProRes), and the TTS → transcribe → captions chain, invoke the `hyperframes-media` skill.
## Troubleshooting
skills/hyperframes-media/SKILL.md
+136
View File
@@ -0,0 +1,136 @@
---
name: hyperframes-media
description: Asset preprocessing for HyperFrames compositions — text-to-speech narration (Kokoro), audio/video transcription (Whisper), and background removal for transparent overlays (u2net). Use when generating voiceover from text, transcribing speech for captions, removing the background from a video or image to use as a transparent overlay, choosing a TTS voice or whisper model, or chaining these (TTS → transcribe → captions). Each command downloads its own model on first run.
---
# HyperFrames Media Preprocessing
Three CLI commands that produce assets for compositions: `tts` (speech), `transcribe` (timestamps), and `remove-background` (transparent video). Each downloads a model on first run and caches it under `~/.cache/hyperframes/`. Drop the output into the project, then reference it from the composition HTML — see the `hyperframes` skill for the audio/video element conventions.
## Text-to-Speech (`tts`)
Generate speech audio locally with Kokoro-82M. No API key.
```bash
npx hyperframes tts "Text here" --voice af_nova --output narration.wav
npx hyperframes tts script.txt --voice bf_emma --output narration.wav
npx hyperframes tts --list # all 54 voices
```
### Voice Selection
Match voice to content. Default is `af_heart`.
| Content type | Voice | Why |
| ----------------- | --------------------- | ----------------------------- |
| Product demo | `af_heart`/`af_nova` | Warm, professional |
| Tutorial / how-to | `am_adam`/`bf_emma` | Neutral, easy to follow |
| Marketing / promo | `af_sky`/`am_michael` | Energetic or authoritative |
| Documentation | `bf_emma`/`bm_george` | Clear British English, formal |
| Casual / social | `af_heart`/`af_sky` | Approachable, natural |
### Multilingual
Voice IDs encode language in the first letter: `a`=American English, `b`=British English, `e`=Spanish, `f`=French, `h`=Hindi, `i`=Italian, `j`=Japanese, `p`=Brazilian Portuguese, `z`=Mandarin. The CLI auto-detects the phonemizer locale from the prefix — no `--lang` needed when the voice matches the text.
```bash
npx hyperframes tts "La reunión empieza a las nueve" --voice ef_dora --output es.wav
npx hyperframes tts "今日はいい天気ですね" --voice jf_alpha --output ja.wav
```
Use `--lang` only to override auto-detection (stylized accents). Valid codes: `en-us`, `en-gb`, `es`, `fr-fr`, `hi`, `it`, `pt-br`, `ja`, `zh`. Non-English phonemization requires `espeak-ng` system-wide (`brew install espeak-ng` / `apt-get install espeak-ng`).
### Speed
- `0.7-0.8` — tutorial, complex content, accessibility
- `1.0` — natural pace (default)
- `1.1-1.2` — intros, transitions, upbeat content
- `1.5+` — rarely appropriate; test carefully
### Long Scripts
For more than a few paragraphs, write to a `.txt` file and pass the path. Inputs over ~5 minutes of speech may benefit from splitting into segments.
### Requirements
Python 3.8+ with `kokoro-onnx` and `soundfile` (`pip install kokoro-onnx soundfile`). Model downloads on first use (~311 MB + ~27 MB voices, cached in `~/.cache/hyperframes/tts/`).
## Transcription (`transcribe`)
Produce a normalized `transcript.json` with word-level timestamps.
```bash
npx hyperframes transcribe audio.mp3
npx hyperframes transcribe video.mp4 --model small --language es
npx hyperframes transcribe subtitles.srt # import existing
npx hyperframes transcribe subtitles.vtt
npx hyperframes transcribe openai-response.json
```
### Language Rule (Non-Negotiable)
**Never use `.en` models unless the user explicitly states the audio is English.** `.en` models (`small.en`, `medium.en`) **translate** non-English audio into English instead of transcribing it. This silently destroys the original language.
1. Language known and non-English → `--model small --language <code>` (no `.en` suffix)
2. Language known and English → `--model small.en`
3. Language unknown → `--model small` (no `.en`, no `--language`) — whisper auto-detects
**Default model is `small`, not `small.en`.**
### Model Sizes
| Model | Size | Speed | When to use |
| ---------- | ------ | -------- | ------------------------------------- |
| `tiny` | 75 MB | Fastest | Quick previews, testing pipeline |
| `base` | 142 MB | Fast | Short clips, clear audio |
| `small` | 466 MB | Moderate | **Default** — most content |
| `medium` | 1.5 GB | Slow | Important content, noisy audio, music |
| `large-v3` | 3.1 GB | Slowest | Production quality |
Music with vocals: start at `medium` minimum; produced tracks often need manual SRT/VTT import. For caption-quality checks (mandatory after every transcription), the cleaning JS, retry rules, and the OpenAI/Groq API import path, see [hyperframes/references/transcript-guide.md](../hyperframes/references/transcript-guide.md).
### Output Shape
Compositions consume a flat array of word objects. The `id` field (`w0`, `w1`, ...) is added during normalization for stable references in caption overrides; it's optional for backwards compatibility.
```json
[
{ "id": "w0", "text": "Hello", "start": 0.0, "end": 0.5 },
{ "id": "w1", "text": "world.", "start": 0.6, "end": 1.2 }
]
```
## Background Removal (`remove-background`)
Remove the background from a video or image so it can sit as a transparent overlay in a composition (e.g. an avatar floating on a background plate).
```bash
npx hyperframes remove-background avatar.mp4 -o transparent.webm # default: VP9 alpha WebM
npx hyperframes remove-background avatar.mp4 -o transparent.mov # ProRes 4444 (editing)
npx hyperframes remove-background portrait.jpg -o cutout.png # single-image cutout
npx hyperframes remove-background avatar.mp4 -o transparent.webm --device cpu
npx hyperframes remove-background --info # detected providers
```
Uses `u2net_human_seg` (MIT). First run downloads ~168 MB of weights to `~/.cache/hyperframes/background-removal/models/`.
### Output Format
| Format | When |
| --------------------- | ------------------------------------------------------------- |
| `.webm` (VP9 + alpha) | Default. Compositions play this directly via `<video>`. |
| `.mov` (ProRes 4444) | Editing in DaVinci/Premiere/FCP. Large files. |
| `.png` | Single-image cutout (still subject, layered over a backdrop). |
Chrome decodes VP9 alpha natively, so the `.webm` plugs into a composition like any other muted-autoplay video — see the `hyperframes` skill for the `<video>` track conventions.
## TTS → Transcribe → Captions
When there's no pre-recorded voiceover, generate one and transcribe it back to get word-level timestamps for captions:
```bash
npx hyperframes tts script.txt --voice af_heart --output narration.wav
npx hyperframes transcribe narration.wav # → transcript.json
```
Whisper extracts precise word boundaries from the generated audio, so caption timing matches delivery without hand-tuning.
skills/hyperframes/SKILL.md
+2 -3
View File
@@ -1,6 +1,6 @@
---
name: hyperframes
description: Create video compositions, animations, title cards, overlays, captions, voiceovers, audio-reactive visuals, and scene transitions in HyperFrames HTML. Use when asked to build any HTML-based video content, add captions or subtitles synced to audio, generate text-to-speech narration, create audio-reactive animation (beat sync, glow, pulse driven by music), add animated text highlighting (marker sweeps, hand-drawn circles, burst lines, scribble, sketchout), or add transitions between scenes (crossfades, wipes, reveals, shader transitions). Covers composition authoring, timing, media, and the full video production workflow. For CLI commands (init, lint, preview, render, transcribe, tts) see the hyperframes-cli skill.
description: Create video compositions, animations, title cards, overlays, captions, voiceovers, audio-reactive visuals, and scene transitions in HyperFrames HTML. Use when asked to build any HTML-based video content, add captions or subtitles synced to audio, generate text-to-speech narration, create audio-reactive animation (beat sync, glow, pulse driven by music), add animated text highlighting (marker sweeps, hand-drawn circles, burst lines, scribble, sketchout), or add transitions between scenes (crossfades, wipes, reveals, shader transitions). Covers composition authoring, timing, media, and the full video production workflow. For dev-loop CLI commands (init, lint, inspect, preview, render) see the hyperframes-cli skill; for asset preprocessing commands (tts, transcribe, remove-background) see the hyperframes-media skill.
---
# HyperFrames
@@ -467,7 +467,6 @@ Skip on small edits (fixing a color, adjusting one duration). Run on new composi
## References (loaded on demand)
- **[references/captions.md](references/captions.md)** — Captions, subtitles, lyrics, karaoke synced to audio. Tone-adaptive style detection, per-word styling, text overflow prevention, caption exit guarantees, word grouping. Read when adding any text synced to audio timing.
- **[references/tts.md](references/tts.md)** — Text-to-speech with Kokoro-82M. Voice selection, speed tuning, TTS+captions workflow. Read when generating narration or voiceover.
- **[references/audio-reactive.md](references/audio-reactive.md)** — Audio-reactive animation: map frequency bands and amplitude to GSAP properties. Read when visuals should respond to music, voice, or sound.
- **[references/css-patterns.md](references/css-patterns.md)** — CSS+GSAP marker highlighting: highlight, circle, burst, scribble, sketchout. Deterministic, fully seekable. Read when adding visual emphasis to text.
- **[references/video-composition.md](references/video-composition.md)** — Video-medium rules: density, color presence, scale, frame composition, design.md as brand not layout. **Always read** — these override web instincts.
@@ -481,7 +480,7 @@ Skip on small edits (fixing a color, adjusting one duration). Run on new composi
- **[house-style.md](house-style.md)** — Default motion, sizing, and color palettes when no design.md is specified.
- **[patterns.md](patterns.md)** — PiP, title cards, slide show patterns.
- **[data-in-motion.md](data-in-motion.md)** — Data, stats, and infographic patterns.
- **[references/transcript-guide.md](references/transcript-guide.md)** — Transcription commands, whisper models, external APIs, troubleshooting.
- **[references/transcript-guide.md](references/transcript-guide.md)** — Caption-side transcript handling: input formats, mandatory quality check, cleaning JS, OpenAI/Groq API fallback, "if no transcript exists" flow. (For the `transcribe` CLI invocation, model selection rules, and the `.en` gotcha, see the `hyperframes-media` skill.)
- **[references/dynamic-techniques.md](references/dynamic-techniques.md)** — Dynamic caption animation techniques (karaoke, clip-path, slam, scatter, elastic, 3D).
- **[references/transitions.md](references/transitions.md)** — Scene transitions: crossfades, wipes, reveals, shader transitions. Energy/mood selection, CSS vs WebGL guidance. **Always read for multi-scene compositions** — scenes without transitions feel like jump cuts.
skills/hyperframes/references/transcript-guide.md
+1 -45
View File
@@ -1,24 +1,6 @@
# Transcript Guide
## How Transcripts Are Generated
`hyperframes transcribe` handles both transcription and format conversion:
```bash
# Transcribe audio/video (uses whisper.cpp locally, no API key needed)
npx hyperframes transcribe audio.mp3
# Use a larger model for better accuracy
npx hyperframes transcribe audio.mp3 --model medium.en
# Filter to English only (skips non-English speech)
npx hyperframes transcribe audio.mp3 --language en
# Import an existing transcript from another tool
npx hyperframes transcribe captions.srt
npx hyperframes transcribe captions.vtt
npx hyperframes transcribe openai-response.json
```
For the `transcribe` CLI invocation, the `.en`-translates-non-English rule, and whisper model selection, see the `hyperframes-media` skill. This file covers what to do with the resulting transcript when authoring captions: input formats, mandatory quality checks, cleaning code, external-API fallbacks.
## Supported Input Formats
@@ -34,32 +16,6 @@ The CLI auto-detects and normalizes these formats:
**Word-level timestamps produce better captions.** SRT/VTT give phrase-level timing, which works but can't do per-word animation effects.
## Whisper Model Guide
The default model (`small.en`) balances accuracy and speed. For better results, use a larger model:
| Model | Size | Speed | Accuracy | When to use |
| ---------- | ------ | -------- | --------- | ------------------------------------- |
| `tiny` | 75 MB | Fastest | Low | Quick previews, testing pipeline |
| `base` | 142 MB | Fast | Fair | Short clips, clear audio |
| `small` | 466 MB | Moderate | Good | **Default** — good for most content |
| `medium` | 1.5 GB | Slow | Very good | Important content, noisy audio, music |
| `large-v3` | 3.1 GB | Slowest | Best | Production quality |
**Only add `.en` suffix when the user explicitly says the audio is English.** `.en` models are slightly more accurate for English but will TRANSLATE non-English audio instead of transcribing it.
**Critical: `.en` models translate non-English audio into English** — they don't transcribe it. If the audio might not be English, always use a model without the `.en` suffix and pass `--language` to specify the source language. If you're unsure of the language, use `small` (not `small.en`) without `--language` — whisper will auto-detect.
```bash
# Spanish audio
npx hyperframes transcribe audio.mp3 --model small --language es
# Unknown language — let whisper auto-detect
npx hyperframes transcribe audio.mp3 --model small
```
**Music and vocals over instrumentation**: `small.en` will misidentify lyrics — use `medium.en` as the minimum, or import lyrics manually. Even `medium.en` struggles with heavily produced tracks; for music videos, providing known lyrics as an SRT/VTT and importing with `hyperframes transcribe lyrics.srt` will always beat automated transcription.
## Transcript Quality Check (Mandatory)
After every transcription, **read the transcript and check for quality issues before proceeding.** Bad transcripts produce nonsensical captions. Never skip this step.
skills/hyperframes/references/tts.md
-75
View File
@@ -1,75 +0,0 @@
# Text-to-Speech
Generate speech audio locally using Kokoro-82M (no API key, runs on CPU).
## Voice Selection
Match voice to content. Default is `af_heart`.
| Content type | Voice | Why |
| ------------- | --------------------- | -------------------------- |
| Product demo | `af_heart`/`af_nova` | Warm, professional |
| Tutorial | `am_adam`/`bf_emma` | Neutral, easy to follow |
| Marketing | `af_sky`/`am_michael` | Energetic or authoritative |
| Documentation | `bf_emma`/`bm_george` | Clear British English |
| Casual | `af_heart`/`af_sky` | Approachable, natural |
Run `npx hyperframes tts --list` for all 54 voices (8 languages).
## Multilingual Phonemization
Kokoro voice IDs encode language in the first letter: `a`=American English, `b`=British English, `e`=Spanish, `f`=French, `h`=Hindi, `i`=Italian, `j`=Japanese, `p`=Brazilian Portuguese, `z`=Mandarin. The CLI auto-detects the phonemizer locale from that prefix — you don't need to pass `--lang` when the voice matches the text.
```bash
npx hyperframes tts "La reunión empieza a las nueve" --voice ef_dora --output es.wav
npx hyperframes tts "今日はいい天気ですね" --voice jf_alpha --output ja.wav
```
Use `--lang` only to override auto-detection (e.g. stylized accents):
```bash
npx hyperframes tts "Hello there" --voice af_heart --lang fr-fr --output accented.wav
```
Valid `--lang` codes: `en-us`, `en-gb`, `es`, `fr-fr`, `hi`, `it`, `pt-br`, `ja`, `zh`.
Non-English phonemization requires `espeak-ng` installed system-wide (`brew install espeak-ng` on macOS, `apt-get install espeak-ng` on Debian/Ubuntu).
## Speed Tuning
- **0.7-0.8** — Tutorial, complex content
- **1.0** — Natural pace (default)
- **1.1-1.2** — Intros, upbeat content
- **1.5+** — Rarely appropriate
## Usage
```bash
npx hyperframes tts "Your script here" --voice af_nova --output narration.wav
npx hyperframes tts script.txt --voice bf_emma --output narration.wav
```
In compositions:
```html
<audio
id="narration"
data-start="0"
data-duration="auto"
data-track-index="2"
src="narration.wav"
data-volume="1"
></audio>
```
## TTS + Captions Workflow
```bash
npx hyperframes tts script.txt --voice af_heart --output narration.wav
npx hyperframes transcribe narration.wav # → transcript.json with word-level timestamps
```
## Requirements
- Python 3.8+ with `kokoro-onnx` and `soundfile`
- Model downloads on first use (~311 MB + ~27 MB voices, cached in `~/.cache/hyperframes/tts/`)