mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-12 07:09:59 +00:00
refactor(skills): consolidate 15 skills into 3 (#211)
* refactor(skills): consolidate 15 skills into 3 for better trigger reliability Merge 9 GSAP skills (core, timeline, scrolltrigger, plugins, utils, react, frameworks, performance, effects) and 6 HyperFrames skills (compose, captions, tts, audio-reactive, marker-highlight, cli) into 3 consolidated skills: - `gsap` — core API + timelines + performance in SKILL.md; scrolltrigger, plugins, utils, react, frameworks, effects in references/ - `hyperframes` — composition authoring rules in SKILL.md; captions, tts, audio-reactive, marker-highlight in references/ - `hyperframes-cli` — CLI commands (init, lint, preview, render, etc.) Why: With 15 separate skills, agents must correctly trigger the right subset for any task. "Create an animated video with captions" needed 6+ skills to fire — each with ~90% trigger accuracy means ~53% chance of getting all of them. With 3 skills, that same task needs just `hyperframes` + `gsap` (~90% both fire). Progressive disclosure still works via references/ files loaded on demand. Also fixes: CLAUDE.md referenced `window.__GSAP_TIMELINE` (incorrect) — corrected to `window.__timelines`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(cli): add --skip-skills flag to init command Allow skipping the AI coding skills installation prompt during `hyperframes init` with `--skip-skills`. Useful when skills are already installed or when the user wants to scaffold without them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(skills): address code review feedback on consolidation Restore content lost during over-compression: - captions: fix overflow to `visible` (not hidden — clips glow effects), add container pattern warning, scale headroom formula, and self-lint placement guidance - audio-reactive: restore sampling frequency pattern (per-frame tl.call loop vs single tween) and textShadow-on-container gotcha - effects/typewriter: restore word rotation, appending words, spacing with static text, and multi-line cursor handoff patterns - effects/audio-visualizer: restore spatial mapping conventions, fetch vs inline loading, WebGL/DOM rendering approaches, and canvas layering - hyperframes-cli: restore --strict-all flag in render flags table Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): update build:copy and template for consolidated skill names - build:copy: reference skills/hyperframes, skills/hyperframes-cli, skills/gsap instead of the old 15 skill directory names - _shared/CLAUDE.md template: update skill table to consolidated names Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
James Russo
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
5655dabff6
commit
0a0d5d3654
@@ -0,0 +1,76 @@
|
||||
# Audio-Reactive Animation
|
||||
|
||||
Drive visuals from music, voice, or sound. Any GSAP-animatable property can respond to pre-extracted audio data.
|
||||
|
||||
## Audio Data Format
|
||||
|
||||
```js
|
||||
var AUDIO_DATA = {
|
||||
fps: 30,
|
||||
totalFrames: 900,
|
||||
frames: [{ bands: [0.82, 0.45, 0.31, ...] }, ...]
|
||||
};
|
||||
```
|
||||
|
||||
- `frames[i].bands[]` — frequency band amplitudes, 0-1. Index 0 = bass, higher = treble.
|
||||
- Each band normalized independently across the full track.
|
||||
|
||||
## Mapping Audio to Visuals
|
||||
|
||||
| Audio signal | Visual property | Effect |
|
||||
| ---------------------- | --------------------------------- | -------------------------- |
|
||||
| Bass (bands[0]) | `scale` | Pulse on beat |
|
||||
| Treble (bands[12-14]) | `textShadow`, `boxShadow` | Glow intensity |
|
||||
| Overall amplitude | `opacity`, `y`, `backgroundColor` | Breathe, lift, color shift |
|
||||
| Mid-range (bands[4-8]) | `borderRadius`, `width` | Shape morphing |
|
||||
|
||||
Any GSAP-tweenable property works — `clipPath`, `filter`, SVG attributes, CSS custom properties.
|
||||
|
||||
## Content, Not Medium
|
||||
|
||||
Audio provides **timing and intensity**. The visual vocabulary comes from the narrative.
|
||||
|
||||
**Never add:** equalizer bars, spectrum analyzers, waveform displays, musical notes clip art, generic particle systems, rainbow color cycling, strobing white on beats, abstract pulsing orbs.
|
||||
|
||||
**Instead:** Let content guide the visual and audio drive its behavior. Bass makes warmth _swell_. Treble sharpens _contrast_. The visual choice comes from "what does this piece feel like?"
|
||||
|
||||
## Sampling Pattern
|
||||
|
||||
Audio reactivity requires per-frame sampling via a `for` loop with `tl.call()`, not a single tween:
|
||||
|
||||
```js
|
||||
// ✅ Correct — sample every frame
|
||||
for (var f = 0; f < AUDIO_DATA.totalFrames; f++) {
|
||||
tl.call(
|
||||
(function (frame) {
|
||||
return function () {
|
||||
draw(frame);
|
||||
};
|
||||
})(AUDIO_DATA.frames[f]),
|
||||
[],
|
||||
f / AUDIO_DATA.fps,
|
||||
);
|
||||
}
|
||||
|
||||
// ❌ Wrong — single tween, doesn't react to audio
|
||||
gsap.to(".el", { scale: 1.2, duration: totalDuration });
|
||||
```
|
||||
|
||||
Without per-frame sampling, the composition doesn't actually react to audio.
|
||||
|
||||
## textShadow Gotcha
|
||||
|
||||
`textShadow` on a parent container with semi-transparent children (e.g., inactive caption words at `rgba(255,255,255,0.3)`) renders a visible glow rectangle behind all children. Fix: apply `scale` to the container for beat pulse, but apply `textShadow` to individual active words only.
|
||||
|
||||
## Guidelines
|
||||
|
||||
- **Subtlety for text** — 3-6% scale variation, soft glow. Heavy pulsing makes text unreadable.
|
||||
- **Go bigger on non-text** — backgrounds and shapes can handle 10-30% swings.
|
||||
- **Match the energy** — corporate = subtle; music video = dramatic.
|
||||
- **Deterministic** — pre-extracted data, no Web Audio API, no runtime analysis.
|
||||
|
||||
## Constraints
|
||||
|
||||
- All audio data must be pre-extracted (use `extract-audio-data.py` from the gsap skill's scripts/)
|
||||
- No `Math.random()` or `Date.now()`
|
||||
- Audio reactivity runs on the same GSAP timeline as everything else
|
||||
@@ -0,0 +1,132 @@
|
||||
# Captions
|
||||
|
||||
## Language Rule (Non-Negotiable)
|
||||
|
||||
**Never use `.en` models unless the user explicitly states the audio is English.** `.en` models TRANSLATE non-English audio into English instead of transcribing it.
|
||||
|
||||
1. User says the language → `--model small --language <code>` (no `.en`)
|
||||
2. User says English → `--model small.en`
|
||||
3. Language unknown → `--model small` (no `.en`, no `--language`) — auto-detects
|
||||
|
||||
---
|
||||
|
||||
Analyze spoken content to determine caption style. If user specifies a style, use that. Otherwise, detect tone from the transcript.
|
||||
|
||||
## Transcript Source
|
||||
|
||||
```json
|
||||
[
|
||||
{ "text": "Hello", "start": 0.0, "end": 0.5 },
|
||||
{ "text": "world.", "start": 0.6, "end": 1.2 }
|
||||
]
|
||||
```
|
||||
|
||||
For transcription commands, whisper models, external APIs, see [transcript-guide.md](transcript-guide.md).
|
||||
|
||||
## Style Detection (When No Style Specified)
|
||||
|
||||
Read the full transcript before choosing. Four dimensions:
|
||||
|
||||
**1. Visual feel** — corporate→clean; energetic→bold; storytelling→elegant; technical→precise; social→playful.
|
||||
|
||||
**2. Color palette** — dark+bright for energy; muted for professional; high contrast for clarity; one accent color.
|
||||
|
||||
**3. Font mood** — heavy/condensed for impact; clean sans for modern; rounded for friendly; serif for elegance.
|
||||
|
||||
**4. Animation character** — scale-pop for punchy; gentle fade for calm; word-by-word for emphasis; typewriter for technical.
|
||||
|
||||
## Per-Word Styling
|
||||
|
||||
Scan for words deserving distinct treatment:
|
||||
|
||||
- **Brand/product names** — larger size, unique color
|
||||
- **ALL CAPS** — scale boost, flash, accent color
|
||||
- **Numbers/statistics** — bold weight, accent color
|
||||
- **Emotional keywords** — exaggerated animation (overshoot, bounce)
|
||||
- **Call-to-action** — highlight, underline, color pop
|
||||
- **Marker highlight** — for beyond-color emphasis, see [marker-highlight.md](marker-highlight.md)
|
||||
|
||||
## Script-to-Style Mapping
|
||||
|
||||
| Tone | Font mood | Animation | Color | Size |
|
||||
| ------------ | ------------------------ | ---------------------------------- | --------------------------- | ------- |
|
||||
| Hype/launch | Heavy condensed, 800-900 | Scale-pop, back.out(1.7), 0.1-0.2s | Bright on dark | 72-96px |
|
||||
| Corporate | Clean sans, 600-700 | Fade+slide, power3.out, 0.3s | White/neutral, muted accent | 56-72px |
|
||||
| Tutorial | Mono/clean sans, 500-600 | Typewriter/fade, 0.4-0.5s | High contrast, minimal | 48-64px |
|
||||
| Storytelling | Serif/elegant, 400-500 | Slow fade, power2.out, 0.5-0.6s | Warm muted tones | 44-56px |
|
||||
| Social | Rounded sans, 700-800 | Bounce, elastic.out, word-by-word | Playful, colored pills | 56-80px |
|
||||
|
||||
## Word Grouping
|
||||
|
||||
- **High energy:** 2-3 words. Quick turnover.
|
||||
- **Conversational:** 3-5 words. Natural phrases.
|
||||
- **Measured/calm:** 4-6 words. Longer groups.
|
||||
|
||||
Break on sentence boundaries, 150ms+ pauses, or max word count.
|
||||
|
||||
## Positioning
|
||||
|
||||
- **Landscape (1920x1080):** Bottom 80-120px, centered
|
||||
- **Portrait (1080x1920):** Lower middle ~600-700px from bottom, centered
|
||||
- Never cover the subject's face
|
||||
- `position: absolute` — never relative
|
||||
- One caption group visible at a time
|
||||
|
||||
## Text Overflow Prevention
|
||||
|
||||
Use `window.__hyperframes.fitTextFontSize()`:
|
||||
|
||||
```js
|
||||
var result = window.__hyperframes.fitTextFontSize(group.text.toUpperCase(), {
|
||||
fontFamily: "Outfit",
|
||||
fontWeight: 900,
|
||||
maxWidth: 1600,
|
||||
});
|
||||
el.style.fontSize = result.fontSize + "px";
|
||||
```
|
||||
|
||||
Options: `maxWidth` (1600 landscape, 900 portrait), `baseFontSize` (78), `minFontSize` (42), `fontWeight`, `fontFamily`, `step` (2).
|
||||
|
||||
CSS safety nets: `max-width` on container, `overflow: visible` (**not** `hidden` — hidden clips scaled emphasis words and glow effects), `position: absolute`, explicit `height`. When per-word styling uses `scale > 1.0`, compute `maxWidth = safeWidth / maxScale` to leave headroom.
|
||||
|
||||
**Container pattern:** Full-width absolute container, centered. Do **not** use `left: 50%; transform: translateX(-50%)` — causes clipping at composition edges.
|
||||
|
||||
## Caption Exit Guarantee
|
||||
|
||||
Every group **must** have a hard kill after exit animation:
|
||||
|
||||
```js
|
||||
tl.to(groupEl, { opacity: 0, scale: 0.95, duration: 0.12, ease: "power2.in" }, group.end - 0.12);
|
||||
tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end); // deterministic kill
|
||||
```
|
||||
|
||||
Self-lint after building timeline — place **before** `window.__timelines[id] = tl` so it runs at composition init:
|
||||
|
||||
```js
|
||||
GROUPS.forEach(function (group, gi) {
|
||||
var el = document.getElementById("cg-" + gi);
|
||||
if (!el) return;
|
||||
tl.seek(group.end + 0.01);
|
||||
var computed = window.getComputedStyle(el);
|
||||
if (computed.opacity !== "0" && computed.visibility !== "hidden") {
|
||||
console.warn(
|
||||
"[caption-lint] group " + gi + " still visible at t=" + (group.end + 0.01).toFixed(2) + "s",
|
||||
);
|
||||
}
|
||||
});
|
||||
tl.seek(0);
|
||||
```
|
||||
|
||||
## Further References
|
||||
|
||||
- [dynamic-techniques.md](dynamic-techniques.md) — karaoke, clip-path reveals, slam words, scatter exits, elastic, 3D rotation
|
||||
- [transcript-guide.md](transcript-guide.md) — transcription commands, whisper models, external APIs
|
||||
- [marker-highlight.md](marker-highlight.md) — animated text emphasis paired with per-word styling
|
||||
|
||||
## Constraints
|
||||
|
||||
- Deterministic. No `Math.random()`, no `Date.now()`.
|
||||
- Sync to transcript timestamps.
|
||||
- One group visible at a time.
|
||||
- Every group must have a hard `tl.set` kill at `group.end`.
|
||||
- Check project root for font files before defaulting to Google Fonts.
|
||||
@@ -0,0 +1,371 @@
|
||||
# CSS Patterns for Marker Highlighting
|
||||
|
||||
Pure CSS + GSAP implementations of all five MarkerHighlight.js drawing modes. Use these for deterministic rendering in HyperFrames compositions — no external library dependency, full GSAP timeline control.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [1. Highlight Mode](#1-highlight-mode) — Yellow marker sweep behind text
|
||||
- [2. Circle Mode](#2-circle-mode) — Hand-drawn ellipse around text
|
||||
- [3. Burst Mode](#3-burst-mode) — Radiating lines from text
|
||||
- [4. Scribble Mode](#4-scribble-mode) — Chaotic scribble over text
|
||||
- [5. Sketchout Mode](#5-sketchout-mode) — Rough rectangle outline
|
||||
|
||||
## 1. Highlight Mode
|
||||
|
||||
Yellow marker sweep behind text. The most common mode.
|
||||
|
||||
```html
|
||||
<div class="mh-highlight-wrap">
|
||||
<div class="mh-highlight-bar" id="hl-1"></div>
|
||||
<span class="mh-highlight-text">highlighted text</span>
|
||||
</div>
|
||||
```
|
||||
|
||||
```css
|
||||
.mh-highlight-wrap {
|
||||
position: relative;
|
||||
display: inline-block;
|
||||
}
|
||||
.mh-highlight-bar {
|
||||
position: absolute;
|
||||
top: 0;
|
||||
left: -6px;
|
||||
right: -6px;
|
||||
bottom: 0;
|
||||
background: #fdd835;
|
||||
opacity: 0.35;
|
||||
transform: scaleX(0);
|
||||
transform-origin: left center;
|
||||
border-radius: 3px;
|
||||
z-index: 0;
|
||||
}
|
||||
.mh-highlight-text {
|
||||
position: relative;
|
||||
z-index: 1;
|
||||
}
|
||||
```
|
||||
|
||||
```js
|
||||
// Sweep in from left
|
||||
tl.to("#hl-1", { scaleX: 1, duration: 0.5, ease: "power2.out" }, 0.6);
|
||||
|
||||
// Optional: skew for hand-drawn feel
|
||||
// gsap.set("#hl-1", { skewX: -2 });
|
||||
```
|
||||
|
||||
### Multi-line Highlight
|
||||
|
||||
Stagger bars across multiple lines:
|
||||
|
||||
```js
|
||||
tl.to(
|
||||
".mh-highlight-bar",
|
||||
{
|
||||
scaleX: 1,
|
||||
duration: 0.5,
|
||||
ease: "power2.out",
|
||||
stagger: 0.3,
|
||||
},
|
||||
0.6,
|
||||
);
|
||||
```
|
||||
|
||||
## 2. Circle Mode
|
||||
|
||||
Hand-drawn circle around text. Use `border-radius: 50%` with a slight rotation for organic feel.
|
||||
|
||||
```html
|
||||
<div class="mh-circle-wrap">
|
||||
<span class="mh-circle-text" id="circle-word">IMPORTANT</span>
|
||||
<div class="mh-circle-ring" id="circle-1"></div>
|
||||
</div>
|
||||
```
|
||||
|
||||
```css
|
||||
.mh-circle-wrap {
|
||||
position: relative;
|
||||
display: inline-block;
|
||||
}
|
||||
.mh-circle-text {
|
||||
position: relative;
|
||||
z-index: 1;
|
||||
}
|
||||
.mh-circle-ring {
|
||||
position: absolute;
|
||||
top: 50%;
|
||||
left: 50%;
|
||||
width: 130%;
|
||||
height: 160%;
|
||||
transform: translate(-50%, -50%) rotate(-3deg) scale(0);
|
||||
border: 3px solid #e53935;
|
||||
border-radius: 50%;
|
||||
pointer-events: none;
|
||||
z-index: 0;
|
||||
}
|
||||
```
|
||||
|
||||
```js
|
||||
// Circle scales in with a wobble
|
||||
tl.to(
|
||||
"#circle-1",
|
||||
{
|
||||
scale: 1,
|
||||
rotation: -3,
|
||||
duration: 0.6,
|
||||
ease: "back.out(1.7)",
|
||||
transformOrigin: "center center",
|
||||
},
|
||||
0.7,
|
||||
);
|
||||
```
|
||||
|
||||
### Variations
|
||||
|
||||
```css
|
||||
/* Tighter circle (for short words) */
|
||||
.mh-circle-ring.tight {
|
||||
width: 150%;
|
||||
height: 180%;
|
||||
}
|
||||
|
||||
/* Squared circle (rounded rectangle) */
|
||||
.mh-circle-ring.rounded {
|
||||
border-radius: 30%;
|
||||
width: 120%;
|
||||
height: 140%;
|
||||
}
|
||||
|
||||
/* Ellipse (wider than tall) */
|
||||
.mh-circle-ring.ellipse {
|
||||
width: 150%;
|
||||
height: 130%;
|
||||
border-radius: 50%;
|
||||
}
|
||||
```
|
||||
|
||||
## 3. Burst Mode
|
||||
|
||||
Radiating lines from text center. Each line is a positioned div rotated to its angle.
|
||||
|
||||
```html
|
||||
<div class="mh-burst-wrap">
|
||||
<span class="mh-burst-text">WOW</span>
|
||||
<div class="mh-burst-container" id="burst-1">
|
||||
<div class="mh-burst-line" style="--angle: 0deg; --len: 70px;"></div>
|
||||
<div class="mh-burst-line" style="--angle: 30deg; --len: 55px;"></div>
|
||||
<div class="mh-burst-line" style="--angle: 60deg; --len: 80px;"></div>
|
||||
<div class="mh-burst-line" style="--angle: 90deg; --len: 45px;"></div>
|
||||
<div class="mh-burst-line" style="--angle: 120deg; --len: 65px;"></div>
|
||||
<div class="mh-burst-line" style="--angle: 150deg; --len: 75px;"></div>
|
||||
<div class="mh-burst-line" style="--angle: 180deg; --len: 50px;"></div>
|
||||
<div class="mh-burst-line" style="--angle: 210deg; --len: 60px;"></div>
|
||||
<div class="mh-burst-line" style="--angle: 240deg; --len: 80px;"></div>
|
||||
<div class="mh-burst-line" style="--angle: 270deg; --len: 40px;"></div>
|
||||
<div class="mh-burst-line" style="--angle: 300deg; --len: 70px;"></div>
|
||||
<div class="mh-burst-line" style="--angle: 330deg; --len: 55px;"></div>
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
```css
|
||||
.mh-burst-wrap {
|
||||
position: relative;
|
||||
display: inline-block;
|
||||
}
|
||||
.mh-burst-text {
|
||||
position: relative;
|
||||
z-index: 2;
|
||||
}
|
||||
.mh-burst-container {
|
||||
position: absolute;
|
||||
top: 50%;
|
||||
left: 50%;
|
||||
width: 0;
|
||||
height: 0;
|
||||
z-index: 1;
|
||||
}
|
||||
.mh-burst-line {
|
||||
position: absolute;
|
||||
width: 3px;
|
||||
height: var(--len);
|
||||
background: #1e88e5;
|
||||
left: -1.5px;
|
||||
top: calc(-1 * var(--len));
|
||||
transform: rotate(var(--angle));
|
||||
transform-origin: bottom center;
|
||||
opacity: 0;
|
||||
}
|
||||
```
|
||||
|
||||
```js
|
||||
// All lines burst outward simultaneously with slight stagger
|
||||
tl.fromTo(
|
||||
"#burst-1 .mh-burst-line",
|
||||
{ scaleY: 0, opacity: 0 },
|
||||
{ scaleY: 1, opacity: 1, duration: 0.4, ease: "power2.out", stagger: 0.03 },
|
||||
0.7,
|
||||
);
|
||||
```
|
||||
|
||||
**Vary line lengths** (40-80px range) for an organic, hand-drawn feel. Equal lengths look mechanical.
|
||||
|
||||
## 4. Scribble Mode
|
||||
|
||||
Wavy SVG underlines and strikethroughs that draw themselves via `stroke-dashoffset`.
|
||||
|
||||
```html
|
||||
<div class="mh-scribble-wrap">
|
||||
<span class="mh-scribble-text">underlined text</span>
|
||||
<svg class="mh-scribble-svg" viewBox="0 0 500 24" preserveAspectRatio="none">
|
||||
<path
|
||||
id="scribble-1"
|
||||
d="M0,12 Q31,0 62,12 Q93,24 125,12 Q156,0 187,12 Q218,24 250,12 Q281,0 312,12 Q343,24 375,12 Q406,0 437,12 Q468,24 500,12"
|
||||
fill="none"
|
||||
stroke="#FDD835"
|
||||
stroke-width="3"
|
||||
stroke-linecap="round"
|
||||
/>
|
||||
</svg>
|
||||
</div>
|
||||
```
|
||||
|
||||
```css
|
||||
.mh-scribble-wrap {
|
||||
position: relative;
|
||||
display: inline-block;
|
||||
}
|
||||
.mh-scribble-text {
|
||||
position: relative;
|
||||
z-index: 1;
|
||||
}
|
||||
.mh-scribble-svg {
|
||||
position: absolute;
|
||||
left: 0;
|
||||
bottom: -6px;
|
||||
width: 100%;
|
||||
height: 24px;
|
||||
z-index: 0;
|
||||
}
|
||||
```
|
||||
|
||||
```js
|
||||
// Measure path length and set initial dash state
|
||||
var path = document.querySelector("#scribble-1");
|
||||
var len = path.getTotalLength();
|
||||
gsap.set(path, { strokeDasharray: len, strokeDashoffset: len });
|
||||
|
||||
// Draw the line
|
||||
tl.to(
|
||||
"#scribble-1",
|
||||
{
|
||||
strokeDashoffset: 0,
|
||||
duration: 0.8,
|
||||
ease: "power1.inOut",
|
||||
},
|
||||
0.7,
|
||||
);
|
||||
```
|
||||
|
||||
### Strikethrough Variant
|
||||
|
||||
Position the SVG at `top: 50%; transform: translateY(-50%)` instead of `bottom: -6px`.
|
||||
|
||||
### Wavy Path Generator
|
||||
|
||||
Scale the path's viewBox width to match text width. The wave pattern `Q x1,y1 x2,y2` alternates between `y=0` and `y=24` for a natural wobble. Adjust the control points for tighter or looser waves:
|
||||
|
||||
- **Tight waves**: smaller x-increments (25px per half-wave)
|
||||
- **Loose waves**: larger x-increments (50px per half-wave)
|
||||
- **Amplitude**: change the y range (0-24 for standard, 0-16 for subtle)
|
||||
|
||||
## 5. Sketchout Mode
|
||||
|
||||
Cross-hatch lines over de-emphasized text. Multiple angled lines create a "crossed out" effect.
|
||||
|
||||
```html
|
||||
<div class="mh-sketchout-wrap">
|
||||
<span class="mh-sketchout-text">old price</span>
|
||||
<div class="mh-sketchout-lines" id="sketchout-1">
|
||||
<div class="mh-sketchout-line mh-sketchout-fwd"></div>
|
||||
<div class="mh-sketchout-line mh-sketchout-bwd"></div>
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
```css
|
||||
.mh-sketchout-wrap {
|
||||
position: relative;
|
||||
display: inline-block;
|
||||
}
|
||||
.mh-sketchout-text {
|
||||
position: relative;
|
||||
z-index: 0;
|
||||
}
|
||||
.mh-sketchout-lines {
|
||||
position: absolute;
|
||||
top: 0;
|
||||
left: -4px;
|
||||
right: -4px;
|
||||
bottom: 0;
|
||||
overflow: hidden;
|
||||
z-index: 1;
|
||||
}
|
||||
.mh-sketchout-line {
|
||||
position: absolute;
|
||||
top: 50%;
|
||||
left: 0;
|
||||
width: 100%;
|
||||
height: 2px;
|
||||
background: #e53935;
|
||||
transform-origin: left center;
|
||||
transform: scaleX(0);
|
||||
}
|
||||
.mh-sketchout-fwd {
|
||||
transform: scaleX(0) rotate(-12deg);
|
||||
}
|
||||
.mh-sketchout-bwd {
|
||||
transform: scaleX(0) rotate(12deg);
|
||||
}
|
||||
```
|
||||
|
||||
```js
|
||||
// Forward slash draws first
|
||||
tl.to(
|
||||
"#sketchout-1 .mh-sketchout-fwd",
|
||||
{
|
||||
scaleX: 1,
|
||||
duration: 0.3,
|
||||
ease: "power2.out",
|
||||
},
|
||||
1.0,
|
||||
);
|
||||
|
||||
// Backward slash follows
|
||||
tl.to(
|
||||
"#sketchout-1 .mh-sketchout-bwd",
|
||||
{
|
||||
scaleX: 1,
|
||||
duration: 0.3,
|
||||
ease: "power2.out",
|
||||
},
|
||||
1.15,
|
||||
);
|
||||
```
|
||||
|
||||
## Combining Modes in Captions
|
||||
|
||||
Use mode cycling for visual variety across caption groups:
|
||||
|
||||
```js
|
||||
var MODES = ["highlight", "circle", "burst", "scribble"];
|
||||
|
||||
GROUPS.forEach(function (group, gi) {
|
||||
var mode = MODES[gi % MODES.length];
|
||||
// Apply the mode's CSS pattern to emphasis words in this group
|
||||
group.emphasisWords.forEach(function (word) {
|
||||
applyMode(word.el, mode, tl, word.start);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
Cycle every 2-3 groups for high energy, every 3-4 for medium, every 4-5 for low.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Dynamic Caption Techniques
|
||||
|
||||
You are here because SKILL.md told you to read this file before writing animation code. Pick your technique combination from the table below based on the energy level you detected from the transcript, then implement using standard GSAP patterns.
|
||||
|
||||
## Technique Selection by Energy
|
||||
|
||||
| Energy level | Highlight | Exit | Cycle pattern |
|
||||
| ------------ | ------------------------------------- | ------------------- | ----------------------------------------- |
|
||||
| High | Karaoke with accent glow + scale pop | Scatter or drop | Alternate highlight styles every 2 groups |
|
||||
| Medium-high | Karaoke with color pop | Scatter or collapse | Alternate every 3 groups |
|
||||
| Medium | Karaoke (subtle, white only) | Fade + slide | Alternate every 3 groups |
|
||||
| Medium-low | Karaoke (minimal scale change) | Fade | Single style, vary ease per group |
|
||||
| Low | Karaoke (warm tones, slow transition) | Collapse | Alternate every 4 groups |
|
||||
|
||||
**All energy levels use karaoke highlight as the baseline.** The difference is intensity — high energy gets accent color + glow + 15% scale pop on active words, low energy gets a gentle white shift with 3% scale.
|
||||
|
||||
**Emphasis words always break the pattern.** When a word is flagged as emphasis (emotional keyword, ALL CAPS, brand name), give it a stronger animation than surrounding words (larger scale, accent color, overshoot ease). This creates contrast.
|
||||
|
||||
**Marker highlight modes add a visual layer on top of karaoke.** For emphasis words that need more than color/scale, add a marker-style effect — highlight sweep, circle, burst, or scribble — using the `/marker-highlight` skill. Match mode to energy: burst for hype, circle for key terms, highlight for standard, scribble for subtle.
|
||||
|
||||
## Audio-Reactive Captions (Mandatory for Music)
|
||||
|
||||
**If the source audio is music (vocals over instrumentation, beats, any musical content), you MUST extract audio data and add audio-reactive animations.** This is not optional — music without audio reactivity looks disconnected. Even low-energy ballads get subtle bass pulse and treble glow.
|
||||
|
||||
No special wiring is needed. The group loop already iterates over every caption group to build entrance, karaoke, and exit tweens. At that point, read the audio data for each group's time range and use it to modulate the group's animation intensity with regular GSAP tweens.
|
||||
|
||||
```js
|
||||
// Load audio data inline (same pattern as TRANSCRIPT)
|
||||
var AUDIO = JSON.parse(audioDataJson); // { fps, totalFrames, frames: [{ bands: [...] }] }
|
||||
|
||||
GROUPS.forEach(function (group, gi) {
|
||||
var groupEl = document.getElementById("cg-" + gi);
|
||||
if (!groupEl) return;
|
||||
|
||||
// Read peak energy for this group's time range
|
||||
var startFrame = Math.floor(group.start * AUDIO.fps);
|
||||
var endFrame = Math.min(Math.floor(group.end * AUDIO.fps), AUDIO.totalFrames - 1);
|
||||
var peakBass = 0;
|
||||
var peakTreble = 0;
|
||||
for (var f = startFrame; f <= endFrame; f++) {
|
||||
var frame = AUDIO.frames[f];
|
||||
if (!frame) continue;
|
||||
peakBass = Math.max(peakBass, frame.bands[0] || 0, frame.bands[1] || 0);
|
||||
peakTreble = Math.max(peakTreble, frame.bands[6] || 0, frame.bands[7] || 0);
|
||||
}
|
||||
|
||||
// Modulate entrance — louder groups enter bigger and glowier
|
||||
tl.to(
|
||||
groupEl,
|
||||
{
|
||||
scale: 1 + peakBass * 0.06,
|
||||
textShadow:
|
||||
"0 0 " + Math.round(peakTreble * 12) + "px rgba(255,255,255," + peakTreble * 0.4 + ")",
|
||||
duration: 0.3,
|
||||
ease: "power2.out",
|
||||
},
|
||||
group.start,
|
||||
);
|
||||
|
||||
// Reset at exit so audio-driven values don't persist
|
||||
tl.set(groupEl, { scale: 1, textShadow: "none" }, group.end - 0.15);
|
||||
});
|
||||
```
|
||||
|
||||
This shapes the animation at build time, not playback time — no per-frame callbacks, no `tl.call()` loops, no async fetch timing issues. Loud groups come in with more weight and glow; quiet groups come in soft. The audio data modulates _how much_, the content determines _what_.
|
||||
|
||||
Keep audio reactivity subtle — 3-6% scale variation and soft glow. Heavy pulsing makes text unreadable.
|
||||
|
||||
To generate the audio data file:
|
||||
|
||||
```bash
|
||||
python3 skills/gsap-effects/scripts/extract-audio-data.py audio.mp3 --fps 30 --bands 8 -o audio-data.json
|
||||
```
|
||||
|
||||
## Combining Techniques
|
||||
|
||||
Don't use the same highlight animation on every group — cycle through styles using the group index. Don't combine multiple competing animations on the same word at the same timestamp. Vary techniques across groups to match the content's pace changes.
|
||||
|
||||
**Marker highlight effects** (from the `/marker-highlight` skill) layer well with karaoke — use karaoke for the word-by-word reveal, then add a marker effect on emphasis words only. For example: karaoke highlights each word in white, but brand names get a yellow highlight sweep and stats get a red circle. Cycle marker modes across groups for visual variety (see the mode-to-energy mapping in the marker-highlight skill).
|
||||
|
||||
## Available Tools
|
||||
|
||||
These tools are available in the HyperFrames runtime. Use them when they solve a real problem — not every composition needs all of them.
|
||||
|
||||
| Tool | What it does | Access | When it's useful |
|
||||
| ------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| **pretext** | Pure-arithmetic text measurement without DOM reflow. 0.0002ms per call. | `window.__hyperframes.pretext.prepare(text, font)` / `.layout(prepared, maxWidth, lineHeight)` | Per-frame text reflow, shrinkwrap containers, computing layout before render |
|
||||
| **fitTextFontSize** | Finds the largest font size that fits text on one line. Built on pretext. | `window.__hyperframes.fitTextFontSize(text, { maxWidth, fontFamily, fontWeight })` | Overflow prevention for long phrases, portrait mode, large base sizes |
|
||||
| **audio data** | Pre-extracted per-frame RMS energy and frequency bands. | Extract with `extract-audio-data.py`, load inline or via `fetch("audio-data.json")` | Audio-reactive visuals — modulate intensity based on the music |
|
||||
| **GSAP** | Animation timeline with tweens and callbacks. | `gsap.to()`, `gsap.set()`, `tl.to()`, `tl.set()` | All caption animation |
|
||||
@@ -0,0 +1,146 @@
|
||||
# Marker Highlight Examples
|
||||
|
||||
## Recipes
|
||||
|
||||
### Underline
|
||||
|
||||
```html
|
||||
<mark
|
||||
data-height="0.15"
|
||||
data-offset="0.8"
|
||||
data-padding="0"
|
||||
data-highlight='{"amplitude":0.2,"wavelength":5,"roughEnds":0}'
|
||||
data-color="rgba(30, 136, 229, 0.6)"
|
||||
>important</mark
|
||||
>
|
||||
```
|
||||
|
||||
### Strikethrough
|
||||
|
||||
```html
|
||||
<mark
|
||||
data-drawing-mode="highlight"
|
||||
data-height="0.1"
|
||||
data-offset="0"
|
||||
data-highlight='{"amplitude":0.1,"wavelength":3}'
|
||||
data-color="rgba(229, 57, 53, 0.8)"
|
||||
>wrong answer</mark
|
||||
>
|
||||
```
|
||||
|
||||
### Circled Annotation
|
||||
|
||||
```html
|
||||
<mark
|
||||
data-drawing-mode="circle"
|
||||
data-circle='{"curve":0.8,"wobble":0.4,"loops":2,"thickness":3}'
|
||||
data-animation-speed="1200"
|
||||
data-color="rgba(229, 57, 53, 0.6)"
|
||||
>this one</mark
|
||||
>
|
||||
```
|
||||
|
||||
## Full Example in a Composition
|
||||
|
||||
```html
|
||||
<div data-composition-id="highlight-demo" data-width="1920" data-height="1080">
|
||||
<div
|
||||
id="content"
|
||||
style="
|
||||
position: absolute; inset: 0;
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
font-family: 'Inter', sans-serif; font-size: 72px; color: #fff;
|
||||
background: #111;
|
||||
"
|
||||
>
|
||||
<p id="hero">
|
||||
The <mark id="m1" data-color="rgba(255, 220, 50, 0.5)">fastest</mark> way to
|
||||
<mark
|
||||
id="m2"
|
||||
data-drawing-mode="circle"
|
||||
data-circle='{"curve":0.8,"wobble":0.3,"loops":2,"thickness":3}'
|
||||
data-color="rgba(229, 57, 53, 0.6)"
|
||||
>ship</mark
|
||||
>
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
[data-composition-id="highlight-demo"] mark {
|
||||
background-color: transparent;
|
||||
color: inherit;
|
||||
}
|
||||
</style>
|
||||
|
||||
<script src="https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js"></script>
|
||||
<script src="marker-highlight.global.js"></script>
|
||||
<script>
|
||||
window.__timelines = window.__timelines || {};
|
||||
var tl = gsap.timeline({ paused: true });
|
||||
|
||||
// Set colors via data attribute (no visible flash)
|
||||
document.querySelectorAll("mark[data-color]").forEach(function (m) {
|
||||
m.setAttribute("data-original-bgcolor", m.getAttribute("data-color"));
|
||||
});
|
||||
|
||||
// Init once after fonts, then hide all canvases
|
||||
var hl;
|
||||
document.fonts.ready.then(function () {
|
||||
setTimeout(function () {
|
||||
hl = new MarkerHighlighter(document.getElementById("hero"), {
|
||||
animate: false,
|
||||
animationSpeed: 800,
|
||||
padding: 0.3,
|
||||
highlight: { amplitude: 0.3, wavelength: 5 },
|
||||
});
|
||||
setTimeout(function () {
|
||||
document.querySelectorAll(".highlight").forEach(function (d) {
|
||||
d.style.opacity = "0";
|
||||
});
|
||||
}, 100);
|
||||
}, 50);
|
||||
});
|
||||
|
||||
function addHighlight(markId, time) {
|
||||
tl.to(
|
||||
{},
|
||||
{
|
||||
duration: 0.001,
|
||||
onStart: function () {
|
||||
var mark = document.getElementById(markId);
|
||||
var ref = mark.getAttribute("data-mark-ref");
|
||||
if (!ref || !hl) return;
|
||||
mark.parentElement
|
||||
.querySelectorAll('.highlight[data-mark-id="' + ref + '"]')
|
||||
.forEach(function (div) {
|
||||
var c = div.querySelector("canvas");
|
||||
if (c) c.getContext("2d").clearRect(0, 0, c.width, c.height);
|
||||
div.style.opacity = "1";
|
||||
});
|
||||
hl.reanimateMark(mark);
|
||||
},
|
||||
onReverseComplete: function () {
|
||||
var mark = document.getElementById(markId);
|
||||
var ref = mark.getAttribute("data-mark-ref");
|
||||
if (!ref) return;
|
||||
mark.parentElement
|
||||
.querySelectorAll('.highlight[data-mark-id="' + ref + '"]')
|
||||
.forEach(function (div) {
|
||||
div.style.opacity = "0";
|
||||
});
|
||||
},
|
||||
},
|
||||
time,
|
||||
);
|
||||
}
|
||||
|
||||
gsap.set("#hero", { opacity: 0 });
|
||||
tl.to("#hero", { opacity: 1, duration: 0.6 }, 0);
|
||||
|
||||
addHighlight("m1", 0.8);
|
||||
addHighlight("m2", 1.6);
|
||||
|
||||
window.__timelines["highlight-demo"] = tl;
|
||||
</script>
|
||||
</div>
|
||||
```
|
||||
@@ -0,0 +1,158 @@
|
||||
# Marker Highlight
|
||||
|
||||
Animated canvas-based text highlighting using MarkerHighlight.js. Wraps text in `<mark>` tags and renders effects (marker pen, circle, burst, scribble, sketchout) on a canvas overlay without modifying text DOM.
|
||||
|
||||
The library runs its own requestAnimationFrame loop — **not** GSAP-driven. Use `tl.call()` to trigger at specific timeline points.
|
||||
|
||||
## Required Script
|
||||
|
||||
Download and convert to global script:
|
||||
|
||||
```bash
|
||||
curl -sL "https://cdn.jsdelivr.net/gh/Robincodes-Sandbox/marker-highlight@main/dist/marker-highlight.min.js" \
|
||||
| sed 's/export{[^}]*};$/window.MarkerHighlighter=W;/' > marker-highlight.global.js
|
||||
```
|
||||
|
||||
```html
|
||||
<script src="https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js"></script>
|
||||
<script src="marker-highlight.global.js"></script>
|
||||
```
|
||||
|
||||
## Color Setup
|
||||
|
||||
Set via `data-color`, copy to `data-original-bgcolor` before constructing. Never set `background-color` in CSS.
|
||||
|
||||
```css
|
||||
mark {
|
||||
color: inherit;
|
||||
background-color: transparent;
|
||||
}
|
||||
```
|
||||
|
||||
```html
|
||||
<mark id="m1" data-color="rgba(255, 220, 50, 0.5)">highlighted</mark>
|
||||
```
|
||||
|
||||
```js
|
||||
document
|
||||
.querySelectorAll("mark[data-color]")
|
||||
.forEach((m) => m.setAttribute("data-original-bgcolor", m.getAttribute("data-color")));
|
||||
```
|
||||
|
||||
## GSAP Integration Pattern
|
||||
|
||||
ONE MarkerHighlighter per container with `animate: false`, hide all canvases, then clear+show+reanimate per mark at trigger time.
|
||||
|
||||
```js
|
||||
var hl = new MarkerHighlighter(document.getElementById("text-container"), {
|
||||
animate: false,
|
||||
animationSpeed: 800,
|
||||
padding: 0.3,
|
||||
highlight: { amplitude: 0.3, wavelength: 5 },
|
||||
});
|
||||
|
||||
setTimeout(function () {
|
||||
document.querySelectorAll(".highlight").forEach((div) => (div.style.opacity = "0"));
|
||||
}, 100);
|
||||
|
||||
function addHighlight(highlighter, markId, time) {
|
||||
tl.to(
|
||||
{},
|
||||
{
|
||||
duration: 0.001,
|
||||
onStart: function () {
|
||||
var mark = document.getElementById(markId);
|
||||
var ref = mark.getAttribute("data-mark-ref");
|
||||
var divs = mark.parentElement.querySelectorAll('.highlight[data-mark-id="' + ref + '"]');
|
||||
divs.forEach(function (div) {
|
||||
var canvas = div.querySelector("canvas");
|
||||
if (canvas) canvas.getContext("2d").clearRect(0, 0, canvas.width, canvas.height);
|
||||
div.style.opacity = "1";
|
||||
});
|
||||
highlighter.reanimateMark(mark);
|
||||
},
|
||||
onReverseComplete: function () {
|
||||
var mark = document.getElementById(markId);
|
||||
var ref = mark.getAttribute("data-mark-ref");
|
||||
mark.parentElement
|
||||
.querySelectorAll('.highlight[data-mark-id="' + ref + '"]')
|
||||
.forEach((div) => (div.style.opacity = "0"));
|
||||
},
|
||||
},
|
||||
time,
|
||||
);
|
||||
}
|
||||
|
||||
addHighlight(hl, "m1", 1.0);
|
||||
```
|
||||
|
||||
## Drawing Modes
|
||||
|
||||
| Mode | Effect | Best for |
|
||||
| ----------- | ---------------------------- | -------------------------- |
|
||||
| `highlight` | Wavy marker stroke (default) | Phrases, key terms |
|
||||
| `circle` | Hand-drawn ellipse | Single words, annotations |
|
||||
| `burst` | Radiating lines/curves/puffs | Excitement, energy |
|
||||
| `scribble` | Chaotic scribble | Crossing out, messy energy |
|
||||
| `sketchout` | Rough rectangle outline | Boxed callouts, blueprint |
|
||||
|
||||
```html
|
||||
<mark data-drawing-mode="circle" data-color="rgba(229, 57, 53, 0.6)">critical</mark>
|
||||
<mark
|
||||
data-drawing-mode="burst"
|
||||
data-burst='{"style":"cloud","count":20}'
|
||||
data-color="rgba(255, 220, 50, 0.5)"
|
||||
>amazing</mark
|
||||
>
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Global (constructor)
|
||||
|
||||
| Option | Default | Description |
|
||||
| ---------------- | ------------- | ------------------------- |
|
||||
| `animate` | `true` | `false` to defer for GSAP |
|
||||
| `animationSpeed` | `5000` | Duration in ms |
|
||||
| `drawingMode` | `"highlight"` | Default mode |
|
||||
| `height` | `1` | Relative to line height |
|
||||
| `offset` | `0` | Vertical shift |
|
||||
| `padding` | `0` | Horizontal padding |
|
||||
|
||||
### Per-Mode
|
||||
|
||||
**highlight**: `amplitude` (0.25), `wavelength` (1), `roughEnds` (5), `jitter` (0.1)
|
||||
**circle**: `curve` (0.5), `wobble` (0.3), `loops` (3), `thickness` (5)
|
||||
**burst**: `style` ("lines"/"curve"/"cloud"), `count` (10), `power` (1), `randomness` (0.5)
|
||||
|
||||
### Named Styles
|
||||
|
||||
```js
|
||||
MarkerHighlighter.defineStyle("underline", {
|
||||
animationSpeed: 400,
|
||||
height: 0.15,
|
||||
offset: 0.8,
|
||||
padding: 0,
|
||||
highlight: { amplitude: 0.2, wavelength: 5, roughEnds: 0 },
|
||||
});
|
||||
```
|
||||
|
||||
## Mode-to-Caption Energy Mapping
|
||||
|
||||
| Energy | Mode | Use for |
|
||||
| ----------- | --------------------- | ------------------- |
|
||||
| High | `burst` + `highlight` | Launches, hype |
|
||||
| Medium-high | `circle` | Key stats, terms |
|
||||
| Medium | `highlight` | Standard emphasis |
|
||||
| Medium-low | `scribble` | Subtle, tutorials |
|
||||
| Low | `sketchout` | Contrast, blueprint |
|
||||
|
||||
## Notes
|
||||
|
||||
- One highlighter per container (clears all `.highlight` divs on init)
|
||||
- Canvas pre-draw + clear pattern for clean reveals
|
||||
- rAF-based — not seekable mid-stroke
|
||||
- Use `onReverseComplete` for rewind support
|
||||
|
||||
For CSS+GSAP fallback (no library, fully seekable), see [css-patterns.md](css-patterns.md).
|
||||
For full examples, see [examples.md](examples.md).
|
||||
@@ -0,0 +1,151 @@
|
||||
# Transcript Guide
|
||||
|
||||
## How Transcripts Are Generated
|
||||
|
||||
`hyperframes transcribe` handles both transcription and format conversion:
|
||||
|
||||
```bash
|
||||
# Transcribe audio/video (uses whisper.cpp locally, no API key needed)
|
||||
npx hyperframes transcribe audio.mp3
|
||||
|
||||
# Use a larger model for better accuracy
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
|
||||
# Filter to English only (skips non-English speech)
|
||||
npx hyperframes transcribe audio.mp3 --language en
|
||||
|
||||
# Import an existing transcript from another tool
|
||||
npx hyperframes transcribe captions.srt
|
||||
npx hyperframes transcribe captions.vtt
|
||||
npx hyperframes transcribe openai-response.json
|
||||
```
|
||||
|
||||
## Supported Input Formats
|
||||
|
||||
The CLI auto-detects and normalizes these formats:
|
||||
|
||||
| Format | Extension | Source | Word-level? |
|
||||
| --------------------- | --------- | --------------------------------------------------------------------------- | ----------------- |
|
||||
| whisper.cpp JSON | `.json` | `hyperframes init --video`, `hyperframes transcribe` | Yes |
|
||||
| OpenAI Whisper API | `.json` | `openai.audio.transcriptions.create({ timestamp_granularities: ["word"] })` | Yes |
|
||||
| SRT subtitles | `.srt` | Video editors, subtitle tools, YouTube | No (phrase-level) |
|
||||
| VTT subtitles | `.vtt` | Web players, YouTube, transcription services | No (phrase-level) |
|
||||
| Normalized word array | `.json` | Pre-processed by any tool | Yes |
|
||||
|
||||
**Word-level timestamps produce better captions.** SRT/VTT give phrase-level timing, which works but can't do per-word animation effects.
|
||||
|
||||
## Whisper Model Guide
|
||||
|
||||
The default model (`small.en`) balances accuracy and speed. For better results, use a larger model:
|
||||
|
||||
| Model | Size | Speed | Accuracy | When to use |
|
||||
| ---------- | ------ | -------- | --------- | ------------------------------------- |
|
||||
| `tiny` | 75 MB | Fastest | Low | Quick previews, testing pipeline |
|
||||
| `base` | 142 MB | Fast | Fair | Short clips, clear audio |
|
||||
| `small` | 466 MB | Moderate | Good | **Default** — good for most content |
|
||||
| `medium` | 1.5 GB | Slow | Very good | Important content, noisy audio, music |
|
||||
| `large-v3` | 3.1 GB | Slowest | Best | Production quality |
|
||||
|
||||
**Only add `.en` suffix when the user explicitly says the audio is English.** `.en` models are slightly more accurate for English but will TRANSLATE non-English audio instead of transcribing it.
|
||||
|
||||
**Critical: `.en` models translate non-English audio into English** — they don't transcribe it. If the audio might not be English, always use a model without the `.en` suffix and pass `--language` to specify the source language. If you're unsure of the language, use `small` (not `small.en`) without `--language` — whisper will auto-detect.
|
||||
|
||||
```bash
|
||||
# Spanish audio
|
||||
npx hyperframes transcribe audio.mp3 --model small --language es
|
||||
|
||||
# Unknown language — let whisper auto-detect
|
||||
npx hyperframes transcribe audio.mp3 --model small
|
||||
```
|
||||
|
||||
**Music and vocals over instrumentation**: `small.en` will misidentify lyrics — use `medium.en` as the minimum, or import lyrics manually. Even `medium.en` struggles with heavily produced tracks; for music videos, providing known lyrics as an SRT/VTT and importing with `hyperframes transcribe lyrics.srt` will always beat automated transcription.
|
||||
|
||||
## Transcript Quality Check (Mandatory)
|
||||
|
||||
After every transcription, **read the transcript and check for quality issues before proceeding.** Bad transcripts produce nonsensical captions. Never skip this step.
|
||||
|
||||
### What to look for
|
||||
|
||||
| Signal | Example | Cause |
|
||||
| ---------------------------- | -------------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| Music note tokens (`♪`, `�`) | `{ "text": "♪" }` or `{ "text": "�" }` | Whisper detected music, not speech |
|
||||
| Garbled / nonsense words | "Do a chin", "Get so gay", "huh" | Model misheard lyrics or background noise |
|
||||
| Long gaps with no words | 20+ seconds of only `♪` tokens | Instrumental section — expected, but high ratio means speech is being missed |
|
||||
| Repeated filler | Many "huh", "uh", "oh" entries | Model is hallucinating on music |
|
||||
| Very short word spans | Words with `end - start < 0.05` | Unreliable timestamp alignment |
|
||||
|
||||
### Automatic retry rules
|
||||
|
||||
**If more than 20% of entries are `♪`/`�` tokens, or the transcript contains obvious nonsense words, the transcription failed.** Do not proceed with the bad transcript. Instead:
|
||||
|
||||
1. **Retry with `medium.en`** if the original used `small.en` or smaller:
|
||||
```bash
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
```
|
||||
2. **If `medium.en` also fails** (still >20% music tokens or garbled), tell the user the audio is too noisy for local transcription and suggest:
|
||||
- Providing lyrics manually as an SRT/VTT file
|
||||
- Using an external API (OpenAI or Groq Whisper — see below)
|
||||
3. **Always clean the transcript** before building captions — filter out `♪`/`�` tokens and entries where `text` is a single non-word character. Only real words should reach the caption composition.
|
||||
|
||||
### Cleaning a transcript
|
||||
|
||||
After transcription (even with a good model), strip non-word entries:
|
||||
|
||||
```js
|
||||
var raw = JSON.parse(transcriptJson);
|
||||
var words = raw.filter(function (w) {
|
||||
if (!w.text || w.text.trim().length === 0) return false;
|
||||
if (/^[♪�\u266a\u266b\u266c\u266d\u266e\u266f]+$/.test(w.text)) return false;
|
||||
if (/^(huh|uh|um|ah|oh)$/i.test(w.text) && w.end - w.start < 0.1) return false;
|
||||
return true;
|
||||
});
|
||||
```
|
||||
|
||||
### When to use which model (decision tree)
|
||||
|
||||
1. **Is this speech over silence/light background?** → `small.en` is fine
|
||||
2. **Is this speech over music, or music with vocals?** → Start with `medium.en`
|
||||
3. **Is this a produced music track (vocals + full instrumentation)?** → Start with `medium.en`, expect to need manual lyrics or an external API
|
||||
4. **Is this multilingual?** → Use `medium` or `large-v3` (no `.en` suffix)
|
||||
|
||||
## Using External Transcription APIs
|
||||
|
||||
For the best accuracy, use an external API and import the result:
|
||||
|
||||
**OpenAI Whisper API** (recommended for quality):
|
||||
|
||||
```bash
|
||||
# Generate with word timestamps, then import
|
||||
curl https://api.openai.com/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-1 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-openai.json
|
||||
|
||||
npx hyperframes transcribe transcript-openai.json
|
||||
```
|
||||
|
||||
**Groq Whisper API** (fast, free tier available):
|
||||
|
||||
```bash
|
||||
curl https://api.groq.com/openai/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $GROQ_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-large-v3 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-groq.json
|
||||
|
||||
npx hyperframes transcribe transcript-groq.json
|
||||
```
|
||||
|
||||
## If No Transcript Exists
|
||||
|
||||
1. Check the project root for `transcript.json`, `.srt`, or `.vtt` files
|
||||
2. If none found, run transcription — pick the starting model based on the content type:
|
||||
- Speech/voiceover → `small.en`
|
||||
- Music with vocals → `medium.en`
|
||||
```bash
|
||||
npx hyperframes transcribe <audio-or-video-file> --model medium.en
|
||||
```
|
||||
3. **Read the transcript and run the quality check** (see above). If it fails, retry with a larger model or suggest manual lyrics.
|
||||
@@ -0,0 +1,56 @@
|
||||
# Text-to-Speech
|
||||
|
||||
Generate speech audio locally using Kokoro-82M (no API key, runs on CPU).
|
||||
|
||||
## Voice Selection
|
||||
|
||||
Match voice to content. Default is `af_heart`.
|
||||
|
||||
| Content type | Voice | Why |
|
||||
| ------------- | --------------------- | -------------------------- |
|
||||
| Product demo | `af_heart`/`af_nova` | Warm, professional |
|
||||
| Tutorial | `am_adam`/`bf_emma` | Neutral, easy to follow |
|
||||
| Marketing | `af_sky`/`am_michael` | Energetic or authoritative |
|
||||
| Documentation | `bf_emma`/`bm_george` | Clear British English |
|
||||
| Casual | `af_heart`/`af_sky` | Approachable, natural |
|
||||
|
||||
Run `npx hyperframes tts --list` for all 54 voices (8 languages).
|
||||
|
||||
## Speed Tuning
|
||||
|
||||
- **0.7-0.8** — Tutorial, complex content
|
||||
- **1.0** — Natural pace (default)
|
||||
- **1.1-1.2** — Intros, upbeat content
|
||||
- **1.5+** — Rarely appropriate
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
npx hyperframes tts "Your script here" --voice af_nova --output narration.wav
|
||||
npx hyperframes tts script.txt --voice bf_emma --output narration.wav
|
||||
```
|
||||
|
||||
In compositions:
|
||||
|
||||
```html
|
||||
<audio
|
||||
id="narration"
|
||||
data-start="0"
|
||||
data-duration="auto"
|
||||
data-track-index="2"
|
||||
src="narration.wav"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
```
|
||||
|
||||
## TTS + Captions Workflow
|
||||
|
||||
```bash
|
||||
npx hyperframes tts script.txt --voice af_heart --output narration.wav
|
||||
npx hyperframes transcribe narration.wav # → transcript.json with word-level timestamps
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.8+ with `kokoro-onnx` and `soundfile`
|
||||
- Model downloads on first use (~311 MB + ~27 MB voices, cached in `~/.cache/hyperframes/tts/`)
|
||||
Reference in New Issue
Block a user