mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
feat(skills): add dynamic caption techniques and split captions skill into references (#173)
## Summary - Split the captions skill from a single 611-line file into focused references: `SKILL.md` (core rules), `transcript-guide.md` (whisper/transcription), `dynamic-techniques.md` (animation patterns) - Add `audio-reactive` skill with "Content, Not Medium" constraint — steers away from generic visualizations (equalizer bars, spectrum analyzers, waveforms) toward content-grounded animation where audio drives *when* and *how much*, not *what to show* - Add initial dynamic caption technique selection by energy level ## Test plan - [ ] All skill files render correctly as markdown - [ ] Cross-references between files use correct relative paths - [ ] `audio-reactive/SKILL.md` contains the anti-pattern list and content-grounded examples 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This commit is contained in:
Vance Ingalls
@@ -0,0 +1,74 @@
|
||||
---
|
||||
name: audio-reactive
|
||||
description: Drive any visual element in a HyperFrames composition from audio data — captions, backgrounds, shapes, overlays, anything GSAP can animate. Use when a composition should respond to music, voice, or sound.
|
||||
trigger: Use when a composition involves music, beat-synced animation, audio visualization, or any visual element that should react to sound.
|
||||
---
|
||||
|
||||
# Audio-Reactive Animation
|
||||
|
||||
When audio data is available (extracted via `extract-audio-data.py` or loaded from `audio-data.json`), any visual element in the composition can be driven by the music — captions, backgrounds, shapes, overlays, anything GSAP can animate.
|
||||
|
||||
## Audio Data Format
|
||||
|
||||
```js
|
||||
var AUDIO_DATA = {
|
||||
fps: 30, // frame rate of the analysis
|
||||
totalFrames: 900, // total analyzed frames
|
||||
frames: [
|
||||
{ bands: [0.82, 0.45, 0.31, ...] }, // per-frame frequency band amplitudes
|
||||
// ...
|
||||
]
|
||||
};
|
||||
```
|
||||
|
||||
- `frames[i].bands[]` — frequency band amplitudes, normalized 0–1. Index 0 = bass, higher indices = mids and treble.
|
||||
- `fps` — frame rate of the analysis (matches composition frame rate)
|
||||
- `totalFrames` — total number of analyzed frames
|
||||
|
||||
## Mapping Audio to Visuals
|
||||
|
||||
Map frequency bands and amplitude to any GSAP-animatable property. The creative choice is yours — these are common mappings:
|
||||
|
||||
| Audio signal | Visual property | Effect |
|
||||
| ---------------------- | --------------------------------- | -------------------------- |
|
||||
| Bass (bands[0]) | `scale` | Pulse on beat |
|
||||
| Treble (bands[12–14]) | `textShadow`, `boxShadow` | Glow intensity |
|
||||
| Overall amplitude | `opacity`, `y`, `backgroundColor` | Breathe, lift, color shift |
|
||||
| Beat onset | `scale`, `color`, `rotation` | Flash or pop on hits |
|
||||
| Mid-range (bands[4–8]) | `borderRadius`, `width`, `height` | Shape morphing |
|
||||
|
||||
These are starting points. Any property GSAP can tween is fair game — `clipPath`, `filter`, `backgroundPosition`, SVG attributes, custom CSS properties.
|
||||
|
||||
## Content, Not Medium
|
||||
|
||||
Audio data provides **timing and intensity** for visuals grounded in the content. It tells the animation _when_ and _how much_ — not _what to show_. The visual vocabulary comes from the narrative, theme, and emotion of the piece. A funeral dirge and a party anthem should produce completely different visuals even though their audio data has the same shape.
|
||||
|
||||
**Never add these — they represent the medium, not the content:**
|
||||
|
||||
- Frequency equalizer bars, spectrum analyzers, radial spectrum rings — technical readouts that just say "audio exists"
|
||||
- Waveform displays, oscilloscope lines, VU meters — diagnostic tools, not creative choices
|
||||
- Musical notes, vinyl records, turntable imagery — clip art signaling "music is playing"
|
||||
- Generic particle systems driven by amplitude — interchangeable across any song or mood
|
||||
- Background color cycling through rainbow hues on frequency — rave aesthetic regardless of content
|
||||
- Strobing/flashing white on beat hits — lazy beat sync (also an accessibility problem)
|
||||
- Abstract pulsing orbs, breathing geometric wireframes, concentric rings — would look identical on a lullaby or a metal track
|
||||
|
||||
**Instead, let the content guide the visual and the audio drive its behavior:**
|
||||
|
||||
- If the scene is warm, bass makes the warmth _swell_ (slight scale, deeper color saturation)
|
||||
- If the mood is tense, treble sharpens _contrast_ or tightens _letterSpacing_
|
||||
- If text is the focus, bass gives it subtle _weight_ (shadow depth, y offset) while treble adds _shimmer_ (glow, lightness)
|
||||
- The visual choice comes from asking "what does this piece feel like?" — audio data just animates the answer
|
||||
|
||||
## Guidelines
|
||||
|
||||
- **Subtlety for text.** Captions and readable text should stay in the 3–6% scale variation range with soft glow. Heavy pulsing makes text unreadable.
|
||||
- **Go bigger on non-text elements.** Backgrounds, shapes, and decorative elements can handle 10–30% scale swings, full color shifts, and dramatic transforms.
|
||||
- **Match the energy.** A corporate explainer needs subtle reactivity. A music video can go hard.
|
||||
- **Deterministic.** Audio data is pre-extracted — no Web Audio API, no `AnalyserNode`, no runtime mic input. The data is static JSON, the animation is repeatable.
|
||||
|
||||
## Constraints
|
||||
|
||||
- All audio data must be pre-extracted — no runtime audio analysis
|
||||
- No `Math.random()` or `Date.now()` — deterministic rendering applies
|
||||
- Audio reactivity runs on the same GSAP timeline as everything else
|
||||
@@ -25,98 +25,7 @@ This is the only format the captions composition consumes. Use it directly:
|
||||
const words = JSON.parse(transcriptJson); // [{ text, start, end }]
|
||||
```
|
||||
|
||||
### How transcripts are generated
|
||||
|
||||
`hyperframes transcribe` handles both transcription and format conversion:
|
||||
|
||||
```bash
|
||||
# Transcribe audio/video (uses whisper.cpp locally, no API key needed)
|
||||
npx hyperframes transcribe audio.mp3
|
||||
|
||||
# Use a larger model for better accuracy
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
|
||||
# Filter to English only (skips non-English speech)
|
||||
npx hyperframes transcribe audio.mp3 --language en
|
||||
|
||||
# Import an existing transcript from another tool
|
||||
npx hyperframes transcribe captions.srt
|
||||
npx hyperframes transcribe captions.vtt
|
||||
npx hyperframes transcribe openai-response.json
|
||||
```
|
||||
|
||||
### Supported input formats
|
||||
|
||||
The CLI auto-detects and normalizes these formats:
|
||||
|
||||
| Format | Extension | Source | Word-level? |
|
||||
| --------------------- | --------- | --------------------------------------------------------------------------- | ----------------- |
|
||||
| whisper.cpp JSON | `.json` | `hyperframes init --video`, `hyperframes transcribe` | Yes |
|
||||
| OpenAI Whisper API | `.json` | `openai.audio.transcriptions.create({ timestamp_granularities: ["word"] })` | Yes |
|
||||
| SRT subtitles | `.srt` | Video editors, subtitle tools, YouTube | No (phrase-level) |
|
||||
| VTT subtitles | `.vtt` | Web players, YouTube, transcription services | No (phrase-level) |
|
||||
| Normalized word array | `.json` | Pre-processed by any tool | Yes |
|
||||
|
||||
**Word-level timestamps produce better captions.** SRT/VTT give phrase-level timing, which works but can't do per-word animation effects.
|
||||
|
||||
### Whisper model guide
|
||||
|
||||
The default model (`small.en`) balances accuracy and speed. For better results, use a larger model:
|
||||
|
||||
| Model | Size | Speed | Accuracy | When to use |
|
||||
| ----------- | ------ | -------- | --------- | ------------------------------------- |
|
||||
| `tiny.en` | 75 MB | Fastest | Low | Quick previews, testing pipeline |
|
||||
| `base.en` | 142 MB | Fast | Fair | Short clips, clear audio |
|
||||
| `small.en` | 466 MB | Moderate | Good | **Default** — good for most content |
|
||||
| `medium.en` | 1.5 GB | Slow | Very good | Important content, noisy audio, music |
|
||||
| `large-v3` | 3.1 GB | Slowest | Best | Multilingual, production captions |
|
||||
|
||||
`.en` models are English-only and more accurate for English. Drop the `.en` suffix for multilingual (e.g., `medium` instead of `medium.en`).
|
||||
|
||||
**Music and vocals over instrumentation**: `small.en` will misidentify lyrics — use `medium.en` as the minimum, or import lyrics manually. Even `medium.en` struggles with heavily produced tracks; for music videos, providing known lyrics as an SRT/VTT and importing with `hyperframes transcribe lyrics.srt` will always beat automated transcription.
|
||||
|
||||
### Using external transcription APIs
|
||||
|
||||
For the best accuracy, use an external API and import the result:
|
||||
|
||||
**OpenAI Whisper API** (recommended for quality):
|
||||
|
||||
```bash
|
||||
# Generate with word timestamps, then import
|
||||
curl https://api.openai.com/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-1 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-openai.json
|
||||
|
||||
npx hyperframes transcribe transcript-openai.json
|
||||
```
|
||||
|
||||
**Groq Whisper API** (fast, free tier available):
|
||||
|
||||
```bash
|
||||
curl https://api.groq.com/openai/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $GROQ_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-large-v3 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-groq.json
|
||||
|
||||
npx hyperframes transcribe transcript-groq.json
|
||||
```
|
||||
|
||||
### If no transcript exists
|
||||
|
||||
1. Check the project root for `transcript.json`, `.srt`, or `.vtt` files
|
||||
2. If none found, ask the user to provide one or run:
|
||||
```bash
|
||||
npx hyperframes transcribe <audio-or-video-file>
|
||||
```
|
||||
3. If transcription quality is poor (words at wrong times, gibberish), suggest upgrading the model:
|
||||
```bash
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
```
|
||||
For transcription commands, whisper model selection, external APIs (OpenAI, Groq), and supported input formats, see [transcript-guide.md](./transcript-guide.md).
|
||||
|
||||
## Style Detection (Default — When No Style Is Specified)
|
||||
|
||||
@@ -207,28 +116,19 @@ Break groups on sentence boundaries (period, question mark, exclamation), pauses
|
||||
|
||||
Use `window.__hyperframes.fitTextFontSize()` to measure actual rendered text width and compute the correct font size. This replaces character-count heuristics with pixel-accurate measurement powered by [pretext](https://github.com/chenglou/pretext).
|
||||
|
||||
**Usage in composition scripts:**
|
||||
|
||||
```js
|
||||
GROUPS.forEach(function (group, gi) {
|
||||
// Measure with text-transform applied (captions typically uppercase)
|
||||
var result = window.__hyperframes.fitTextFontSize(group.text.toUpperCase(), {
|
||||
fontFamily: "Outfit",
|
||||
fontWeight: 900,
|
||||
maxWidth: 1600,
|
||||
});
|
||||
|
||||
// Apply computed font size to all word spans
|
||||
wordEls.forEach(function (el) {
|
||||
el.style.fontSize = result.fontSize + "px";
|
||||
});
|
||||
|
||||
// If result.fits is false, text exceeds minFontSize — overflow: hidden catches it
|
||||
});
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
| Option | Default | Description |
|
||||
| -------------- | ---------- | ---------------------------------------------------- |
|
||||
| `maxWidth` | `1600` | Container width in px (1600 landscape, 900 portrait) |
|
||||
@@ -238,7 +138,7 @@ GROUPS.forEach(function (group, gi) {
|
||||
| `fontFamily` | `"Outfit"` | Must match the CSS font-family |
|
||||
| `step` | `2` | Decrement step in px per iteration |
|
||||
|
||||
**Important:** The `fontWeight` and `fontFamily` options must match the CSS applied to the text elements exactly, or measurements will be inaccurate.
|
||||
`fontWeight` and `fontFamily` must match the CSS applied to the text elements exactly, or measurements will be inaccurate.
|
||||
|
||||
**Safety nets (still required in CSS):**
|
||||
|
||||
@@ -251,8 +151,6 @@ GROUPS.forEach(function (group, gi) {
|
||||
|
||||
Captions that stick on screen are the most common caption bug. Every caption group **must** have a hard kill after its exit animation.
|
||||
|
||||
**The pattern:**
|
||||
|
||||
```js
|
||||
// Animate exit (soft — can fail if tweens conflict)
|
||||
tl.to(groupEl, { opacity: 0, scale: 0.95, duration: 0.12, ease: "power2.in" }, group.end - 0.12);
|
||||
@@ -261,18 +159,11 @@ tl.to(groupEl, { opacity: 0, scale: 0.95, duration: 0.12, ease: "power2.in" }, g
|
||||
tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end);
|
||||
```
|
||||
|
||||
**Why both?** The `tl.to` exit can fail to fully hide a group when:
|
||||
**Why both?** The `tl.to` exit can fail to fully hide a group when karaoke word-level tweens conflict with the parent exit tween, `fromTo` entrance tweens lock values that override later tweens, or timeline scrubbing lands between the exit start and end. The `tl.set` at `group.end` is a deterministic kill — it fires at an exact time, doesn't animate, and can't be overridden.
|
||||
|
||||
- Karaoke word-level tweens (`scale`, `color`) on child elements conflict with the parent exit tween
|
||||
- `fromTo` entrance tweens lock start/end values that override later tweens on the same property
|
||||
- Timeline scrubbing lands between the exit start and end
|
||||
|
||||
The `tl.set` at `group.end` is a deterministic kill — it fires at an exact time, doesn't animate, and can't be overridden by other tweens at different times.
|
||||
|
||||
**Self-lint rule:** After building the timeline, verify every caption group has a hard kill. Run this check before registering the timeline:
|
||||
**Self-lint rule:** After building the timeline, verify every caption group has a hard kill:
|
||||
|
||||
```js
|
||||
// Caption lint: verify every group has a hard kill
|
||||
GROUPS.forEach(function (group, gi) {
|
||||
var el = document.getElementById("cg-" + gi);
|
||||
if (!el) return;
|
||||
@@ -280,20 +171,20 @@ GROUPS.forEach(function (group, gi) {
|
||||
var computed = window.getComputedStyle(el);
|
||||
if (computed.opacity !== "0" && computed.visibility !== "hidden") {
|
||||
console.warn(
|
||||
"[caption-lint] group " +
|
||||
gi +
|
||||
" ('" +
|
||||
group.text +
|
||||
"') still visible at t=" +
|
||||
(group.end + 0.01).toFixed(2) +
|
||||
"s",
|
||||
"[caption-lint] group " + gi + " still visible at t=" + (group.end + 0.01).toFixed(2) + "s",
|
||||
);
|
||||
}
|
||||
});
|
||||
tl.seek(0); // reset after lint
|
||||
tl.seek(0);
|
||||
```
|
||||
|
||||
Place this **before** `window.__timelines[id] = tl` so it runs at composition init. Warnings appear in the browser console during `hyperframes preview`.
|
||||
Place this **before** `window.__timelines[id] = tl` so it runs at composition init.
|
||||
|
||||
## References
|
||||
|
||||
For dynamic animation techniques (karaoke, clip-path reveals, slam words, scatter exits, elastic entrances, 3D rotation, audio-reactive captions, pretext-based positioning and grouping), see [dynamic-techniques.md](./dynamic-techniques.md).
|
||||
|
||||
For transcription commands, whisper models, external APIs, and troubleshooting, see [transcript-guide.md](./transcript-guide.md).
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -0,0 +1,327 @@
|
||||
# Dynamic Caption Techniques
|
||||
|
||||
The default caption pattern — fade group in, hold, fade out — works but looks like subtitles. These techniques make captions feel designed and intentional. Mix them based on the content's energy. Every technique below is deterministic and works with HyperFrames' frame-by-frame rendering.
|
||||
|
||||
## Per-Word Staggered Entrances
|
||||
|
||||
Each word in a group enters individually with staggered timing. The stagger creates a wave that follows the speaker's rhythm.
|
||||
|
||||
```js
|
||||
// Words enter one by one, timed to their speech timestamps
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.set(wordEl, { opacity: 0, y: 30, scale: 0.85 }, group.start);
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
opacity: 1,
|
||||
y: 0,
|
||||
scale: 1,
|
||||
duration: 0.18,
|
||||
ease: "back.out(1.7)",
|
||||
},
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
Vary the entrance per word role. Content words (nouns, verbs) get scale + y. Function words (the, a, and) get opacity only — they shouldn't compete for attention.
|
||||
|
||||
## Karaoke Highlight
|
||||
|
||||
All words in the group are visible from the start but muted. Each word transitions to full brightness as it's spoken. This gives the viewer reading context while directing attention to the current word.
|
||||
|
||||
```js
|
||||
// Show all words muted at group start
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.set(wordEl, { opacity: 0.3, scale: 0.95, color: "rgba(255,255,255,0.4)" }, group.start);
|
||||
// Light up when spoken
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
opacity: 1,
|
||||
scale: 1.05,
|
||||
color: "#ffffff",
|
||||
duration: 0.1,
|
||||
ease: "power2.out",
|
||||
},
|
||||
word.start,
|
||||
);
|
||||
// Settle after speaking
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
scale: 1,
|
||||
color: "rgba(255,255,255,0.85)",
|
||||
duration: 0.2,
|
||||
ease: "power1.out",
|
||||
},
|
||||
word.end,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
For high-energy content, add a color accent to the active word (`color: accentColor`) and a subtle glow (`textShadow: "0 0 20px " + accentColor`).
|
||||
|
||||
## Clip-Path Reveals
|
||||
|
||||
Words or groups reveal through an animated clip-path rather than fading. This creates a physical, tactile feeling — like text being uncovered.
|
||||
|
||||
```js
|
||||
// Horizontal wipe: text sweeps in from left
|
||||
tl.fromTo(
|
||||
groupEl,
|
||||
{ clipPath: "inset(0 100% 0 0)" },
|
||||
{ clipPath: "inset(0 0% 0 0)", duration: 0.4, ease: "power3.out" },
|
||||
group.start,
|
||||
);
|
||||
|
||||
// Per-word vertical reveal: each word drops in from behind a mask
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ clipPath: "inset(100% 0 0 0)", y: -10 },
|
||||
{ clipPath: "inset(0% 0 0 0)", y: 0, duration: 0.2, ease: "power2.out" },
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
|
||||
// Circle reveal: text appears through an expanding circle
|
||||
tl.fromTo(
|
||||
groupEl,
|
||||
{ clipPath: "circle(0% at 50% 50%)" },
|
||||
{ clipPath: "circle(100% at 50% 50%)", duration: 0.35, ease: "expo.out" },
|
||||
group.start,
|
||||
);
|
||||
```
|
||||
|
||||
## Slam / Impact Words
|
||||
|
||||
Hero words slam onto the screen — they arrive fast, overshoot, and settle with weight. Reserve this for emphasis words (1-2 per group max). Over-using it kills the impact.
|
||||
|
||||
```js
|
||||
var isHeroWord = /^(LAUNCH|FREE|NOW|NEW|HUGE|INSANE)$/i.test(word.text);
|
||||
if (isHeroWord) {
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ scale: 2.5, opacity: 0, rotation: -8 },
|
||||
{ scale: 1, opacity: 1, rotation: 0, duration: 0.25, ease: "back.out(2.5)" },
|
||||
word.start,
|
||||
);
|
||||
// Micro-shake on impact
|
||||
tl.to(wordEl, { x: 4, duration: 0.03 }, word.start + 0.25);
|
||||
tl.to(wordEl, { x: -3, duration: 0.03 }, word.start + 0.28);
|
||||
tl.to(wordEl, { x: 0, duration: 0.04, ease: "power2.out" }, word.start + 0.31);
|
||||
} else {
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ opacity: 0, y: 20 },
|
||||
{ opacity: 1, y: 0, duration: 0.15, ease: "power2.out" },
|
||||
word.start,
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Absolute-Positioned Word Layout with Pretext
|
||||
|
||||
Position every word with `position: absolute` using pretext-measured widths. This unlocks animation paths that CSS inline flow can't do — words can fly in from any direction to their reading position.
|
||||
|
||||
```js
|
||||
var FONT = "900 72px Outfit";
|
||||
var GAP = 14; // px between words
|
||||
var containerWidth = 1600;
|
||||
|
||||
// Measure each word and compute its x position
|
||||
var xCursor = 0;
|
||||
var wordPositions = [];
|
||||
group.words.forEach(function (word) {
|
||||
var prepared = window.__hyperframes.pretext.prepare(word.text.toUpperCase(), FONT);
|
||||
var measured = window.__hyperframes.pretext.layout(prepared, 9999, 72 * 1.2);
|
||||
var w = measured.height / 1.2;
|
||||
wordPositions.push({ x: xCursor, width: w });
|
||||
xCursor += w + GAP;
|
||||
});
|
||||
|
||||
// Center the whole group
|
||||
var totalWidth = xCursor - GAP;
|
||||
var offsetX = (containerWidth - totalWidth) / 2;
|
||||
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
var finalX = wordPositions[wi].x + offsetX;
|
||||
wordEl.style.position = "absolute";
|
||||
wordEl.style.left = finalX + "px";
|
||||
|
||||
// Scatter entrance: each word arrives from a unique direction
|
||||
var angle = (wi / group.words.length) * Math.PI * 2;
|
||||
var radius = 300;
|
||||
var startX = finalX + Math.cos(angle) * radius;
|
||||
var startY = Math.sin(angle) * radius;
|
||||
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ x: startX - finalX, y: startY, opacity: 0, scale: 0.5 },
|
||||
{ x: 0, y: 0, opacity: 1, scale: 1, duration: 0.35, ease: "back.out(1.4)" },
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Elastic / Spring Entrances
|
||||
|
||||
Words arrive with physics — they overshoot their target and oscillate before settling. Different spring constants per word create an organic, staggered feeling.
|
||||
|
||||
```js
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
// Vary elasticity by word position — earlier words bouncier
|
||||
var elasticity = 0.3 + wi * 0.05;
|
||||
var amplitude = 1.2 - wi * 0.1;
|
||||
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ y: 60, opacity: 0, scaleY: 1.3, scaleX: 0.85 },
|
||||
{
|
||||
y: 0,
|
||||
opacity: 1,
|
||||
scaleY: 1,
|
||||
scaleX: 1,
|
||||
duration: 0.5,
|
||||
ease: "elastic.out(" + amplitude + ", " + elasticity + ")",
|
||||
},
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Rotation & 3D Perspective
|
||||
|
||||
Words rotate into view on the X or Y axis, creating a sense of depth. Requires `transformPerspective` on the parent for 3D effect.
|
||||
|
||||
```js
|
||||
// Set perspective on the group container
|
||||
gsap.set(groupEl, { transformPerspective: 800 });
|
||||
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
// Alternate rotation direction per word
|
||||
var rotDir = wi % 2 === 0 ? 90 : -90;
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ rotationX: rotDir, opacity: 0, transformOrigin: "50% 100%" },
|
||||
{ rotationX: 0, opacity: 1, duration: 0.3, ease: "power3.out" },
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Kinetic Exit Patterns
|
||||
|
||||
Exits are as important as entrances. Don't always fade out — give words somewhere to go.
|
||||
|
||||
```js
|
||||
// Scatter exit: words fly apart when the group ends
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
var angle = (wi / group.words.length) * Math.PI * 2;
|
||||
var exitX = Math.cos(angle) * 200;
|
||||
var exitY = Math.sin(angle) * 150;
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
x: exitX,
|
||||
y: exitY,
|
||||
opacity: 0,
|
||||
scale: 0.6,
|
||||
rotation: wi % 2 ? 15 : -15,
|
||||
duration: 0.2,
|
||||
ease: "power3.in",
|
||||
},
|
||||
group.end - 0.2,
|
||||
);
|
||||
});
|
||||
// Hard kill still required
|
||||
tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end);
|
||||
|
||||
// Collapse exit: words squeeze together then vanish
|
||||
tl.to(
|
||||
groupEl.querySelectorAll("span"),
|
||||
{
|
||||
letterSpacing: "-0.15em",
|
||||
scaleX: 0.7,
|
||||
opacity: 0,
|
||||
duration: 0.15,
|
||||
ease: "power2.in",
|
||||
stagger: { each: 0.02, from: "edges" },
|
||||
},
|
||||
group.end - 0.2,
|
||||
);
|
||||
|
||||
// Drop exit: words fall with gravity
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
y: 300,
|
||||
rotation: 10 + wi * 5,
|
||||
opacity: 0,
|
||||
duration: 0.3,
|
||||
ease: "power2.in",
|
||||
},
|
||||
group.end - 0.3 + wi * 0.03,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Combining Techniques
|
||||
|
||||
The best dynamic captions layer 2-3 techniques together. A few combinations that work:
|
||||
|
||||
| Combination | Energy | Best for |
|
||||
| ---------------------------------------- | ----------- | ------------------------------- |
|
||||
| Karaoke highlight + audio reactivity | Medium-high | Music videos, lyric videos |
|
||||
| Staggered entrance + scatter exit | High | Hype content, trailers |
|
||||
| Clip-path reveal + fade exit | Medium | Corporate, storytelling |
|
||||
| Slam heroes + elastic others + drop exit | Very high | Product launches, announcements |
|
||||
| 3D rotation entrance + collapse exit | Medium-high | Tech, modern brands |
|
||||
|
||||
Don't combine slam entrances with elastic entrances on the same group — pick one motion personality per group. You can vary techniques across groups to match the content's pace changes.
|
||||
|
||||
## Width-Aware Grouping with Pretext
|
||||
|
||||
Instead of grouping by word count alone, use pretext to group by visual width. This prevents some groups from filling the frame while others use 30%.
|
||||
|
||||
```js
|
||||
var FONT = "900 72px Outfit";
|
||||
var MAX_WIDTH = 1500; // slightly under container to leave padding
|
||||
|
||||
var groups = [];
|
||||
var currentGroup = { words: [], text: "" };
|
||||
|
||||
words.forEach(function (word) {
|
||||
var testText = (currentGroup.text + " " + word.text).trim().toUpperCase();
|
||||
var result = window.__hyperframes.fitTextFontSize(testText, {
|
||||
fontFamily: "Outfit",
|
||||
fontWeight: 900,
|
||||
maxWidth: MAX_WIDTH,
|
||||
baseFontSize: 72,
|
||||
minFontSize: 72,
|
||||
step: 2,
|
||||
});
|
||||
|
||||
if (!result.fits && currentGroup.words.length > 0) {
|
||||
// Adding this word would overflow — start new group
|
||||
groups.push(currentGroup);
|
||||
currentGroup = { words: [word], text: word.text };
|
||||
} else {
|
||||
currentGroup.words.push(word);
|
||||
currentGroup.text = testText;
|
||||
}
|
||||
});
|
||||
if (currentGroup.words.length > 0) groups.push(currentGroup);
|
||||
```
|
||||
|
||||
This replaces the fixed "3-5 words per group" heuristic with pixel-accurate measurement. "I" and "EXTRAORDINARY" take very different widths — pretext accounts for that.
|
||||
@@ -0,0 +1,94 @@
|
||||
# Transcript Guide
|
||||
|
||||
## How Transcripts Are Generated
|
||||
|
||||
`hyperframes transcribe` handles both transcription and format conversion:
|
||||
|
||||
```bash
|
||||
# Transcribe audio/video (uses whisper.cpp locally, no API key needed)
|
||||
npx hyperframes transcribe audio.mp3
|
||||
|
||||
# Use a larger model for better accuracy
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
|
||||
# Filter to English only (skips non-English speech)
|
||||
npx hyperframes transcribe audio.mp3 --language en
|
||||
|
||||
# Import an existing transcript from another tool
|
||||
npx hyperframes transcribe captions.srt
|
||||
npx hyperframes transcribe captions.vtt
|
||||
npx hyperframes transcribe openai-response.json
|
||||
```
|
||||
|
||||
## Supported Input Formats
|
||||
|
||||
The CLI auto-detects and normalizes these formats:
|
||||
|
||||
| Format | Extension | Source | Word-level? |
|
||||
| --------------------- | --------- | --------------------------------------------------------------------------- | ----------------- |
|
||||
| whisper.cpp JSON | `.json` | `hyperframes init --video`, `hyperframes transcribe` | Yes |
|
||||
| OpenAI Whisper API | `.json` | `openai.audio.transcriptions.create({ timestamp_granularities: ["word"] })` | Yes |
|
||||
| SRT subtitles | `.srt` | Video editors, subtitle tools, YouTube | No (phrase-level) |
|
||||
| VTT subtitles | `.vtt` | Web players, YouTube, transcription services | No (phrase-level) |
|
||||
| Normalized word array | `.json` | Pre-processed by any tool | Yes |
|
||||
|
||||
**Word-level timestamps produce better captions.** SRT/VTT give phrase-level timing, which works but can't do per-word animation effects.
|
||||
|
||||
## Whisper Model Guide
|
||||
|
||||
The default model (`small.en`) balances accuracy and speed. For better results, use a larger model:
|
||||
|
||||
| Model | Size | Speed | Accuracy | When to use |
|
||||
| ----------- | ------ | -------- | --------- | ------------------------------------- |
|
||||
| `tiny.en` | 75 MB | Fastest | Low | Quick previews, testing pipeline |
|
||||
| `base.en` | 142 MB | Fast | Fair | Short clips, clear audio |
|
||||
| `small.en` | 466 MB | Moderate | Good | **Default** — good for most content |
|
||||
| `medium.en` | 1.5 GB | Slow | Very good | Important content, noisy audio, music |
|
||||
| `large-v3` | 3.1 GB | Slowest | Best | Multilingual, production captions |
|
||||
|
||||
`.en` models are English-only and more accurate for English. Drop the `.en` suffix for multilingual (e.g., `medium` instead of `medium.en`).
|
||||
|
||||
**Music and vocals over instrumentation**: `small.en` will misidentify lyrics — use `medium.en` as the minimum, or import lyrics manually. Even `medium.en` struggles with heavily produced tracks; for music videos, providing known lyrics as an SRT/VTT and importing with `hyperframes transcribe lyrics.srt` will always beat automated transcription.
|
||||
|
||||
## Using External Transcription APIs
|
||||
|
||||
For the best accuracy, use an external API and import the result:
|
||||
|
||||
**OpenAI Whisper API** (recommended for quality):
|
||||
|
||||
```bash
|
||||
# Generate with word timestamps, then import
|
||||
curl https://api.openai.com/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-1 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-openai.json
|
||||
|
||||
npx hyperframes transcribe transcript-openai.json
|
||||
```
|
||||
|
||||
**Groq Whisper API** (fast, free tier available):
|
||||
|
||||
```bash
|
||||
curl https://api.groq.com/openai/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $GROQ_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-large-v3 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-groq.json
|
||||
|
||||
npx hyperframes transcribe transcript-groq.json
|
||||
```
|
||||
|
||||
## If No Transcript Exists
|
||||
|
||||
1. Check the project root for `transcript.json`, `.srt`, or `.vtt` files
|
||||
2. If none found, ask the user to provide one or run:
|
||||
```bash
|
||||
npx hyperframes transcribe <audio-or-video-file>
|
||||
```
|
||||
3. If transcription quality is poor (words at wrong times, gibberish), suggest upgrading the model:
|
||||
```bash
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
```
|
||||
@@ -140,6 +140,7 @@ Video must be `muted playsinline`. Audio is always a separate `<audio>` element:
|
||||
For PiP, title cards, and slide show patterns, see [patterns.md](./patterns.md).
|
||||
For data, stats, and infographics, see [data-in-motion.md](./data-in-motion.md).
|
||||
For typewriter text and other GSAP animation effects, see the `gsap-effects` skill.
|
||||
For audio-driven animation (beat sync, glow, pulse), see the `audio-reactive` skill.
|
||||
|
||||
## Output Checklist
|
||||
|
||||
|
||||
Reference in New Issue
Block a user