mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
feat: dynamic captions skill
This commit is contained in:
Vance Ingalls
@@ -25,98 +25,7 @@ This is the only format the captions composition consumes. Use it directly:
|
||||
const words = JSON.parse(transcriptJson); // [{ text, start, end }]
|
||||
```
|
||||
|
||||
### How transcripts are generated
|
||||
|
||||
`hyperframes transcribe` handles both transcription and format conversion:
|
||||
|
||||
```bash
|
||||
# Transcribe audio/video (uses whisper.cpp locally, no API key needed)
|
||||
npx hyperframes transcribe audio.mp3
|
||||
|
||||
# Use a larger model for better accuracy
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
|
||||
# Filter to English only (skips non-English speech)
|
||||
npx hyperframes transcribe audio.mp3 --language en
|
||||
|
||||
# Import an existing transcript from another tool
|
||||
npx hyperframes transcribe captions.srt
|
||||
npx hyperframes transcribe captions.vtt
|
||||
npx hyperframes transcribe openai-response.json
|
||||
```
|
||||
|
||||
### Supported input formats
|
||||
|
||||
The CLI auto-detects and normalizes these formats:
|
||||
|
||||
| Format | Extension | Source | Word-level? |
|
||||
| --------------------- | --------- | --------------------------------------------------------------------------- | ----------------- |
|
||||
| whisper.cpp JSON | `.json` | `hyperframes init --video`, `hyperframes transcribe` | Yes |
|
||||
| OpenAI Whisper API | `.json` | `openai.audio.transcriptions.create({ timestamp_granularities: ["word"] })` | Yes |
|
||||
| SRT subtitles | `.srt` | Video editors, subtitle tools, YouTube | No (phrase-level) |
|
||||
| VTT subtitles | `.vtt` | Web players, YouTube, transcription services | No (phrase-level) |
|
||||
| Normalized word array | `.json` | Pre-processed by any tool | Yes |
|
||||
|
||||
**Word-level timestamps produce better captions.** SRT/VTT give phrase-level timing, which works but can't do per-word animation effects.
|
||||
|
||||
### Whisper model guide
|
||||
|
||||
The default model (`small.en`) balances accuracy and speed. For better results, use a larger model:
|
||||
|
||||
| Model | Size | Speed | Accuracy | When to use |
|
||||
| ----------- | ------ | -------- | --------- | ------------------------------------- |
|
||||
| `tiny.en` | 75 MB | Fastest | Low | Quick previews, testing pipeline |
|
||||
| `base.en` | 142 MB | Fast | Fair | Short clips, clear audio |
|
||||
| `small.en` | 466 MB | Moderate | Good | **Default** — good for most content |
|
||||
| `medium.en` | 1.5 GB | Slow | Very good | Important content, noisy audio, music |
|
||||
| `large-v3` | 3.1 GB | Slowest | Best | Multilingual, production captions |
|
||||
|
||||
`.en` models are English-only and more accurate for English. Drop the `.en` suffix for multilingual (e.g., `medium` instead of `medium.en`).
|
||||
|
||||
**Music and vocals over instrumentation**: `small.en` will misidentify lyrics — use `medium.en` as the minimum, or import lyrics manually. Even `medium.en` struggles with heavily produced tracks; for music videos, providing known lyrics as an SRT/VTT and importing with `hyperframes transcribe lyrics.srt` will always beat automated transcription.
|
||||
|
||||
### Using external transcription APIs
|
||||
|
||||
For the best accuracy, use an external API and import the result:
|
||||
|
||||
**OpenAI Whisper API** (recommended for quality):
|
||||
|
||||
```bash
|
||||
# Generate with word timestamps, then import
|
||||
curl https://api.openai.com/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-1 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-openai.json
|
||||
|
||||
npx hyperframes transcribe transcript-openai.json
|
||||
```
|
||||
|
||||
**Groq Whisper API** (fast, free tier available):
|
||||
|
||||
```bash
|
||||
curl https://api.groq.com/openai/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $GROQ_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-large-v3 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-groq.json
|
||||
|
||||
npx hyperframes transcribe transcript-groq.json
|
||||
```
|
||||
|
||||
### If no transcript exists
|
||||
|
||||
1. Check the project root for `transcript.json`, `.srt`, or `.vtt` files
|
||||
2. If none found, ask the user to provide one or run:
|
||||
```bash
|
||||
npx hyperframes transcribe <audio-or-video-file>
|
||||
```
|
||||
3. If transcription quality is poor (words at wrong times, gibberish), suggest upgrading the model:
|
||||
```bash
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
```
|
||||
For transcription commands, whisper model selection, external APIs (OpenAI, Groq), and supported input formats, see [transcript-guide.md](./transcript-guide.md).
|
||||
|
||||
## Style Detection (Default — When No Style Is Specified)
|
||||
|
||||
@@ -207,28 +116,19 @@ Break groups on sentence boundaries (period, question mark, exclamation), pauses
|
||||
|
||||
Use `window.__hyperframes.fitTextFontSize()` to measure actual rendered text width and compute the correct font size. This replaces character-count heuristics with pixel-accurate measurement powered by [pretext](https://github.com/chenglou/pretext).
|
||||
|
||||
**Usage in composition scripts:**
|
||||
|
||||
```js
|
||||
GROUPS.forEach(function (group, gi) {
|
||||
// Measure with text-transform applied (captions typically uppercase)
|
||||
var result = window.__hyperframes.fitTextFontSize(group.text.toUpperCase(), {
|
||||
fontFamily: "Outfit",
|
||||
fontWeight: 900,
|
||||
maxWidth: 1600,
|
||||
});
|
||||
|
||||
// Apply computed font size to all word spans
|
||||
wordEls.forEach(function (el) {
|
||||
el.style.fontSize = result.fontSize + "px";
|
||||
});
|
||||
|
||||
// If result.fits is false, text exceeds minFontSize — overflow: hidden catches it
|
||||
});
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
| Option | Default | Description |
|
||||
| -------------- | ---------- | ---------------------------------------------------- |
|
||||
| `maxWidth` | `1600` | Container width in px (1600 landscape, 900 portrait) |
|
||||
@@ -238,7 +138,7 @@ GROUPS.forEach(function (group, gi) {
|
||||
| `fontFamily` | `"Outfit"` | Must match the CSS font-family |
|
||||
| `step` | `2` | Decrement step in px per iteration |
|
||||
|
||||
**Important:** The `fontWeight` and `fontFamily` options must match the CSS applied to the text elements exactly, or measurements will be inaccurate.
|
||||
`fontWeight` and `fontFamily` must match the CSS applied to the text elements exactly, or measurements will be inaccurate.
|
||||
|
||||
**Safety nets (still required in CSS):**
|
||||
|
||||
@@ -251,8 +151,6 @@ GROUPS.forEach(function (group, gi) {
|
||||
|
||||
Captions that stick on screen are the most common caption bug. Every caption group **must** have a hard kill after its exit animation.
|
||||
|
||||
**The pattern:**
|
||||
|
||||
```js
|
||||
// Animate exit (soft — can fail if tweens conflict)
|
||||
tl.to(groupEl, { opacity: 0, scale: 0.95, duration: 0.12, ease: "power2.in" }, group.end - 0.12);
|
||||
@@ -261,18 +159,11 @@ tl.to(groupEl, { opacity: 0, scale: 0.95, duration: 0.12, ease: "power2.in" }, g
|
||||
tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end);
|
||||
```
|
||||
|
||||
**Why both?** The `tl.to` exit can fail to fully hide a group when:
|
||||
**Why both?** The `tl.to` exit can fail to fully hide a group when karaoke word-level tweens conflict with the parent exit tween, `fromTo` entrance tweens lock values that override later tweens, or timeline scrubbing lands between the exit start and end. The `tl.set` at `group.end` is a deterministic kill — it fires at an exact time, doesn't animate, and can't be overridden.
|
||||
|
||||
- Karaoke word-level tweens (`scale`, `color`) on child elements conflict with the parent exit tween
|
||||
- `fromTo` entrance tweens lock start/end values that override later tweens on the same property
|
||||
- Timeline scrubbing lands between the exit start and end
|
||||
|
||||
The `tl.set` at `group.end` is a deterministic kill — it fires at an exact time, doesn't animate, and can't be overridden by other tweens at different times.
|
||||
|
||||
**Self-lint rule:** After building the timeline, verify every caption group has a hard kill. Run this check before registering the timeline:
|
||||
**Self-lint rule:** After building the timeline, verify every caption group has a hard kill:
|
||||
|
||||
```js
|
||||
// Caption lint: verify every group has a hard kill
|
||||
GROUPS.forEach(function (group, gi) {
|
||||
var el = document.getElementById("cg-" + gi);
|
||||
if (!el) return;
|
||||
@@ -280,20 +171,20 @@ GROUPS.forEach(function (group, gi) {
|
||||
var computed = window.getComputedStyle(el);
|
||||
if (computed.opacity !== "0" && computed.visibility !== "hidden") {
|
||||
console.warn(
|
||||
"[caption-lint] group " +
|
||||
gi +
|
||||
" ('" +
|
||||
group.text +
|
||||
"') still visible at t=" +
|
||||
(group.end + 0.01).toFixed(2) +
|
||||
"s",
|
||||
"[caption-lint] group " + gi + " still visible at t=" + (group.end + 0.01).toFixed(2) + "s",
|
||||
);
|
||||
}
|
||||
});
|
||||
tl.seek(0); // reset after lint
|
||||
tl.seek(0);
|
||||
```
|
||||
|
||||
Place this **before** `window.__timelines[id] = tl` so it runs at composition init. Warnings appear in the browser console during `hyperframes preview`.
|
||||
Place this **before** `window.__timelines[id] = tl` so it runs at composition init.
|
||||
|
||||
## References
|
||||
|
||||
For dynamic animation techniques (karaoke, clip-path reveals, slam words, scatter exits, elastic entrances, 3D rotation, audio-reactive captions, pretext-based positioning and grouping), see [dynamic-techniques.md](./dynamic-techniques.md).
|
||||
|
||||
For transcription commands, whisper models, external APIs, and troubleshooting, see [transcript-guide.md](./transcript-guide.md).
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -0,0 +1,360 @@
|
||||
# Dynamic Caption Techniques
|
||||
|
||||
The default caption pattern — fade group in, hold, fade out — works but looks like subtitles. These techniques make captions feel designed and intentional. Mix them based on the content's energy. Every technique below is deterministic and works with HyperFrames' frame-by-frame rendering.
|
||||
|
||||
## Per-Word Staggered Entrances
|
||||
|
||||
Each word in a group enters individually with staggered timing. The stagger creates a wave that follows the speaker's rhythm.
|
||||
|
||||
```js
|
||||
// Words enter one by one, timed to their speech timestamps
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.set(wordEl, { opacity: 0, y: 30, scale: 0.85 }, group.start);
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
opacity: 1,
|
||||
y: 0,
|
||||
scale: 1,
|
||||
duration: 0.18,
|
||||
ease: "back.out(1.7)",
|
||||
},
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
Vary the entrance per word role. Content words (nouns, verbs) get scale + y. Function words (the, a, and) get opacity only — they shouldn't compete for attention.
|
||||
|
||||
## Karaoke Highlight
|
||||
|
||||
All words in the group are visible from the start but muted. Each word transitions to full brightness as it's spoken. This gives the viewer reading context while directing attention to the current word.
|
||||
|
||||
```js
|
||||
// Show all words muted at group start
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.set(wordEl, { opacity: 0.3, scale: 0.95, color: "rgba(255,255,255,0.4)" }, group.start);
|
||||
// Light up when spoken
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
opacity: 1,
|
||||
scale: 1.05,
|
||||
color: "#ffffff",
|
||||
duration: 0.1,
|
||||
ease: "power2.out",
|
||||
},
|
||||
word.start,
|
||||
);
|
||||
// Settle after speaking
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
scale: 1,
|
||||
color: "rgba(255,255,255,0.85)",
|
||||
duration: 0.2,
|
||||
ease: "power1.out",
|
||||
},
|
||||
word.end,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
For high-energy content, add a color accent to the active word (`color: accentColor`) and a subtle glow (`textShadow: "0 0 20px " + accentColor`).
|
||||
|
||||
## Clip-Path Reveals
|
||||
|
||||
Words or groups reveal through an animated clip-path rather than fading. This creates a physical, tactile feeling — like text being uncovered.
|
||||
|
||||
```js
|
||||
// Horizontal wipe: text sweeps in from left
|
||||
tl.fromTo(
|
||||
groupEl,
|
||||
{ clipPath: "inset(0 100% 0 0)" },
|
||||
{ clipPath: "inset(0 0% 0 0)", duration: 0.4, ease: "power3.out" },
|
||||
group.start,
|
||||
);
|
||||
|
||||
// Per-word vertical reveal: each word drops in from behind a mask
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ clipPath: "inset(100% 0 0 0)", y: -10 },
|
||||
{ clipPath: "inset(0% 0 0 0)", y: 0, duration: 0.2, ease: "power2.out" },
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
|
||||
// Circle reveal: text appears through an expanding circle
|
||||
tl.fromTo(
|
||||
groupEl,
|
||||
{ clipPath: "circle(0% at 50% 50%)" },
|
||||
{ clipPath: "circle(100% at 50% 50%)", duration: 0.35, ease: "expo.out" },
|
||||
group.start,
|
||||
);
|
||||
```
|
||||
|
||||
## Slam / Impact Words
|
||||
|
||||
Hero words slam onto the screen — they arrive fast, overshoot, and settle with weight. Reserve this for emphasis words (1-2 per group max). Over-using it kills the impact.
|
||||
|
||||
```js
|
||||
var isHeroWord = /^(LAUNCH|FREE|NOW|NEW|HUGE|INSANE)$/i.test(word.text);
|
||||
if (isHeroWord) {
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ scale: 2.5, opacity: 0, rotation: -8 },
|
||||
{ scale: 1, opacity: 1, rotation: 0, duration: 0.25, ease: "back.out(2.5)" },
|
||||
word.start,
|
||||
);
|
||||
// Micro-shake on impact
|
||||
tl.to(wordEl, { x: 4, duration: 0.03 }, word.start + 0.25);
|
||||
tl.to(wordEl, { x: -3, duration: 0.03 }, word.start + 0.28);
|
||||
tl.to(wordEl, { x: 0, duration: 0.04, ease: "power2.out" }, word.start + 0.31);
|
||||
} else {
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ opacity: 0, y: 20 },
|
||||
{ opacity: 1, y: 0, duration: 0.15, ease: "power2.out" },
|
||||
word.start,
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Absolute-Positioned Word Layout with Pretext
|
||||
|
||||
Position every word with `position: absolute` using pretext-measured widths. This unlocks animation paths that CSS inline flow can't do — words can fly in from any direction to their reading position.
|
||||
|
||||
```js
|
||||
var FONT = "900 72px Outfit";
|
||||
var GAP = 14; // px between words
|
||||
var containerWidth = 1600;
|
||||
|
||||
// Measure each word and compute its x position
|
||||
var xCursor = 0;
|
||||
var wordPositions = [];
|
||||
group.words.forEach(function (word) {
|
||||
var prepared = window.__hyperframes.pretext.prepare(word.text.toUpperCase(), FONT);
|
||||
var measured = window.__hyperframes.pretext.layout(prepared, 9999, 72 * 1.2);
|
||||
var w = measured.height / 1.2;
|
||||
wordPositions.push({ x: xCursor, width: w });
|
||||
xCursor += w + GAP;
|
||||
});
|
||||
|
||||
// Center the whole group
|
||||
var totalWidth = xCursor - GAP;
|
||||
var offsetX = (containerWidth - totalWidth) / 2;
|
||||
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
var finalX = wordPositions[wi].x + offsetX;
|
||||
wordEl.style.position = "absolute";
|
||||
wordEl.style.left = finalX + "px";
|
||||
|
||||
// Scatter entrance: each word arrives from a unique direction
|
||||
var angle = (wi / group.words.length) * Math.PI * 2;
|
||||
var radius = 300;
|
||||
var startX = finalX + Math.cos(angle) * radius;
|
||||
var startY = Math.sin(angle) * radius;
|
||||
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ x: startX - finalX, y: startY, opacity: 0, scale: 0.5 },
|
||||
{ x: 0, y: 0, opacity: 1, scale: 1, duration: 0.35, ease: "back.out(1.4)" },
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Elastic / Spring Entrances
|
||||
|
||||
Words arrive with physics — they overshoot their target and oscillate before settling. Different spring constants per word create an organic, staggered feeling.
|
||||
|
||||
```js
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
// Vary elasticity by word position — earlier words bouncier
|
||||
var elasticity = 0.3 + wi * 0.05;
|
||||
var amplitude = 1.2 - wi * 0.1;
|
||||
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ y: 60, opacity: 0, scaleY: 1.3, scaleX: 0.85 },
|
||||
{
|
||||
y: 0,
|
||||
opacity: 1,
|
||||
scaleY: 1,
|
||||
scaleX: 1,
|
||||
duration: 0.5,
|
||||
ease: "elastic.out(" + amplitude + ", " + elasticity + ")",
|
||||
},
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Rotation & 3D Perspective
|
||||
|
||||
Words rotate into view on the X or Y axis, creating a sense of depth. Requires `transformPerspective` on the parent for 3D effect.
|
||||
|
||||
```js
|
||||
// Set perspective on the group container
|
||||
gsap.set(groupEl, { transformPerspective: 800 });
|
||||
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
// Alternate rotation direction per word
|
||||
var rotDir = wi % 2 === 0 ? 90 : -90;
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ rotationX: rotDir, opacity: 0, transformOrigin: "50% 100%" },
|
||||
{ rotationX: 0, opacity: 1, duration: 0.3, ease: "power3.out" },
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Kinetic Exit Patterns
|
||||
|
||||
Exits are as important as entrances. Don't always fade out — give words somewhere to go.
|
||||
|
||||
```js
|
||||
// Scatter exit: words fly apart when the group ends
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
var angle = (wi / group.words.length) * Math.PI * 2;
|
||||
var exitX = Math.cos(angle) * 200;
|
||||
var exitY = Math.sin(angle) * 150;
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
x: exitX,
|
||||
y: exitY,
|
||||
opacity: 0,
|
||||
scale: 0.6,
|
||||
rotation: wi % 2 ? 15 : -15,
|
||||
duration: 0.2,
|
||||
ease: "power3.in",
|
||||
},
|
||||
group.end - 0.2,
|
||||
);
|
||||
});
|
||||
// Hard kill still required
|
||||
tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end);
|
||||
|
||||
// Collapse exit: words squeeze together then vanish
|
||||
tl.to(
|
||||
groupEl.querySelectorAll("span"),
|
||||
{
|
||||
letterSpacing: "-0.15em",
|
||||
scaleX: 0.7,
|
||||
opacity: 0,
|
||||
duration: 0.15,
|
||||
ease: "power2.in",
|
||||
stagger: { each: 0.02, from: "edges" },
|
||||
},
|
||||
group.end - 0.2,
|
||||
);
|
||||
|
||||
// Drop exit: words fall with gravity
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
y: 300,
|
||||
rotation: 10 + wi * 5,
|
||||
opacity: 0,
|
||||
duration: 0.3,
|
||||
ease: "power2.in",
|
||||
},
|
||||
group.end - 0.3 + wi * 0.03,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Audio-Reactive Captions
|
||||
|
||||
When audio data is available (extracted via `extract-audio-data.py`), tie caption properties to the music. Captions pulse, glow, and shift with the beat.
|
||||
|
||||
```js
|
||||
// Load audio data alongside the transcript
|
||||
var AUDIO_DATA = /* loaded from audio-data.json */;
|
||||
|
||||
for (var f = 0; f < AUDIO_DATA.totalFrames; f++) {
|
||||
tl.call(function (frameIdx) {
|
||||
return function () {
|
||||
var frame = AUDIO_DATA.frames[frameIdx];
|
||||
if (!frame) return;
|
||||
// Find the currently visible caption group
|
||||
var activeGroup = document.querySelector('.caption-group[data-active="true"]');
|
||||
if (!activeGroup) return;
|
||||
var words = activeGroup.querySelectorAll("span");
|
||||
|
||||
// Bass drives scale pulse on the whole group
|
||||
var bassPulse = 1 + frame.bands[0] * 0.06;
|
||||
gsap.set(activeGroup, { scale: bassPulse });
|
||||
|
||||
// Treble drives glow intensity
|
||||
var treble = Math.max(frame.bands[12] || 0, frame.bands[13] || 0, frame.bands[14] || 0);
|
||||
var glow = Math.round(treble * 15);
|
||||
gsap.set(activeGroup, { textShadow: "0 0 " + glow + "px rgba(255,255,255," + (treble * 0.6) + ")" });
|
||||
};
|
||||
}(f), [], f / AUDIO_DATA.fps);
|
||||
}
|
||||
```
|
||||
|
||||
Keep audio reactivity subtle for captions — 3-6% scale variation and soft glow. Heavy pulsing makes text unreadable. Audio reactivity works best as a background texture, not the main event.
|
||||
|
||||
## Combining Techniques
|
||||
|
||||
The best dynamic captions layer 2-3 techniques together. A few combinations that work:
|
||||
|
||||
| Combination | Energy | Best for |
|
||||
| ---------------------------------------- | ----------- | ------------------------------- |
|
||||
| Karaoke highlight + bass pulse | Medium-high | Music videos, lyric videos |
|
||||
| Staggered entrance + scatter exit | High | Hype content, trailers |
|
||||
| Clip-path reveal + fade exit | Medium | Corporate, storytelling |
|
||||
| Slam heroes + elastic others + drop exit | Very high | Product launches, announcements |
|
||||
| 3D rotation entrance + collapse exit | Medium-high | Tech, modern brands |
|
||||
|
||||
Don't combine slam entrances with elastic entrances on the same group — pick one motion personality per group. You can vary techniques across groups to match the content's pace changes.
|
||||
|
||||
## Width-Aware Grouping with Pretext
|
||||
|
||||
Instead of grouping by word count alone, use pretext to group by visual width. This prevents some groups from filling the frame while others use 30%.
|
||||
|
||||
```js
|
||||
var FONT = "900 72px Outfit";
|
||||
var MAX_WIDTH = 1500; // slightly under container to leave padding
|
||||
|
||||
var groups = [];
|
||||
var currentGroup = { words: [], text: "" };
|
||||
|
||||
words.forEach(function (word) {
|
||||
var testText = (currentGroup.text + " " + word.text).trim().toUpperCase();
|
||||
var result = window.__hyperframes.fitTextFontSize(testText, {
|
||||
fontFamily: "Outfit",
|
||||
fontWeight: 900,
|
||||
maxWidth: MAX_WIDTH,
|
||||
baseFontSize: 72,
|
||||
minFontSize: 72,
|
||||
step: 2,
|
||||
});
|
||||
|
||||
if (!result.fits && currentGroup.words.length > 0) {
|
||||
// Adding this word would overflow — start new group
|
||||
groups.push(currentGroup);
|
||||
currentGroup = { words: [word], text: word.text };
|
||||
} else {
|
||||
currentGroup.words.push(word);
|
||||
currentGroup.text = testText;
|
||||
}
|
||||
});
|
||||
if (currentGroup.words.length > 0) groups.push(currentGroup);
|
||||
```
|
||||
|
||||
This replaces the fixed "3-5 words per group" heuristic with pixel-accurate measurement. "I" and "EXTRAORDINARY" take very different widths — pretext accounts for that.
|
||||
@@ -0,0 +1,94 @@
|
||||
# Transcript Guide
|
||||
|
||||
## How Transcripts Are Generated
|
||||
|
||||
`hyperframes transcribe` handles both transcription and format conversion:
|
||||
|
||||
```bash
|
||||
# Transcribe audio/video (uses whisper.cpp locally, no API key needed)
|
||||
npx hyperframes transcribe audio.mp3
|
||||
|
||||
# Use a larger model for better accuracy
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
|
||||
# Filter to English only (skips non-English speech)
|
||||
npx hyperframes transcribe audio.mp3 --language en
|
||||
|
||||
# Import an existing transcript from another tool
|
||||
npx hyperframes transcribe captions.srt
|
||||
npx hyperframes transcribe captions.vtt
|
||||
npx hyperframes transcribe openai-response.json
|
||||
```
|
||||
|
||||
## Supported Input Formats
|
||||
|
||||
The CLI auto-detects and normalizes these formats:
|
||||
|
||||
| Format | Extension | Source | Word-level? |
|
||||
| --------------------- | --------- | --------------------------------------------------------------------------- | ----------------- |
|
||||
| whisper.cpp JSON | `.json` | `hyperframes init --video`, `hyperframes transcribe` | Yes |
|
||||
| OpenAI Whisper API | `.json` | `openai.audio.transcriptions.create({ timestamp_granularities: ["word"] })` | Yes |
|
||||
| SRT subtitles | `.srt` | Video editors, subtitle tools, YouTube | No (phrase-level) |
|
||||
| VTT subtitles | `.vtt` | Web players, YouTube, transcription services | No (phrase-level) |
|
||||
| Normalized word array | `.json` | Pre-processed by any tool | Yes |
|
||||
|
||||
**Word-level timestamps produce better captions.** SRT/VTT give phrase-level timing, which works but can't do per-word animation effects.
|
||||
|
||||
## Whisper Model Guide
|
||||
|
||||
The default model (`small.en`) balances accuracy and speed. For better results, use a larger model:
|
||||
|
||||
| Model | Size | Speed | Accuracy | When to use |
|
||||
| ----------- | ------ | -------- | --------- | ------------------------------------- |
|
||||
| `tiny.en` | 75 MB | Fastest | Low | Quick previews, testing pipeline |
|
||||
| `base.en` | 142 MB | Fast | Fair | Short clips, clear audio |
|
||||
| `small.en` | 466 MB | Moderate | Good | **Default** — good for most content |
|
||||
| `medium.en` | 1.5 GB | Slow | Very good | Important content, noisy audio, music |
|
||||
| `large-v3` | 3.1 GB | Slowest | Best | Multilingual, production captions |
|
||||
|
||||
`.en` models are English-only and more accurate for English. Drop the `.en` suffix for multilingual (e.g., `medium` instead of `medium.en`).
|
||||
|
||||
**Music and vocals over instrumentation**: `small.en` will misidentify lyrics — use `medium.en` as the minimum, or import lyrics manually. Even `medium.en` struggles with heavily produced tracks; for music videos, providing known lyrics as an SRT/VTT and importing with `hyperframes transcribe lyrics.srt` will always beat automated transcription.
|
||||
|
||||
## Using External Transcription APIs
|
||||
|
||||
For the best accuracy, use an external API and import the result:
|
||||
|
||||
**OpenAI Whisper API** (recommended for quality):
|
||||
|
||||
```bash
|
||||
# Generate with word timestamps, then import
|
||||
curl https://api.openai.com/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-1 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-openai.json
|
||||
|
||||
npx hyperframes transcribe transcript-openai.json
|
||||
```
|
||||
|
||||
**Groq Whisper API** (fast, free tier available):
|
||||
|
||||
```bash
|
||||
curl https://api.groq.com/openai/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $GROQ_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-large-v3 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-groq.json
|
||||
|
||||
npx hyperframes transcribe transcript-groq.json
|
||||
```
|
||||
|
||||
## If No Transcript Exists
|
||||
|
||||
1. Check the project root for `transcript.json`, `.srt`, or `.vtt` files
|
||||
2. If none found, ask the user to provide one or run:
|
||||
```bash
|
||||
npx hyperframes transcribe <audio-or-video-file>
|
||||
```
|
||||
3. If transcription quality is poor (words at wrong times, gibberish), suggest upgrading the model:
|
||||
```bash
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
```
|
||||
Reference in New Issue
Block a user