mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
feat(whisper+captions): language detection, audio-reactive captions, multilingual defaults (#175)
## Summary **Whisper improvements:** - Auto-detect language and switch from `.en` to multilingual model when needed - Detect speech onset in WAV to strip hallucinated words before speech begins - Merge whisper-cpp token fragments: contractions (`didn` + `'t` → `didn't`), split capitals (`C` + `aught` → `Caught`), dropped-g (`shin` + `in'` → `shinin'`) - Interpolate zero-duration word clusters for reliable karaoke timing **Captions skill updates (folded from #176):** - Rewrite script-to-style mapping as an energy detection table (high → low) with mandatory animation requirements: karaoke baseline, 2+ highlight techniques, kinetic exits - Replace `tl.call()` per-frame audio-reactive pattern with group-level GSAP tweens — read peak bass/treble for each group's time range and modulate entrance intensity at build time - Add transcript quality check with automatic retry rules (>20% music tokens = retry with larger model) - Add caption word structure lint rule (`.caption-group` + `<span>`) for studio editor compatibility **Multilingual defaults (folded from #186):** - Default whisper model changed from `small.en` to `small` to prevent silent translation of non-English audio - Added non-negotiable language rule to captions skill ## Test plan - [ ] `pnpm test` passes (contraction merging, fragment merging, zero-duration interpolation, speech onset) - [ ] Transcribe non-English audio — verify it transcribes in original language, not translates - [ ] Skill files render correctly, cross-references resolve - [ ] `dynamic-techniques.md` audio-reactive section uses `tl.to()`/`tl.set()` only, no `tl.call()` loops 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This commit is contained in:
Vance Ingalls
@@ -95,13 +95,13 @@ Default is `small.en`. Upgrade for better accuracy:
|
||||
|
||||
| Model | Size | Use case |
|
||||
| ----------- | ------ | -------------------------------- |
|
||||
| `tiny.en` | 75 MB | Quick testing |
|
||||
| `base.en` | 142 MB | Short clips, clear audio |
|
||||
| `small.en` | 466 MB | **Default** — most content |
|
||||
| `medium.en` | 1.5 GB | Important content, noisy audio |
|
||||
| `large-v3` | 3.1 GB | Multilingual, production quality |
|
||||
| `tiny` | 75 MB | Quick testing |
|
||||
| `base` | 142 MB | Short clips, clear audio |
|
||||
| `small` | 466 MB | **Default** — most content |
|
||||
| `medium` | 1.5 GB | Important content, noisy audio |
|
||||
| `large-v3` | 3.1 GB | Production quality |
|
||||
|
||||
Use `.en` suffix for English-only (more accurate). Drop it for multilingual content.
|
||||
**Only use `.en` suffix when you know the audio is English.** `.en` models translate non-English audio into English instead of transcribing it.
|
||||
|
||||
### Supported transcript formats
|
||||
|
||||
|
||||
@@ -103,7 +103,8 @@ async function transcribeAudio(
|
||||
opts: { model?: string; language?: string; json?: boolean },
|
||||
): Promise<void> {
|
||||
const { transcribe } = await import("../whisper/transcribe.js");
|
||||
const { loadTranscript, patchCaptionHtml } = await import("../whisper/normalize.js");
|
||||
const { loadTranscript, patchCaptionHtml, stripBeforeOnset } =
|
||||
await import("../whisper/normalize.js");
|
||||
|
||||
const model = opts.model ?? DEFAULT_MODEL;
|
||||
const spin = opts.json ? null : clack.spinner();
|
||||
@@ -116,7 +117,19 @@ async function transcribeAudio(
|
||||
onProgress: spin ? (msg) => spin.message(msg) : undefined,
|
||||
});
|
||||
|
||||
const { words } = loadTranscript(result.transcriptPath);
|
||||
let { words } = loadTranscript(result.transcriptPath);
|
||||
|
||||
if (result.speechOnsetSeconds != null) {
|
||||
const before = words.length;
|
||||
words = stripBeforeOnset(words, result.speechOnsetSeconds);
|
||||
const stripped = before - words.length;
|
||||
if (stripped > 0 && !opts.json) {
|
||||
spin?.message(
|
||||
`Stripped ${stripped} words before speech onset at ${result.speechOnsetSeconds.toFixed(1)}s`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
writeFileSync(result.transcriptPath, JSON.stringify(words, null, 2));
|
||||
patchCaptionHtml(dir, words);
|
||||
|
||||
@@ -127,13 +140,18 @@ async function transcribeAudio(
|
||||
model,
|
||||
wordCount: words.length,
|
||||
durationSeconds: result.durationSeconds,
|
||||
speechOnsetSeconds: result.speechOnsetSeconds,
|
||||
transcriptPath: result.transcriptPath,
|
||||
}),
|
||||
);
|
||||
} else {
|
||||
spin!.stop(
|
||||
const onsetNote =
|
||||
result.speechOnsetSeconds != null
|
||||
? ` — speech detected at ${result.speechOnsetSeconds.toFixed(1)}s`
|
||||
: "";
|
||||
spin?.stop(
|
||||
c.success(
|
||||
`Transcribed ${c.accent(String(words.length))} words (${result.durationSeconds.toFixed(1)}s)`,
|
||||
`Transcribed ${c.accent(String(words.length))} words (${result.durationSeconds.toFixed(1)}s${onsetNote})`,
|
||||
),
|
||||
);
|
||||
}
|
||||
@@ -142,7 +160,7 @@ async function transcribeAudio(
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify({ ok: false, error: message }));
|
||||
} else {
|
||||
spin!.stop(c.error(`Transcription failed: ${message}`));
|
||||
spin?.stop(c.error(`Transcription failed: ${message}`));
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
@@ -2,7 +2,8 @@ import { describe, it, expect, afterEach } from "vitest";
|
||||
import { writeFileSync, readFileSync, mkdirSync, rmSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { loadTranscript, detectFormat, patchCaptionHtml } from "./normalize.js";
|
||||
import { loadTranscript, detectFormat, patchCaptionHtml, stripBeforeOnset } from "./normalize.js";
|
||||
import { detectSpeechOnset } from "./transcribe.js";
|
||||
|
||||
function tmpFile(name: string, content: string): string {
|
||||
const dir = join(tmpdir(), `hf-normalize-test-${Date.now()}`);
|
||||
@@ -117,7 +118,7 @@ describe("loadTranscript", () => {
|
||||
);
|
||||
const { words } = loadTranscript(path);
|
||||
expect(words).toHaveLength(1);
|
||||
expect(words[0]!.text).toBe("Hello");
|
||||
expect(words[0]?.text).toBe("Hello");
|
||||
});
|
||||
|
||||
it("parses OpenAI Whisper API response", () => {
|
||||
@@ -183,8 +184,8 @@ Short format
|
||||
`;
|
||||
const path = tmpFile("short.vtt", vtt);
|
||||
const { words } = loadTranscript(path);
|
||||
expect(words[0]!.start).toBeCloseTo(83.456, 2);
|
||||
expect(words[0]!.end).toBe(120.0);
|
||||
expect(words[0]?.start).toBeCloseTo(83.456, 2);
|
||||
expect(words[0]?.end).toBe(120.0);
|
||||
});
|
||||
|
||||
it("strips HTML tags from SRT/VTT", () => {
|
||||
@@ -194,7 +195,7 @@ Short format
|
||||
`;
|
||||
const path = tmpFile("tags.srt", srt);
|
||||
const { words } = loadTranscript(path);
|
||||
expect(words[0]!.text).toBe("Bold and italic");
|
||||
expect(words[0]?.text).toBe("Bold and italic");
|
||||
});
|
||||
|
||||
it("passes through normalized word arrays", () => {
|
||||
@@ -209,6 +210,172 @@ Short format
|
||||
});
|
||||
});
|
||||
|
||||
describe("whisper-cpp contraction merging", () => {
|
||||
it("merges didn + 't into didn't", () => {
|
||||
const path = tmpFile(
|
||||
"contractions.json",
|
||||
JSON.stringify({
|
||||
transcription: [
|
||||
{
|
||||
tokens: [
|
||||
{ text: " I", offsets: { from: 0, to: 200 } },
|
||||
{ text: " didn", offsets: { from: 200, to: 500 } },
|
||||
{ text: "'t", offsets: { from: 500, to: 700 } },
|
||||
{ text: " know", offsets: { from: 700, to: 1000 } },
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const { words } = loadTranscript(path);
|
||||
expect(words).toEqual([
|
||||
{ text: "I", start: 0, end: 0.2 },
|
||||
{ text: "didn't", start: 0.2, end: 0.7 },
|
||||
{ text: "know", start: 0.7, end: 1 },
|
||||
]);
|
||||
});
|
||||
|
||||
it("merges I + 'm into I'm", () => {
|
||||
const path = tmpFile(
|
||||
"im.json",
|
||||
JSON.stringify({
|
||||
transcription: [
|
||||
{
|
||||
tokens: [
|
||||
{ text: " I", offsets: { from: 0, to: 100 } },
|
||||
{ text: "'m", offsets: { from: 100, to: 300 } },
|
||||
{ text: " done", offsets: { from: 300, to: 600 } },
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const { words } = loadTranscript(path);
|
||||
expect(words[0]?.text).toBe("I'm");
|
||||
expect(words[0]?.end).toBe(0.3);
|
||||
});
|
||||
|
||||
it("merges could + 've into could've", () => {
|
||||
const path = tmpFile(
|
||||
"couldve.json",
|
||||
JSON.stringify({
|
||||
transcription: [
|
||||
{
|
||||
tokens: [
|
||||
{ text: " could", offsets: { from: 0, to: 400 } },
|
||||
{ text: "'ve", offsets: { from: 400, to: 600 } },
|
||||
{ text: " been", offsets: { from: 600, to: 900 } },
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const { words } = loadTranscript(path);
|
||||
expect(words[0]?.text).toBe("could've");
|
||||
});
|
||||
});
|
||||
|
||||
describe("whisper-cpp fragment merging", () => {
|
||||
it("merges single capital + lowercase: C + aught -> Caught", () => {
|
||||
const path = tmpFile(
|
||||
"fragments.json",
|
||||
JSON.stringify({
|
||||
transcription: [
|
||||
{
|
||||
tokens: [
|
||||
{ text: " C", offsets: { from: 0, to: 100 } },
|
||||
{ text: "aught", offsets: { from: 100, to: 500 } },
|
||||
{ text: " a", offsets: { from: 500, to: 600 } },
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const { words } = loadTranscript(path);
|
||||
expect(words[0]?.text).toBe("Caught");
|
||||
expect(words[0]?.end).toBe(0.5);
|
||||
expect(words).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("merges consonant + in': shin + in' -> shinin'", () => {
|
||||
const path = tmpFile(
|
||||
"dropg.json",
|
||||
JSON.stringify({
|
||||
transcription: [
|
||||
{
|
||||
tokens: [
|
||||
{ text: " shin", offsets: { from: 0, to: 300 } },
|
||||
{ text: "in'", offsets: { from: 300, to: 500 } },
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const { words } = loadTranscript(path);
|
||||
expect(words).toHaveLength(1);
|
||||
expect(words[0]?.text).toBe("shinin'");
|
||||
});
|
||||
});
|
||||
|
||||
describe("whisper-cpp zero-duration interpolation", () => {
|
||||
it("interpolates a cluster of zero-duration words", () => {
|
||||
const path = tmpFile(
|
||||
"zerodur.json",
|
||||
JSON.stringify({
|
||||
transcription: [
|
||||
{
|
||||
tokens: [
|
||||
{ text: " hello", offsets: { from: 0, to: 500 } },
|
||||
{ text: " we", offsets: { from: 1000, to: 1000 } },
|
||||
{ text: " are", offsets: { from: 1000, to: 1000 } },
|
||||
{ text: " here", offsets: { from: 1000, to: 1000 } },
|
||||
{ text: " now", offsets: { from: 1500, to: 2000 } },
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const { words } = loadTranscript(path);
|
||||
expect(words).toHaveLength(5);
|
||||
// The three zero-duration words should be spread between 0.5 and 1.5
|
||||
const we = words[1] ?? { start: 0, end: 0, text: "" };
|
||||
const are = words[2] ?? { start: 0, end: 0, text: "" };
|
||||
const here = words[3] ?? { start: 0, end: 0, text: "" };
|
||||
expect(we.start).toBeCloseTo(0.5, 1);
|
||||
expect(we.end).toBeCloseTo(0.833, 1);
|
||||
expect(are.start).toBeCloseTo(0.833, 1);
|
||||
expect(are.end).toBeCloseTo(1.167, 1);
|
||||
expect(here.start).toBeCloseTo(1.167, 1);
|
||||
expect(here.end).toBeCloseTo(1.5, 1);
|
||||
// Each should have positive duration
|
||||
expect(we.end).toBeGreaterThan(we.start);
|
||||
expect(are.end).toBeGreaterThan(are.start);
|
||||
expect(here.end).toBeGreaterThan(here.start);
|
||||
});
|
||||
|
||||
it("handles isolated zero-duration word", () => {
|
||||
const path = tmpFile(
|
||||
"singlezero.json",
|
||||
JSON.stringify({
|
||||
transcription: [
|
||||
{
|
||||
tokens: [
|
||||
{ text: " hello", offsets: { from: 0, to: 500 } },
|
||||
{ text: " I", offsets: { from: 800, to: 800 } },
|
||||
{ text: " know", offsets: { from: 1000, to: 1500 } },
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const { words } = loadTranscript(path);
|
||||
const iWord = words[1] ?? { start: 0, end: 0, text: "" };
|
||||
expect(iWord.end).toBeGreaterThan(iWord.start);
|
||||
expect(iWord.start).toBeCloseTo(0.5, 1);
|
||||
expect(iWord.end).toBeCloseTo(1, 1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("patchCaptionHtml", () => {
|
||||
it("replaces const script = [] in HTML files", () => {
|
||||
const dir = join(tmpdir(), `hf-patch-test-${Date.now()}`);
|
||||
@@ -276,3 +443,105 @@ describe("patchCaptionHtml", () => {
|
||||
expect(result).toBe(html);
|
||||
});
|
||||
});
|
||||
|
||||
describe("detectSpeechOnset", () => {
|
||||
function makeSyntheticWav(
|
||||
sampleRate: number,
|
||||
durationSeconds: number,
|
||||
energyFn: (t: number) => number,
|
||||
): string {
|
||||
const numSamples = Math.floor(sampleRate * durationSeconds);
|
||||
const dataSize = numSamples * 2;
|
||||
const buf = Buffer.alloc(44 + dataSize);
|
||||
// RIFF header
|
||||
buf.write("RIFF", 0);
|
||||
buf.writeUInt32LE(36 + dataSize, 4);
|
||||
buf.write("WAVE", 8);
|
||||
buf.write("fmt ", 12);
|
||||
buf.writeUInt32LE(16, 16); // chunk size
|
||||
buf.writeUInt16LE(1, 20); // PCM
|
||||
buf.writeUInt16LE(1, 22); // mono
|
||||
buf.writeUInt32LE(sampleRate, 24);
|
||||
buf.writeUInt32LE(sampleRate * 2, 28); // byte rate
|
||||
buf.writeUInt16LE(2, 32); // block align
|
||||
buf.writeUInt16LE(16, 34); // bits per sample
|
||||
buf.write("data", 36);
|
||||
buf.writeUInt32LE(dataSize, 40);
|
||||
for (let i = 0; i < numSamples; i++) {
|
||||
const t = i / sampleRate;
|
||||
const amplitude = energyFn(t);
|
||||
buf.writeInt16LE(Math.round(amplitude * 32767), 44 + i * 2);
|
||||
}
|
||||
const path = join(tmpdir(), `hf-wav-test-${Date.now()}-${Math.floor(Math.random() * 1e6)}.wav`);
|
||||
writeFileSync(path, buf);
|
||||
dirs.push(path);
|
||||
return path;
|
||||
}
|
||||
|
||||
it("detects onset when silence transitions to loud", () => {
|
||||
const wavPath = makeSyntheticWav(16000, 15, (t) => (t < 5 ? 0.01 : 0.8));
|
||||
const onset = detectSpeechOnset(wavPath);
|
||||
expect(onset).not.toBeNull();
|
||||
expect(onset!).toBeGreaterThanOrEqual(4);
|
||||
expect(onset!).toBeLessThanOrEqual(7);
|
||||
});
|
||||
|
||||
it("returns null for consistent energy throughout", () => {
|
||||
const wavPath = makeSyntheticWav(16000, 10, () => 0.5);
|
||||
const onset = detectSpeechOnset(wavPath);
|
||||
expect(onset).toBeNull();
|
||||
});
|
||||
|
||||
it("returns null for very short audio", () => {
|
||||
const wavPath = makeSyntheticWav(16000, 2, () => 0.5);
|
||||
const onset = detectSpeechOnset(wavPath);
|
||||
expect(onset).toBeNull();
|
||||
});
|
||||
|
||||
it("returns null when onset is too early (< 3s)", () => {
|
||||
const wavPath = makeSyntheticWav(16000, 10, (t) => (t < 1 ? 0.01 : 0.8));
|
||||
const onset = detectSpeechOnset(wavPath);
|
||||
expect(onset).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("stripBeforeOnset", () => {
|
||||
it("removes words before onset time", () => {
|
||||
const words = [
|
||||
{ text: "ghost", start: 0.5, end: 2.0 },
|
||||
{ text: "alone", start: 3.0, end: 5.0 },
|
||||
{ text: "Given", start: 19.0, end: 19.5 },
|
||||
{ text: "the", start: 19.5, end: 20.0 },
|
||||
];
|
||||
const result = stripBeforeOnset(words, 18.5);
|
||||
expect(result).toHaveLength(2);
|
||||
expect(result[0]!.text).toBe("Given");
|
||||
});
|
||||
|
||||
it("keeps words within 0.5s tolerance of onset", () => {
|
||||
const words = [
|
||||
{ text: "hello", start: 18.2, end: 18.8 },
|
||||
{ text: "world", start: 19.0, end: 19.5 },
|
||||
];
|
||||
const result = stripBeforeOnset(words, 18.5);
|
||||
expect(result).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("keeps everything when onset is 0", () => {
|
||||
const words = [
|
||||
{ text: "hello", start: 0.1, end: 0.5 },
|
||||
{ text: "world", start: 0.6, end: 1.0 },
|
||||
];
|
||||
const result = stripBeforeOnset(words, 0);
|
||||
expect(result).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("returns empty array when all words are before onset", () => {
|
||||
const words = [
|
||||
{ text: "ghost", start: 0.5, end: 2.0 },
|
||||
{ text: "alone", start: 3.0, end: 5.0 },
|
||||
];
|
||||
const result = stripBeforeOnset(words, 20.0);
|
||||
expect(result).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -43,6 +43,67 @@ function detectJsonFormat(raw: unknown): TranscriptFormat {
|
||||
// Parsers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Rejoin word fragments that whisper splits across tokens:
|
||||
* - Single capital + lowercase continuation: C + aught -> Caught, G + onna -> Gonna
|
||||
* - Word ending in consonant + in': shin + in' -> shinin', hid + in' -> hidin'
|
||||
*/
|
||||
function mergeFragments(words: Word[]): void {
|
||||
for (let i = 0; i < words.length - 1; i++) {
|
||||
const curr = words[i];
|
||||
const next = words[i + 1];
|
||||
if (!curr || !next) continue;
|
||||
const isSingleLetterFragment =
|
||||
curr.text.length === 1 &&
|
||||
/^[A-Z]$/.test(curr.text) &&
|
||||
!/^[IAO]$/.test(curr.text) &&
|
||||
/^[a-z]/.test(next.text);
|
||||
const shouldMerge =
|
||||
isSingleLetterFragment || (/[a-z]$/.test(curr.text) && /^in'$/i.test(next.text));
|
||||
if (shouldMerge) {
|
||||
curr.text += next.text;
|
||||
curr.end = next.end;
|
||||
words.splice(i + 1, 1);
|
||||
i--;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Distribute timestamps evenly across zero-duration word clusters.
|
||||
* Whisper sometimes assigns identical start/end to sequences of words,
|
||||
* making karaoke highlights flash through them instantly.
|
||||
*
|
||||
* Also handles malformed timestamps where start > end — these are treated
|
||||
* the same as zero-duration and get interpolated from surrounding words.
|
||||
*/
|
||||
function interpolateZeroDuration(words: Word[]): void {
|
||||
for (let i = 0; i < words.length; i++) {
|
||||
const wi = words[i];
|
||||
if (!wi || wi.start < wi.end) continue;
|
||||
let j = i;
|
||||
while (j < words.length) {
|
||||
const wj = words[j];
|
||||
if (!wj || wj.start < wj.end) break;
|
||||
j++;
|
||||
}
|
||||
const clusterLen = j - i;
|
||||
const prev = i > 0 ? words[i - 1] : undefined;
|
||||
const prevEnd = prev ? prev.end : wi.start;
|
||||
const nextWord = j < words.length ? words[j] : undefined;
|
||||
const nextStart = nextWord ? nextWord.start : prevEnd + clusterLen * 0.3;
|
||||
const span = nextStart - prevEnd;
|
||||
const perWord = span / clusterLen;
|
||||
for (let k = i; k < j; k++) {
|
||||
const wk = words[k];
|
||||
if (!wk) continue;
|
||||
wk.start = round3(prevEnd + (k - i) * perWord);
|
||||
wk.end = round3(prevEnd + (k - i + 1) * perWord);
|
||||
}
|
||||
i = j - 1;
|
||||
}
|
||||
}
|
||||
|
||||
function parseWhisperCpp(data: Record<string, unknown>): Word[] {
|
||||
const words: Word[] = [];
|
||||
const transcription = data.transcription as Array<{
|
||||
@@ -54,13 +115,21 @@ function parseWhisperCpp(data: Record<string, unknown>): Word[] {
|
||||
|
||||
for (const seg of transcription ?? []) {
|
||||
for (const token of seg.tokens ?? []) {
|
||||
const text = (token.text ?? "").trim();
|
||||
const rawText = token.text ?? "";
|
||||
const text = rawText.trim();
|
||||
if (!text || text.startsWith("[_") || text.startsWith("[BLANK")) continue;
|
||||
|
||||
// Merge punctuation with the previous word
|
||||
const isPunctuation = /^[.,!?;:'")\]}>…–—-]+$/.test(text);
|
||||
const lastWord = words[words.length - 1];
|
||||
if (isPunctuation && lastWord) {
|
||||
|
||||
// Merge into previous word when the token is a sub-word continuation,
|
||||
// trailing punctuation, or a contraction suffix.
|
||||
// Whisper uses leading spaces to mark word boundaries in all languages.
|
||||
const shouldMerge =
|
||||
lastWord &&
|
||||
(!rawText.startsWith(" ") ||
|
||||
/^[.,!?;:'")\]}>…–—¡¿-]+$/.test(text) ||
|
||||
/^'(t|m|s|ve|re|ll|d)$/i.test(text));
|
||||
if (shouldMerge) {
|
||||
lastWord.text += text;
|
||||
lastWord.end = round3((token.offsets?.to ?? 0) / 1000);
|
||||
continue;
|
||||
@@ -73,6 +142,10 @@ function parseWhisperCpp(data: Record<string, unknown>): Word[] {
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
mergeFragments(words);
|
||||
interpolateZeroDuration(words);
|
||||
|
||||
return words;
|
||||
}
|
||||
|
||||
@@ -229,9 +302,15 @@ export function loadTranscript(filePath: string): { words: Word[]; format: Trans
|
||||
}
|
||||
|
||||
/**
|
||||
* Patch caption HTML files in a project directory with transcript words.
|
||||
* Replaces `const script = [...]` or `const TRANSCRIPT = [...]` in <script> blocks.
|
||||
* Remove words that fall before the detected speech onset.
|
||||
* Whisper can hallucinate words over non-speech sections at the start of audio.
|
||||
*/
|
||||
export function stripBeforeOnset(words: Word[], onsetSeconds: number): Word[] {
|
||||
// 0.5s tolerance: keep words whose timestamps straddle the onset boundary,
|
||||
// since whisper may assign a slightly early start to the first spoken word.
|
||||
return words.filter((w) => w.start >= onsetSeconds - 0.5);
|
||||
}
|
||||
|
||||
export function patchCaptionHtml(dir: string, words: Word[]): void {
|
||||
if (words.length === 0) return;
|
||||
|
||||
|
||||
@@ -4,6 +4,98 @@ import { join, extname } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { ensureWhisper, ensureModel, hasFFmpeg, DEFAULT_MODEL } from "./manager.js";
|
||||
|
||||
/**
|
||||
* Detect the language of a WAV file using whisper's built-in language detection.
|
||||
* Returns an ISO 639-1 code (e.g. "en", "es", "hi") or null if detection fails.
|
||||
*/
|
||||
function detectLanguage(whisperPath: string, modelPath: string, wavPath: string): string | null {
|
||||
try {
|
||||
const output = execFileSync(whisperPath, ["--model", modelPath, "--detect-language", wavPath], {
|
||||
encoding: "utf-8",
|
||||
timeout: 30_000,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
const match = output.match(/auto-detected language:\s*(\w+)/);
|
||||
return match?.[1] ?? null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function findWavDataChunk(buf: Buffer): { offset: number; size: number } | null {
|
||||
if (buf.length < 12) return null;
|
||||
let pos = 12; // skip RIFF header
|
||||
while (pos + 8 < buf.length) {
|
||||
const id = buf.toString("ascii", pos, pos + 4);
|
||||
const size = buf.readUInt32LE(pos + 4);
|
||||
if (id === "data") return { offset: pos + 8, size: Math.min(size, buf.length - pos - 8) };
|
||||
pos += 8 + size;
|
||||
if (size % 2 !== 0) pos++; // RIFF chunks are word-aligned
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect when speech begins in a 16kHz mono WAV by finding the first
|
||||
* sustained energy jump above the track's median RMS. Returns onset time in
|
||||
* seconds, or null if the track has consistent energy throughout.
|
||||
*/
|
||||
export function detectSpeechOnset(wavPath: string): number | null {
|
||||
const SAMPLE_RATE = 16000;
|
||||
const WINDOW_SECONDS = 0.5;
|
||||
const WINDOW_SAMPLES = SAMPLE_RATE * WINDOW_SECONDS;
|
||||
const SUSTAINED_WINDOWS = 3; // 1.5s above threshold to count as onset
|
||||
const SILENCE_THRESHOLD_RATIO = 0.6;
|
||||
const MIN_INTRO_SECONDS = 3; // don't strip if onset is very early
|
||||
|
||||
try {
|
||||
const buf = readFileSync(wavPath);
|
||||
const dataChunk = findWavDataChunk(buf);
|
||||
if (!dataChunk) return null;
|
||||
const pcm = new Int16Array(buf.buffer, buf.byteOffset + dataChunk.offset, dataChunk.size / 2);
|
||||
const totalWindows = Math.floor(pcm.length / WINDOW_SAMPLES);
|
||||
if (totalWindows < 10) return null;
|
||||
|
||||
const rmsValues: number[] = [];
|
||||
for (let i = 0; i < totalWindows; i++) {
|
||||
const start = i * WINDOW_SAMPLES;
|
||||
let sumSq = 0;
|
||||
for (let j = start; j < start + WINDOW_SAMPLES; j++) {
|
||||
const sample = pcm[j] ?? 0;
|
||||
sumSq += sample * sample;
|
||||
}
|
||||
rmsValues.push(Math.sqrt(sumSq / WINDOW_SAMPLES));
|
||||
}
|
||||
|
||||
const sorted = [...rmsValues].sort((a, b) => a - b);
|
||||
const median = sorted[Math.floor(sorted.length / 2)] ?? 0;
|
||||
const threshold = median * SILENCE_THRESHOLD_RATIO;
|
||||
|
||||
// Check if energy is fairly consistent (no clear intro) — ratio of
|
||||
// first 10s average to median. If it's already close, no onset to detect.
|
||||
const introAvg =
|
||||
rmsValues.slice(0, Math.min(20, rmsValues.length)).reduce((a, b) => a + b, 0) /
|
||||
Math.min(20, rmsValues.length);
|
||||
if (introAvg >= threshold) return null;
|
||||
|
||||
let consecutive = 0;
|
||||
for (let i = 0; i < rmsValues.length; i++) {
|
||||
if ((rmsValues[i] ?? 0) >= threshold) {
|
||||
consecutive++;
|
||||
if (consecutive >= SUSTAINED_WINDOWS) {
|
||||
const onsetSeconds = (i - SUSTAINED_WINDOWS + 1) * WINDOW_SECONDS;
|
||||
return onsetSeconds >= MIN_INTRO_SECONDS ? onsetSeconds : null;
|
||||
}
|
||||
} else {
|
||||
consecutive = 0;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Can't read WAV — skip onset detection
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
const AUDIO_EXTENSIONS = new Set([".mp3", ".wav", ".m4a", ".aac", ".ogg", ".flac"]);
|
||||
const VIDEO_EXTENSIONS = new Set([".mp4", ".webm", ".mov", ".mkv", ".avi"]);
|
||||
|
||||
@@ -17,6 +109,7 @@ export interface TranscribeResult {
|
||||
transcriptPath: string;
|
||||
wordCount: number;
|
||||
durationSeconds: number;
|
||||
speechOnsetSeconds: number | null;
|
||||
}
|
||||
|
||||
function isAudioFile(filePath: string): boolean {
|
||||
@@ -121,29 +214,53 @@ export async function transcribe(
|
||||
throw new Error(`Unsupported file type: ${ext}`);
|
||||
}
|
||||
|
||||
// 4. Run whisper
|
||||
// 4. Detect language and ensure correct model
|
||||
let effectiveModel = model;
|
||||
let effectiveModelPath = modelPath;
|
||||
let detectedLanguage = options?.language ?? null;
|
||||
|
||||
// Only auto-detect language when using a multilingual model.
|
||||
// .en models always report "en" regardless of actual language, so detection
|
||||
// would be a no-op. If the user chose .en, they want English.
|
||||
if (!detectedLanguage && !effectiveModel.endsWith(".en")) {
|
||||
options?.onProgress?.("Detecting language...");
|
||||
detectedLanguage = detectLanguage(whisper.executablePath, effectiveModelPath, wavPath);
|
||||
}
|
||||
|
||||
if (detectedLanguage && detectedLanguage !== "en" && effectiveModel.endsWith(".en")) {
|
||||
const multilingualModel = effectiveModel.replace(/\.en$/, "");
|
||||
options?.onProgress?.(
|
||||
`Detected ${detectedLanguage} — switching to ${multilingualModel} model...`,
|
||||
);
|
||||
effectiveModelPath = await ensureModel(multilingualModel, {
|
||||
onProgress: options?.onProgress,
|
||||
});
|
||||
effectiveModel = multilingualModel;
|
||||
}
|
||||
|
||||
// 5. Run whisper
|
||||
options?.onProgress?.("Transcribing...");
|
||||
const outputBase = join(outputDir, "transcript");
|
||||
mkdirSync(outputDir, { recursive: true });
|
||||
|
||||
const whisperArgs = [
|
||||
"--model",
|
||||
modelPath,
|
||||
effectiveModelPath,
|
||||
"--output-json-full",
|
||||
"--output-file",
|
||||
outputBase,
|
||||
"--dtw",
|
||||
model,
|
||||
effectiveModel,
|
||||
"--suppress-nst",
|
||||
];
|
||||
if (options?.language) {
|
||||
whisperArgs.push("--language", options.language);
|
||||
if (detectedLanguage) {
|
||||
whisperArgs.push("--language", detectedLanguage);
|
||||
}
|
||||
whisperArgs.push(wavPath);
|
||||
|
||||
execFileSync(whisper.executablePath, whisperArgs, { stdio: "ignore", timeout: 300_000 });
|
||||
|
||||
// 5. Read and validate output
|
||||
// 6. Read and validate output
|
||||
const transcriptPath = `${outputBase}.json`;
|
||||
if (!existsSync(transcriptPath)) {
|
||||
throw new Error("Whisper did not produce output. Check the input file.");
|
||||
@@ -162,6 +279,10 @@ export async function transcribe(
|
||||
}
|
||||
}
|
||||
|
||||
// 7. Detect speech onset before cleaning up the WAV
|
||||
options?.onProgress?.("Detecting speech onset...");
|
||||
const speechOnsetSeconds = detectSpeechOnset(wavPath);
|
||||
|
||||
// Clean up temp WAV if we created one
|
||||
if (wavPath !== inputPath) {
|
||||
try {
|
||||
@@ -175,6 +296,7 @@ export async function transcribe(
|
||||
transcriptPath,
|
||||
wordCount,
|
||||
durationSeconds: maxEnd / 1000,
|
||||
speechOnsetSeconds,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -55,6 +55,59 @@ export const captionRules: Array<(ctx: LintContext) => HyperframeLintFinding[]>
|
||||
return findings;
|
||||
},
|
||||
|
||||
// caption_transcript_not_inline
|
||||
({ scripts, styles, options }) => {
|
||||
const findings: HyperframeLintFinding[] = [];
|
||||
// Only check files that look like caption compositions
|
||||
const isCaptionFile =
|
||||
(options.filePath && /caption/i.test(options.filePath)) ||
|
||||
styles.some((s) => /\.caption[-_]?(?:group|word)/i.test(s.content));
|
||||
if (!isCaptionFile) return findings;
|
||||
|
||||
const allScript = scripts.map((s) => s.content).join("\n");
|
||||
const hasInlineTranscript = /(?:const|let|var)\s+(?:TRANSCRIPT|script)\s*=\s*\[/.test(
|
||||
allScript,
|
||||
);
|
||||
const hasFetchTranscript = /fetch\s*\(\s*["'][^"']*transcript/i.test(allScript);
|
||||
|
||||
if (!hasInlineTranscript && hasFetchTranscript) {
|
||||
findings.push({
|
||||
code: "caption_transcript_not_inline",
|
||||
severity: "warning",
|
||||
message:
|
||||
"Captions composition loads transcript via fetch(). The studio caption editor " +
|
||||
"requires an inline `var TRANSCRIPT = [...]` array to detect and edit captions.",
|
||||
fixHint:
|
||||
'Embed the transcript as `var TRANSCRIPT = [{ "text": "...", "start": 0, "end": 1 }, ...]` ' +
|
||||
"with JSON-quoted property keys. See the captions skill for details.",
|
||||
});
|
||||
}
|
||||
|
||||
if (hasInlineTranscript) {
|
||||
// Verify the inline transcript can be parsed
|
||||
const varPattern = /(?:const|let|var)\s+(?:TRANSCRIPT|script)\s*=\s*(\[[\s\S]*?\]);/;
|
||||
const match = allScript.match(varPattern);
|
||||
if (match?.[1]) {
|
||||
try {
|
||||
JSON.parse(match[1]);
|
||||
} catch {
|
||||
findings.push({
|
||||
code: "caption_transcript_parse_error",
|
||||
severity: "warning",
|
||||
message:
|
||||
"Inline TRANSCRIPT array is not valid JSON. The studio caption editor may fail " +
|
||||
"to parse it. Common cause: unquoted property keys with apostrophes in text.",
|
||||
fixHint:
|
||||
'Use JSON-quoted keys: { "text": "don\'t", "start": 0, "end": 1 } instead of ' +
|
||||
'{ text: "don\'t", start: 0, end: 1 }.',
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return findings;
|
||||
},
|
||||
|
||||
// caption_container_relative_position
|
||||
({ styles }) => {
|
||||
const findings: HyperframeLintFinding[] = [];
|
||||
|
||||
@@ -6,6 +6,19 @@ trigger: Use this skill whenever a task involves syncing text to audio timing. T
|
||||
|
||||
# Captions
|
||||
|
||||
## Language Rule (Non-Negotiable)
|
||||
|
||||
**Never use `.en` models unless the user explicitly states the audio is English.** `.en` models (small.en, medium.en) TRANSLATE non-English audio into English instead of transcribing it. This silently destroys the original language.
|
||||
|
||||
When transcribing:
|
||||
1. If the user says the language → use `--model small --language <code>` (no `.en` suffix)
|
||||
2. If the user says it's English → use `--model small.en`
|
||||
3. If the language is unknown → use `--model small` (no `.en`, no `--language`) — whisper auto-detects
|
||||
|
||||
**Default model is `small` (not `small.en`).** Only add `.en` when explicitly told the audio is English.
|
||||
|
||||
---
|
||||
|
||||
Analyze the spoken content to determine caption style. If the user specifies a style, use that. Otherwise, detect tone from the transcript.
|
||||
|
||||
## Transcript Source
|
||||
@@ -25,7 +38,7 @@ This is the only format the captions composition consumes. Use it directly:
|
||||
const words = JSON.parse(transcriptJson); // [{ text, start, end }]
|
||||
```
|
||||
|
||||
For transcription commands, whisper model selection, external APIs (OpenAI, Groq), and supported input formats, see [transcript-guide.md](./transcript-guide.md).
|
||||
For transcription commands, whisper model selection, external APIs (OpenAI, Groq), and supported input formats, see [transcript-guide.md](./transcript-guide.md). **After every transcription, read the transcript and run the quality check** — bad transcripts (music tokens, garbled words) must be retried with a larger model before proceeding.
|
||||
|
||||
## Style Detection (Default — When No Style Is Specified)
|
||||
|
||||
@@ -86,13 +99,42 @@ For each detected word, specify:
|
||||
|
||||
## Script-to-Style Mapping
|
||||
|
||||
| Script tone | Font mood | Animation | Color | Size |
|
||||
| -------------------- | ------------------------------------- | --------------------------------------- | -------------------------------------------- | -------------------- |
|
||||
| Hype/launch | Heavy condensed, 800-900 weight | Scale-pop, back.out(1.7), fast 0.1-0.2s | Bright accent on dark (cyan, yellow, lime) | Large 72-96px |
|
||||
| Corporate/pitch | Clean sans-serif, 600-700 weight | Fade + slide-up, power3.out, 0.3s | White/neutral on dark, single muted accent | Medium 56-72px |
|
||||
| Tutorial/educational | Mono or clean sans, 500-600 weight | Typewriter or gentle fade, 0.4-0.5s | High contrast, minimal color | Medium 48-64px |
|
||||
| Storytelling/brand | Serif or elegant sans, 400-500 weight | Slow fade, power2.out, 0.5-0.6s | Warm muted tones, low opacity (0.85-0.9) | Smaller 44-56px |
|
||||
| Social/casual | Rounded sans, 700-800 weight | Bounce, elastic.out, word-by-word | Playful colors, colored backgrounds on pills | Medium-large 56-80px |
|
||||
Read the transcript. Detect the energy. The tone determines everything — typography, color, animation techniques. Use the table below to select your full animation stack.
|
||||
|
||||
| Detected energy | Font mood | Color | Entrance | Highlight | Exit |
|
||||
| ------------------------------------ | ------------------------------ | --------------------------- | ---------------------------- | ------------------------ | ------------------- |
|
||||
| High (hype, launch, music, anthem) | Heavy condensed, 800-900 | Bright accent on dark | Slam heroes + elastic others | Karaoke with accent glow | Scatter or drop |
|
||||
| Medium-high (social, casual, upbeat) | Rounded sans, 700-800 | Playful, colored pills | Elastic springs + staggered | Karaoke with color pop | Scatter or collapse |
|
||||
| Medium (corporate, pitch, explainer) | Clean sans, 600-700 | White on dark, muted accent | Clip-path reveal | Karaoke (subtle) | Fade + slide |
|
||||
| Medium-low (tutorial, educational) | Mono or clean sans, 500-600 | High contrast, minimal | Staggered entrance | Karaoke (minimal scale) | Fade |
|
||||
| Low (storytelling, cinematic, brand) | Serif or elegant sans, 400-500 | Warm muted tones | 3D rotation | Karaoke (warm tones) | Collapse |
|
||||
|
||||
**How to detect energy from the transcript:**
|
||||
|
||||
- High energy: short sentences, exclamations, repetition ("up, up, up"), emotional vocabulary ("dream", "shine", "believe", "fire"), song lyrics, fast delivery (many words per second)
|
||||
- Medium energy: declarative statements, product descriptions, mixed sentence length, moderate pacing
|
||||
- Low energy: long flowing sentences, reflective/introspective language, slow pacing (few words per second), narrative arcs
|
||||
|
||||
When in doubt, **bias toward higher energy**. Boring captions are worse than slightly over-animated ones.
|
||||
|
||||
## Animation Design (Mandatory)
|
||||
|
||||
Before writing any animation code, read [dynamic-techniques.md](./dynamic-techniques.md) for the implementation patterns referenced in the table above.
|
||||
|
||||
**Minimum requirements — every caption composition must have:**
|
||||
|
||||
- At least **2 distinct highlight techniques** — cycle them across groups (e.g., odd groups get elastic pop, even groups get clip-path wipe)
|
||||
- At least **1 kinetic exit** (scatter, collapse, or drop) — fade-out alone is not acceptable for medium energy or above
|
||||
- **Karaoke highlight** on every composition — all words visible but muted, each lights up when spoken. This is the baseline, not optional.
|
||||
- **Emphasis words get special treatment** — words flagged by per-word styling (emotional keywords, ALL CAPS, brand names) must use a different animation than surrounding words (slam, scale-pop with overshoot, or 3D flip)
|
||||
|
||||
**Technique cycling:** never use the same entrance on more than 3 consecutive groups. Rotate techniques using the group index to create variety. Higher energy content should cycle through more techniques.
|
||||
|
||||
**Energy scaling:** the detected energy level controls animation intensity:
|
||||
|
||||
- High: large overshoot (back.out(2.5)), fast timing (0.1-0.2s), 3+ techniques per composition, scatter/drop exits
|
||||
- Medium: moderate motion (back.out(1.4)), standard timing (0.2-0.4s), 2 techniques, clip-path + fade exits
|
||||
- Low: gentle reveals (power2.out), slow timing (0.4-0.6s), 1-2 techniques, collapse/fade exits
|
||||
|
||||
## Word Grouping by Tone
|
||||
|
||||
@@ -180,11 +222,14 @@ tl.seek(0);
|
||||
|
||||
Place this **before** `window.__timelines[id] = tl` so it runs at composition init.
|
||||
|
||||
## References
|
||||
## Studio Caption Editor Compatibility
|
||||
|
||||
For dynamic animation techniques (karaoke, clip-path reveals, slam words, scatter exits, elastic entrances, 3D rotation, audio-reactive captions, pretext-based positioning and grouping), see [dynamic-techniques.md](./dynamic-techniques.md).
|
||||
The HyperFrames Studio can edit captions in real time, but only if the composition follows these rules:
|
||||
|
||||
For transcription commands, whisper models, external APIs, and troubleshooting, see [transcript-guide.md](./transcript-guide.md).
|
||||
- **Inline the transcript as `var TRANSCRIPT = [...]`** — the studio's parser extracts the transcript by matching this variable name in the composition source. Using `fetch()` to load transcript data at runtime will NOT be detected.
|
||||
- **Use JSON-quoted property keys** — write `{ "text": "hello", "start": 0, "end": 1 }` not `{ text: "hello", start: 0, end: 1 }`. The parser's fallback normalization for unquoted keys breaks on apostrophes in words like `didn't`.
|
||||
- **Use `.caption-group` and `.caption-word` CSS classes** — the studio detects caption elements by these class names.
|
||||
- **Audio data can be inline or fetched** — only the transcript must be inline. Audio data loaded via `fetch("audio-data.json")` or embedded as `var AUDIO = {...}` both work.
|
||||
|
||||
## Constraints
|
||||
|
||||
@@ -192,4 +237,6 @@ For transcription commands, whisper models, external APIs, and troubleshooting,
|
||||
- **Sync to transcript timestamps.** Words appear when spoken.
|
||||
- **One group visible at a time.** No overlapping caption groups.
|
||||
- **Every caption group must have a hard `tl.set` kill at `group.end`.** Exit animations alone are not sufficient.
|
||||
- **Never `overflow: hidden` on caption containers or groups.** Glow, shadow, and scale effects paint outside the box — clipping them creates hard visual cutoffs. Always use `overflow: visible`.
|
||||
- **Music requires audio-reactive captions.** If the source audio is music (any genre, any energy level), extract audio data with `extract-audio-data.py` and use it to modulate group entrance intensity (scale, glow) in the group loop. No special wiring needed — see [dynamic-techniques.md](./dynamic-techniques.md). This is not optional.
|
||||
- **Check project root** for font files before defaulting to Google Fonts.
|
||||
|
||||
@@ -1,327 +1,86 @@
|
||||
# Dynamic Caption Techniques
|
||||
|
||||
The default caption pattern — fade group in, hold, fade out — works but looks like subtitles. These techniques make captions feel designed and intentional. Mix them based on the content's energy. Every technique below is deterministic and works with HyperFrames' frame-by-frame rendering.
|
||||
You are here because SKILL.md told you to read this file before writing animation code. Pick your technique combination from the table below based on the energy level you detected from the transcript, then implement using standard GSAP patterns.
|
||||
|
||||
## Per-Word Staggered Entrances
|
||||
## Technique Selection by Energy
|
||||
|
||||
Each word in a group enters individually with staggered timing. The stagger creates a wave that follows the speaker's rhythm.
|
||||
| Energy level | Highlight | Exit | Cycle pattern |
|
||||
| ------------ | ------------------------------------- | ------------------- | ----------------------------------------- |
|
||||
| High | Karaoke with accent glow + scale pop | Scatter or drop | Alternate highlight styles every 2 groups |
|
||||
| Medium-high | Karaoke with color pop | Scatter or collapse | Alternate every 3 groups |
|
||||
| Medium | Karaoke (subtle, white only) | Fade + slide | Alternate every 3 groups |
|
||||
| Medium-low | Karaoke (minimal scale change) | Fade | Single style, vary ease per group |
|
||||
| Low | Karaoke (warm tones, slow transition) | Collapse | Alternate every 4 groups |
|
||||
|
||||
**All energy levels use karaoke highlight as the baseline.** The difference is intensity — high energy gets accent color + glow + 15% scale pop on active words, low energy gets a gentle white shift with 3% scale.
|
||||
|
||||
**Emphasis words always break the pattern.** When a word is flagged as emphasis (emotional keyword, ALL CAPS, brand name), give it a stronger animation than surrounding words (larger scale, accent color, overshoot ease). This creates contrast.
|
||||
|
||||
## Audio-Reactive Captions (Mandatory for Music)
|
||||
|
||||
**If the source audio is music (vocals over instrumentation, beats, any musical content), you MUST extract audio data and add audio-reactive animations.** This is not optional — music without audio reactivity looks disconnected. Even low-energy ballads get subtle bass pulse and treble glow.
|
||||
|
||||
No special wiring is needed. The group loop already iterates over every caption group to build entrance, karaoke, and exit tweens. At that point, read the audio data for each group's time range and use it to modulate the group's animation intensity with regular GSAP tweens.
|
||||
|
||||
```js
|
||||
// Words enter one by one, timed to their speech timestamps
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.set(wordEl, { opacity: 0, y: 30, scale: 0.85 }, group.start);
|
||||
// Load audio data inline (same pattern as TRANSCRIPT)
|
||||
var AUDIO = JSON.parse(audioDataJson); // { fps, totalFrames, frames: [{ bands: [...] }] }
|
||||
|
||||
GROUPS.forEach(function (group, gi) {
|
||||
var groupEl = document.getElementById("cg-" + gi);
|
||||
if (!groupEl) return;
|
||||
|
||||
// Read peak energy for this group's time range
|
||||
var startFrame = Math.floor(group.start * AUDIO.fps);
|
||||
var endFrame = Math.min(Math.floor(group.end * AUDIO.fps), AUDIO.totalFrames - 1);
|
||||
var peakBass = 0;
|
||||
var peakTreble = 0;
|
||||
for (var f = startFrame; f <= endFrame; f++) {
|
||||
var frame = AUDIO.frames[f];
|
||||
if (!frame) continue;
|
||||
peakBass = Math.max(peakBass, frame.bands[0] || 0, frame.bands[1] || 0);
|
||||
peakTreble = Math.max(peakTreble, frame.bands[6] || 0, frame.bands[7] || 0);
|
||||
}
|
||||
|
||||
// Modulate entrance — louder groups enter bigger and glowier
|
||||
tl.to(
|
||||
wordEl,
|
||||
groupEl,
|
||||
{
|
||||
opacity: 1,
|
||||
y: 0,
|
||||
scale: 1,
|
||||
duration: 0.18,
|
||||
ease: "back.out(1.7)",
|
||||
},
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
Vary the entrance per word role. Content words (nouns, verbs) get scale + y. Function words (the, a, and) get opacity only — they shouldn't compete for attention.
|
||||
|
||||
## Karaoke Highlight
|
||||
|
||||
All words in the group are visible from the start but muted. Each word transitions to full brightness as it's spoken. This gives the viewer reading context while directing attention to the current word.
|
||||
|
||||
```js
|
||||
// Show all words muted at group start
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.set(wordEl, { opacity: 0.3, scale: 0.95, color: "rgba(255,255,255,0.4)" }, group.start);
|
||||
// Light up when spoken
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
opacity: 1,
|
||||
scale: 1.05,
|
||||
color: "#ffffff",
|
||||
duration: 0.1,
|
||||
scale: 1 + peakBass * 0.06,
|
||||
textShadow:
|
||||
"0 0 " + Math.round(peakTreble * 12) + "px rgba(255,255,255," + peakTreble * 0.4 + ")",
|
||||
duration: 0.3,
|
||||
ease: "power2.out",
|
||||
},
|
||||
word.start,
|
||||
);
|
||||
// Settle after speaking
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
scale: 1,
|
||||
color: "rgba(255,255,255,0.85)",
|
||||
duration: 0.2,
|
||||
ease: "power1.out",
|
||||
},
|
||||
word.end,
|
||||
group.start,
|
||||
);
|
||||
|
||||
// Reset at exit so audio-driven values don't persist
|
||||
tl.set(groupEl, { scale: 1, textShadow: "none" }, group.end - 0.15);
|
||||
});
|
||||
```
|
||||
|
||||
For high-energy content, add a color accent to the active word (`color: accentColor`) and a subtle glow (`textShadow: "0 0 20px " + accentColor`).
|
||||
This shapes the animation at build time, not playback time — no per-frame callbacks, no `tl.call()` loops, no async fetch timing issues. Loud groups come in with more weight and glow; quiet groups come in soft. The audio data modulates _how much_, the content determines _what_.
|
||||
|
||||
## Clip-Path Reveals
|
||||
Keep audio reactivity subtle — 3-6% scale variation and soft glow. Heavy pulsing makes text unreadable.
|
||||
|
||||
Words or groups reveal through an animated clip-path rather than fading. This creates a physical, tactile feeling — like text being uncovered.
|
||||
To generate the audio data file:
|
||||
|
||||
```js
|
||||
// Horizontal wipe: text sweeps in from left
|
||||
tl.fromTo(
|
||||
groupEl,
|
||||
{ clipPath: "inset(0 100% 0 0)" },
|
||||
{ clipPath: "inset(0 0% 0 0)", duration: 0.4, ease: "power3.out" },
|
||||
group.start,
|
||||
);
|
||||
|
||||
// Per-word vertical reveal: each word drops in from behind a mask
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ clipPath: "inset(100% 0 0 0)", y: -10 },
|
||||
{ clipPath: "inset(0% 0 0 0)", y: 0, duration: 0.2, ease: "power2.out" },
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
|
||||
// Circle reveal: text appears through an expanding circle
|
||||
tl.fromTo(
|
||||
groupEl,
|
||||
{ clipPath: "circle(0% at 50% 50%)" },
|
||||
{ clipPath: "circle(100% at 50% 50%)", duration: 0.35, ease: "expo.out" },
|
||||
group.start,
|
||||
);
|
||||
```
|
||||
|
||||
## Slam / Impact Words
|
||||
|
||||
Hero words slam onto the screen — they arrive fast, overshoot, and settle with weight. Reserve this for emphasis words (1-2 per group max). Over-using it kills the impact.
|
||||
|
||||
```js
|
||||
var isHeroWord = /^(LAUNCH|FREE|NOW|NEW|HUGE|INSANE)$/i.test(word.text);
|
||||
if (isHeroWord) {
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ scale: 2.5, opacity: 0, rotation: -8 },
|
||||
{ scale: 1, opacity: 1, rotation: 0, duration: 0.25, ease: "back.out(2.5)" },
|
||||
word.start,
|
||||
);
|
||||
// Micro-shake on impact
|
||||
tl.to(wordEl, { x: 4, duration: 0.03 }, word.start + 0.25);
|
||||
tl.to(wordEl, { x: -3, duration: 0.03 }, word.start + 0.28);
|
||||
tl.to(wordEl, { x: 0, duration: 0.04, ease: "power2.out" }, word.start + 0.31);
|
||||
} else {
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ opacity: 0, y: 20 },
|
||||
{ opacity: 1, y: 0, duration: 0.15, ease: "power2.out" },
|
||||
word.start,
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Absolute-Positioned Word Layout with Pretext
|
||||
|
||||
Position every word with `position: absolute` using pretext-measured widths. This unlocks animation paths that CSS inline flow can't do — words can fly in from any direction to their reading position.
|
||||
|
||||
```js
|
||||
var FONT = "900 72px Outfit";
|
||||
var GAP = 14; // px between words
|
||||
var containerWidth = 1600;
|
||||
|
||||
// Measure each word and compute its x position
|
||||
var xCursor = 0;
|
||||
var wordPositions = [];
|
||||
group.words.forEach(function (word) {
|
||||
var prepared = window.__hyperframes.pretext.prepare(word.text.toUpperCase(), FONT);
|
||||
var measured = window.__hyperframes.pretext.layout(prepared, 9999, 72 * 1.2);
|
||||
var w = measured.height / 1.2;
|
||||
wordPositions.push({ x: xCursor, width: w });
|
||||
xCursor += w + GAP;
|
||||
});
|
||||
|
||||
// Center the whole group
|
||||
var totalWidth = xCursor - GAP;
|
||||
var offsetX = (containerWidth - totalWidth) / 2;
|
||||
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
var finalX = wordPositions[wi].x + offsetX;
|
||||
wordEl.style.position = "absolute";
|
||||
wordEl.style.left = finalX + "px";
|
||||
|
||||
// Scatter entrance: each word arrives from a unique direction
|
||||
var angle = (wi / group.words.length) * Math.PI * 2;
|
||||
var radius = 300;
|
||||
var startX = finalX + Math.cos(angle) * radius;
|
||||
var startY = Math.sin(angle) * radius;
|
||||
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ x: startX - finalX, y: startY, opacity: 0, scale: 0.5 },
|
||||
{ x: 0, y: 0, opacity: 1, scale: 1, duration: 0.35, ease: "back.out(1.4)" },
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Elastic / Spring Entrances
|
||||
|
||||
Words arrive with physics — they overshoot their target and oscillate before settling. Different spring constants per word create an organic, staggered feeling.
|
||||
|
||||
```js
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
// Vary elasticity by word position — earlier words bouncier
|
||||
var elasticity = 0.3 + wi * 0.05;
|
||||
var amplitude = 1.2 - wi * 0.1;
|
||||
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ y: 60, opacity: 0, scaleY: 1.3, scaleX: 0.85 },
|
||||
{
|
||||
y: 0,
|
||||
opacity: 1,
|
||||
scaleY: 1,
|
||||
scaleX: 1,
|
||||
duration: 0.5,
|
||||
ease: "elastic.out(" + amplitude + ", " + elasticity + ")",
|
||||
},
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Rotation & 3D Perspective
|
||||
|
||||
Words rotate into view on the X or Y axis, creating a sense of depth. Requires `transformPerspective` on the parent for 3D effect.
|
||||
|
||||
```js
|
||||
// Set perspective on the group container
|
||||
gsap.set(groupEl, { transformPerspective: 800 });
|
||||
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
// Alternate rotation direction per word
|
||||
var rotDir = wi % 2 === 0 ? 90 : -90;
|
||||
tl.fromTo(
|
||||
wordEl,
|
||||
{ rotationX: rotDir, opacity: 0, transformOrigin: "50% 100%" },
|
||||
{ rotationX: 0, opacity: 1, duration: 0.3, ease: "power3.out" },
|
||||
word.start,
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Kinetic Exit Patterns
|
||||
|
||||
Exits are as important as entrances. Don't always fade out — give words somewhere to go.
|
||||
|
||||
```js
|
||||
// Scatter exit: words fly apart when the group ends
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
var angle = (wi / group.words.length) * Math.PI * 2;
|
||||
var exitX = Math.cos(angle) * 200;
|
||||
var exitY = Math.sin(angle) * 150;
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
x: exitX,
|
||||
y: exitY,
|
||||
opacity: 0,
|
||||
scale: 0.6,
|
||||
rotation: wi % 2 ? 15 : -15,
|
||||
duration: 0.2,
|
||||
ease: "power3.in",
|
||||
},
|
||||
group.end - 0.2,
|
||||
);
|
||||
});
|
||||
// Hard kill still required
|
||||
tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end);
|
||||
|
||||
// Collapse exit: words squeeze together then vanish
|
||||
tl.to(
|
||||
groupEl.querySelectorAll("span"),
|
||||
{
|
||||
letterSpacing: "-0.15em",
|
||||
scaleX: 0.7,
|
||||
opacity: 0,
|
||||
duration: 0.15,
|
||||
ease: "power2.in",
|
||||
stagger: { each: 0.02, from: "edges" },
|
||||
},
|
||||
group.end - 0.2,
|
||||
);
|
||||
|
||||
// Drop exit: words fall with gravity
|
||||
group.words.forEach(function (word, wi) {
|
||||
var wordEl = document.getElementById("w-" + gi + "-" + wi);
|
||||
tl.to(
|
||||
wordEl,
|
||||
{
|
||||
y: 300,
|
||||
rotation: 10 + wi * 5,
|
||||
opacity: 0,
|
||||
duration: 0.3,
|
||||
ease: "power2.in",
|
||||
},
|
||||
group.end - 0.3 + wi * 0.03,
|
||||
);
|
||||
});
|
||||
```bash
|
||||
python3 skills/gsap-effects/scripts/extract-audio-data.py audio.mp3 --fps 30 --bands 8 -o audio-data.json
|
||||
```
|
||||
|
||||
## Combining Techniques
|
||||
|
||||
The best dynamic captions layer 2-3 techniques together. A few combinations that work:
|
||||
Don't use the same highlight animation on every group — cycle through styles using the group index. Don't combine multiple competing animations on the same word at the same timestamp. Vary techniques across groups to match the content's pace changes.
|
||||
|
||||
| Combination | Energy | Best for |
|
||||
| ---------------------------------------- | ----------- | ------------------------------- |
|
||||
| Karaoke highlight + audio reactivity | Medium-high | Music videos, lyric videos |
|
||||
| Staggered entrance + scatter exit | High | Hype content, trailers |
|
||||
| Clip-path reveal + fade exit | Medium | Corporate, storytelling |
|
||||
| Slam heroes + elastic others + drop exit | Very high | Product launches, announcements |
|
||||
| 3D rotation entrance + collapse exit | Medium-high | Tech, modern brands |
|
||||
## Available Tools
|
||||
|
||||
Don't combine slam entrances with elastic entrances on the same group — pick one motion personality per group. You can vary techniques across groups to match the content's pace changes.
|
||||
These tools are available in the HyperFrames runtime. Use them when they solve a real problem — not every composition needs all of them.
|
||||
|
||||
## Width-Aware Grouping with Pretext
|
||||
|
||||
Instead of grouping by word count alone, use pretext to group by visual width. This prevents some groups from filling the frame while others use 30%.
|
||||
|
||||
```js
|
||||
var FONT = "900 72px Outfit";
|
||||
var MAX_WIDTH = 1500; // slightly under container to leave padding
|
||||
|
||||
var groups = [];
|
||||
var currentGroup = { words: [], text: "" };
|
||||
|
||||
words.forEach(function (word) {
|
||||
var testText = (currentGroup.text + " " + word.text).trim().toUpperCase();
|
||||
var result = window.__hyperframes.fitTextFontSize(testText, {
|
||||
fontFamily: "Outfit",
|
||||
fontWeight: 900,
|
||||
maxWidth: MAX_WIDTH,
|
||||
baseFontSize: 72,
|
||||
minFontSize: 72,
|
||||
step: 2,
|
||||
});
|
||||
|
||||
if (!result.fits && currentGroup.words.length > 0) {
|
||||
// Adding this word would overflow — start new group
|
||||
groups.push(currentGroup);
|
||||
currentGroup = { words: [word], text: word.text };
|
||||
} else {
|
||||
currentGroup.words.push(word);
|
||||
currentGroup.text = testText;
|
||||
}
|
||||
});
|
||||
if (currentGroup.words.length > 0) groups.push(currentGroup);
|
||||
```
|
||||
|
||||
This replaces the fixed "3-5 words per group" heuristic with pixel-accurate measurement. "I" and "EXTRAORDINARY" take very different widths — pretext accounts for that.
|
||||
| Tool | What it does | Access | When it's useful |
|
||||
| ------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| **pretext** | Pure-arithmetic text measurement without DOM reflow. 0.0002ms per call. | `window.__hyperframes.pretext.prepare(text, font)` / `.layout(prepared, maxWidth, lineHeight)` | Per-frame text reflow, shrinkwrap containers, computing layout before render |
|
||||
| **fitTextFontSize** | Finds the largest font size that fits text on one line. Built on pretext. | `window.__hyperframes.fitTextFontSize(text, { maxWidth, fontFamily, fontWeight })` | Overflow prevention for long phrases, portrait mode, large base sizes |
|
||||
| **audio data** | Pre-extracted per-frame RMS energy and frequency bands. | Extract with `extract-audio-data.py`, load inline or via `fetch("audio-data.json")` | Audio-reactive visuals — modulate intensity based on the music |
|
||||
| **GSAP** | Animation timeline with tweens and callbacks. | `gsap.to()`, `gsap.set()`, `tl.to()`, `tl.set()` | All caption animation |
|
||||
|
||||
@@ -40,16 +40,74 @@ The default model (`small.en`) balances accuracy and speed. For better results,
|
||||
|
||||
| Model | Size | Speed | Accuracy | When to use |
|
||||
| ----------- | ------ | -------- | --------- | ------------------------------------- |
|
||||
| `tiny.en` | 75 MB | Fastest | Low | Quick previews, testing pipeline |
|
||||
| `base.en` | 142 MB | Fast | Fair | Short clips, clear audio |
|
||||
| `small.en` | 466 MB | Moderate | Good | **Default** — good for most content |
|
||||
| `medium.en` | 1.5 GB | Slow | Very good | Important content, noisy audio, music |
|
||||
| `large-v3` | 3.1 GB | Slowest | Best | Multilingual, production captions |
|
||||
| `tiny` | 75 MB | Fastest | Low | Quick previews, testing pipeline |
|
||||
| `base` | 142 MB | Fast | Fair | Short clips, clear audio |
|
||||
| `small` | 466 MB | Moderate | Good | **Default** — good for most content |
|
||||
| `medium` | 1.5 GB | Slow | Very good | Important content, noisy audio, music |
|
||||
| `large-v3` | 3.1 GB | Slowest | Best | Production quality |
|
||||
|
||||
`.en` models are English-only and more accurate for English. Drop the `.en` suffix for multilingual (e.g., `medium` instead of `medium.en`).
|
||||
**Only add `.en` suffix when the user explicitly says the audio is English.** `.en` models are slightly more accurate for English but will TRANSLATE non-English audio instead of transcribing it.
|
||||
|
||||
**Critical: `.en` models translate non-English audio into English** — they don't transcribe it. If the audio might not be English, always use a model without the `.en` suffix and pass `--language` to specify the source language. If you're unsure of the language, use `small` (not `small.en`) without `--language` — whisper will auto-detect.
|
||||
|
||||
```bash
|
||||
# Spanish audio
|
||||
npx hyperframes transcribe audio.mp3 --model small --language es
|
||||
|
||||
# Unknown language — let whisper auto-detect
|
||||
npx hyperframes transcribe audio.mp3 --model small
|
||||
```
|
||||
|
||||
**Music and vocals over instrumentation**: `small.en` will misidentify lyrics — use `medium.en` as the minimum, or import lyrics manually. Even `medium.en` struggles with heavily produced tracks; for music videos, providing known lyrics as an SRT/VTT and importing with `hyperframes transcribe lyrics.srt` will always beat automated transcription.
|
||||
|
||||
## Transcript Quality Check (Mandatory)
|
||||
|
||||
After every transcription, **read the transcript and check for quality issues before proceeding.** Bad transcripts produce nonsensical captions. Never skip this step.
|
||||
|
||||
### What to look for
|
||||
|
||||
| Signal | Example | Cause |
|
||||
| ---------------------------- | -------------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| Music note tokens (`♪`, `�`) | `{ "text": "♪" }` or `{ "text": "�" }` | Whisper detected music, not speech |
|
||||
| Garbled / nonsense words | "Do a chin", "Get so gay", "huh" | Model misheard lyrics or background noise |
|
||||
| Long gaps with no words | 20+ seconds of only `♪` tokens | Instrumental section — expected, but high ratio means speech is being missed |
|
||||
| Repeated filler | Many "huh", "uh", "oh" entries | Model is hallucinating on music |
|
||||
| Very short word spans | Words with `end - start < 0.05` | Unreliable timestamp alignment |
|
||||
|
||||
### Automatic retry rules
|
||||
|
||||
**If more than 20% of entries are `♪`/`�` tokens, or the transcript contains obvious nonsense words, the transcription failed.** Do not proceed with the bad transcript. Instead:
|
||||
|
||||
1. **Retry with `medium.en`** if the original used `small.en` or smaller:
|
||||
```bash
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
```
|
||||
2. **If `medium.en` also fails** (still >20% music tokens or garbled), tell the user the audio is too noisy for local transcription and suggest:
|
||||
- Providing lyrics manually as an SRT/VTT file
|
||||
- Using an external API (OpenAI or Groq Whisper — see below)
|
||||
3. **Always clean the transcript** before building captions — filter out `♪`/`�` tokens and entries where `text` is a single non-word character. Only real words should reach the caption composition.
|
||||
|
||||
### Cleaning a transcript
|
||||
|
||||
After transcription (even with a good model), strip non-word entries:
|
||||
|
||||
```js
|
||||
var raw = JSON.parse(transcriptJson);
|
||||
var words = raw.filter(function (w) {
|
||||
if (!w.text || w.text.trim().length === 0) return false;
|
||||
if (/^[♪�\u266a\u266b\u266c\u266d\u266e\u266f]+$/.test(w.text)) return false;
|
||||
if (/^(huh|uh|um|ah|oh)$/i.test(w.text) && w.end - w.start < 0.1) return false;
|
||||
return true;
|
||||
});
|
||||
```
|
||||
|
||||
### When to use which model (decision tree)
|
||||
|
||||
1. **Is this speech over silence/light background?** → `small.en` is fine
|
||||
2. **Is this speech over music, or music with vocals?** → Start with `medium.en`
|
||||
3. **Is this a produced music track (vocals + full instrumentation)?** → Start with `medium.en`, expect to need manual lyrics or an external API
|
||||
4. **Is this multilingual?** → Use `medium` or `large-v3` (no `.en` suffix)
|
||||
|
||||
## Using External Transcription APIs
|
||||
|
||||
For the best accuracy, use an external API and import the result:
|
||||
@@ -84,11 +142,10 @@ npx hyperframes transcribe transcript-groq.json
|
||||
## If No Transcript Exists
|
||||
|
||||
1. Check the project root for `transcript.json`, `.srt`, or `.vtt` files
|
||||
2. If none found, ask the user to provide one or run:
|
||||
2. If none found, run transcription — pick the starting model based on the content type:
|
||||
- Speech/voiceover → `small.en`
|
||||
- Music with vocals → `medium.en`
|
||||
```bash
|
||||
npx hyperframes transcribe <audio-or-video-file>
|
||||
```
|
||||
3. If transcription quality is poor (words at wrong times, gibberish), suggest upgrading the model:
|
||||
```bash
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
npx hyperframes transcribe <audio-or-video-file> --model medium.en
|
||||
```
|
||||
3. **Read the transcript and run the quality check** (see above). If it fails, retry with a larger model or suggest manual lyrics.
|
||||
|
||||
Reference in New Issue
Block a user