diff --git a/CLAUDE.md b/CLAUDE.md index d77f482b5..d71f20cd3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,7 +9,7 @@ This repo ships skills that are installed globally via `npx hyperframes skills` | Skill | Invoke with | When to use | | ------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **hyperframes-compose** | `/hyperframes-compose` | Creating ANY HTML composition — videos, animations, title cards, overlays. Contains required HTML structure, `class="clip"` rules, GSAP timeline patterns, and rendering constraints. | -| **hyperframes-captions** | `/hyperframes-captions` | Building tone-adaptive captions from whisper transcripts — style detection, per-word styling, positioning. | +| **hyperframes-captions** | `/hyperframes-captions` | Any task involving text synced to audio: captions, subtitles, lyrics, lyric videos, karaoke. Also covers transcription strategy (whisper model selection, transcript format). | ### GSAP Skills (from [greensock/gsap-skills](https://github.com/greensock/gsap-skills)) @@ -29,9 +29,12 @@ The skills encode HyperFrames-specific patterns (e.g., required `class="clip"` o ### Rules - When creating or modifying HTML compositions → invoke `/hyperframes-compose` BEFORE writing any code -- When adding captions → invoke `/hyperframes-captions` BEFORE writing any code +- When adding captions, subtitles, lyrics, or any text synced to audio → invoke `/hyperframes-captions` BEFORE writing any code +- When transcribing audio or choosing a whisper model → invoke `/hyperframes-captions` BEFORE running any transcription tool +- When creating a video from audio (music video, lyric video, audio visualizer with text) → invoke BOTH `/hyperframes-compose` AND `/hyperframes-captions` - When writing GSAP animations → invoke `/gsap-core` and `/gsap-timeline` BEFORE writing any code - When optimizing animation performance → invoke `/gsap-performance` BEFORE making changes +- After creating or editing any `.html` composition → run `npx hyperframes lint` and fix all errors before considering the task complete ### Installing skills @@ -68,3 +71,48 @@ pnpm test # Run tests - **Frame Adapters** bridge animation runtimes (GSAP, Lottie, CSS) to the capture engine - **Producer** orchestrates capture → encode → audio mix into final MP4 - **BeginFrame rendering** uses `HeadlessExperimental.beginFrame` for deterministic frame capture + +## Transcription + +HyperFrames uses word-level timestamps for captions. The `hyperframes transcribe` command handles both transcription and format conversion. + +### Quick reference + +```bash +# Transcribe audio/video (local whisper.cpp, no API key) +npx hyperframes transcribe audio.mp3 +npx hyperframes transcribe video.mp4 --model medium.en --language en + +# Import existing transcript from another tool +npx hyperframes transcribe subtitles.srt +npx hyperframes transcribe subtitles.vtt +npx hyperframes transcribe openai-response.json +``` + +### Whisper models + +Default is `small.en`. Upgrade for better accuracy: + +| Model | Size | Use case | +| ----------- | ------ | -------------------------------- | +| `tiny.en` | 75 MB | Quick testing | +| `base.en` | 142 MB | Short clips, clear audio | +| `small.en` | 466 MB | **Default** — most content | +| `medium.en` | 1.5 GB | Important content, noisy audio | +| `large-v3` | 3.1 GB | Multilingual, production quality | + +Use `.en` suffix for English-only (more accurate). Drop it for multilingual content. + +### Supported transcript formats + +The CLI auto-detects and normalizes: whisper.cpp JSON, OpenAI Whisper API JSON, SRT, VTT, and pre-normalized `[{text, start, end}]` arrays. + +### Improving transcription quality + +If captions are inaccurate (wrong words, bad timing): + +1. **Upgrade the model**: `--model medium.en` or `--model large-v3` +2. **Set language**: `--language en` to filter non-target speech +3. **Use an external API**: Transcribe via OpenAI or Groq Whisper API, then import the JSON with `hyperframes transcribe response.json` + +See the `/hyperframes-captions` skill for full details on model selection and API usage. diff --git a/docs/packages/cli.mdx b/docs/packages/cli.mdx index 9ab8ac448..300b1542e 100644 --- a/docs/packages/cli.mdx +++ b/docs/packages/cli.mdx @@ -156,6 +156,8 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_ | `--audio, -a` | Path to an audio file (MP3, WAV, M4A) | | `--skip-skills` | Skip AI coding skills installation | | `--skip-transcribe` | Skip automatic whisper transcription | + | `--model` | Whisper model for transcription (e.g. `small.en`, `medium.en`, `large-v3`) | + | `--language` | Language code for transcription (e.g. `en`, `es`, `ja`). Filters non-target speech. | | `--human-friendly` | Enable interactive terminal UI with prompts | | Template | Description | @@ -185,6 +187,45 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_ | `--json` | Output as JSON | Shows each composition's ID, duration, resolution, and element count. + + ### `transcribe` + + Transcribe audio/video to word-level timestamps, or import an existing transcript: + + ```bash + # Transcribe audio/video with local whisper.cpp + npx hyperframes transcribe audio.mp3 + npx hyperframes transcribe video.mp4 --model medium.en --language en + + # Import existing transcripts from other tools + npx hyperframes transcribe subtitles.srt + npx hyperframes transcribe captions.vtt + npx hyperframes transcribe openai-response.json + ``` + + | Flag | Description | + |------|-------------| + | `--dir, -d` | Project directory (default: current directory) | + | `--model, -m` | Whisper model (default: `small.en`). Options: `tiny.en`, `base.en`, `small.en`, `medium.en`, `large-v3` | + | `--language, -l` | Language code (e.g. `en`, `es`, `ja`). Filters out non-target language speech. | + | `--json` | Output result as JSON | + + The command auto-detects the input type. Audio/video files are transcribed with whisper.cpp. Transcript files (`.json`, `.srt`, `.vtt`) are normalized and imported. + + **Supported transcript formats:** + + | Format | Source | + |--------|--------| + | whisper.cpp JSON | `hyperframes init --video`, `hyperframes transcribe` | + | OpenAI Whisper API JSON | `openai.audio.transcriptions.create()` with word timestamps | + | SRT subtitles | Video editors, YouTube, subtitle tools | + | VTT subtitles | Web players, YouTube, transcription services | + + All formats are normalized to a standard `[{text, start, end}]` word array and saved as `transcript.json`. If the project has caption HTML files, they are automatically patched with the transcript data. + + + For music or noisy audio, use `--model medium.en` for better accuracy. For the best results with production content, transcribe via the OpenAI or Groq Whisper API and import the JSON. + ### `dev` diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index 952303292..e994de17d 100644 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -26,6 +26,7 @@ const subCommands = { benchmark: () => import("./commands/benchmark.js").then((m) => m.default), browser: () => import("./commands/browser.js").then((m) => m.default), skills: () => import("./commands/install-skills.js").then((m) => m.default), + transcribe: () => import("./commands/transcribe.js").then((m) => m.default), docs: () => import("./commands/docs.js").then((m) => m.default), doctor: () => import("./commands/doctor.js").then((m) => m.default), upgrade: () => import("./commands/upgrade.js").then((m) => m.default), diff --git a/packages/cli/src/commands/init.ts b/packages/cli/src/commands/init.ts index 4c43bcfcb..38a01bfeb 100644 --- a/packages/cli/src/commands/init.ts +++ b/packages/cli/src/commands/init.ts @@ -246,60 +246,11 @@ function patchVideoSrc( } } -function patchTranscript(dir: string, transcriptPath: string): void { - // Read the whisper transcript and normalize to [{text, start, end}] - const raw = JSON.parse(readFileSync(transcriptPath, "utf-8")); - const words: { text: string; start: number; end: number }[] = []; - for (const seg of raw.transcription ?? []) { - for (const token of seg.tokens ?? []) { - const text = (token.text ?? "").trim(); - if (!text || text.startsWith("[_") || text.startsWith("[BLANK")) continue; - - // Merge punctuation with the previous word - const isPunctuation = /^[.,!?;:'")\]}>…–—-]+$/.test(text); - const lastWord = words[words.length - 1]; - if (isPunctuation && lastWord) { - lastWord.text += text; - lastWord.end = Math.round(((token.offsets?.to ?? 0) / 1000) * 1000) / 1000; - continue; - } - - words.push({ - text, - start: Math.round(((token.offsets?.from ?? 0) / 1000) * 1000) / 1000, - end: Math.round(((token.offsets?.to ?? 0) / 1000) * 1000) / 1000, - }); - } - } - +async function patchTranscript(dir: string, transcriptPath: string): Promise { + const { loadTranscript, patchCaptionHtml } = await import("../whisper/normalize.js"); + const { words } = loadTranscript(transcriptPath); if (words.length === 0) return; - - const wordsJson = JSON.stringify(words, null, 10) - .replace(/^\[/, "[") - .replace(/\n {10}/g, "\n "); - - // Find captions HTML files and replace the hardcoded script array - const htmlFiles = readdirSync(dir, { withFileTypes: true, recursive: true }) - .filter((e) => e.isFile() && e.name.endsWith(".html")) - .map((e) => join(e.parentPath ?? e.path, e.name)); - - for (const file of htmlFiles) { - let content = readFileSync(file, "utf-8"); - // Match within `; + writeFileSync(join(dir, "captions.html"), html); + + const words = [ + { text: "Hello", start: 1.0, end: 1.5 }, + { text: "world", start: 2.0, end: 2.5 }, + ]; + patchCaptionHtml(dir, words); + + const result = readFileSync(join(dir, "captions.html"), "utf-8"); + expect(result).toContain('"Hello"'); + expect(result).toContain('"world"'); + expect(result).not.toContain("const script = [];"); + }); + + it("replaces const TRANSCRIPT = [] variant", () => { + const dir = join(tmpdir(), `hf-patch-test-${Date.now()}`); + mkdirSync(dir, { recursive: true }); + dirs.push(dir); + + const html = ``; + writeFileSync(join(dir, "index.html"), html); + + patchCaptionHtml(dir, [{ text: "Hi", start: 0, end: 1 }]); + + const result = readFileSync(join(dir, "index.html"), "utf-8"); + expect(result).toContain("const TRANSCRIPT = "); + expect(result).toContain('"Hi"'); + }); + + it("does not modify HTML files without matching script patterns", () => { + const dir = join(tmpdir(), `hf-patch-test-${Date.now()}`); + mkdirSync(dir, { recursive: true }); + dirs.push(dir); + + const html = ``; + writeFileSync(join(dir, "page.html"), html); + + patchCaptionHtml(dir, [{ text: "Hi", start: 0, end: 1 }]); + + const result = readFileSync(join(dir, "page.html"), "utf-8"); + expect(result).toBe(html); + }); + + it("skips empty word arrays", () => { + const dir = join(tmpdir(), `hf-patch-test-${Date.now()}`); + mkdirSync(dir, { recursive: true }); + dirs.push(dir); + + const html = ``; + writeFileSync(join(dir, "captions.html"), html); + + patchCaptionHtml(dir, []); + + const result = readFileSync(join(dir, "captions.html"), "utf-8"); + expect(result).toBe(html); + }); +}); diff --git a/packages/cli/src/whisper/normalize.ts b/packages/cli/src/whisper/normalize.ts new file mode 100644 index 000000000..4975501a2 --- /dev/null +++ b/packages/cli/src/whisper/normalize.ts @@ -0,0 +1,266 @@ +import { readFileSync, readdirSync, writeFileSync } from "node:fs"; +import { extname, join } from "node:path"; + +export interface Word { + text: string; + start: number; + end: number; +} + +// --------------------------------------------------------------------------- +// Format detection + parsing +// --------------------------------------------------------------------------- + +export type TranscriptFormat = "whisper-cpp" | "openai" | "srt" | "vtt" | "words-json"; + +/** + * Detect the format of a transcript file from its extension and content. + */ +export function detectFormat(filePath: string): TranscriptFormat { + const ext = extname(filePath).toLowerCase(); + if (ext === ".srt") return "srt"; + if (ext === ".vtt") return "vtt"; + if (ext === ".json") return detectJsonFormat(JSON.parse(readFileSync(filePath, "utf-8"))); + throw new Error(`Unsupported transcript file extension: ${ext}. Use .json, .srt, or .vtt`); +} + +function detectJsonFormat(raw: unknown): TranscriptFormat { + if (raw && typeof raw === "object" && !Array.isArray(raw)) { + const obj = raw as Record; + if (obj.transcription && Array.isArray(obj.transcription)) return "whisper-cpp"; + if (obj.words && Array.isArray(obj.words)) return "openai"; + } + if (Array.isArray(raw) && raw[0]?.text !== undefined && raw[0]?.start !== undefined) { + return "words-json"; + } + throw new Error( + "Unrecognized JSON transcript format. Expected whisper.cpp (transcription[].tokens), " + + "OpenAI API (words[]), or normalized ([{text, start, end}]).", + ); +} + +// --------------------------------------------------------------------------- +// Parsers +// --------------------------------------------------------------------------- + +function parseWhisperCpp(data: Record): Word[] { + const words: Word[] = []; + const transcription = data.transcription as Array<{ + tokens?: Array<{ + text?: string; + offsets?: { from?: number; to?: number }; + }>; + }>; + + for (const seg of transcription ?? []) { + for (const token of seg.tokens ?? []) { + const text = (token.text ?? "").trim(); + if (!text || text.startsWith("[_") || text.startsWith("[BLANK")) continue; + + // Merge punctuation with the previous word + const isPunctuation = /^[.,!?;:'")\]}>…–—-]+$/.test(text); + const lastWord = words[words.length - 1]; + if (isPunctuation && lastWord) { + lastWord.text += text; + lastWord.end = round3((token.offsets?.to ?? 0) / 1000); + continue; + } + + words.push({ + text, + start: round3((token.offsets?.from ?? 0) / 1000), + end: round3((token.offsets?.to ?? 0) / 1000), + }); + } + } + return words; +} + +function parseOpenAI(data: Record): Word[] { + const rawWords = (data.words ?? []) as Array<{ + word?: string; + text?: string; + start?: number; + end?: number; + }>; + return rawWords + .map((w) => ({ + text: (w.word ?? w.text ?? "").trim(), + start: round3(w.start ?? 0), + end: round3(w.end ?? 0), + })) + .filter((w) => w.text.length > 0); +} + +function parseSrt(content: string): Word[] { + // SRT doesn't have word-level timestamps — parse as phrase-level entries. + // Each cue becomes one "word" entry (the full phrase). + const blocks = content.trim().split(/\n\n+/); + const words: Word[] = []; + + for (const block of blocks) { + const lines = block.trim().split("\n"); + // SRT format: index, timestamp line, text lines + const timeLine = lines.find((l) => l.includes("-->")); + if (!timeLine) continue; + + const [startStr, endStr] = timeLine.split("-->").map((s) => s.trim()); + if (!startStr || !endStr) continue; + + const text = lines + .slice(lines.indexOf(timeLine) + 1) + .join(" ") + .replace(/<[^>]+>/g, "") // strip HTML tags + .trim(); + if (!text) continue; + + words.push({ + text, + start: parseSrtTimestamp(startStr), + end: parseSrtTimestamp(endStr), + }); + } + return words; +} + +function parseVtt(content: string): Word[] { + // Strip the WEBVTT header and any metadata blocks + const body = content.replace(/^WEBVTT[^\n]*\n/, "").replace(/^[A-Z-]+:.*\n/gm, ""); + // VTT is structurally similar to SRT (without numeric indices) + const blocks = body.trim().split(/\n\n+/); + const words: Word[] = []; + + for (const block of blocks) { + const lines = block.trim().split("\n"); + const timeLine = lines.find((l) => l.includes("-->")); + if (!timeLine) continue; + + const [startStr, endStr] = timeLine.split("-->").map((s) => s.trim()); + if (!startStr || !endStr) continue; + + const text = lines + .slice(lines.indexOf(timeLine) + 1) + .join(" ") + .replace(/<[^>]+>/g, "") // strip HTML tags + .trim(); + if (!text) continue; + + words.push({ + text, + start: parseVttTimestamp(startStr), + end: parseVttTimestamp(endStr), + }); + } + return words; +} + +// --------------------------------------------------------------------------- +// Timestamp helpers +// --------------------------------------------------------------------------- + +/** Parse SRT timestamp: 00:01:23,456 → seconds */ +function parseSrtTimestamp(ts: string): number { + const m = ts.match(/(\d+):(\d+):(\d+)[,.](\d+)/); + if (!m) return 0; + return ( + parseInt(m[1]!, 10) * 3600 + + parseInt(m[2]!, 10) * 60 + + parseInt(m[3]!, 10) + + parseInt(m[4]!.padEnd(3, "0"), 10) / 1000 + ); +} + +/** Parse VTT timestamp: 00:01:23.456 or 01:23.456 → seconds */ +function parseVttTimestamp(ts: string): number { + const parts = ts.split(":"); + if (parts.length === 3) return parseSrtTimestamp(ts); + // MM:SS.mmm + if (parts.length === 2) { + const [min, secMs] = parts; + const [sec, ms] = (secMs ?? "0.0").split("."); + return ( + parseInt(min!, 10) * 60 + parseInt(sec!, 10) + parseInt((ms ?? "0").padEnd(3, "0"), 10) / 1000 + ); + } + return 0; +} + +function round3(n: number): number { + return Math.round(n * 1000) / 1000; +} + +// --------------------------------------------------------------------------- +// Public API +// --------------------------------------------------------------------------- + +/** + * Load and normalize a transcript file to a standard word array. + * + * Supports: + * - whisper.cpp JSON (--output-json-full with --dtw) + * - OpenAI Whisper API response (verbose_json with word timestamps) + * - SRT subtitle files (phrase-level, not word-level) + * - VTT subtitle files (phrase-level, not word-level) + * - Pre-normalized JSON array ([{text, start, end}]) + */ +export function loadTranscript(filePath: string): { words: Word[]; format: TranscriptFormat } { + const ext = extname(filePath).toLowerCase(); + const content = readFileSync(filePath, "utf-8"); + + if (ext === ".srt") return { words: parseSrt(content), format: "srt" }; + if (ext === ".vtt") return { words: parseVtt(content), format: "vtt" }; + + // JSON formats — parse once, detect, then extract words + const parsed = JSON.parse(content); + const format = detectJsonFormat(parsed); + + const words = + format === "whisper-cpp" + ? parseWhisperCpp(parsed) + : format === "openai" + ? parseOpenAI(parsed) + : (parsed as Word[]).map((w) => ({ + text: w.text.trim(), + start: round3(w.start), + end: round3(w.end), + })); + + return { words, format }; +} + +/** + * Patch caption HTML files in a project directory with transcript words. + * Replaces `const script = [...]` or `const TRANSCRIPT = [...]` in + +`; + const result = lintHyperframeHtml(html); + const finding = result.findings.find((f) => f.code === "caption_exit_missing_hard_kill"); + expect(finding).toBeDefined(); + expect(finding?.severity).toBe("warning"); + }); + + it("does not warn when caption exit has hard kill tl.set", () => { + const html = ` + +
+
+ +
+`; + const result = lintHyperframeHtml(html); + const finding = result.findings.find((f) => f.code === "caption_exit_missing_hard_kill"); + expect(finding).toBeUndefined(); + }); + + it("warns when caption group has nowrap without max-width", () => { + const html = ` + +
+ + +
+`; + const result = lintHyperframeHtml(html); + const finding = result.findings.find((f) => f.code === "caption_text_overflow_risk"); + expect(finding).toBeDefined(); + expect(finding?.severity).toBe("warning"); + }); + + it("does not warn when caption group has nowrap with max-width", () => { + const html = ` + +
+ + +
+`; + const result = lintHyperframeHtml(html); + const finding = result.findings.find( + (f) => f.code === "caption_text_overflow_risk" && f.severity === "warning", + ); + expect(finding).toBeUndefined(); + }); + + it("warns when caption container uses position: relative", () => { + const html = ` + +
+ + +
+`; + const result = lintHyperframeHtml(html); + const finding = result.findings.find((f) => f.code === "caption_container_relative_position"); + expect(finding).toBeDefined(); + expect(finding?.severity).toBe("warning"); + }); }); diff --git a/packages/core/src/lint/hyperframeLinter.ts b/packages/core/src/lint/hyperframeLinter.ts index db878ccb9..13b34e5f8 100644 --- a/packages/core/src/lint/hyperframeLinter.ts +++ b/packages/core/src/lint/hyperframeLinter.ts @@ -710,6 +710,79 @@ export function lintHyperframeHtml( } } + // ── Caption lint rules ────────────────────────────────────────────────── + + // Rule: caption_exit_missing_hard_kill + // Exit tweens (tl.to with opacity: 0) can fail when karaoke word-level tweens + // conflict, leaving captions stuck on screen. A hard tl.set kill is needed. + for (const script of scripts) { + const content = script.content; + const hasExitTween = /\.to\s*\([^,]+,\s*\{[^}]*opacity\s*:\s*0/.test(content); + const hasHardKill = + /\.set\s*\([^,]+,\s*\{[^}]*(?:visibility\s*:\s*["']hidden["']|opacity\s*:\s*0)/.test(content); + const hasCaptionLoop = + /forEach|\.forEach\s*\(/.test(content) && /createElement|caption|group|cg-/.test(content); + + if (hasCaptionLoop && hasExitTween && !hasHardKill) { + pushFinding({ + code: "caption_exit_missing_hard_kill", + severity: "warning", + message: + "Caption exit animations (tl.to with opacity: 0) detected without a hard tl.set kill. " + + "Exit tweens can fail when karaoke word-level tweens conflict, leaving captions stuck on screen.", + fixHint: + 'Add `tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end)` after every ' + + "exit tl.to animation as a deterministic kill.", + }); + } + } + + // Rule: caption_text_overflow_risk + // Captions with nowrap text and no max-width will clip off-screen. + for (const style of styles) { + const content = style.content; + const captionBlocks = content.matchAll( + /(\.caption[-_]?(?:group|container|text|line|word)|#caption[-_]?container)\s*\{([^}]+)\}/gi, + ); + for (const [, selector, body] of captionBlocks) { + if (!body) continue; + const hasNowrap = /white-space\s*:\s*nowrap/i.test(body); + const hasMaxWidth = /max-width/i.test(body); + + if (hasNowrap && !hasMaxWidth) { + pushFinding({ + code: "caption_text_overflow_risk", + severity: "warning", + selector: (selector ?? "").trim(), + message: `Caption selector "${(selector ?? "").trim()}" has white-space: nowrap but no max-width. Long phrases will clip off-screen.`, + fixHint: + "Add max-width: 1600px (landscape) or max-width: 900px (portrait) and overflow: hidden.", + }); + } + } + } + + // Rule: caption_container_relative_position + // position: relative on caption containers causes overflow and stacking issues. + for (const style of styles) { + const content = style.content; + const captionBlocks = content.matchAll( + /(\.caption[-_]?(?:group|container|text|line)|#caption[-_]?container)\s*\{([^}]+)\}/gi, + ); + for (const [, selector, body] of captionBlocks) { + if (!body) continue; + if (/position\s*:\s*relative/i.test(body)) { + pushFinding({ + code: "caption_container_relative_position", + severity: "warning", + selector: (selector ?? "").trim(), + message: `Caption selector "${(selector ?? "").trim()}" uses position: relative which causes overflow and breaks caption stacking.`, + fixHint: "Use position: absolute for all caption elements.", + }); + } + } + } + // ── External CDN script dependency check ──────────────────────────────── // Compositions that load CDN libraries via