mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
feat(cli,core): standalone transcribe command, transcript normalization, caption lint rules (#151)
* feat(cli,core): add standalone transcribe command, transcript normalization, and caption lint rules
- Add `hyperframes transcribe` command for transcribing audio/video and importing
existing transcripts (SRT, VTT, OpenAI Whisper API JSON, whisper.cpp JSON)
- Add transcript format normalizer (normalize.ts) with auto-detection and
conversion to standard [{text, start, end}] word arrays
- Upgrade default whisper model from base.en to small.en for better accuracy
- Add --model and --language flags to both `transcribe` and `init` commands
- Extract shared patchCaptionHtml() to eliminate duplication between init.ts
and transcribe.ts (init.ts reduced by ~55 lines)
- Add 3 caption lint rules: caption_exit_missing_hard_kill,
caption_text_overflow_risk, caption_container_relative_position
- Update captions skill with model guide, format docs, music guidance,
text overflow prevention, caption exit guarantee pattern
- Expand captions skill trigger to cover lyrics, karaoke, lyric videos
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(cli): add transcribe command and --model/--language flags to CLI docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): fix blank template lint issues
- blank/index.html: remove data-start from video (was nested in timed parent),
add class="clip" for initial hidden state
- blank/captions.html: add max-width + overflow:hidden to prevent text clipping,
add tl.set hard kill after exit tween to prevent stuck captions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: add lint-after-edit rule to repo and project CLAUDE.md
Agents must run `npx hyperframes lint` after editing compositions.
Also expand captions skill description in project template.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style: format _shared/CLAUDE.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
James Russo
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
7b18c0352e
commit
2f99e33bbe
@@ -9,7 +9,7 @@ This repo ships skills that are installed globally via `npx hyperframes skills`
|
|||||||
| Skill | Invoke with | When to use |
|
| Skill | Invoke with | When to use |
|
||||||
| ------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
| ------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
| **hyperframes-compose** | `/hyperframes-compose` | Creating ANY HTML composition — videos, animations, title cards, overlays. Contains required HTML structure, `class="clip"` rules, GSAP timeline patterns, and rendering constraints. |
|
| **hyperframes-compose** | `/hyperframes-compose` | Creating ANY HTML composition — videos, animations, title cards, overlays. Contains required HTML structure, `class="clip"` rules, GSAP timeline patterns, and rendering constraints. |
|
||||||
| **hyperframes-captions** | `/hyperframes-captions` | Building tone-adaptive captions from whisper transcripts — style detection, per-word styling, positioning. |
|
| **hyperframes-captions** | `/hyperframes-captions` | Any task involving text synced to audio: captions, subtitles, lyrics, lyric videos, karaoke. Also covers transcription strategy (whisper model selection, transcript format). |
|
||||||
|
|
||||||
### GSAP Skills (from [greensock/gsap-skills](https://github.com/greensock/gsap-skills))
|
### GSAP Skills (from [greensock/gsap-skills](https://github.com/greensock/gsap-skills))
|
||||||
|
|
||||||
@@ -29,9 +29,12 @@ The skills encode HyperFrames-specific patterns (e.g., required `class="clip"` o
|
|||||||
### Rules
|
### Rules
|
||||||
|
|
||||||
- When creating or modifying HTML compositions → invoke `/hyperframes-compose` BEFORE writing any code
|
- When creating or modifying HTML compositions → invoke `/hyperframes-compose` BEFORE writing any code
|
||||||
- When adding captions → invoke `/hyperframes-captions` BEFORE writing any code
|
- When adding captions, subtitles, lyrics, or any text synced to audio → invoke `/hyperframes-captions` BEFORE writing any code
|
||||||
|
- When transcribing audio or choosing a whisper model → invoke `/hyperframes-captions` BEFORE running any transcription tool
|
||||||
|
- When creating a video from audio (music video, lyric video, audio visualizer with text) → invoke BOTH `/hyperframes-compose` AND `/hyperframes-captions`
|
||||||
- When writing GSAP animations → invoke `/gsap-core` and `/gsap-timeline` BEFORE writing any code
|
- When writing GSAP animations → invoke `/gsap-core` and `/gsap-timeline` BEFORE writing any code
|
||||||
- When optimizing animation performance → invoke `/gsap-performance` BEFORE making changes
|
- When optimizing animation performance → invoke `/gsap-performance` BEFORE making changes
|
||||||
|
- After creating or editing any `.html` composition → run `npx hyperframes lint` and fix all errors before considering the task complete
|
||||||
|
|
||||||
### Installing skills
|
### Installing skills
|
||||||
|
|
||||||
@@ -68,3 +71,48 @@ pnpm test # Run tests
|
|||||||
- **Frame Adapters** bridge animation runtimes (GSAP, Lottie, CSS) to the capture engine
|
- **Frame Adapters** bridge animation runtimes (GSAP, Lottie, CSS) to the capture engine
|
||||||
- **Producer** orchestrates capture → encode → audio mix into final MP4
|
- **Producer** orchestrates capture → encode → audio mix into final MP4
|
||||||
- **BeginFrame rendering** uses `HeadlessExperimental.beginFrame` for deterministic frame capture
|
- **BeginFrame rendering** uses `HeadlessExperimental.beginFrame` for deterministic frame capture
|
||||||
|
|
||||||
|
## Transcription
|
||||||
|
|
||||||
|
HyperFrames uses word-level timestamps for captions. The `hyperframes transcribe` command handles both transcription and format conversion.
|
||||||
|
|
||||||
|
### Quick reference
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Transcribe audio/video (local whisper.cpp, no API key)
|
||||||
|
npx hyperframes transcribe audio.mp3
|
||||||
|
npx hyperframes transcribe video.mp4 --model medium.en --language en
|
||||||
|
|
||||||
|
# Import existing transcript from another tool
|
||||||
|
npx hyperframes transcribe subtitles.srt
|
||||||
|
npx hyperframes transcribe subtitles.vtt
|
||||||
|
npx hyperframes transcribe openai-response.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Whisper models
|
||||||
|
|
||||||
|
Default is `small.en`. Upgrade for better accuracy:
|
||||||
|
|
||||||
|
| Model | Size | Use case |
|
||||||
|
| ----------- | ------ | -------------------------------- |
|
||||||
|
| `tiny.en` | 75 MB | Quick testing |
|
||||||
|
| `base.en` | 142 MB | Short clips, clear audio |
|
||||||
|
| `small.en` | 466 MB | **Default** — most content |
|
||||||
|
| `medium.en` | 1.5 GB | Important content, noisy audio |
|
||||||
|
| `large-v3` | 3.1 GB | Multilingual, production quality |
|
||||||
|
|
||||||
|
Use `.en` suffix for English-only (more accurate). Drop it for multilingual content.
|
||||||
|
|
||||||
|
### Supported transcript formats
|
||||||
|
|
||||||
|
The CLI auto-detects and normalizes: whisper.cpp JSON, OpenAI Whisper API JSON, SRT, VTT, and pre-normalized `[{text, start, end}]` arrays.
|
||||||
|
|
||||||
|
### Improving transcription quality
|
||||||
|
|
||||||
|
If captions are inaccurate (wrong words, bad timing):
|
||||||
|
|
||||||
|
1. **Upgrade the model**: `--model medium.en` or `--model large-v3`
|
||||||
|
2. **Set language**: `--language en` to filter non-target speech
|
||||||
|
3. **Use an external API**: Transcribe via OpenAI or Groq Whisper API, then import the JSON with `hyperframes transcribe response.json`
|
||||||
|
|
||||||
|
See the `/hyperframes-captions` skill for full details on model selection and API usage.
|
||||||
|
|||||||
@@ -156,6 +156,8 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_
|
|||||||
| `--audio, -a` | Path to an audio file (MP3, WAV, M4A) |
|
| `--audio, -a` | Path to an audio file (MP3, WAV, M4A) |
|
||||||
| `--skip-skills` | Skip AI coding skills installation |
|
| `--skip-skills` | Skip AI coding skills installation |
|
||||||
| `--skip-transcribe` | Skip automatic whisper transcription |
|
| `--skip-transcribe` | Skip automatic whisper transcription |
|
||||||
|
| `--model` | Whisper model for transcription (e.g. `small.en`, `medium.en`, `large-v3`) |
|
||||||
|
| `--language` | Language code for transcription (e.g. `en`, `es`, `ja`). Filters non-target speech. |
|
||||||
| `--human-friendly` | Enable interactive terminal UI with prompts |
|
| `--human-friendly` | Enable interactive terminal UI with prompts |
|
||||||
|
|
||||||
| Template | Description |
|
| Template | Description |
|
||||||
@@ -185,6 +187,45 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_
|
|||||||
| `--json` | Output as JSON |
|
| `--json` | Output as JSON |
|
||||||
|
|
||||||
Shows each composition's ID, duration, resolution, and element count.
|
Shows each composition's ID, duration, resolution, and element count.
|
||||||
|
|
||||||
|
### `transcribe`
|
||||||
|
|
||||||
|
Transcribe audio/video to word-level timestamps, or import an existing transcript:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Transcribe audio/video with local whisper.cpp
|
||||||
|
npx hyperframes transcribe audio.mp3
|
||||||
|
npx hyperframes transcribe video.mp4 --model medium.en --language en
|
||||||
|
|
||||||
|
# Import existing transcripts from other tools
|
||||||
|
npx hyperframes transcribe subtitles.srt
|
||||||
|
npx hyperframes transcribe captions.vtt
|
||||||
|
npx hyperframes transcribe openai-response.json
|
||||||
|
```
|
||||||
|
|
||||||
|
| Flag | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `--dir, -d` | Project directory (default: current directory) |
|
||||||
|
| `--model, -m` | Whisper model (default: `small.en`). Options: `tiny.en`, `base.en`, `small.en`, `medium.en`, `large-v3` |
|
||||||
|
| `--language, -l` | Language code (e.g. `en`, `es`, `ja`). Filters out non-target language speech. |
|
||||||
|
| `--json` | Output result as JSON |
|
||||||
|
|
||||||
|
The command auto-detects the input type. Audio/video files are transcribed with whisper.cpp. Transcript files (`.json`, `.srt`, `.vtt`) are normalized and imported.
|
||||||
|
|
||||||
|
**Supported transcript formats:**
|
||||||
|
|
||||||
|
| Format | Source |
|
||||||
|
|--------|--------|
|
||||||
|
| whisper.cpp JSON | `hyperframes init --video`, `hyperframes transcribe` |
|
||||||
|
| OpenAI Whisper API JSON | `openai.audio.transcriptions.create()` with word timestamps |
|
||||||
|
| SRT subtitles | Video editors, YouTube, subtitle tools |
|
||||||
|
| VTT subtitles | Web players, YouTube, transcription services |
|
||||||
|
|
||||||
|
All formats are normalized to a standard `[{text, start, end}]` word array and saved as `transcript.json`. If the project has caption HTML files, they are automatically patched with the transcript data.
|
||||||
|
|
||||||
|
<Tip>
|
||||||
|
For music or noisy audio, use `--model medium.en` for better accuracy. For the best results with production content, transcribe via the OpenAI or Groq Whisper API and import the JSON.
|
||||||
|
</Tip>
|
||||||
</Tab>
|
</Tab>
|
||||||
<Tab title="Develop">
|
<Tab title="Develop">
|
||||||
### `dev`
|
### `dev`
|
||||||
|
|||||||
@@ -26,6 +26,7 @@ const subCommands = {
|
|||||||
benchmark: () => import("./commands/benchmark.js").then((m) => m.default),
|
benchmark: () => import("./commands/benchmark.js").then((m) => m.default),
|
||||||
browser: () => import("./commands/browser.js").then((m) => m.default),
|
browser: () => import("./commands/browser.js").then((m) => m.default),
|
||||||
skills: () => import("./commands/install-skills.js").then((m) => m.default),
|
skills: () => import("./commands/install-skills.js").then((m) => m.default),
|
||||||
|
transcribe: () => import("./commands/transcribe.js").then((m) => m.default),
|
||||||
docs: () => import("./commands/docs.js").then((m) => m.default),
|
docs: () => import("./commands/docs.js").then((m) => m.default),
|
||||||
doctor: () => import("./commands/doctor.js").then((m) => m.default),
|
doctor: () => import("./commands/doctor.js").then((m) => m.default),
|
||||||
upgrade: () => import("./commands/upgrade.js").then((m) => m.default),
|
upgrade: () => import("./commands/upgrade.js").then((m) => m.default),
|
||||||
|
|||||||
@@ -246,60 +246,11 @@ function patchVideoSrc(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function patchTranscript(dir: string, transcriptPath: string): void {
|
async function patchTranscript(dir: string, transcriptPath: string): Promise<void> {
|
||||||
// Read the whisper transcript and normalize to [{text, start, end}]
|
const { loadTranscript, patchCaptionHtml } = await import("../whisper/normalize.js");
|
||||||
const raw = JSON.parse(readFileSync(transcriptPath, "utf-8"));
|
const { words } = loadTranscript(transcriptPath);
|
||||||
const words: { text: string; start: number; end: number }[] = [];
|
|
||||||
for (const seg of raw.transcription ?? []) {
|
|
||||||
for (const token of seg.tokens ?? []) {
|
|
||||||
const text = (token.text ?? "").trim();
|
|
||||||
if (!text || text.startsWith("[_") || text.startsWith("[BLANK")) continue;
|
|
||||||
|
|
||||||
// Merge punctuation with the previous word
|
|
||||||
const isPunctuation = /^[.,!?;:'")\]}>…–—-]+$/.test(text);
|
|
||||||
const lastWord = words[words.length - 1];
|
|
||||||
if (isPunctuation && lastWord) {
|
|
||||||
lastWord.text += text;
|
|
||||||
lastWord.end = Math.round(((token.offsets?.to ?? 0) / 1000) * 1000) / 1000;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
words.push({
|
|
||||||
text,
|
|
||||||
start: Math.round(((token.offsets?.from ?? 0) / 1000) * 1000) / 1000,
|
|
||||||
end: Math.round(((token.offsets?.to ?? 0) / 1000) * 1000) / 1000,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (words.length === 0) return;
|
if (words.length === 0) return;
|
||||||
|
patchCaptionHtml(dir, words);
|
||||||
const wordsJson = JSON.stringify(words, null, 10)
|
|
||||||
.replace(/^\[/, "[")
|
|
||||||
.replace(/\n {10}/g, "\n ");
|
|
||||||
|
|
||||||
// Find captions HTML files and replace the hardcoded script array
|
|
||||||
const htmlFiles = readdirSync(dir, { withFileTypes: true, recursive: true })
|
|
||||||
.filter((e) => e.isFile() && e.name.endsWith(".html"))
|
|
||||||
.map((e) => join(e.parentPath ?? e.path, e.name));
|
|
||||||
|
|
||||||
for (const file of htmlFiles) {
|
|
||||||
let content = readFileSync(file, "utf-8");
|
|
||||||
// Match within <script> blocks only to avoid crossing block boundaries
|
|
||||||
const scriptBlocks = content.match(/<script>[\s\S]*?<\/script>/g) ?? [];
|
|
||||||
let scriptMatch: RegExpMatchArray | null = null;
|
|
||||||
let transcriptMatch: RegExpMatchArray | null = null;
|
|
||||||
for (const block of scriptBlocks) {
|
|
||||||
scriptMatch = scriptMatch ?? block.match(/const script = \[[\s\S]*?\];/);
|
|
||||||
transcriptMatch = transcriptMatch ?? block.match(/const TRANSCRIPT = \[[\s\S]*?\];/);
|
|
||||||
}
|
|
||||||
const match = scriptMatch ?? transcriptMatch;
|
|
||||||
if (match) {
|
|
||||||
const varName = scriptMatch ? "script" : "TRANSCRIPT";
|
|
||||||
content = content.replace(match[0], `const ${varName} = ${wordsJson};`);
|
|
||||||
writeFileSync(file, content, "utf-8");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -543,6 +494,16 @@ Examples:
|
|||||||
type: "boolean",
|
type: "boolean",
|
||||||
description: "Skip whisper transcription",
|
description: "Skip whisper transcription",
|
||||||
},
|
},
|
||||||
|
model: {
|
||||||
|
type: "string",
|
||||||
|
description:
|
||||||
|
"Whisper model for transcription (e.g. tiny.en, base.en, small.en, medium.en, large)",
|
||||||
|
},
|
||||||
|
language: {
|
||||||
|
type: "string",
|
||||||
|
description:
|
||||||
|
"Language code for transcription (e.g. en, es, ja). Filters out non-target speech.",
|
||||||
|
},
|
||||||
"non-interactive": {
|
"non-interactive": {
|
||||||
type: "boolean",
|
type: "boolean",
|
||||||
description: "Disable interactive prompts (for CI/agents)",
|
description: "Disable interactive prompts (for CI/agents)",
|
||||||
@@ -555,6 +516,8 @@ Examples:
|
|||||||
const skipSkills = args["skip-skills"] === true;
|
const skipSkills = args["skip-skills"] === true;
|
||||||
const skipTranscribe = args["skip-transcribe"] === true;
|
const skipTranscribe = args["skip-transcribe"] === true;
|
||||||
const nonInteractive = args["non-interactive"] === true;
|
const nonInteractive = args["non-interactive"] === true;
|
||||||
|
const modelFlag = args.model;
|
||||||
|
const languageFlag = args.language;
|
||||||
const interactive = !nonInteractive && process.stdout.isTTY === true;
|
const interactive = !nonInteractive && process.stdout.isTTY === true;
|
||||||
|
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
@@ -615,10 +578,13 @@ Examples:
|
|||||||
try {
|
try {
|
||||||
const { ensureWhisper, ensureModel } = await import("../whisper/manager.js");
|
const { ensureWhisper, ensureModel } = await import("../whisper/manager.js");
|
||||||
await ensureWhisper();
|
await ensureWhisper();
|
||||||
await ensureModel();
|
await ensureModel(modelFlag);
|
||||||
console.log("Transcribing...");
|
console.log("Transcribing...");
|
||||||
const { transcribe: runTranscribe } = await import("../whisper/transcribe.js");
|
const { transcribe: runTranscribe } = await import("../whisper/transcribe.js");
|
||||||
const result = await runTranscribe(sourceFilePath, destDir);
|
const result = await runTranscribe(sourceFilePath, destDir, {
|
||||||
|
model: modelFlag,
|
||||||
|
language: languageFlag,
|
||||||
|
});
|
||||||
console.log(
|
console.log(
|
||||||
`Transcribed: ${result.wordCount} words (${result.durationSeconds.toFixed(1)}s)`,
|
`Transcribed: ${result.wordCount} words (${result.durationSeconds.toFixed(1)}s)`,
|
||||||
);
|
);
|
||||||
@@ -633,7 +599,7 @@ Examples:
|
|||||||
trackInitTemplate(templateId);
|
trackInitTemplate(templateId);
|
||||||
const transcriptFile = resolve(destDir, "transcript.json");
|
const transcriptFile = resolve(destDir, "transcript.json");
|
||||||
if (existsSync(transcriptFile)) {
|
if (existsSync(transcriptFile)) {
|
||||||
patchTranscript(destDir, transcriptFile);
|
await patchTranscript(destDir, transcriptFile);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Skills
|
// Skills
|
||||||
@@ -796,13 +762,15 @@ Examples:
|
|||||||
await ensureWhisper({
|
await ensureWhisper({
|
||||||
onProgress: (msg) => spin.message(msg),
|
onProgress: (msg) => spin.message(msg),
|
||||||
});
|
});
|
||||||
await ensureModel(undefined, {
|
await ensureModel(modelFlag, {
|
||||||
onProgress: (msg) => spin.message(msg),
|
onProgress: (msg) => spin.message(msg),
|
||||||
});
|
});
|
||||||
|
|
||||||
spin.message("Transcribing audio...");
|
spin.message("Transcribing audio...");
|
||||||
const { transcribe: runTranscribe } = await import("../whisper/transcribe.js");
|
const { transcribe: runTranscribe } = await import("../whisper/transcribe.js");
|
||||||
const transcribeResult = await runTranscribe(sourceFilePath, destDir, {
|
const transcribeResult = await runTranscribe(sourceFilePath, destDir, {
|
||||||
|
model: modelFlag,
|
||||||
|
language: languageFlag,
|
||||||
onProgress: (msg) => spin.message(msg),
|
onProgress: (msg) => spin.message(msg),
|
||||||
});
|
});
|
||||||
spin.stop(
|
spin.stop(
|
||||||
|
|||||||
@@ -0,0 +1,149 @@
|
|||||||
|
import { defineCommand } from "citty";
|
||||||
|
import { existsSync, writeFileSync } from "node:fs";
|
||||||
|
import { resolve, join, extname } from "node:path";
|
||||||
|
import * as clack from "@clack/prompts";
|
||||||
|
import { c } from "../ui/colors.js";
|
||||||
|
import { DEFAULT_MODEL } from "../whisper/manager.js";
|
||||||
|
|
||||||
|
export default defineCommand({
|
||||||
|
meta: {
|
||||||
|
name: "transcribe",
|
||||||
|
description:
|
||||||
|
"Transcribe audio/video to word-level timestamps, or import an existing transcript",
|
||||||
|
},
|
||||||
|
args: {
|
||||||
|
input: {
|
||||||
|
type: "positional",
|
||||||
|
description:
|
||||||
|
"Audio/video file to transcribe, or transcript file to import (.json, .srt, .vtt)",
|
||||||
|
required: true,
|
||||||
|
},
|
||||||
|
dir: {
|
||||||
|
type: "string",
|
||||||
|
description: "Project directory (default: current directory)",
|
||||||
|
alias: "d",
|
||||||
|
},
|
||||||
|
model: {
|
||||||
|
type: "string",
|
||||||
|
description: `Whisper model (default: ${DEFAULT_MODEL}). Options: tiny.en, base.en, small.en, medium.en, large-v3`,
|
||||||
|
alias: "m",
|
||||||
|
},
|
||||||
|
language: {
|
||||||
|
type: "string",
|
||||||
|
description: "Language code (e.g. en, es, ja). Filters out non-target language speech.",
|
||||||
|
alias: "l",
|
||||||
|
},
|
||||||
|
json: {
|
||||||
|
type: "boolean",
|
||||||
|
description: "Output result as JSON",
|
||||||
|
default: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
async run({ args }) {
|
||||||
|
const inputPath = resolve(args.input);
|
||||||
|
if (!existsSync(inputPath)) {
|
||||||
|
console.error(c.error(`File not found: ${args.input}`));
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
const dir = resolve(args.dir ?? ".");
|
||||||
|
const ext = extname(inputPath).toLowerCase();
|
||||||
|
|
||||||
|
// ── Import mode: convert existing transcript ──────────────────────────
|
||||||
|
const isImport = ext === ".json" || ext === ".srt" || ext === ".vtt";
|
||||||
|
|
||||||
|
if (isImport) {
|
||||||
|
return importTranscript(inputPath, dir, args.json);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Transcribe mode: run whisper ─────────────────────────────────────
|
||||||
|
return transcribeAudio(inputPath, dir, {
|
||||||
|
model: args.model,
|
||||||
|
language: args.language,
|
||||||
|
json: args.json,
|
||||||
|
});
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Import existing transcript
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
async function importTranscript(inputPath: string, dir: string, json: boolean): Promise<void> {
|
||||||
|
const { loadTranscript, patchCaptionHtml } = await import("../whisper/normalize.js");
|
||||||
|
const { words, format } = loadTranscript(inputPath);
|
||||||
|
|
||||||
|
if (words.length === 0) {
|
||||||
|
console.error(c.error("No words found in transcript."));
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
const outPath = join(dir, "transcript.json");
|
||||||
|
writeFileSync(outPath, JSON.stringify(words, null, 2));
|
||||||
|
patchCaptionHtml(dir, words);
|
||||||
|
|
||||||
|
if (json) {
|
||||||
|
console.log(
|
||||||
|
JSON.stringify({ ok: true, format, wordCount: words.length, transcriptPath: outPath }),
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
console.log(
|
||||||
|
`${c.success("◇")} Imported ${c.accent(String(words.length))} words from ${c.accent(format)} format → ${c.accent("transcript.json")}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Transcribe audio/video with whisper
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
async function transcribeAudio(
|
||||||
|
inputPath: string,
|
||||||
|
dir: string,
|
||||||
|
opts: { model?: string; language?: string; json?: boolean },
|
||||||
|
): Promise<void> {
|
||||||
|
const { transcribe } = await import("../whisper/transcribe.js");
|
||||||
|
const { loadTranscript, patchCaptionHtml } = await import("../whisper/normalize.js");
|
||||||
|
|
||||||
|
const model = opts.model ?? DEFAULT_MODEL;
|
||||||
|
const spin = opts.json ? null : clack.spinner();
|
||||||
|
spin?.start(`Transcribing with ${c.accent(model)}...`);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await transcribe(inputPath, dir, {
|
||||||
|
model,
|
||||||
|
language: opts.language,
|
||||||
|
onProgress: spin ? (msg) => spin.message(msg) : undefined,
|
||||||
|
});
|
||||||
|
|
||||||
|
const { words } = loadTranscript(result.transcriptPath);
|
||||||
|
writeFileSync(result.transcriptPath, JSON.stringify(words, null, 2));
|
||||||
|
patchCaptionHtml(dir, words);
|
||||||
|
|
||||||
|
if (opts.json) {
|
||||||
|
console.log(
|
||||||
|
JSON.stringify({
|
||||||
|
ok: true,
|
||||||
|
model,
|
||||||
|
wordCount: words.length,
|
||||||
|
durationSeconds: result.durationSeconds,
|
||||||
|
transcriptPath: result.transcriptPath,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
spin!.stop(
|
||||||
|
c.success(
|
||||||
|
`Transcribed ${c.accent(String(words.length))} words (${result.durationSeconds.toFixed(1)}s)`,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
const message = err instanceof Error ? err.message : String(err);
|
||||||
|
if (opts.json) {
|
||||||
|
console.log(JSON.stringify({ ok: false, error: message }));
|
||||||
|
} else {
|
||||||
|
spin!.stop(c.error(`Transcription failed: ${message}`));
|
||||||
|
}
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,7 +7,7 @@
|
|||||||
| Skill | Command | When to use |
|
| Skill | Command | When to use |
|
||||||
| ------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------ |
|
| ------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------ |
|
||||||
| **hyperframes-compose** | `/hyperframes-compose` | Creating or editing ANY HTML composition — videos, animations, title cards, overlays, sub-compositions |
|
| **hyperframes-compose** | `/hyperframes-compose` | Creating or editing ANY HTML composition — videos, animations, title cards, overlays, sub-compositions |
|
||||||
| **hyperframes-captions** | `/hyperframes-captions` | Building captions from whisper transcripts — style detection, per-word styling |
|
| **hyperframes-captions** | `/hyperframes-captions` | Any text synced to audio: captions, subtitles, lyrics, karaoke. Also covers transcription strategy. |
|
||||||
| **gsap-core** | `/gsap-core` | GSAP tweens: `gsap.to()`, `from()`, `fromTo()`, easing, stagger, defaults |
|
| **gsap-core** | `/gsap-core` | GSAP tweens: `gsap.to()`, `from()`, `fromTo()`, easing, stagger, defaults |
|
||||||
| **gsap-timeline** | `/gsap-timeline` | Timeline sequencing, position parameter, labels, nesting |
|
| **gsap-timeline** | `/gsap-timeline` | Timeline sequencing, position parameter, labels, nesting |
|
||||||
| **gsap-performance** | `/gsap-performance` | Animation performance — transforms over layout props, will-change, batching |
|
| **gsap-performance** | `/gsap-performance` | Animation performance — transforms over layout props, will-change, batching |
|
||||||
@@ -50,6 +50,16 @@ https://hyperframes.heygen.com/llms.txt
|
|||||||
- `meta.json` — project metadata (id, name)
|
- `meta.json` — project metadata (id, name)
|
||||||
- `transcript.json` — whisper word-level transcript (if generated)
|
- `transcript.json` — whisper word-level transcript (if generated)
|
||||||
|
|
||||||
|
## Linting — ALWAYS RUN AFTER CHANGES
|
||||||
|
|
||||||
|
After creating or editing any `.html` composition, **always** run the linter before considering the task complete:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npx hyperframes lint
|
||||||
|
```
|
||||||
|
|
||||||
|
Fix all errors before presenting the result. Warnings are informational and usually safe to ignore.
|
||||||
|
|
||||||
## Key Rules
|
## Key Rules
|
||||||
|
|
||||||
1. Every timed element needs `data-start`, `data-duration`, and `data-track-index`
|
1. Every timed element needs `data-start`, `data-duration`, and `data-track-index`
|
||||||
|
|||||||
@@ -37,6 +37,8 @@
|
|||||||
0 2px 8px rgba(0, 0, 0, 0.8),
|
0 2px 8px rgba(0, 0, 0, 0.8),
|
||||||
0 0 2px rgba(0, 0, 0, 0.9);
|
0 0 2px rgba(0, 0, 0, 0.9);
|
||||||
white-space: nowrap;
|
white-space: nowrap;
|
||||||
|
max-width: 1600px;
|
||||||
|
overflow: hidden;
|
||||||
}
|
}
|
||||||
</style>
|
</style>
|
||||||
|
|
||||||
@@ -83,6 +85,7 @@
|
|||||||
index < lines.length - 1 ? Math.min(line.end, lines[index + 1].start) : line.end;
|
index < lines.length - 1 ? Math.min(line.end, lines[index + 1].start) : line.end;
|
||||||
|
|
||||||
tl.to(el, { opacity: 0, y: -10, duration: 0.25, ease: "power2.in" }, hideTime - 0.25);
|
tl.to(el, { opacity: 0, y: -10, duration: 0.25, ease: "power2.in" }, hideTime - 0.25);
|
||||||
|
tl.set(el, { opacity: 0, visibility: "hidden" }, hideTime);
|
||||||
});
|
});
|
||||||
|
|
||||||
window.__timelines["captions"] = tl;
|
window.__timelines["captions"] = tl;
|
||||||
|
|||||||
@@ -31,10 +31,10 @@
|
|||||||
>
|
>
|
||||||
<video
|
<video
|
||||||
id="a-roll"
|
id="a-roll"
|
||||||
|
class="clip"
|
||||||
src="__VIDEO_SRC__"
|
src="__VIDEO_SRC__"
|
||||||
muted
|
muted
|
||||||
playsinline
|
playsinline
|
||||||
data-start="0"
|
|
||||||
data-duration="__VIDEO_DURATION__"
|
data-duration="__VIDEO_DURATION__"
|
||||||
data-track-index="0"
|
data-track-index="0"
|
||||||
style="position: absolute; inset: 0; width: 100%; height: 100%; object-fit: cover"
|
style="position: absolute; inset: 0; width: 100%; height: 100%; object-fit: cover"
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ import { get as httpsGet } from "node:https";
|
|||||||
import { pipeline } from "node:stream/promises";
|
import { pipeline } from "node:stream/promises";
|
||||||
|
|
||||||
const MODELS_DIR = join(homedir(), ".cache", "hyperframes", "whisper", "models");
|
const MODELS_DIR = join(homedir(), ".cache", "hyperframes", "whisper", "models");
|
||||||
const DEFAULT_MODEL = "base.en";
|
const DEFAULT_MODEL = "small.en";
|
||||||
|
|
||||||
export type WhisperSource = "env" | "system" | "brew" | "build";
|
export type WhisperSource = "env" | "system" | "brew" | "build";
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,278 @@
|
|||||||
|
import { describe, it, expect, afterEach } from "vitest";
|
||||||
|
import { writeFileSync, readFileSync, mkdirSync, rmSync } from "node:fs";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { loadTranscript, detectFormat, patchCaptionHtml } from "./normalize.js";
|
||||||
|
|
||||||
|
function tmpFile(name: string, content: string): string {
|
||||||
|
const dir = join(tmpdir(), `hf-normalize-test-${Date.now()}`);
|
||||||
|
mkdirSync(dir, { recursive: true });
|
||||||
|
dirs.push(dir);
|
||||||
|
const path = join(dir, name);
|
||||||
|
writeFileSync(path, content);
|
||||||
|
return path;
|
||||||
|
}
|
||||||
|
|
||||||
|
let dirs: string[] = [];
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
for (const d of dirs) rmSync(d, { recursive: true, force: true });
|
||||||
|
dirs = [];
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("detectFormat", () => {
|
||||||
|
it("detects SRT by extension", () => {
|
||||||
|
const path = tmpFile("test.srt", "1\n00:00:01,000 --> 00:00:02,000\nHello\n");
|
||||||
|
expect(detectFormat(path)).toBe("srt");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("detects VTT by extension", () => {
|
||||||
|
const path = tmpFile("test.vtt", "WEBVTT\n\n00:00:01.000 --> 00:00:02.000\nHello\n");
|
||||||
|
expect(detectFormat(path)).toBe("vtt");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("detects whisper-cpp JSON", () => {
|
||||||
|
const path = tmpFile(
|
||||||
|
"transcript.json",
|
||||||
|
JSON.stringify({
|
||||||
|
transcription: [
|
||||||
|
{
|
||||||
|
offsets: { from: 0, to: 2000 },
|
||||||
|
text: " Hello world.",
|
||||||
|
tokens: [
|
||||||
|
{ text: " Hello", offsets: { from: 0, to: 1000 }, p: 0.98 },
|
||||||
|
{ text: " world", offsets: { from: 1000, to: 2000 }, p: 0.95 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
expect(detectFormat(path)).toBe("whisper-cpp");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("detects OpenAI JSON", () => {
|
||||||
|
const path = tmpFile(
|
||||||
|
"openai.json",
|
||||||
|
JSON.stringify({
|
||||||
|
words: [
|
||||||
|
{ word: "Hello", start: 0.0, end: 0.5 },
|
||||||
|
{ word: "world", start: 0.6, end: 1.2 },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
expect(detectFormat(path)).toBe("openai");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("detects normalized word array", () => {
|
||||||
|
const path = tmpFile(
|
||||||
|
"words.json",
|
||||||
|
JSON.stringify([
|
||||||
|
{ text: "Hello", start: 0.0, end: 0.5 },
|
||||||
|
{ text: "world", start: 0.6, end: 1.2 },
|
||||||
|
]),
|
||||||
|
);
|
||||||
|
expect(detectFormat(path)).toBe("words-json");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("loadTranscript", () => {
|
||||||
|
it("parses whisper-cpp JSON with punctuation merging", () => {
|
||||||
|
const path = tmpFile(
|
||||||
|
"transcript.json",
|
||||||
|
JSON.stringify({
|
||||||
|
transcription: [
|
||||||
|
{
|
||||||
|
tokens: [
|
||||||
|
{ text: " Hello", offsets: { from: 0, to: 500 } },
|
||||||
|
{ text: ",", offsets: { from: 500, to: 550 } },
|
||||||
|
{ text: " world", offsets: { from: 600, to: 1200 } },
|
||||||
|
{ text: ".", offsets: { from: 1200, to: 1250 } },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const { words, format } = loadTranscript(path);
|
||||||
|
expect(format).toBe("whisper-cpp");
|
||||||
|
expect(words).toEqual([
|
||||||
|
{ text: "Hello,", start: 0, end: 0.55 },
|
||||||
|
{ text: "world.", start: 0.6, end: 1.25 },
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("filters whisper-cpp non-speech tokens", () => {
|
||||||
|
const path = tmpFile(
|
||||||
|
"transcript.json",
|
||||||
|
JSON.stringify({
|
||||||
|
transcription: [
|
||||||
|
{
|
||||||
|
tokens: [
|
||||||
|
{ text: "[_BEG_]", offsets: { from: 0, to: 0 } },
|
||||||
|
{ text: " Hello", offsets: { from: 100, to: 500 } },
|
||||||
|
{ text: "[BLANK_AUDIO]", offsets: { from: 500, to: 1000 } },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const { words } = loadTranscript(path);
|
||||||
|
expect(words).toHaveLength(1);
|
||||||
|
expect(words[0]!.text).toBe("Hello");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses OpenAI Whisper API response", () => {
|
||||||
|
const path = tmpFile(
|
||||||
|
"openai.json",
|
||||||
|
JSON.stringify({
|
||||||
|
text: "Hello world",
|
||||||
|
words: [
|
||||||
|
{ word: "Hello", start: 0.0, end: 0.5 },
|
||||||
|
{ word: "world", start: 0.6, end: 1.2 },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const { words, format } = loadTranscript(path);
|
||||||
|
expect(format).toBe("openai");
|
||||||
|
expect(words).toEqual([
|
||||||
|
{ text: "Hello", start: 0, end: 0.5 },
|
||||||
|
{ text: "world", start: 0.6, end: 1.2 },
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses SRT files", () => {
|
||||||
|
const srt = `1
|
||||||
|
00:00:01,000 --> 00:00:03,500
|
||||||
|
Hello world
|
||||||
|
|
||||||
|
2
|
||||||
|
00:00:04,000 --> 00:00:06,000
|
||||||
|
How are you
|
||||||
|
`;
|
||||||
|
const path = tmpFile("captions.srt", srt);
|
||||||
|
const { words, format } = loadTranscript(path);
|
||||||
|
expect(format).toBe("srt");
|
||||||
|
expect(words).toEqual([
|
||||||
|
{ text: "Hello world", start: 1.0, end: 3.5 },
|
||||||
|
{ text: "How are you", start: 4.0, end: 6.0 },
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses VTT files", () => {
|
||||||
|
const vtt = `WEBVTT
|
||||||
|
|
||||||
|
00:00:01.000 --> 00:00:03.500
|
||||||
|
Hello world
|
||||||
|
|
||||||
|
00:00:04.000 --> 00:00:06.000
|
||||||
|
How are you
|
||||||
|
`;
|
||||||
|
const path = tmpFile("captions.vtt", vtt);
|
||||||
|
const { words, format } = loadTranscript(path);
|
||||||
|
expect(format).toBe("vtt");
|
||||||
|
expect(words).toEqual([
|
||||||
|
{ text: "Hello world", start: 1.0, end: 3.5 },
|
||||||
|
{ text: "How are you", start: 4.0, end: 6.0 },
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses VTT with short timestamps (MM:SS.mmm)", () => {
|
||||||
|
const vtt = `WEBVTT
|
||||||
|
|
||||||
|
01:23.456 --> 02:00.000
|
||||||
|
Short format
|
||||||
|
`;
|
||||||
|
const path = tmpFile("short.vtt", vtt);
|
||||||
|
const { words } = loadTranscript(path);
|
||||||
|
expect(words[0]!.start).toBeCloseTo(83.456, 2);
|
||||||
|
expect(words[0]!.end).toBe(120.0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("strips HTML tags from SRT/VTT", () => {
|
||||||
|
const srt = `1
|
||||||
|
00:00:01,000 --> 00:00:03,000
|
||||||
|
<b>Bold</b> and <i>italic</i>
|
||||||
|
`;
|
||||||
|
const path = tmpFile("tags.srt", srt);
|
||||||
|
const { words } = loadTranscript(path);
|
||||||
|
expect(words[0]!.text).toBe("Bold and italic");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("passes through normalized word arrays", () => {
|
||||||
|
const input = [
|
||||||
|
{ text: "Hello", start: 0.0, end: 0.5 },
|
||||||
|
{ text: "world", start: 0.6, end: 1.2 },
|
||||||
|
];
|
||||||
|
const path = tmpFile("normalized.json", JSON.stringify(input));
|
||||||
|
const { words, format } = loadTranscript(path);
|
||||||
|
expect(format).toBe("words-json");
|
||||||
|
expect(words).toEqual(input);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("patchCaptionHtml", () => {
|
||||||
|
it("replaces const script = [] in HTML files", () => {
|
||||||
|
const dir = join(tmpdir(), `hf-patch-test-${Date.now()}`);
|
||||||
|
mkdirSync(dir, { recursive: true });
|
||||||
|
dirs.push(dir);
|
||||||
|
|
||||||
|
const html = `<html><body><script>
|
||||||
|
const script = [];
|
||||||
|
console.log(script);
|
||||||
|
</script></body></html>`;
|
||||||
|
writeFileSync(join(dir, "captions.html"), html);
|
||||||
|
|
||||||
|
const words = [
|
||||||
|
{ text: "Hello", start: 1.0, end: 1.5 },
|
||||||
|
{ text: "world", start: 2.0, end: 2.5 },
|
||||||
|
];
|
||||||
|
patchCaptionHtml(dir, words);
|
||||||
|
|
||||||
|
const result = readFileSync(join(dir, "captions.html"), "utf-8");
|
||||||
|
expect(result).toContain('"Hello"');
|
||||||
|
expect(result).toContain('"world"');
|
||||||
|
expect(result).not.toContain("const script = [];");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("replaces const TRANSCRIPT = [] variant", () => {
|
||||||
|
const dir = join(tmpdir(), `hf-patch-test-${Date.now()}`);
|
||||||
|
mkdirSync(dir, { recursive: true });
|
||||||
|
dirs.push(dir);
|
||||||
|
|
||||||
|
const html = `<script>const TRANSCRIPT = [];</script>`;
|
||||||
|
writeFileSync(join(dir, "index.html"), html);
|
||||||
|
|
||||||
|
patchCaptionHtml(dir, [{ text: "Hi", start: 0, end: 1 }]);
|
||||||
|
|
||||||
|
const result = readFileSync(join(dir, "index.html"), "utf-8");
|
||||||
|
expect(result).toContain("const TRANSCRIPT = ");
|
||||||
|
expect(result).toContain('"Hi"');
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not modify HTML files without matching script patterns", () => {
|
||||||
|
const dir = join(tmpdir(), `hf-patch-test-${Date.now()}`);
|
||||||
|
mkdirSync(dir, { recursive: true });
|
||||||
|
dirs.push(dir);
|
||||||
|
|
||||||
|
const html = `<html><body><script>console.log("hello");</script></body></html>`;
|
||||||
|
writeFileSync(join(dir, "page.html"), html);
|
||||||
|
|
||||||
|
patchCaptionHtml(dir, [{ text: "Hi", start: 0, end: 1 }]);
|
||||||
|
|
||||||
|
const result = readFileSync(join(dir, "page.html"), "utf-8");
|
||||||
|
expect(result).toBe(html);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("skips empty word arrays", () => {
|
||||||
|
const dir = join(tmpdir(), `hf-patch-test-${Date.now()}`);
|
||||||
|
mkdirSync(dir, { recursive: true });
|
||||||
|
dirs.push(dir);
|
||||||
|
|
||||||
|
const html = `<script>const script = [];</script>`;
|
||||||
|
writeFileSync(join(dir, "captions.html"), html);
|
||||||
|
|
||||||
|
patchCaptionHtml(dir, []);
|
||||||
|
|
||||||
|
const result = readFileSync(join(dir, "captions.html"), "utf-8");
|
||||||
|
expect(result).toBe(html);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,266 @@
|
|||||||
|
import { readFileSync, readdirSync, writeFileSync } from "node:fs";
|
||||||
|
import { extname, join } from "node:path";
|
||||||
|
|
||||||
|
export interface Word {
|
||||||
|
text: string;
|
||||||
|
start: number;
|
||||||
|
end: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Format detection + parsing
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
export type TranscriptFormat = "whisper-cpp" | "openai" | "srt" | "vtt" | "words-json";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Detect the format of a transcript file from its extension and content.
|
||||||
|
*/
|
||||||
|
export function detectFormat(filePath: string): TranscriptFormat {
|
||||||
|
const ext = extname(filePath).toLowerCase();
|
||||||
|
if (ext === ".srt") return "srt";
|
||||||
|
if (ext === ".vtt") return "vtt";
|
||||||
|
if (ext === ".json") return detectJsonFormat(JSON.parse(readFileSync(filePath, "utf-8")));
|
||||||
|
throw new Error(`Unsupported transcript file extension: ${ext}. Use .json, .srt, or .vtt`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function detectJsonFormat(raw: unknown): TranscriptFormat {
|
||||||
|
if (raw && typeof raw === "object" && !Array.isArray(raw)) {
|
||||||
|
const obj = raw as Record<string, unknown>;
|
||||||
|
if (obj.transcription && Array.isArray(obj.transcription)) return "whisper-cpp";
|
||||||
|
if (obj.words && Array.isArray(obj.words)) return "openai";
|
||||||
|
}
|
||||||
|
if (Array.isArray(raw) && raw[0]?.text !== undefined && raw[0]?.start !== undefined) {
|
||||||
|
return "words-json";
|
||||||
|
}
|
||||||
|
throw new Error(
|
||||||
|
"Unrecognized JSON transcript format. Expected whisper.cpp (transcription[].tokens), " +
|
||||||
|
"OpenAI API (words[]), or normalized ([{text, start, end}]).",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Parsers
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
function parseWhisperCpp(data: Record<string, unknown>): Word[] {
|
||||||
|
const words: Word[] = [];
|
||||||
|
const transcription = data.transcription as Array<{
|
||||||
|
tokens?: Array<{
|
||||||
|
text?: string;
|
||||||
|
offsets?: { from?: number; to?: number };
|
||||||
|
}>;
|
||||||
|
}>;
|
||||||
|
|
||||||
|
for (const seg of transcription ?? []) {
|
||||||
|
for (const token of seg.tokens ?? []) {
|
||||||
|
const text = (token.text ?? "").trim();
|
||||||
|
if (!text || text.startsWith("[_") || text.startsWith("[BLANK")) continue;
|
||||||
|
|
||||||
|
// Merge punctuation with the previous word
|
||||||
|
const isPunctuation = /^[.,!?;:'")\]}>…–—-]+$/.test(text);
|
||||||
|
const lastWord = words[words.length - 1];
|
||||||
|
if (isPunctuation && lastWord) {
|
||||||
|
lastWord.text += text;
|
||||||
|
lastWord.end = round3((token.offsets?.to ?? 0) / 1000);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
words.push({
|
||||||
|
text,
|
||||||
|
start: round3((token.offsets?.from ?? 0) / 1000),
|
||||||
|
end: round3((token.offsets?.to ?? 0) / 1000),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return words;
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseOpenAI(data: Record<string, unknown>): Word[] {
|
||||||
|
const rawWords = (data.words ?? []) as Array<{
|
||||||
|
word?: string;
|
||||||
|
text?: string;
|
||||||
|
start?: number;
|
||||||
|
end?: number;
|
||||||
|
}>;
|
||||||
|
return rawWords
|
||||||
|
.map((w) => ({
|
||||||
|
text: (w.word ?? w.text ?? "").trim(),
|
||||||
|
start: round3(w.start ?? 0),
|
||||||
|
end: round3(w.end ?? 0),
|
||||||
|
}))
|
||||||
|
.filter((w) => w.text.length > 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseSrt(content: string): Word[] {
|
||||||
|
// SRT doesn't have word-level timestamps — parse as phrase-level entries.
|
||||||
|
// Each cue becomes one "word" entry (the full phrase).
|
||||||
|
const blocks = content.trim().split(/\n\n+/);
|
||||||
|
const words: Word[] = [];
|
||||||
|
|
||||||
|
for (const block of blocks) {
|
||||||
|
const lines = block.trim().split("\n");
|
||||||
|
// SRT format: index, timestamp line, text lines
|
||||||
|
const timeLine = lines.find((l) => l.includes("-->"));
|
||||||
|
if (!timeLine) continue;
|
||||||
|
|
||||||
|
const [startStr, endStr] = timeLine.split("-->").map((s) => s.trim());
|
||||||
|
if (!startStr || !endStr) continue;
|
||||||
|
|
||||||
|
const text = lines
|
||||||
|
.slice(lines.indexOf(timeLine) + 1)
|
||||||
|
.join(" ")
|
||||||
|
.replace(/<[^>]+>/g, "") // strip HTML tags
|
||||||
|
.trim();
|
||||||
|
if (!text) continue;
|
||||||
|
|
||||||
|
words.push({
|
||||||
|
text,
|
||||||
|
start: parseSrtTimestamp(startStr),
|
||||||
|
end: parseSrtTimestamp(endStr),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return words;
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseVtt(content: string): Word[] {
|
||||||
|
// Strip the WEBVTT header and any metadata blocks
|
||||||
|
const body = content.replace(/^WEBVTT[^\n]*\n/, "").replace(/^[A-Z-]+:.*\n/gm, "");
|
||||||
|
// VTT is structurally similar to SRT (without numeric indices)
|
||||||
|
const blocks = body.trim().split(/\n\n+/);
|
||||||
|
const words: Word[] = [];
|
||||||
|
|
||||||
|
for (const block of blocks) {
|
||||||
|
const lines = block.trim().split("\n");
|
||||||
|
const timeLine = lines.find((l) => l.includes("-->"));
|
||||||
|
if (!timeLine) continue;
|
||||||
|
|
||||||
|
const [startStr, endStr] = timeLine.split("-->").map((s) => s.trim());
|
||||||
|
if (!startStr || !endStr) continue;
|
||||||
|
|
||||||
|
const text = lines
|
||||||
|
.slice(lines.indexOf(timeLine) + 1)
|
||||||
|
.join(" ")
|
||||||
|
.replace(/<[^>]+>/g, "") // strip HTML tags
|
||||||
|
.trim();
|
||||||
|
if (!text) continue;
|
||||||
|
|
||||||
|
words.push({
|
||||||
|
text,
|
||||||
|
start: parseVttTimestamp(startStr),
|
||||||
|
end: parseVttTimestamp(endStr),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return words;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Timestamp helpers
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** Parse SRT timestamp: 00:01:23,456 → seconds */
|
||||||
|
function parseSrtTimestamp(ts: string): number {
|
||||||
|
const m = ts.match(/(\d+):(\d+):(\d+)[,.](\d+)/);
|
||||||
|
if (!m) return 0;
|
||||||
|
return (
|
||||||
|
parseInt(m[1]!, 10) * 3600 +
|
||||||
|
parseInt(m[2]!, 10) * 60 +
|
||||||
|
parseInt(m[3]!, 10) +
|
||||||
|
parseInt(m[4]!.padEnd(3, "0"), 10) / 1000
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Parse VTT timestamp: 00:01:23.456 or 01:23.456 → seconds */
|
||||||
|
function parseVttTimestamp(ts: string): number {
|
||||||
|
const parts = ts.split(":");
|
||||||
|
if (parts.length === 3) return parseSrtTimestamp(ts);
|
||||||
|
// MM:SS.mmm
|
||||||
|
if (parts.length === 2) {
|
||||||
|
const [min, secMs] = parts;
|
||||||
|
const [sec, ms] = (secMs ?? "0.0").split(".");
|
||||||
|
return (
|
||||||
|
parseInt(min!, 10) * 60 + parseInt(sec!, 10) + parseInt((ms ?? "0").padEnd(3, "0"), 10) / 1000
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
function round3(n: number): number {
|
||||||
|
return Math.round(n * 1000) / 1000;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Public API
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Load and normalize a transcript file to a standard word array.
|
||||||
|
*
|
||||||
|
* Supports:
|
||||||
|
* - whisper.cpp JSON (--output-json-full with --dtw)
|
||||||
|
* - OpenAI Whisper API response (verbose_json with word timestamps)
|
||||||
|
* - SRT subtitle files (phrase-level, not word-level)
|
||||||
|
* - VTT subtitle files (phrase-level, not word-level)
|
||||||
|
* - Pre-normalized JSON array ([{text, start, end}])
|
||||||
|
*/
|
||||||
|
export function loadTranscript(filePath: string): { words: Word[]; format: TranscriptFormat } {
|
||||||
|
const ext = extname(filePath).toLowerCase();
|
||||||
|
const content = readFileSync(filePath, "utf-8");
|
||||||
|
|
||||||
|
if (ext === ".srt") return { words: parseSrt(content), format: "srt" };
|
||||||
|
if (ext === ".vtt") return { words: parseVtt(content), format: "vtt" };
|
||||||
|
|
||||||
|
// JSON formats — parse once, detect, then extract words
|
||||||
|
const parsed = JSON.parse(content);
|
||||||
|
const format = detectJsonFormat(parsed);
|
||||||
|
|
||||||
|
const words =
|
||||||
|
format === "whisper-cpp"
|
||||||
|
? parseWhisperCpp(parsed)
|
||||||
|
: format === "openai"
|
||||||
|
? parseOpenAI(parsed)
|
||||||
|
: (parsed as Word[]).map((w) => ({
|
||||||
|
text: w.text.trim(),
|
||||||
|
start: round3(w.start),
|
||||||
|
end: round3(w.end),
|
||||||
|
}));
|
||||||
|
|
||||||
|
return { words, format };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Patch caption HTML files in a project directory with transcript words.
|
||||||
|
* Replaces `const script = [...]` or `const TRANSCRIPT = [...]` in <script> blocks.
|
||||||
|
*/
|
||||||
|
export function patchCaptionHtml(dir: string, words: Word[]): void {
|
||||||
|
if (words.length === 0) return;
|
||||||
|
|
||||||
|
// Indent to 10 spaces to match typical composition script indentation
|
||||||
|
const wordsJson = JSON.stringify(words, null, 2).replace(/\n/g, "\n ");
|
||||||
|
|
||||||
|
let htmlFiles: string[];
|
||||||
|
try {
|
||||||
|
htmlFiles = readdirSync(dir, { withFileTypes: true, recursive: true })
|
||||||
|
.filter((e) => e.isFile() && e.name.endsWith(".html"))
|
||||||
|
.map((e) => join(e.parentPath ?? e.path, e.name));
|
||||||
|
} catch {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const file of htmlFiles) {
|
||||||
|
let content = readFileSync(file, "utf-8");
|
||||||
|
const scriptBlocks = content.match(/<script>[\s\S]*?<\/script>/g) ?? [];
|
||||||
|
let scriptMatch: RegExpMatchArray | null = null;
|
||||||
|
let transcriptMatch: RegExpMatchArray | null = null;
|
||||||
|
for (const block of scriptBlocks) {
|
||||||
|
scriptMatch = scriptMatch ?? block.match(/const script = \[[\s\S]*?\];/);
|
||||||
|
transcriptMatch = transcriptMatch ?? block.match(/const TRANSCRIPT = \[[\s\S]*?\];/);
|
||||||
|
}
|
||||||
|
const match = scriptMatch ?? transcriptMatch;
|
||||||
|
if (match) {
|
||||||
|
const varName = scriptMatch ? "script" : "TRANSCRIPT";
|
||||||
|
content = content.replace(match[0], `const ${varName} = ${wordsJson};`);
|
||||||
|
writeFileSync(file, content, "utf-8");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -9,6 +9,7 @@ const VIDEO_EXTENSIONS = new Set([".mp4", ".webm", ".mov", ".mkv", ".avi"]);
|
|||||||
|
|
||||||
export interface TranscribeOptions {
|
export interface TranscribeOptions {
|
||||||
model?: string;
|
model?: string;
|
||||||
|
language?: string;
|
||||||
onProgress?: (message: string) => void;
|
onProgress?: (message: string) => void;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -125,21 +126,22 @@ export async function transcribe(
|
|||||||
const outputBase = join(outputDir, "transcript");
|
const outputBase = join(outputDir, "transcript");
|
||||||
mkdirSync(outputDir, { recursive: true });
|
mkdirSync(outputDir, { recursive: true });
|
||||||
|
|
||||||
execFileSync(
|
const whisperArgs = [
|
||||||
whisper.executablePath,
|
"--model",
|
||||||
[
|
modelPath,
|
||||||
"--model",
|
"--output-json-full",
|
||||||
modelPath,
|
"--output-file",
|
||||||
"--output-json-full",
|
outputBase,
|
||||||
"--output-file",
|
"--dtw",
|
||||||
outputBase,
|
model,
|
||||||
"--dtw",
|
"--suppress-nst",
|
||||||
model,
|
];
|
||||||
"--suppress-nst",
|
if (options?.language) {
|
||||||
wavPath,
|
whisperArgs.push("--language", options.language);
|
||||||
],
|
}
|
||||||
{ stdio: "ignore", timeout: 300_000 },
|
whisperArgs.push(wavPath);
|
||||||
);
|
|
||||||
|
execFileSync(whisper.executablePath, whisperArgs, { stdio: "ignore", timeout: 300_000 });
|
||||||
|
|
||||||
// 5. Read and validate output
|
// 5. Read and validate output
|
||||||
const transcriptPath = `${outputBase}.json`;
|
const transcriptPath = `${outputBase}.json`;
|
||||||
|
|||||||
@@ -603,4 +603,126 @@ describe("template_literal_selector rule", () => {
|
|||||||
const finding = result.findings.find((f) => f.code === "template_literal_selector");
|
const finding = result.findings.find((f) => f.code === "template_literal_selector");
|
||||||
expect(finding).toBeUndefined();
|
expect(finding).toBeUndefined();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ── Caption lint rules ────────────────────────────────────────────────
|
||||||
|
|
||||||
|
it("warns when caption exit has no hard kill tl.set", () => {
|
||||||
|
const html = `
|
||||||
|
<html><body>
|
||||||
|
<div data-composition-id="captions" data-width="1920" data-height="1080">
|
||||||
|
<div id="caption-container"></div>
|
||||||
|
<script>
|
||||||
|
window.__timelines = window.__timelines || {};
|
||||||
|
var tl = gsap.timeline({ paused: true });
|
||||||
|
GROUPS.forEach(function(group, gi) {
|
||||||
|
var groupEl = document.createElement("div");
|
||||||
|
groupEl.id = "cg-" + gi;
|
||||||
|
tl.set(groupEl, { opacity: 1 }, group.start);
|
||||||
|
tl.to(groupEl, { opacity: 0, duration: 0.12 }, group.end - 0.12);
|
||||||
|
});
|
||||||
|
window.__timelines["captions"] = tl;
|
||||||
|
</script>
|
||||||
|
</div>
|
||||||
|
</body></html>`;
|
||||||
|
const result = lintHyperframeHtml(html);
|
||||||
|
const finding = result.findings.find((f) => f.code === "caption_exit_missing_hard_kill");
|
||||||
|
expect(finding).toBeDefined();
|
||||||
|
expect(finding?.severity).toBe("warning");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not warn when caption exit has hard kill tl.set", () => {
|
||||||
|
const html = `
|
||||||
|
<html><body>
|
||||||
|
<div data-composition-id="captions" data-width="1920" data-height="1080">
|
||||||
|
<div id="caption-container"></div>
|
||||||
|
<script>
|
||||||
|
window.__timelines = window.__timelines || {};
|
||||||
|
var tl = gsap.timeline({ paused: true });
|
||||||
|
GROUPS.forEach(function(group, gi) {
|
||||||
|
var groupEl = document.createElement("div");
|
||||||
|
groupEl.id = "cg-" + gi;
|
||||||
|
tl.set(groupEl, { opacity: 1 }, group.start);
|
||||||
|
tl.to(groupEl, { opacity: 0, duration: 0.12 }, group.end - 0.12);
|
||||||
|
tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end);
|
||||||
|
});
|
||||||
|
window.__timelines["captions"] = tl;
|
||||||
|
</script>
|
||||||
|
</div>
|
||||||
|
</body></html>`;
|
||||||
|
const result = lintHyperframeHtml(html);
|
||||||
|
const finding = result.findings.find((f) => f.code === "caption_exit_missing_hard_kill");
|
||||||
|
expect(finding).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("warns when caption group has nowrap without max-width", () => {
|
||||||
|
const html = `
|
||||||
|
<html><body>
|
||||||
|
<div data-composition-id="captions" data-width="1920" data-height="1080">
|
||||||
|
<style>
|
||||||
|
.caption-group {
|
||||||
|
position: absolute;
|
||||||
|
white-space: nowrap;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
<script>
|
||||||
|
window.__timelines = window.__timelines || {};
|
||||||
|
var tl = gsap.timeline({ paused: true });
|
||||||
|
window.__timelines["captions"] = tl;
|
||||||
|
</script>
|
||||||
|
</div>
|
||||||
|
</body></html>`;
|
||||||
|
const result = lintHyperframeHtml(html);
|
||||||
|
const finding = result.findings.find((f) => f.code === "caption_text_overflow_risk");
|
||||||
|
expect(finding).toBeDefined();
|
||||||
|
expect(finding?.severity).toBe("warning");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not warn when caption group has nowrap with max-width", () => {
|
||||||
|
const html = `
|
||||||
|
<html><body>
|
||||||
|
<div data-composition-id="captions" data-width="1920" data-height="1080">
|
||||||
|
<style>
|
||||||
|
.caption-group {
|
||||||
|
position: absolute;
|
||||||
|
white-space: nowrap;
|
||||||
|
max-width: 1600px;
|
||||||
|
overflow: hidden;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
<script>
|
||||||
|
window.__timelines = window.__timelines || {};
|
||||||
|
var tl = gsap.timeline({ paused: true });
|
||||||
|
window.__timelines["captions"] = tl;
|
||||||
|
</script>
|
||||||
|
</div>
|
||||||
|
</body></html>`;
|
||||||
|
const result = lintHyperframeHtml(html);
|
||||||
|
const finding = result.findings.find(
|
||||||
|
(f) => f.code === "caption_text_overflow_risk" && f.severity === "warning",
|
||||||
|
);
|
||||||
|
expect(finding).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("warns when caption container uses position: relative", () => {
|
||||||
|
const html = `
|
||||||
|
<html><body>
|
||||||
|
<div data-composition-id="captions" data-width="1920" data-height="1080">
|
||||||
|
<style>
|
||||||
|
.caption-group {
|
||||||
|
position: relative;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
<script>
|
||||||
|
window.__timelines = window.__timelines || {};
|
||||||
|
var tl = gsap.timeline({ paused: true });
|
||||||
|
window.__timelines["captions"] = tl;
|
||||||
|
</script>
|
||||||
|
</div>
|
||||||
|
</body></html>`;
|
||||||
|
const result = lintHyperframeHtml(html);
|
||||||
|
const finding = result.findings.find((f) => f.code === "caption_container_relative_position");
|
||||||
|
expect(finding).toBeDefined();
|
||||||
|
expect(finding?.severity).toBe("warning");
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -710,6 +710,79 @@ export function lintHyperframeHtml(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── Caption lint rules ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
// Rule: caption_exit_missing_hard_kill
|
||||||
|
// Exit tweens (tl.to with opacity: 0) can fail when karaoke word-level tweens
|
||||||
|
// conflict, leaving captions stuck on screen. A hard tl.set kill is needed.
|
||||||
|
for (const script of scripts) {
|
||||||
|
const content = script.content;
|
||||||
|
const hasExitTween = /\.to\s*\([^,]+,\s*\{[^}]*opacity\s*:\s*0/.test(content);
|
||||||
|
const hasHardKill =
|
||||||
|
/\.set\s*\([^,]+,\s*\{[^}]*(?:visibility\s*:\s*["']hidden["']|opacity\s*:\s*0)/.test(content);
|
||||||
|
const hasCaptionLoop =
|
||||||
|
/forEach|\.forEach\s*\(/.test(content) && /createElement|caption|group|cg-/.test(content);
|
||||||
|
|
||||||
|
if (hasCaptionLoop && hasExitTween && !hasHardKill) {
|
||||||
|
pushFinding({
|
||||||
|
code: "caption_exit_missing_hard_kill",
|
||||||
|
severity: "warning",
|
||||||
|
message:
|
||||||
|
"Caption exit animations (tl.to with opacity: 0) detected without a hard tl.set kill. " +
|
||||||
|
"Exit tweens can fail when karaoke word-level tweens conflict, leaving captions stuck on screen.",
|
||||||
|
fixHint:
|
||||||
|
'Add `tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end)` after every ' +
|
||||||
|
"exit tl.to animation as a deterministic kill.",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rule: caption_text_overflow_risk
|
||||||
|
// Captions with nowrap text and no max-width will clip off-screen.
|
||||||
|
for (const style of styles) {
|
||||||
|
const content = style.content;
|
||||||
|
const captionBlocks = content.matchAll(
|
||||||
|
/(\.caption[-_]?(?:group|container|text|line|word)|#caption[-_]?container)\s*\{([^}]+)\}/gi,
|
||||||
|
);
|
||||||
|
for (const [, selector, body] of captionBlocks) {
|
||||||
|
if (!body) continue;
|
||||||
|
const hasNowrap = /white-space\s*:\s*nowrap/i.test(body);
|
||||||
|
const hasMaxWidth = /max-width/i.test(body);
|
||||||
|
|
||||||
|
if (hasNowrap && !hasMaxWidth) {
|
||||||
|
pushFinding({
|
||||||
|
code: "caption_text_overflow_risk",
|
||||||
|
severity: "warning",
|
||||||
|
selector: (selector ?? "").trim(),
|
||||||
|
message: `Caption selector "${(selector ?? "").trim()}" has white-space: nowrap but no max-width. Long phrases will clip off-screen.`,
|
||||||
|
fixHint:
|
||||||
|
"Add max-width: 1600px (landscape) or max-width: 900px (portrait) and overflow: hidden.",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rule: caption_container_relative_position
|
||||||
|
// position: relative on caption containers causes overflow and stacking issues.
|
||||||
|
for (const style of styles) {
|
||||||
|
const content = style.content;
|
||||||
|
const captionBlocks = content.matchAll(
|
||||||
|
/(\.caption[-_]?(?:group|container|text|line)|#caption[-_]?container)\s*\{([^}]+)\}/gi,
|
||||||
|
);
|
||||||
|
for (const [, selector, body] of captionBlocks) {
|
||||||
|
if (!body) continue;
|
||||||
|
if (/position\s*:\s*relative/i.test(body)) {
|
||||||
|
pushFinding({
|
||||||
|
code: "caption_container_relative_position",
|
||||||
|
severity: "warning",
|
||||||
|
selector: (selector ?? "").trim(),
|
||||||
|
message: `Caption selector "${(selector ?? "").trim()}" uses position: relative which causes overflow and breaks caption stacking.`,
|
||||||
|
fixHint: "Use position: absolute for all caption elements.",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ── External CDN script dependency check ────────────────────────────────
|
// ── External CDN script dependency check ────────────────────────────────
|
||||||
// Compositions that load CDN libraries via <script src="https://..."> work
|
// Compositions that load CDN libraries via <script src="https://..."> work
|
||||||
// correctly in bundled mode (bundleToSingleHtml auto-hoists them to the parent
|
// correctly in bundled mode (bundleToSingleHtml auto-hoists them to the parent
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
---
|
---
|
||||||
name: hyperframes-captions
|
name: hyperframes-captions
|
||||||
description: Build tone-adaptive captions from whisper transcripts. Detects script energy (hype, corporate, tutorial, storytelling, social) and applies matching typography, color, and animation. Supports per-word styling for brand names, ALL CAPS, numbers, and CTAs. Use when adding captions or subtitles to a HyperFrames composition.
|
description: Build tone-adaptive captions from whisper transcripts. Detects script energy (hype, corporate, tutorial, storytelling, social) and applies matching typography, color, and animation. Supports per-word styling for brand names, ALL CAPS, numbers, and CTAs. Use when adding captions, subtitles, or lyrics to a HyperFrames composition. Lyric videos ARE captions — any text synced to audio uses this skill.
|
||||||
|
trigger: Use this skill whenever a task involves syncing text to audio timing. This includes captions, subtitles, lyrics, karaoke, transcription overlays, and any word-level or phrase-level text timed to speech or music.
|
||||||
---
|
---
|
||||||
|
|
||||||
# Captions
|
# Captions
|
||||||
@@ -9,41 +10,113 @@ Analyze the spoken content to determine caption style. If the user specifies a s
|
|||||||
|
|
||||||
## Transcript Source
|
## Transcript Source
|
||||||
|
|
||||||
The project's `transcript.json` contains word-level timestamps from whisper.cpp (`--output-json-full` with `--dtw`):
|
The project's `transcript.json` contains a normalized word array with word-level timestamps:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
[
|
||||||
"transcription": [
|
{ "text": "Hello", "start": 0.0, "end": 0.5 },
|
||||||
{
|
{ "text": "world.", "start": 0.6, "end": 1.2 }
|
||||||
"offsets": { "from": 0, "to": 5000 },
|
]
|
||||||
"text": " Hello world.",
|
|
||||||
"tokens": [
|
|
||||||
{ "text": " Hello", "offsets": { "from": 0, "to": 1000 }, "p": 0.98 },
|
|
||||||
{ "text": " world", "offsets": { "from": 1000, "to": 2000 }, "p": 0.95 }
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Normalize tokens into a word array before grouping:
|
This is the only format the captions composition consumes. Use it directly:
|
||||||
|
|
||||||
```js
|
```js
|
||||||
const words = [];
|
const words = JSON.parse(transcriptJson); // [{ text, start, end }]
|
||||||
for (const segment of transcript.transcription) {
|
|
||||||
for (const token of segment.tokens || []) {
|
|
||||||
const text = token.text.trim();
|
|
||||||
if (!text) continue;
|
|
||||||
words.push({
|
|
||||||
text,
|
|
||||||
start: token.offsets.from / 1000,
|
|
||||||
end: token.offsets.to / 1000,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
If no `transcript.json` exists, check for `.srt` or `.vtt` files. If no transcript is available, ask the user to provide one or run `hyperframes transcribe` (when available).
|
### How transcripts are generated
|
||||||
|
|
||||||
|
`hyperframes transcribe` handles both transcription and format conversion:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Transcribe audio/video (uses whisper.cpp locally, no API key needed)
|
||||||
|
npx hyperframes transcribe audio.mp3
|
||||||
|
|
||||||
|
# Use a larger model for better accuracy
|
||||||
|
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||||
|
|
||||||
|
# Filter to English only (skips non-English speech)
|
||||||
|
npx hyperframes transcribe audio.mp3 --language en
|
||||||
|
|
||||||
|
# Import an existing transcript from another tool
|
||||||
|
npx hyperframes transcribe captions.srt
|
||||||
|
npx hyperframes transcribe captions.vtt
|
||||||
|
npx hyperframes transcribe openai-response.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Supported input formats
|
||||||
|
|
||||||
|
The CLI auto-detects and normalizes these formats:
|
||||||
|
|
||||||
|
| Format | Extension | Source | Word-level? |
|
||||||
|
| --------------------- | --------- | --------------------------------------------------------------------------- | ----------------- |
|
||||||
|
| whisper.cpp JSON | `.json` | `hyperframes init --video`, `hyperframes transcribe` | Yes |
|
||||||
|
| OpenAI Whisper API | `.json` | `openai.audio.transcriptions.create({ timestamp_granularities: ["word"] })` | Yes |
|
||||||
|
| SRT subtitles | `.srt` | Video editors, subtitle tools, YouTube | No (phrase-level) |
|
||||||
|
| VTT subtitles | `.vtt` | Web players, YouTube, transcription services | No (phrase-level) |
|
||||||
|
| Normalized word array | `.json` | Pre-processed by any tool | Yes |
|
||||||
|
|
||||||
|
**Word-level timestamps produce better captions.** SRT/VTT give phrase-level timing, which works but can't do per-word animation effects.
|
||||||
|
|
||||||
|
### Whisper model guide
|
||||||
|
|
||||||
|
The default model (`small.en`) balances accuracy and speed. For better results, use a larger model:
|
||||||
|
|
||||||
|
| Model | Size | Speed | Accuracy | When to use |
|
||||||
|
| ----------- | ------ | -------- | --------- | ------------------------------------- |
|
||||||
|
| `tiny.en` | 75 MB | Fastest | Low | Quick previews, testing pipeline |
|
||||||
|
| `base.en` | 142 MB | Fast | Fair | Short clips, clear audio |
|
||||||
|
| `small.en` | 466 MB | Moderate | Good | **Default** — good for most content |
|
||||||
|
| `medium.en` | 1.5 GB | Slow | Very good | Important content, noisy audio, music |
|
||||||
|
| `large-v3` | 3.1 GB | Slowest | Best | Multilingual, production captions |
|
||||||
|
|
||||||
|
`.en` models are English-only and more accurate for English. Drop the `.en` suffix for multilingual (e.g., `medium` instead of `medium.en`).
|
||||||
|
|
||||||
|
**Music and vocals over instrumentation**: `small.en` will misidentify lyrics — use `medium.en` as the minimum, or import lyrics manually. Even `medium.en` struggles with heavily produced tracks; for music videos, providing known lyrics as an SRT/VTT and importing with `hyperframes transcribe lyrics.srt` will always beat automated transcription.
|
||||||
|
|
||||||
|
### Using external transcription APIs
|
||||||
|
|
||||||
|
For the best accuracy, use an external API and import the result:
|
||||||
|
|
||||||
|
**OpenAI Whisper API** (recommended for quality):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate with word timestamps, then import
|
||||||
|
curl https://api.openai.com/v1/audio/transcriptions \
|
||||||
|
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||||||
|
-F file=@audio.mp3 -F model=whisper-1 \
|
||||||
|
-F response_format=verbose_json \
|
||||||
|
-F "timestamp_granularities[]=word" \
|
||||||
|
-o transcript-openai.json
|
||||||
|
|
||||||
|
npx hyperframes transcribe transcript-openai.json
|
||||||
|
```
|
||||||
|
|
||||||
|
**Groq Whisper API** (fast, free tier available):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl https://api.groq.com/openai/v1/audio/transcriptions \
|
||||||
|
-H "Authorization: Bearer $GROQ_API_KEY" \
|
||||||
|
-F file=@audio.mp3 -F model=whisper-large-v3 \
|
||||||
|
-F response_format=verbose_json \
|
||||||
|
-F "timestamp_granularities[]=word" \
|
||||||
|
-o transcript-groq.json
|
||||||
|
|
||||||
|
npx hyperframes transcribe transcript-groq.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### If no transcript exists
|
||||||
|
|
||||||
|
1. Check the project root for `transcript.json`, `.srt`, or `.vtt` files
|
||||||
|
2. If none found, ask the user to provide one or run:
|
||||||
|
```bash
|
||||||
|
npx hyperframes transcribe <audio-or-video-file>
|
||||||
|
```
|
||||||
|
3. If transcription quality is poor (words at wrong times, gibberish), suggest upgrading the model:
|
||||||
|
```bash
|
||||||
|
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||||
|
```
|
||||||
|
|
||||||
## Style Detection (Default — When No Style Is Specified)
|
## Style Detection (Default — When No Style Is Specified)
|
||||||
|
|
||||||
@@ -130,9 +203,46 @@ Break groups on sentence boundaries (`.` `?` `!`), pauses (>150ms gap), or max w
|
|||||||
- Use `position: absolute` — never relative (causes overflow)
|
- Use `position: absolute` — never relative (causes overflow)
|
||||||
- One caption group visible at a time
|
- One caption group visible at a time
|
||||||
|
|
||||||
|
## Text Overflow Prevention
|
||||||
|
|
||||||
|
Captions must never clip off-screen. Apply these rules:
|
||||||
|
|
||||||
|
- Set `max-width: 1600px` (landscape) or `max-width: 900px` (portrait) on caption container
|
||||||
|
- Add `overflow: hidden` as a safety net
|
||||||
|
- **Auto-scale font size** based on character count:
|
||||||
|
- ≤18 chars → full size (e.g., 78px)
|
||||||
|
- 19–25 chars → reduce ~15% (e.g., 68px)
|
||||||
|
- 26+ chars → reduce ~25% (e.g., 58px)
|
||||||
|
- Reduce `letter-spacing` for long text (switch from `-0.02em` to `-0.04em`)
|
||||||
|
- Give the caption container an explicit `height` (e.g., `200px`) — don't rely on content sizing with absolute children
|
||||||
|
- Use `position: absolute` on all caption elements — `position: relative` causes overflow
|
||||||
|
|
||||||
|
## Caption Exit Guarantee
|
||||||
|
|
||||||
|
Captions that stick on screen are the most common caption bug. Every caption group **must** have a hard kill after its exit animation.
|
||||||
|
|
||||||
|
**The pattern:**
|
||||||
|
|
||||||
|
```js
|
||||||
|
// Animate exit (soft — can fail if tweens conflict)
|
||||||
|
tl.to(groupEl, { opacity: 0, scale: 0.95, duration: 0.12, ease: "power2.in" }, group.end - 0.12);
|
||||||
|
|
||||||
|
// Hard kill at group.end (deterministic — guarantees invisible)
|
||||||
|
tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end);
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why both?** The `tl.to` exit can fail to fully hide a group when:
|
||||||
|
|
||||||
|
- Karaoke word-level tweens (`scale`, `color`) on child elements conflict with the parent exit tween
|
||||||
|
- `fromTo` entrance tweens lock start/end values that override later tweens on the same property
|
||||||
|
- Timeline scrubbing lands between the exit start and end
|
||||||
|
|
||||||
|
The `tl.set` at `group.end` is a deterministic kill — it fires at an exact time, doesn't animate, and can't be overridden by other tweens at different times.
|
||||||
|
|
||||||
## Constraints
|
## Constraints
|
||||||
|
|
||||||
- **Deterministic.** No `Math.random()`, no `Date.now()`.
|
- **Deterministic.** No `Math.random()`, no `Date.now()`.
|
||||||
- **Sync to transcript timestamps.** Words appear when spoken.
|
- **Sync to transcript timestamps.** Words appear when spoken.
|
||||||
- **One group visible at a time.** No overlapping caption groups.
|
- **One group visible at a time.** No overlapping caption groups.
|
||||||
|
- **Every caption group must have a hard `tl.set` kill at `group.end`.** Exit animations alone are not sufficient.
|
||||||
- **Check project root** for font files before defaulting to Google Fonts.
|
- **Check project root** for font files before defaulting to Google Fonts.
|
||||||
|
|||||||
Reference in New Issue
Block a user