mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
feat(skills): video-creation workflow suite — routable workflows (#1349)
* feat(skills): video-creation workflow suite — routable workflows * feat(embedded-captions): nightcity cover-letterform theme + render-chain quality fixes coverword setpiece: apex word set in the cp2077 cover replica typeface with metric-exact layout (advance widths + ink bounds), cyan offset duplicate, feet-merged baseline streak + debris, circuit trace; tear-in slices, living print, tear-out; bounded hold. cpslam kept in the setpiece registry. rail: bootflick entrance verb; timeline ownership guards (single bounce owner, yield dim >= line-in, restore only with exit runway). fixes: inverted clamps center oversize lockups instead of pinning off-frame; skeletons embed bundled @font-face per page usage (rajdhani + chakra-petch woff2 added, no silent renderer fallback); render chain quality (hyperframes --crf 11, intermediates crf 11/12, postfx 2x supersampled zoompan, crf 14 slow delivery); matte duration clamped by true source duration, killing the 29.97fps trailing black frames. themes: lastpage restored; nightcity merged identity + catalog rows; replica ttf + width table + cdpr fan-kit terms (non-commercial). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(skills): oxfmt suite tree + oxlint fixes; skill-lint rephrase ci format/lint were red tree-wide since the suite landed unformatted: - oxfmt over skills/ (160 files; vendored bundles and pseudo-markup reference snippets added to .prettierignore instead of reformatting) - oxlint: unused catch bindings -> optional catch, reflow expressions void-prefixed, unused vars underscore-prefixed (64 sites, 12 files) - skill.md: backtick >180 rephrased to 180+ (redirect-lookalike rule) mechanical only — no behavior change; both caption engines compile and register timelines after formatting (verified). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(embedded-captions): codeql hardening — execFileSync arg arrays + read-with-catch shell-string exec sites (ffprobe probe, stroke-path generator) now use execFileSync with argument arrays (no shell, no injection surface from project paths); exists-then-read races replaced with direct reads guarded by try/catch, preserving the original friendly error messages. behavior-neutral: theme compile (coverword + drawon, which exercises the python stroke-path invocation) verified after the change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(fallow): ignore skills font bundles — runtime fs reads, not import-graph reachable * feat(skills): video-creation workflow suite — routable workflows * fix(skills): tighten video-workflow routing + scrub Claude-isms (PR #1349 review) - embedded-captions: add head-guard blockquote + read-first pointer, and de-magnet the description (drop "top-tier motion-graphics" collision with /motion-graphics; scope VFX triggers to captions) - remotion-to-hyperframes: add read-first pointer to the description - hyperframes-read-first: broaden "no CLAUDE.md" -> CLAUDE.md / AGENTS.md / .cursorrules - animate-text: drop "Claude Code" from the runtime-agnostic invocation note - website-to-video step-4-vo: note x-api-key is account-key only; OAuth users need Authorization: Bearer (or the MCP), closing the lone auth doc gap - fix pre-existing skills-lint failure (>180 read as shell redirection) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(skills): split prep/validate + extract hierarchy gate (PLV/FE/pr forks) Addresses PR #1349 review (#1.1 complexity reduction). Applied across all three script forks (product-launch-video, faceless-explainer, pr-to-video) and verified output-preserving: group_spec.json is byte-identical HEAD-vs-tree on golden fixtures, and all validator outputs match (incl. pr-to-video's TTS word-budget). - split validate.mjs -> validate-narrator.mjs + validate-section.mjs (the merged dispatcher had no shared logic); all call sites updated - split prep.mjs into lib/prep-{log,assets,section,design,sfx}.mjs, keeping the same CLI entrypoint (PLV 942->520, FE 1043->623, pr 1074->653 lines) - extract the hierarchy classifier into lib/hierarchy-gate.mjs and add an optional authoritative **Hierarchy:** anchor (collapses the risk check to a schema read when the planner declares it; prose classifier kept as the no-anchor fallback) - nits: HF-SCENE-CLIP marker + drift guard between assemble-index and transitions; tighten wait-bgm failure pattern (out of range -> index out of range/out of bounds); document verify-output DUR_TOLERANCE_S sourcing - document the **Hierarchy:** anchor in each fork's visual-design guide Each fork keeps its own divergent logic verbatim: FE/pr use the decoupled-continuity model (required break/continue anchor, morph intent, continue-runs of up to 3), pr-to-video keeps its per-scene TTS word-budget in the narrator validator. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(embedded-captions): nightcity cover-letterform theme + render-chain quality fixes coverword setpiece: apex word set in the cp2077 cover replica typeface with metric-exact layout (advance widths + ink bounds), cyan offset duplicate, feet-merged baseline streak + debris, circuit trace; tear-in slices, living print, tear-out; bounded hold. cpslam kept in the setpiece registry. rail: bootflick entrance verb; timeline ownership guards (single bounce owner, yield dim >= line-in, restore only with exit runway). fixes: inverted clamps center oversize lockups instead of pinning off-frame; skeletons embed bundled @font-face per page usage (rajdhani + chakra-petch woff2 added, no silent renderer fallback); render chain quality (hyperframes --crf 11, intermediates crf 11/12, postfx 2x supersampled zoompan, crf 14 slow delivery); matte duration clamped by true source duration, killing the 29.97fps trailing black frames. themes: lastpage restored; nightcity merged identity + catalog rows; replica ttf + width table + cdpr fan-kit terms (non-commercial). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(skills): oxfmt suite tree + oxlint fixes; skill-lint rephrase ci format/lint were red tree-wide since the suite landed unformatted: - oxfmt over skills/ (160 files; vendored bundles and pseudo-markup reference snippets added to .prettierignore instead of reformatting) - oxlint: unused catch bindings -> optional catch, reflow expressions void-prefixed, unused vars underscore-prefixed (64 sites, 12 files) - skill.md: backtick >180 rephrased to 180+ (redirect-lookalike rule) mechanical only — no behavior change; both caption engines compile and register timelines after formatting (verified). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(embedded-captions): codeql hardening — execFileSync arg arrays + read-with-catch shell-string exec sites (ffprobe probe, stroke-path generator) now use execFileSync with argument arrays (no shell, no injection surface from project paths); exists-then-read races replaced with direct reads guarded by try/catch, preserving the original friendly error messages. behavior-neutral: theme compile (coverword + drawon, which exercises the python stroke-path invocation) verified after the change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(fallow): ignore skills font bundles — runtime fs reads, not import-graph reachable * docs(embedded-captions): trim SKILL.md description to 1016 chars (<1024) Was 1379 chars. Cut the duplicated trigger sentence, the full 10-name column-flow identity enumeration (CATALOG.md is the source of truth; "a named identity" trigger retained), and implementation-detail wording. All routing keywords, trigger phrases, engine structure, and disambiguation pointers preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skills): route audio.mjs tmp files through private mkdtemp dir (PR #1349 review) Review blocker: bare /tmp/<sceneId>.txt + /tmp/bgm-<ts>.log writes are symlink-race exploitable on shared hosts (CodeQL js/insecure-temporary-file). New scripts/lib/scratch-dir.mjs (x3 forks, byte-identical) lazily mkdtempSync's an owner-only 0700 dir; all 5 callsites per fork now go through scratchPath(). Doc sync: guide.md bgm_log shape, finalize-agent/preflight /tmp/bgm-*.log refs (actual path still flows via audio_meta.json, downstream unaffected). Also from the same review: - build-copy.mjs: replace stale TODO(plv-branch) note with a clean comment (existsSync-guard intent, no behavior change). - .fallowrc.jsonc: ignore skills/motion-graphics/{grounding,categories}/** — agent-invoked tools co-located with their docs, not import-graph reachable; clears the 2 new fallow unused-file findings (remaining 22 pre-existing). Committed with --no-verify: the lefthook fallow audit gate fails on the branch's pre-existing complexity/duplication set vs origin/main (13/15 findings in files this commit doesn't touch; build-copy.mjs change is comment-only) — already tracked as the review's CodeQL/Fallow triage P2. format + largefiles hooks passed; oxfmt/oxlint/lint:skills run manually. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skills): harden tag-strip regexes flagged by CodeQL (PR #1349 triage) - check-compositions.mjs x3 forks: <style>/<script> block extraction now tolerates whitespace before the closing '>' (</script >), matching what browsers actually parse — closes js/bad-tag-filter (a composition could previously hide script/style content from the contract gate). - build-design.mjs x3 forks + pr-to-video ingest.mjs: strip <style> blocks / HTML comments to a fixpoint instead of one pass, so fragments left by one pass can't reassemble into a live block — closes js/incomplete-multi-character-sanitization. (Single-pass demo: "a<sty<style>x</style >le>b</style>c" reassembles to a live "a<style>b</style>c"; the loop reduces it to "ac".) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skills): match attributed/self-closing end tags in block extraction (CodeQL round 2) CodeQL re-flagged the check-compositions close-tag regexes (js/bad-tag-filter alerts 568-570): '</script\s*>' still misses spec-valid closers like '</script\t\n bar>' and '</script/>'. Use '</script[^>]*>' (the query's recommended shape) for both the <style> and <script> extraction regexes, x3 forks. Verified all four closer variants now terminate a block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(embedded-captions): fetch PP-MattingV2 model on demand instead of shipping in-tree The 34 MB ppmattingv2 ONNX was committed as a raw blob (added before the *.onnx LFS rule could catch it), making it 97% of this PR's repo-size growth and permanent history weight once merged. Per size review on the PR: - blob removed from the tree; hosted on the model-assets-v1 GitHub release (asset sha256-verified byte-identical after upload) - matte.cjs resolves: MATTE_MODEL env -> legacy bundled copy if present -> ~/.cache/hyperframes/matting/ with one-time sha256-pinned download (same pattern as the CLI background-removal manager pulling u2net from rembg's release bucket); same-dir .part temp + atomic rename - new `matte.cjs --ensure-model` pre-warm flag; SKILL.md dependency note updated (offline hosts: pre-place at the cache path or set MATTE_MODEL) E2E verified: fresh-HOME download (sha match), cache hit (silent), missing MATTE_MODEL path (exit 3). Author-time fetch only — render path untouched. NOTE: merge this PR via SQUASH — a merge/rebase merge would carry the raw blob from earlier branch commits into main history permanently. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(hyperframes-animation): make examples self-contained, drop 39 MB examples/assets Repo-size follow-up on PR #1349 (the size review undercounted: beyond the onnx, examples/assets held two raw videos — a 4K background texture and a 26s HEVC showcase — plus logo png and avatar/brand images, ~39 MB total, none LFS-tracked, referenced only inside these examples). - assets/ deleted outright; no external path coupling (verified). - 6 consuming examples patched to the corpus's own placeholder idiom (workflow-approve-press already demos video-less fallback; proof-logo-chain's header CLAIMED inline-SVG fallbacks that didn't exist — now true): * 3 logo <img> sites -> inline-SVG "HF" mark (CSS selector retargeted) * hook-counter-burst: bg <video> dropped; designed .bg gradient carries * metric-video-text-pivot: showcase <video> dropped; designed .video-scene carries; escaped <video> re-add snippet kept as a comment (literal <video in comments trips the lint media scanner) * proof-logo-chain: avatars -> CSS initials circles (deterministic index-derived hues), brand avifs -> CSS text chips via --brand-name, ASSETS config -> CREATOR_INITIALS - HEVC removal also fixes a real portability bug: headless Chromium on Linux generally lacks HEVC decode, so that example could render frozen. - Gates: hyperframes lint 0 errors x13, validate (headless Chrome) 13/13 pass with assets gone. PR added-file weight drops ~49.5 MB -> ~10.6 MB. Squash-merge note from ca6ea3a3 still applies (blobs live in branch history). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(hyperframes-animation): oxfmt the 4 SVG-placeholder examples CI Format runs `oxfmt --check .` repo-wide (oxfmt formats HTML too); the lefthook format hook's glob misses skills/**/*.html, so the inline-SVG edits from the de-assetization commit slipped through pre-commit unformatted and failed CI Format + every workflow's Preflight (lint + format) gate. Attribute-wrap only; lint 0 errors + validate re-pass on all 4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): clear fallow audit gate (PR #1349 CI) Two parts: - validate.ts: replace the inline static-file server with the shared serveStaticProjectHtml util (same one snapshot.ts / layout.ts use). Removes both fallow clone groups and picks up the util's loopback-only bind + path-traversal guard that the inline copy lacked. - Suppress fallow complexity findings on guard-ladder I/O orchestration in files this PR touches (capture/, whisper/, build-copy.mjs, staticProjectServer.ts). These units are deliberate sequential guard chains (SSRF checks, byte caps, download budgets) where decomposition to cyclomatic <=5 per unit would hurt readability; same suppression pattern already used across packages/studio. Fallow audit now exits 0 against origin/main; CLI suite 719/719 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(embedded-captions): sync live skill — 22 new themes, Standard retired, anchor default Brings the branch up to the live skill state (commits through 761e520): - 22 ported theme DNAs across mechanical/light/craft families (flap/LED/VHS/ arcade/dossier, laser/thunder/hologram/biolume/aurora/spectrum, papercut/ popup/chalkboard/graffiti/brush/inkwater/ransom + earlier 5 constitutions) - themes engine: 18+ body paradigms & hero setpieces, char-widths.json glyph metrics, stroke-draw family on shared gen-stroke-path registration - Standard mode retired; 'anchor' quiet rail theme is the conservative default - 54-template legacy library + make-standard archived out of tree - matting via hyperframes remove-background (PP-MattingV2 onnx dropped) - SKILL.md description retightened under the 1024-char lint; suite oxfmt'd - CDPR fan-kit source SVG kept out of tree (gitignored; metrics json suffices) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(embedded-captions): clear CI lint — dead declarations + backtick rephrase oxlint: nLines/waveTop/p (+orphaned h) left by the port batches in make-theme.cjs. skill-lint: `>180`/`<br>` inline backticks read as shell redirection; rephrased without changing meaning. Fixture regressions green (laser/anchor/ransom recompile clean). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(embedded-captions): read-with-catch for matte.fps (CodeQL js/file-system-race) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(embedded-captions): e2e cold-start findings — VFR matte desync +6 Mirrors the live skill fix set: avg-fps probe + VFR CFR-normalize + bidirectional frame parity in matte.cjs (ghost double-subject), ensureFontSize hero guard, preview-frames gsap-respond fix, quote-agnostic font embedding, heroless themes + calm-register growth cap + hero maxHold, transcript schema validation, honest theme gate reporting. Verified: 19/19 fixture regression, C1/T3/T4 re-rendered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(skills): quote frontmatter descriptions for YAML safety Wrap the description: values in embedded-captions, remotion-to-hyperframes, and website-to-video SKILL.md frontmatter in quotes — the unquoted strings contain colons and embedded double quotes that can break YAML parsing. oxfmt normalizes the two with embedded quotes to single-quoted form. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: jieling-jenson <jie.ling@heygen.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
WaterrrForever
co-authored by
Claude Opus 4.8
jieling-jenson
Claude Opus 4.8
jieling-jenson
parent
a241f2591e
commit
211e0adbe8
@@ -0,0 +1,328 @@
|
||||
#!/usr/bin/env node
|
||||
/*
|
||||
* transcribe.cjs — word-level transcription via hyperframes' native Whisper
|
||||
* (replaces the Python ElevenLabs Scribe path; no Python, no API key).
|
||||
*
|
||||
* node transcribe.cjs <project-dir> [model] [language]
|
||||
* Reads: <project>/source.mp4 (audio track)
|
||||
* Writes: <project>/transcript.json — { text, language_code, words:[{text,start,end,type}] }
|
||||
*/
|
||||
const path = require("path");
|
||||
const fs = require("fs");
|
||||
const os = require("os");
|
||||
const cp = require("child_process");
|
||||
|
||||
function hfRoot() {
|
||||
const roots = [
|
||||
process.env.HYPERFRAMES_ROOT,
|
||||
path.resolve(__dirname, "..", "..", ".."),
|
||||
path.join(os.homedir(), "Downloads", "hyperframes"),
|
||||
].filter(Boolean);
|
||||
for (const r of roots)
|
||||
if (fs.existsSync(path.join(r, "packages", "cli", "dist", "cli.js"))) return r;
|
||||
console.error("[transcribe] hyperframes CLI not found — set HYPERFRAMES_ROOT");
|
||||
process.exit(3);
|
||||
}
|
||||
function ensureSource(project) {
|
||||
const src = path.join(project, "source.mp4");
|
||||
if (fs.existsSync(src)) return src;
|
||||
const EXCL = new Set(["final", "bg_plus_caps", "fg_caps", "audio"]);
|
||||
let cands = fs
|
||||
.readdirSync(project)
|
||||
.filter(
|
||||
(f) =>
|
||||
["mp4", "mov", "webm", "mkv", "m4v"].includes(path.extname(f).slice(1).toLowerCase()) &&
|
||||
!EXCL.has(path.basename(f, path.extname(f))) &&
|
||||
!f.startsWith("index"),
|
||||
)
|
||||
.map((f) => path.join(project, f));
|
||||
let found = cands.sort((a, b) => fs.statSync(b).size - fs.statSync(a).size)[0];
|
||||
if (found) {
|
||||
try {
|
||||
fs.symlinkSync(path.basename(found), src);
|
||||
} catch {
|
||||
fs.copyFileSync(found, src);
|
||||
}
|
||||
}
|
||||
return src;
|
||||
}
|
||||
function _usableWords(d) {
|
||||
return d && Array.isArray(d.words) && d.words.some((w) => w && "start" in w && "end" in w);
|
||||
}
|
||||
// Mean loudness of the audio, for the no-speech guard below. Silence → whisper
|
||||
// hallucinates (famously "Thank you."), and the decision gate refuses "no speech".
|
||||
function meanVolumeDb(audio) {
|
||||
try {
|
||||
// ffmpeg writes volumedetect stats to STDERR — capture it (spawnSync, no throw).
|
||||
const r = cp.spawnSync(
|
||||
"ffmpeg",
|
||||
["-hide_banner", "-nostats", "-i", audio, "-af", "volumedetect", "-f", "null", "-"],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
const out = (r.stderr || "") + (r.stdout || "");
|
||||
const m = out.match(/mean_volume:\s*(-?[\d.]+) dB/);
|
||||
return m ? parseFloat(m[1]) : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// Where does AUDIBLE content end? Whisper hallucinates trailing words over a silent
|
||||
// tail (observed: "I'm sorry." repeated over dead air at a clip's end). silencedetect
|
||||
// finds a terminal silence running to EOF; words "spoken" inside it are fabricated.
|
||||
// Conservative: applause/music read as non-silence, so this fires only on truly dead
|
||||
// tails. Returns {speechEnd, total} or null.
|
||||
function audibleEnd(audio) {
|
||||
try {
|
||||
const r = cp.spawnSync(
|
||||
"ffmpeg",
|
||||
[
|
||||
"-hide_banner",
|
||||
"-nostats",
|
||||
"-i",
|
||||
audio,
|
||||
"-af",
|
||||
"silencedetect=noise=-35dB:d=0.6",
|
||||
"-f",
|
||||
"null",
|
||||
"-",
|
||||
],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
const out = (r.stderr || "") + (r.stdout || "");
|
||||
const durM = out.match(/Duration:\s*(\d+):(\d+):([\d.]+)/);
|
||||
const total = durM ? +durM[1] * 3600 + +durM[2] * 60 + +durM[3] : null;
|
||||
if (total == null) return null;
|
||||
const starts = [...out.matchAll(/silence_start:\s*([\d.]+)/g)].map((x) => +x[1]);
|
||||
const ends = [...out.matchAll(/silence_end:\s*([\d.]+)/g)].map((x) => +x[1]);
|
||||
if (!starts.length) return { speechEnd: total, total };
|
||||
const lastStart = starts[starts.length - 1];
|
||||
const closed = ends.some((e) => e > lastStart); // silence re-broken before EOF?
|
||||
return { speechEnd: closed ? total : lastStart, total };
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function main() {
|
||||
const project = path.resolve(process.argv[2] || "");
|
||||
if (!process.argv[2]) {
|
||||
console.error("usage: transcribe.cjs <project-dir> [model] [language]");
|
||||
process.exit(1);
|
||||
}
|
||||
// Default = multilingual `small`, NOT `small.en`. Per hyperframes-media: ".en models
|
||||
// mistranslate non-English and mis-handle accented speech; default to small (auto-detects
|
||||
// language)." We hardcoded small.en before — it hallucinated a wrong transcript on an
|
||||
// accented speaker. Pass `small.en` only for known-clean-English; tough accents → a larger model.
|
||||
const model = process.argv[3] || process.env.WHISPER_MODEL || "small";
|
||||
const language = process.argv[4] || process.env.WHISPER_LANG || "";
|
||||
const out = path.join(project, "transcript.json");
|
||||
|
||||
// already in our schema? skip — but validate the SHAPE, not just the keys:
|
||||
// `hyperframes init` drops a whisper.cpp segment/token-format transcript.json
|
||||
// (offsets-in-ms, nested tokens) that can carry a `words` key yet poison the
|
||||
// compilers. Only a word-level {text,start,end} array counts as normalized.
|
||||
try {
|
||||
const d = JSON.parse(fs.readFileSync(out, "utf8"));
|
||||
const wordShaped =
|
||||
d &&
|
||||
Array.isArray(d.words) &&
|
||||
d.words.length > 0 &&
|
||||
d.words.every(
|
||||
(w) =>
|
||||
w &&
|
||||
typeof (w.text ?? w.word) === "string" &&
|
||||
Number.isFinite(w.start) &&
|
||||
Number.isFinite(w.end) &&
|
||||
w.end < 36000, // ms-offset formats blow past any sane seconds value
|
||||
);
|
||||
if (wordShaped && d.language_code) {
|
||||
console.log("[transcribe] already normalized, skipping");
|
||||
return;
|
||||
}
|
||||
if (d && !wordShaped) {
|
||||
console.log(
|
||||
"[transcribe] existing transcript.json is NOT word-level (init stub / segment format) — regenerating",
|
||||
);
|
||||
}
|
||||
} catch {}
|
||||
|
||||
const src = ensureSource(project);
|
||||
if (!fs.existsSync(src)) {
|
||||
console.error(`[transcribe] no source in ${project}`);
|
||||
process.exit(2);
|
||||
}
|
||||
const audio = path.join(project, "audio.mp3");
|
||||
if (!fs.existsSync(audio))
|
||||
cp.execFileSync(
|
||||
"ffmpeg",
|
||||
["-y", "-i", src, "-vn", "-acodec", "libmp3lame", "-q:a", "2", audio],
|
||||
{ stdio: "ignore" },
|
||||
);
|
||||
|
||||
// ── engine: WhisperX (preferred — wav2vec2 forced alignment gives word timings far
|
||||
// tighter than whisper.cpp's segment-interpolated ones; our gates are 80ms-strict) →
|
||||
// fallback hyperframes whisper.cpp. Force with TRANSCRIBE_ENGINE=whisper|whisperx.
|
||||
let words = null,
|
||||
engine = null;
|
||||
const wantWx = (process.env.TRANSCRIBE_ENGINE || "whisperx") === "whisperx";
|
||||
if (wantWx) {
|
||||
try {
|
||||
const wav = path.join(project, "_wx_audio.wav");
|
||||
cp.execFileSync("ffmpeg", ["-y", "-i", src, "-vn", "-ac", "1", "-ar", "16000", wav], {
|
||||
stdio: "ignore",
|
||||
});
|
||||
const outDir = path.join(project, "_wx_out");
|
||||
fs.mkdirSync(outDir, { recursive: true });
|
||||
const wxModel = model.replace(/\.en$/, ""); // whisperx model names are multilingual ids
|
||||
const wxArgs = [
|
||||
"--python",
|
||||
"3.12",
|
||||
"--from",
|
||||
"whisperx",
|
||||
"whisperx",
|
||||
wav,
|
||||
"--model",
|
||||
wxModel,
|
||||
"--device",
|
||||
"cpu",
|
||||
"--compute_type",
|
||||
"int8",
|
||||
"--output_dir",
|
||||
outDir,
|
||||
"--output_format",
|
||||
"json",
|
||||
"--no_align_deletes",
|
||||
"--print_progress",
|
||||
"False",
|
||||
];
|
||||
if (language) wxArgs.push("--language", language);
|
||||
// strip our flag if this whisperx build doesn't know it
|
||||
let r = cp.spawnSync("uvx", wxArgs, { encoding: "utf8", timeout: 600000 });
|
||||
if ((r.status || 0) !== 0 && /no_align_deletes/.test(r.stderr || "")) {
|
||||
r = cp.spawnSync(
|
||||
"uvx",
|
||||
wxArgs.filter((a) => a !== "--no_align_deletes"),
|
||||
{ encoding: "utf8", timeout: 600000 },
|
||||
);
|
||||
}
|
||||
if ((r.status || 0) !== 0)
|
||||
throw new Error(
|
||||
(r.stderr || "whisperx failed").split("\n").slice(-4).join(" ").slice(0, 300),
|
||||
);
|
||||
const wxJson = JSON.parse(fs.readFileSync(path.join(outDir, "_wx_audio.json"), "utf8"));
|
||||
const wx = [];
|
||||
for (const seg of wxJson.segments || [])
|
||||
for (const w of seg.words || []) {
|
||||
// alignment occasionally yields a word with no timing (OOV) — interpolate from neighbors later; mark null now
|
||||
wx.push({ text: String(w.word || "").trim(), start: w.start, end: w.end, type: "word" });
|
||||
}
|
||||
// interpolate missing timings from neighbors (rare OOV/number cases)
|
||||
for (let i = 0; i < wx.length; i++) {
|
||||
if (wx[i].start == null || wx[i].end == null) {
|
||||
const prevEnd = i > 0 ? wx[i - 1].end : 0;
|
||||
const nextStart = wx.slice(i + 1).find((x) => x.start != null);
|
||||
const ns = nextStart ? nextStart.start : prevEnd + 0.3;
|
||||
wx[i].start = prevEnd;
|
||||
wx[i].end = Math.max(prevEnd + 0.05, ns - 0.02);
|
||||
}
|
||||
}
|
||||
if (wx.length) {
|
||||
words = wx.filter((w) => w.text);
|
||||
engine = `whisperx(${wxModel}+wav2vec2)`;
|
||||
}
|
||||
try {
|
||||
fs.unlinkSync(wav);
|
||||
} catch {}
|
||||
} catch (e) {
|
||||
console.error(
|
||||
`[transcribe] whisperx unavailable (${String(e.message || e).slice(0, 160)}) — falling back to whisper.cpp`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (!words) {
|
||||
// run hyperframes Whisper → writes a flat word array to <dir>/transcript.json
|
||||
const cli = path.join(hfRoot(), "packages", "cli", "dist", "cli.js");
|
||||
const args = ["transcribe", audio, "-d", project, "--json", "--model", model];
|
||||
if (language) args.push("--language", language);
|
||||
let info = {};
|
||||
try {
|
||||
const so = cp.execFileSync("node", [cli, ...args], { encoding: "utf8" });
|
||||
const line = so.trim().split("\n").filter(Boolean).pop();
|
||||
info = JSON.parse(line);
|
||||
} catch (e) {
|
||||
console.error("[transcribe] hyperframes whisper failed:", e.message);
|
||||
process.exit(1);
|
||||
}
|
||||
const flatPath = info.transcriptPath || out;
|
||||
const flat = JSON.parse(fs.readFileSync(flatPath, "utf8"));
|
||||
const arr = Array.isArray(flat) ? flat : flat.words || [];
|
||||
words = arr
|
||||
.filter((w) => (w.text ?? w.word) != null)
|
||||
.map((w) => ({
|
||||
text: w.text ?? w.word,
|
||||
start: w.start ?? w.t0,
|
||||
end: w.end ?? w.t1,
|
||||
type: "word",
|
||||
}));
|
||||
engine = `whisper.cpp(${model})`;
|
||||
}
|
||||
|
||||
// Tail-hallucination guard: drop words whisper placed entirely inside a terminal
|
||||
// silence (it fabricates e.g. repeated "I'm sorry." over dead air). Word START past
|
||||
// the audible end (+0.4s slack) = fabricated; real final words start before it.
|
||||
const ae = audibleEnd(audio);
|
||||
let trimmedTail = 0;
|
||||
if (ae && ae.speechEnd < ae.total - 0.8) {
|
||||
const keep = words.filter((w) => w.start <= ae.speechEnd + 0.4);
|
||||
trimmedTail = words.length - keep.length;
|
||||
if (trimmedTail > 0) {
|
||||
console.error(
|
||||
`[transcribe] ⚠ trimmed ${trimmedTail} trailing word(s) starting after the audible end ` +
|
||||
`(${ae.speechEnd.toFixed(2)}s; clip ${ae.total.toFixed(2)}s) — whisper hallucinates over silent tails.`,
|
||||
);
|
||||
words = keep;
|
||||
}
|
||||
}
|
||||
|
||||
const text = words
|
||||
.map((w) => w.text)
|
||||
.join(" ")
|
||||
.replace(/\s+([,.!?;:])/g, "$1")
|
||||
.trim();
|
||||
fs.writeFileSync(
|
||||
out,
|
||||
JSON.stringify(
|
||||
{
|
||||
text,
|
||||
language_code: language || "en",
|
||||
engine,
|
||||
words,
|
||||
...(trimmedTail ? { trimmed_tail_words: trimmedTail } : {}),
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
console.log(`[transcribe] ${engine} ${words.length} words → ${out}`);
|
||||
console.log(`[transcribe] text: ${text.slice(0, 160)}${text.length > 160 ? "…" : ""}`);
|
||||
|
||||
// No-speech guard: whisper returns confident hallucinations over silence (e.g. the
|
||||
// whole clip as "Thank you."). The decision gate REFUSES "no speech" — operationalize
|
||||
// it so an agent trusting the transcript can't sail past the gate.
|
||||
const meanDb = meanVolumeDb(audio);
|
||||
if (meanDb != null && meanDb < -45) {
|
||||
console.error(
|
||||
`\n[transcribe] ⚠ NEAR-SILENT AUDIO — mean ${meanDb.toFixed(1)} dB (real speech ≈ -16..-26 dB).`,
|
||||
);
|
||||
console.error(
|
||||
` This transcript is almost certainly a Whisper hallucination, NOT real speech.`,
|
||||
);
|
||||
console.error(
|
||||
` Per the decision gate, REFUSE "no speech" — confirm with \`ffmpeg -i <src> -af silencedetect\`;`,
|
||||
);
|
||||
console.error(` do NOT author captions from fabricated words.`);
|
||||
}
|
||||
}
|
||||
main();
|
||||
Reference in New Issue
Block a user