mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
feat(cli): add whisper transcription and template improvements (#53)
* feat(cli): add whisper transcription to init flow New modules: - whisper/manager.ts: download/cache whisper.cpp binary + model (~/.cache/hyperframes/whisper/) - whisper/transcribe.ts: extract audio, run whisper, save transcript.json Init flow changes: - "Got a video or audio file?" now accepts audio-only files (mp3, wav, m4a) - "Generate captions from audio?" prompt after file selection - Transcription produces transcript.json in project root - Graceful fallback if whisper/ffmpeg unavailable Supports: macOS ARM64/x86, Linux x86_64. Downloads whisper.cpp v1.7.3 from GitHub releases and ggml-base.en model from Hugging Face. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): use brew/system whisper instead of downloading binaries whisper.cpp doesn't ship pre-built macOS/Linux CLI binaries. Use brew install whisper-cpp on macOS (auto-installs if brew available), system PATH lookup otherwise. Model still downloaded from Hugging Face. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): simplify whisper install — detect or instruct, don't build Remove build-from-source complexity. If whisper-cpp is found on PATH, use it. If not, show install instructions instead of blocking: "To generate captions, install whisper-cpp: brew install whisper-cpp" The transcription prompt only appears when whisper is available. When it's not, the user sees the install command and can re-run init. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(cli): auto-install whisper via brew or build from source ensureWhisper() now tries 4 strategies in order: 1. System PATH (whisper-cli or whisper already installed) 2. Homebrew (macOS: brew install whisper-cpp) 3. Build from source (git clone + cmake, ~30-60s) 4. Show install instructions as last resort Init flow always asks "Generate captions?" — whisper is installed automatically in the background if needed. No user intervention required on macOS with Xcode CLI tools or any system with git+cmake. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): add window.__timelines guard to all templates The studio bundler doesn't always initialize window.__timelines before template scripts run, causing "Cannot set properties of undefined" errors. Add defensive guard to every template. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(cli): patch template captions with actual transcript data After scaffolding, if transcript.json exists, replace the hardcoded word array in the template's captions composition with the real transcript data. The template's caption animation and styling are preserved — only the word data changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): show install notice when whisper needs to be installed When whisper-cpp isn't found, show an info message before the spinner: "whisper-cpp not found — installing automatically..." Then the spinner shows "Installing whisper-cpp (this may take a moment)..." Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): add muted and playsinline to all template video elements The framework requires video elements to have muted and playsinline attributes. All four templates were missing these, causing video to not play in the studio preview. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): flat asset structure + separate audio tracks in templates Assets: video, images, fonts all go at project root (not assets/ or fonts/ subdirectories). The studio preview can't resolve relative paths from subdirectories due to the /preview URL suffix. Audio: added <audio> elements alongside muted <video> in all 4 templates so the video's audio plays back. The framework requires muted video + separate audio element. Removed assets/ and fonts/ directory creation from scaffoldProject. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(studio): inject base tag for asset resolution in preview The preview iframe serves bundled HTML from /api/projects/:id/preview but relative asset paths (video.mp4, font.woff2) resolve to the wrong URL without a <base> tag. Now injects <base href="/api/projects/:id/preview/"> so relative paths route through the static asset handler. Also adds proper MIME types for video, audio, image, and font files served from the preview asset route. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(studio): serve HyperFrames runtime in dev mode The preview runtime script had an empty src — the framework never loaded, so video playback and clip lifecycle didn't work. Now auto-detects packages/cli/dist/hyperframe-runtime.js and serves it at /api/runtime.js. No env var needed in dev mode. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): filter whisper special tokens from transcript Use --output-json instead of --output-json-full to avoid special tokens like [_TT_485] and [BLANK_AUDIO]. Also filter remaining bracket tokens when building the word array for captions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): use --output-json-full for word-level timestamps --output-json only produces segment-level timing (no tokens). --output-json-full is required for word-level timestamps that the captions template needs. Special tokens are filtered out by the patchTranscript function. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(cli): patch template durations to match uploaded video Templates now use __VIDEO_DURATION__ placeholder that gets replaced with the actual probed video duration. All data-duration values on the root composition, video, audio, and caption clips are updated. Without a video, defaults to 10 seconds. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): merge punctuation tokens with preceding word Whisper outputs punctuation (. , ! ?) as separate tokens. These appeared as standalone words in captions, sometimes in the wrong group. Now merged with the preceding word during transcript normalization. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): match both TRANSCRIPT and script variable names in templates Three templates use `const TRANSCRIPT = [...]` while warm-grain uses `const script = [...]`. The patchTranscript function now matches both. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): security and template fixes - Replace shell injection risk (execSync rm) with unlinkSync in transcribe.ts - Add GIT_TERMINAL_PROMPT=0 to whisper buildFromSource git clone - Fix hardcoded data-duration="18" in warm-grain captions template - Add data-start="0" to root compositions in swiss-grid, vignelli, warm-grain - Add data-start="0" to warm-grain grain-overlay composition - Deduplicate hasFFmpeg: remove from init.ts, import from whisper/manager.ts - Add my-video/ and packages/studio/data/ to .gitignore Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): format warm-grain captions and fix TS nullability errors - Format warm-grain/compositions/captions.html - Add optional chaining on token.offsets (may be undefined) - Use intermediate variable for lastWord to satisfy TS strict checks Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(cli): add blank template option, smart defaults for video vs audio - Blank template: minimal scaffolding (root composition, video, audio, GSAP timeline) with __VIDEO_SRC__ and __VIDEO_DURATION__ placeholders - Template defaults: video uploads default to "blank" (user brings their own content), audio-only defaults to "warm-grain" (motion graphics template since there's no video to show) - Audio-only projects now tracked with isAudioOnly flag Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): address whisper review feedback - Clean stale builds: if BUILD_DIR exists but no binary, nuke and retry - Build failures clean up BUILD_DIR so next attempt starts fresh - patchTranscript regex scoped within <script> blocks to prevent matching across block boundaries - Removed hardcoded model size hint (~148MB) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): add missing rmSync import to whisper manager Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove test project and lock file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): address review items 7-12 — execFileSync, build diagnostics, WAV verification - manager.ts: replace all execSync with execFileSync to prevent command injection - manager.ts: capture cmake stderr and include in build failure error message - transcribe.ts: verify WAV is 16kHz mono via ffprobe before passing to whisper - init.ts: replace fragile JSON formatting with JSON.stringify(words, null, 2) - init.ts: fix default duration from "10" to "5" matching DEFAULT_META - init.ts: add probeAudioDuration() and --audio/--skip-transcribe flags - init.ts: extract finalizeProject() to reduce code path duplication - init.ts: wire transcription into non-interactive path Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
5fceab9279
commit
f4367d5726
@@ -10,11 +10,12 @@ import {
|
||||
} from "node:fs";
|
||||
import { resolve, basename, join, dirname } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { execSync, execFileSync, spawn } from "node:child_process";
|
||||
import { execFileSync, spawn } from "node:child_process";
|
||||
import * as clack from "@clack/prompts";
|
||||
import { c } from "../ui/colors.js";
|
||||
import { TEMPLATES, type TemplateId } from "../templates/generators.js";
|
||||
import { trackInitTemplate } from "../telemetry/events.js";
|
||||
import { hasFFmpeg } from "../whisper/manager.js";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Install skills silently after scaffolding
|
||||
@@ -161,15 +162,23 @@ function isWebCompatible(codec: string): boolean {
|
||||
return WEB_CODECS.has(codec.toLowerCase());
|
||||
}
|
||||
|
||||
function hasFFmpeg(): boolean {
|
||||
function probeAudioDuration(filePath: string): number | undefined {
|
||||
try {
|
||||
execSync("ffmpeg -version", { stdio: "ignore", timeout: 5000 });
|
||||
return true;
|
||||
const raw = execFileSync(
|
||||
"ffprobe",
|
||||
["-v", "quiet", "-print_format", "json", "-show_format", filePath],
|
||||
{ encoding: "utf-8", timeout: 15_000 },
|
||||
);
|
||||
const parsed: { format?: { duration?: string } } = JSON.parse(raw);
|
||||
const duration = parseFloat(parsed.format?.duration ?? "");
|
||||
return Number.isNaN(duration) ? undefined : duration;
|
||||
} catch {
|
||||
return false;
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
// hasFFmpeg is imported from whisper/manager.ts to avoid duplication
|
||||
|
||||
function transcodeToMp4(inputPath: string, outputPath: string): Promise<boolean> {
|
||||
return new Promise((resolvePromise) => {
|
||||
const child = spawn(
|
||||
@@ -211,7 +220,11 @@ function getStaticTemplateDir(templateId: string): string {
|
||||
return existsSync(devPath) ? devPath : builtPath;
|
||||
}
|
||||
|
||||
function patchVideoSrc(dir: string, videoFilename: string | undefined): void {
|
||||
function patchVideoSrc(
|
||||
dir: string,
|
||||
videoFilename: string | undefined,
|
||||
durationSeconds?: number,
|
||||
): void {
|
||||
const htmlFiles = readdirSync(dir, { withFileTypes: true, recursive: true })
|
||||
.filter((e) => e.isFile() && e.name.endsWith(".html"))
|
||||
.map((e) => join(e.parentPath ?? e.path, e.name));
|
||||
@@ -224,11 +237,71 @@ function patchVideoSrc(dir: string, videoFilename: string | undefined): void {
|
||||
// Remove video elements with placeholder src
|
||||
content = content.replace(/<video[^>]*src="__VIDEO_SRC__"[^>]*>[\s\S]*?<\/video>/g, "");
|
||||
content = content.replace(/<video[^>]*src="__VIDEO_SRC__"[^>]*>/g, "");
|
||||
// Remove audio elements with placeholder src
|
||||
content = content.replace(/<audio[^>]*src="__VIDEO_SRC__"[^>]*>[\s\S]*?<\/audio>/g, "");
|
||||
content = content.replace(/<audio[^>]*src="__VIDEO_SRC__"[^>]*>/g, "");
|
||||
}
|
||||
// Patch duration — use probed duration or default (matches DEFAULT_META)
|
||||
const dur = durationSeconds ? String(Math.round(durationSeconds * 100) / 100) : "5";
|
||||
content = content.replaceAll("__VIDEO_DURATION__", dur);
|
||||
writeFileSync(file, content, "utf-8");
|
||||
}
|
||||
}
|
||||
|
||||
function patchTranscript(dir: string, transcriptPath: string): void {
|
||||
// Read the whisper transcript and normalize to [{text, start, end}]
|
||||
const raw = JSON.parse(readFileSync(transcriptPath, "utf-8"));
|
||||
const words: { text: string; start: number; end: number }[] = [];
|
||||
for (const seg of raw.transcription ?? []) {
|
||||
for (const token of seg.tokens ?? []) {
|
||||
const text = (token.text ?? "").trim();
|
||||
if (!text || text.startsWith("[_") || text.startsWith("[BLANK")) continue;
|
||||
|
||||
// Merge punctuation with the previous word
|
||||
const isPunctuation = /^[.,!?;:'")\]}>…–—-]+$/.test(text);
|
||||
const lastWord = words[words.length - 1];
|
||||
if (isPunctuation && lastWord) {
|
||||
lastWord.text += text;
|
||||
lastWord.end = Math.round(((token.offsets?.to ?? 0) / 1000) * 1000) / 1000;
|
||||
continue;
|
||||
}
|
||||
|
||||
words.push({
|
||||
text,
|
||||
start: Math.round(((token.offsets?.from ?? 0) / 1000) * 1000) / 1000,
|
||||
end: Math.round(((token.offsets?.to ?? 0) / 1000) * 1000) / 1000,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (words.length === 0) return;
|
||||
|
||||
const wordsJson = JSON.stringify(words, null, 2);
|
||||
|
||||
// Find captions HTML files and replace the hardcoded script array
|
||||
const htmlFiles = readdirSync(dir, { withFileTypes: true, recursive: true })
|
||||
.filter((e) => e.isFile() && e.name.endsWith(".html"))
|
||||
.map((e) => join(e.parentPath ?? e.path, e.name));
|
||||
|
||||
for (const file of htmlFiles) {
|
||||
let content = readFileSync(file, "utf-8");
|
||||
// Match within <script> blocks only to avoid crossing block boundaries
|
||||
const scriptBlocks = content.match(/<script>[\s\S]*?<\/script>/g) ?? [];
|
||||
let scriptMatch: RegExpMatchArray | null = null;
|
||||
let transcriptMatch: RegExpMatchArray | null = null;
|
||||
for (const block of scriptBlocks) {
|
||||
scriptMatch = scriptMatch ?? block.match(/const script = \[[\s\S]*?\];/);
|
||||
transcriptMatch = transcriptMatch ?? block.match(/const TRANSCRIPT = \[[\s\S]*?\];/);
|
||||
}
|
||||
const match = scriptMatch ?? transcriptMatch;
|
||||
if (match) {
|
||||
const varName = scriptMatch ? "script" : "TRANSCRIPT";
|
||||
content = content.replace(match[0], `const ${varName} = ${wordsJson};`);
|
||||
writeFileSync(file, content, "utf-8");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// handleVideoFile — probe, check codec, optionally transcode, copy to destDir
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -276,8 +349,16 @@ async function handleVideoFile(
|
||||
const transcode = await clack.select({
|
||||
message: "Transcode to H.264 MP4 for browser playback?",
|
||||
options: [
|
||||
{ value: "yes", label: "Yes, transcode", hint: "converts to H.264 MP4" },
|
||||
{ value: "no", label: "No, keep original", hint: "video won't play in browser" },
|
||||
{
|
||||
value: "yes",
|
||||
label: "Yes, transcode",
|
||||
hint: "converts to H.264 MP4",
|
||||
},
|
||||
{
|
||||
value: "no",
|
||||
label: "No, keep original",
|
||||
hint: "video won't play in browser",
|
||||
},
|
||||
],
|
||||
});
|
||||
if (clack.isCancel(transcode)) {
|
||||
@@ -329,12 +410,13 @@ function scaffoldProject(
|
||||
name: string,
|
||||
templateId: TemplateId,
|
||||
localVideoName: string | undefined,
|
||||
durationSeconds?: number,
|
||||
): void {
|
||||
mkdirSync(destDir, { recursive: true });
|
||||
|
||||
const templateDir = getStaticTemplateDir(templateId);
|
||||
cpSync(templateDir, destDir, { recursive: true });
|
||||
patchVideoSrc(destDir, localVideoName);
|
||||
patchVideoSrc(destDir, localVideoName, durationSeconds);
|
||||
|
||||
writeFileSync(
|
||||
resolve(destDir, "meta.json"),
|
||||
@@ -351,6 +433,37 @@ function scaffoldProject(
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// finalizeProject — shared scaffold + patch + skills logic
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function finalizeProject(opts: {
|
||||
destDir: string;
|
||||
name: string;
|
||||
templateId: TemplateId;
|
||||
localVideoName?: string;
|
||||
durationSeconds?: number;
|
||||
skipSkills: boolean;
|
||||
interactive: boolean;
|
||||
}): Promise<void> {
|
||||
scaffoldProject(
|
||||
opts.destDir,
|
||||
opts.name,
|
||||
opts.templateId,
|
||||
opts.localVideoName,
|
||||
opts.durationSeconds,
|
||||
);
|
||||
|
||||
const transcriptFile = resolve(opts.destDir, "transcript.json");
|
||||
if (existsSync(transcriptFile)) {
|
||||
patchTranscript(opts.destDir, transcriptFile);
|
||||
}
|
||||
|
||||
if (!opts.skipSkills) {
|
||||
await installSkills(opts.interactive);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// nextStepLoop — "What do you want to do?" loop after scaffolding
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -360,7 +473,11 @@ async function nextStepLoop(destDir: string): Promise<void> {
|
||||
const next = await clack.select({
|
||||
message: "What do you want to do?",
|
||||
options: [
|
||||
{ value: "dev", label: "Open in studio", hint: "full editor with timeline" },
|
||||
{
|
||||
value: "dev",
|
||||
label: "Open in studio",
|
||||
hint: "full editor with timeline",
|
||||
},
|
||||
{ value: "render", label: "Render to MP4", hint: "export video now" },
|
||||
{ value: "done", label: "Done for now" },
|
||||
],
|
||||
@@ -398,19 +515,41 @@ async function nextStepLoop(destDir: string): Promise<void> {
|
||||
export default defineCommand({
|
||||
meta: { name: "init", description: "Scaffold a new composition project" },
|
||||
args: {
|
||||
name: { type: "positional", description: "Project name", required: false },
|
||||
name: {
|
||||
type: "positional",
|
||||
description: "Project name (default: my-video)",
|
||||
required: false,
|
||||
},
|
||||
template: {
|
||||
type: "string",
|
||||
description: `Template: ${ALL_TEMPLATE_IDS.join(", ")}`,
|
||||
description: `Template: ${ALL_TEMPLATE_IDS.join(", ")}. Required for non-interactive mode.`,
|
||||
alias: "t",
|
||||
},
|
||||
video: { type: "string", description: "Path to a source video file", alias: "V" },
|
||||
"skip-skills": { type: "boolean", description: "Skip AI skills installation" },
|
||||
video: {
|
||||
type: "string",
|
||||
description: "Path to a source video file (auto-transcodes if needed)",
|
||||
alias: "V",
|
||||
},
|
||||
audio: {
|
||||
type: "string",
|
||||
description: "Path to a source audio file (cannot combine with --video)",
|
||||
alias: "A",
|
||||
},
|
||||
"skip-skills": {
|
||||
type: "boolean",
|
||||
description: "Skip AI skills installation",
|
||||
},
|
||||
"skip-transcribe": {
|
||||
type: "boolean",
|
||||
description: "Skip whisper transcription (default: transcribes when video/audio provided)",
|
||||
},
|
||||
},
|
||||
async run({ args }) {
|
||||
const templateFlag = args.template;
|
||||
const videoFlag = args.video;
|
||||
const audioFlag = args.audio;
|
||||
const skipSkills = args["skip-skills"] === true;
|
||||
const skipTranscribe = args["skip-transcribe"] === true;
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Non-interactive mode: flags provided
|
||||
@@ -433,6 +572,13 @@ export default defineCommand({
|
||||
mkdirSync(destDir, { recursive: true });
|
||||
|
||||
let localVideoName: string | undefined;
|
||||
let videoDuration: number | undefined;
|
||||
let sourceFilePath: string | undefined;
|
||||
|
||||
if (videoFlag && audioFlag) {
|
||||
console.error(c.error("Cannot specify both --video and --audio"));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (videoFlag) {
|
||||
const videoPath = resolve(videoFlag);
|
||||
@@ -440,16 +586,55 @@ export default defineCommand({
|
||||
console.error(c.error(`Video file not found: ${videoFlag}`));
|
||||
process.exit(1);
|
||||
}
|
||||
sourceFilePath = videoPath;
|
||||
const result = await handleVideoFile(videoPath, destDir, false);
|
||||
localVideoName = result.localVideoName;
|
||||
videoDuration = result.meta.durationSeconds;
|
||||
} else if (audioFlag) {
|
||||
const audioPath = resolve(audioFlag);
|
||||
if (!existsSync(audioPath)) {
|
||||
console.error(c.error(`Audio file not found: ${audioFlag}`));
|
||||
process.exit(1);
|
||||
}
|
||||
sourceFilePath = audioPath;
|
||||
copyFileSync(audioPath, resolve(destDir, basename(audioPath)));
|
||||
videoDuration = probeAudioDuration(audioPath);
|
||||
}
|
||||
|
||||
scaffoldProject(destDir, basename(destDir), templateId, localVideoName);
|
||||
trackInitTemplate(templateId);
|
||||
if (!skipSkills) {
|
||||
await installSkills(false);
|
||||
// Transcribe if we have a source file and transcription isn't skipped
|
||||
if (sourceFilePath && !skipTranscribe) {
|
||||
try {
|
||||
const { ensureWhisper, ensureModel } = await import("../whisper/manager.js");
|
||||
await ensureWhisper({
|
||||
onProgress: (msg) => console.log(c.dim(` ${msg}`)),
|
||||
});
|
||||
await ensureModel(undefined, {
|
||||
onProgress: (msg) => console.log(c.dim(` ${msg}`)),
|
||||
});
|
||||
const { transcribe: runTranscribe } = await import("../whisper/transcribe.js");
|
||||
const result = await runTranscribe(sourceFilePath, destDir, {
|
||||
onProgress: (msg) => console.log(c.dim(` ${msg}`)),
|
||||
});
|
||||
console.log(
|
||||
c.success(
|
||||
`Transcribed ${result.wordCount} words (${result.durationSeconds.toFixed(1)}s)`,
|
||||
),
|
||||
);
|
||||
} catch (err) {
|
||||
console.log(c.dim(`Transcription skipped: ${err instanceof Error ? err.message : err}`));
|
||||
}
|
||||
}
|
||||
|
||||
await finalizeProject({
|
||||
destDir,
|
||||
name: basename(destDir),
|
||||
templateId,
|
||||
localVideoName,
|
||||
durationSeconds: videoDuration,
|
||||
skipSkills,
|
||||
interactive: false,
|
||||
});
|
||||
|
||||
console.log(c.success(`\nCreated ${c.accent(name + "/")}`));
|
||||
for (const f of readdirSync(destDir)) {
|
||||
console.log(` ${c.accent(f)}`);
|
||||
@@ -493,42 +678,47 @@ export default defineCommand({
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Got a video?
|
||||
// 2. Got a video or audio file?
|
||||
let localVideoName: string | undefined;
|
||||
let sourceFilePath: string | undefined;
|
||||
let videoDuration: number | undefined;
|
||||
let isAudioOnly = false;
|
||||
|
||||
if (videoFlag) {
|
||||
// Video supplied via --video flag even in interactive mode
|
||||
const videoPath = resolve(videoFlag);
|
||||
if (!existsSync(videoPath)) {
|
||||
clack.log.error(`Video file not found: ${videoFlag}`);
|
||||
clack.log.error(`File not found: ${videoFlag}`);
|
||||
clack.cancel("Setup cancelled.");
|
||||
process.exit(1);
|
||||
}
|
||||
mkdirSync(destDir, { recursive: true });
|
||||
sourceFilePath = videoPath;
|
||||
const result = await handleVideoFile(videoPath, destDir, true);
|
||||
localVideoName = result.localVideoName;
|
||||
videoDuration = result.meta.durationSeconds;
|
||||
} else {
|
||||
const videoChoice = await clack.select({
|
||||
message: "Got a video file?",
|
||||
const mediaChoice = await clack.select({
|
||||
message: "Got a video or audio file?",
|
||||
options: [
|
||||
{ value: "yes", label: "Yes", hint: "MP4 or WebM recommended" },
|
||||
{ value: "video", label: "Video", hint: "MP4, WebM, MOV" },
|
||||
{ value: "audio", label: "Audio only", hint: "MP3, WAV, M4A" },
|
||||
{
|
||||
value: "no",
|
||||
label: "No",
|
||||
hint: "Start with motion graphics or text",
|
||||
},
|
||||
],
|
||||
initialValue: "no" as "yes" | "no",
|
||||
initialValue: "no" as "video" | "audio" | "no",
|
||||
});
|
||||
if (clack.isCancel(videoChoice)) {
|
||||
if (clack.isCancel(mediaChoice)) {
|
||||
clack.cancel("Setup cancelled.");
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (videoChoice === "yes") {
|
||||
if (mediaChoice === "video" || mediaChoice === "audio") {
|
||||
const pathResult = await clack.text({
|
||||
message: "Path to your video file (drag and drop or paste)",
|
||||
placeholder: "/path/to/video.mp4",
|
||||
message: `Path to your ${mediaChoice} file (drag and drop or paste)`,
|
||||
placeholder: mediaChoice === "video" ? "/path/to/video.mp4" : "/path/to/audio.mp3",
|
||||
validate(val) {
|
||||
const trimmed = val?.trim();
|
||||
if (!trimmed) return "Please enter a file path";
|
||||
@@ -541,15 +731,70 @@ export default defineCommand({
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const videoPath = resolve(String(pathResult).trim());
|
||||
|
||||
const filePath = resolve(String(pathResult).trim());
|
||||
sourceFilePath = filePath;
|
||||
mkdirSync(destDir, { recursive: true });
|
||||
const result = await handleVideoFile(videoPath, destDir, true);
|
||||
localVideoName = result.localVideoName;
|
||||
|
||||
if (mediaChoice === "video") {
|
||||
const result = await handleVideoFile(filePath, destDir, true);
|
||||
localVideoName = result.localVideoName;
|
||||
videoDuration = result.meta.durationSeconds;
|
||||
} else {
|
||||
// Audio file — copy to project root and probe duration
|
||||
isAudioOnly = true;
|
||||
copyFileSync(filePath, resolve(destDir, basename(filePath)));
|
||||
videoDuration = probeAudioDuration(filePath);
|
||||
clack.log.info(`Audio copied to ${c.accent(basename(filePath))}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Pick template — single list for all templates
|
||||
// 2b. Transcribe if we have a source file with audio
|
||||
if (sourceFilePath && !skipTranscribe) {
|
||||
const transcribeChoice = await clack.confirm({
|
||||
message: "Generate captions from audio?",
|
||||
initialValue: true,
|
||||
});
|
||||
if (!clack.isCancel(transcribeChoice) && transcribeChoice) {
|
||||
const { findWhisper } = await import("../whisper/manager.js");
|
||||
const needsInstall = findWhisper() === undefined;
|
||||
if (needsInstall) {
|
||||
clack.log.info(c.dim("whisper-cpp not found — installing automatically..."));
|
||||
}
|
||||
|
||||
const spin = clack.spinner();
|
||||
spin.start(
|
||||
needsInstall
|
||||
? "Installing whisper-cpp (this may take a moment)..."
|
||||
: "Preparing transcription...",
|
||||
);
|
||||
try {
|
||||
const { ensureWhisper, ensureModel } = await import("../whisper/manager.js");
|
||||
await ensureWhisper({
|
||||
onProgress: (msg) => spin.message(msg),
|
||||
});
|
||||
await ensureModel(undefined, {
|
||||
onProgress: (msg) => spin.message(msg),
|
||||
});
|
||||
|
||||
spin.message("Transcribing audio...");
|
||||
const { transcribe: runTranscribe } = await import("../whisper/transcribe.js");
|
||||
const transcribeResult = await runTranscribe(sourceFilePath, destDir, {
|
||||
onProgress: (msg) => spin.message(msg),
|
||||
});
|
||||
spin.stop(
|
||||
c.success(
|
||||
`Transcribed ${transcribeResult.wordCount} words (${transcribeResult.durationSeconds.toFixed(1)}s)`,
|
||||
),
|
||||
);
|
||||
} catch (err) {
|
||||
spin.stop(c.dim(`Transcription skipped: ${err instanceof Error ? err.message : err}`));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Pick template — default depends on media type
|
||||
const defaultTemplate = isAudioOnly ? "warm-grain" : "blank";
|
||||
const templateResult = await clack.select({
|
||||
message: "Pick a template",
|
||||
options: TEMPLATES.map((t) => ({
|
||||
@@ -557,7 +802,7 @@ export default defineCommand({
|
||||
label: t.label,
|
||||
hint: t.hint,
|
||||
})),
|
||||
initialValue: TEMPLATES[0]?.id,
|
||||
initialValue: defaultTemplate as TemplateId,
|
||||
});
|
||||
if (clack.isCancel(templateResult)) {
|
||||
clack.cancel("Setup cancelled.");
|
||||
@@ -566,15 +811,18 @@ export default defineCommand({
|
||||
|
||||
const templateId: TemplateId = templateResult;
|
||||
|
||||
// 4. Copy template and patch
|
||||
scaffoldProject(destDir, name, templateId, localVideoName);
|
||||
// 4. Copy template, patch, and install skills
|
||||
await finalizeProject({
|
||||
destDir,
|
||||
name,
|
||||
templateId,
|
||||
localVideoName,
|
||||
durationSeconds: videoDuration,
|
||||
skipSkills,
|
||||
interactive: true,
|
||||
});
|
||||
trackInitTemplate(templateId);
|
||||
|
||||
// 5. Install AI coding skills
|
||||
if (!skipSkills) {
|
||||
await installSkills(true);
|
||||
}
|
||||
|
||||
const files = readdirSync(destDir);
|
||||
clack.note(files.map((f) => c.accent(f)).join("\n"), c.success(`Created ${name}/`));
|
||||
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
<template id="captions-template">
|
||||
<div
|
||||
data-composition-id="captions"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
>
|
||||
<div id="captions-container"></div>
|
||||
|
||||
<style>
|
||||
[data-composition-id="captions"] {
|
||||
width: 1920px;
|
||||
height: 1080px;
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
[data-composition-id="captions"] #captions-container {
|
||||
position: absolute;
|
||||
bottom: 100px;
|
||||
left: 50%;
|
||||
transform: translateX(-50%);
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
width: 100%;
|
||||
height: 150px;
|
||||
}
|
||||
|
||||
.caption-group {
|
||||
position: absolute;
|
||||
opacity: 0;
|
||||
font-family: "Inter", sans-serif;
|
||||
font-size: 48px;
|
||||
font-weight: 700;
|
||||
color: #ffffff;
|
||||
text-shadow:
|
||||
0 2px 8px rgba(0, 0, 0, 0.8),
|
||||
0 0 2px rgba(0, 0, 0, 0.9);
|
||||
white-space: nowrap;
|
||||
}
|
||||
</style>
|
||||
|
||||
<script src="https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js"></script>
|
||||
<script>
|
||||
(function () {
|
||||
window.__timelines = window.__timelines || {};
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
|
||||
const script = [];
|
||||
|
||||
if (script.length === 0) {
|
||||
window.__timelines["captions"] = tl;
|
||||
return;
|
||||
}
|
||||
|
||||
const container = document.getElementById("captions-container");
|
||||
|
||||
// Group words into lines (max 5 words per line)
|
||||
const lines = [];
|
||||
for (let i = 0; i < script.length; i += 5) {
|
||||
const lineWords = script.slice(i, i + 5);
|
||||
lines.push({
|
||||
text: lineWords.map((w) => w.text).join(" "),
|
||||
start: lineWords[0].start,
|
||||
end: lineWords[lineWords.length - 1].end,
|
||||
});
|
||||
}
|
||||
|
||||
lines.forEach((line, index) => {
|
||||
const el = document.createElement("div");
|
||||
el.className = "caption-group";
|
||||
el.textContent = line.text;
|
||||
container.appendChild(el);
|
||||
|
||||
tl.fromTo(
|
||||
el,
|
||||
{ opacity: 0, y: 20 },
|
||||
{ opacity: 1, y: 0, duration: 0.3, ease: "power3.out" },
|
||||
line.start,
|
||||
);
|
||||
|
||||
const hideTime =
|
||||
index < lines.length - 1 ? Math.min(line.end, lines[index + 1].start) : line.end;
|
||||
|
||||
tl.to(el, { opacity: 0, y: -10, duration: 0.25, ease: "power2.in" }, hideTime - 0.25);
|
||||
});
|
||||
|
||||
window.__timelines["captions"] = tl;
|
||||
})();
|
||||
</script>
|
||||
</div>
|
||||
</template>
|
||||
@@ -0,0 +1,44 @@
|
||||
<div
|
||||
id="root"
|
||||
data-composition-id="main"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
>
|
||||
<video
|
||||
id="a-roll"
|
||||
src="__VIDEO_SRC__"
|
||||
muted
|
||||
playsinline
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="0"
|
||||
></video>
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="__VIDEO_SRC__"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="2"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
|
||||
<div
|
||||
id="captions-comp"
|
||||
data-composition-id="captions"
|
||||
data-composition-src="compositions/captions.html"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="3"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
></div>
|
||||
|
||||
<script src="https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js"></script>
|
||||
<script>
|
||||
window.__timelines = window.__timelines || {};
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
window.__timelines["main"] = tl;
|
||||
</script>
|
||||
</div>
|
||||
@@ -1,4 +1,4 @@
|
||||
export type TemplateId = "warm-grain" | "play-mode" | "swiss-grid" | "vignelli";
|
||||
export type TemplateId = "blank" | "warm-grain" | "play-mode" | "swiss-grid" | "vignelli";
|
||||
|
||||
export interface TemplateOption {
|
||||
id: TemplateId;
|
||||
@@ -7,6 +7,7 @@ export interface TemplateOption {
|
||||
}
|
||||
|
||||
export const TEMPLATES: TemplateOption[] = [
|
||||
{ id: "blank", label: "Blank", hint: "Empty composition — just the scaffolding" },
|
||||
{ id: "warm-grain", label: "Warm Grain", hint: "Cream aesthetic with grain texture" },
|
||||
{ id: "play-mode", label: "Play Mode", hint: "Playful elastic animations" },
|
||||
{ id: "swiss-grid", label: "Swiss Grid", hint: "Structured grid layout" },
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
<template id="captions-template">
|
||||
<div data-composition-id="captions" data-width="1920" data-height="1080" data-duration="16.04">
|
||||
<div
|
||||
data-composition-id="captions"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
>
|
||||
<div id="captions-container"></div>
|
||||
|
||||
<style>
|
||||
@@ -150,6 +155,7 @@
|
||||
);
|
||||
});
|
||||
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["captions"] = tl;
|
||||
})();
|
||||
</script>
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
id="intro-comp"
|
||||
data-composition-id="intro"
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
>
|
||||
@@ -99,6 +99,7 @@
|
||||
ease: "elastic.out(1, 0.3)",
|
||||
});
|
||||
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["intro"] = tl;
|
||||
})();
|
||||
</script>
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
<template id="stats-template">
|
||||
<div data-composition-id="stats" data-width="1920" data-height="1080" data-duration="16.04">
|
||||
<div
|
||||
data-composition-id="stats"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
>
|
||||
<div id="stats-container">
|
||||
<!-- Moment 1: 47% NEED MOTION GRAPHICS -->
|
||||
<div id="moment-1" class="moment" style="opacity: 0; transform: scale(0) rotate(8deg)">
|
||||
@@ -375,6 +380,7 @@
|
||||
}
|
||||
});
|
||||
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["stats"] = tl;
|
||||
})();
|
||||
</script>
|
||||
|
||||
@@ -91,7 +91,7 @@
|
||||
id="main-composition"
|
||||
data-composition-id="main-video"
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
>
|
||||
@@ -100,7 +100,7 @@
|
||||
id="bg-comp"
|
||||
data-composition-id="background"
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-track-index="1"
|
||||
@@ -134,7 +134,7 @@
|
||||
id="aroll-comp"
|
||||
data-composition-id="aroll-layer"
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-track-index="10"
|
||||
@@ -143,12 +143,22 @@
|
||||
<video
|
||||
id="short_mag_cut"
|
||||
src="__VIDEO_SRC__"
|
||||
muted
|
||||
playsinline
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="1"
|
||||
style="width: 100%; height: auto"
|
||||
></video>
|
||||
</div>
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="__VIDEO_SRC__"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="5"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
<style>
|
||||
#aroll-container {
|
||||
position: absolute;
|
||||
@@ -184,7 +194,7 @@
|
||||
data-composition-id="intro"
|
||||
data-composition-src="compositions/intro.html"
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-track-index="100"
|
||||
@@ -194,7 +204,7 @@
|
||||
data-composition-id="captions"
|
||||
data-composition-src="compositions/captions.html"
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-track-index="200"
|
||||
@@ -204,7 +214,7 @@
|
||||
data-composition-id="stats"
|
||||
data-composition-src="compositions/stats.html"
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-track-index="150"
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
<template id="captions-template">
|
||||
<div data-composition-id="captions" data-width="1920" data-height="1080" data-duration="16.04">
|
||||
<div
|
||||
data-composition-id="captions"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
>
|
||||
<div id="caption-container"></div>
|
||||
|
||||
<style>
|
||||
@@ -97,6 +102,7 @@
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
|
||||
if (!TRANSCRIPT || TRANSCRIPT.length === 0) {
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["captions"] = tl;
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -199,6 +199,7 @@
|
||||
S3_END - SLIDE_DUR,
|
||||
);
|
||||
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["graphics"] = tl;
|
||||
})();
|
||||
</script>
|
||||
|
||||
@@ -123,6 +123,7 @@
|
||||
0.7,
|
||||
);
|
||||
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["intro"] = tl;
|
||||
})();
|
||||
</script>
|
||||
|
||||
@@ -81,7 +81,8 @@
|
||||
data-composition-id="master"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-duration="16.04"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
>
|
||||
<!-- Background Grid -->
|
||||
<img
|
||||
@@ -90,7 +91,7 @@
|
||||
src="assets/swiss-grid.svg"
|
||||
alt="Grid"
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="0"
|
||||
/>
|
||||
|
||||
@@ -99,11 +100,21 @@
|
||||
<video
|
||||
id="short_mag_cut"
|
||||
src="__VIDEO_SRC__"
|
||||
muted
|
||||
playsinline
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="1"
|
||||
></video>
|
||||
</div>
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="__VIDEO_SRC__"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="5"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
|
||||
<!-- Intro Sub-composition -->
|
||||
<div
|
||||
@@ -123,7 +134,7 @@
|
||||
data-composition-id="graphics"
|
||||
data-composition-src="compositions/graphics.html"
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="3"
|
||||
></div>
|
||||
|
||||
@@ -134,7 +145,7 @@
|
||||
data-composition-id="captions"
|
||||
data-composition-src="compositions/captions.html"
|
||||
data-start="0"
|
||||
data-duration="16.04"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="4"
|
||||
></div>
|
||||
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
<template id="captions-template">
|
||||
<div data-composition-id="captions" data-width="1080" data-height="1920" data-duration="13.88">
|
||||
<div
|
||||
data-composition-id="captions"
|
||||
data-width="1080"
|
||||
data-height="1920"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
>
|
||||
<div id="captions-container"></div>
|
||||
|
||||
<style>
|
||||
@@ -168,6 +173,7 @@
|
||||
);
|
||||
});
|
||||
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["captions"] = tl;
|
||||
})();
|
||||
</script>
|
||||
|
||||
@@ -316,6 +316,7 @@
|
||||
);
|
||||
tl.from("#branding .logo-underline", { scaleX: 0, duration: 0.8, ease: "expo.out" }, 13.04);
|
||||
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["overlays"] = tl;
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -105,18 +105,29 @@
|
||||
data-composition-id="main-comp"
|
||||
data-width="1080"
|
||||
data-height="1920"
|
||||
data-duration="13.88"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
>
|
||||
<!-- A-Roll -->
|
||||
<div id="a-roll-wrapper">
|
||||
<video
|
||||
id="a-roll"
|
||||
src="__VIDEO_SRC__"
|
||||
muted
|
||||
playsinline
|
||||
data-start="0"
|
||||
data-duration="13.88"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="0"
|
||||
></video>
|
||||
</div>
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="__VIDEO_SRC__"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="5"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
|
||||
<!-- Transitions -->
|
||||
<div id="curtain-black" class="curtain"></div>
|
||||
@@ -128,7 +139,7 @@
|
||||
data-composition-id="overlays"
|
||||
data-composition-src="compositions/overlays.html"
|
||||
data-start="0"
|
||||
data-duration="13.88"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="2"
|
||||
></div>
|
||||
|
||||
@@ -138,7 +149,7 @@
|
||||
data-composition-id="captions"
|
||||
data-composition-src="compositions/captions.html"
|
||||
data-start="0"
|
||||
data-duration="13.88"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="3"
|
||||
></div>
|
||||
|
||||
@@ -171,6 +182,7 @@
|
||||
tl.to("#a-roll-wrapper", { opacity: 1, duration: 0 }, 7.3);
|
||||
tl.to("#curtain-red", { left: "100%", duration: 0.4, ease: "power2.inOut" }, 7.3);
|
||||
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["main-comp"] = tl;
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
<template id="captions-template">
|
||||
<div data-composition-id="captions" data-width="1920" data-height="1080" data-duration="18">
|
||||
<div
|
||||
data-composition-id="captions"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
>
|
||||
<div class="captions-container">
|
||||
<div id="caption-box" class="caption-box">
|
||||
<span id="caption-text" class="caption-text"></span>
|
||||
@@ -134,6 +139,7 @@
|
||||
);
|
||||
});
|
||||
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["captions"] = tl;
|
||||
})();
|
||||
</script>
|
||||
|
||||
@@ -156,6 +156,7 @@
|
||||
// Hold until end (14s)
|
||||
tl.to("#moment-3", { opacity: 0, duration: 0.5, ease: "power2.in" }, 13.5);
|
||||
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["graphics"] = tl;
|
||||
})();
|
||||
</script>
|
||||
|
||||
@@ -78,6 +78,7 @@
|
||||
1.8,
|
||||
);
|
||||
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["intro"] = tl;
|
||||
})();
|
||||
</script>
|
||||
|
||||
@@ -104,7 +104,8 @@
|
||||
data-composition-id="main-video"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-duration="17"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
>
|
||||
<!-- Background Layer -->
|
||||
<div
|
||||
@@ -112,24 +113,36 @@
|
||||
data-composition-id="grain-overlay"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
data-duration="17"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="100"
|
||||
>
|
||||
<div class="grain-texture"></div>
|
||||
<script>
|
||||
const grainTl = gsap.timeline({ paused: true });
|
||||
window.__timelines = window.__timelines || {};
|
||||
window.__timelines["grain-overlay"] = grainTl;
|
||||
</script>
|
||||
</div>
|
||||
|
||||
<!-- A-Roll Video -->
|
||||
<!-- A-Roll Video (muted — audio is separate) -->
|
||||
<video
|
||||
id="a-roll"
|
||||
src="__VIDEO_SRC__"
|
||||
muted
|
||||
playsinline
|
||||
data-start="0"
|
||||
data-duration="17"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="0"
|
||||
></video>
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="__VIDEO_SRC__"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="5"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
|
||||
<!-- Compositions -->
|
||||
<div
|
||||
@@ -148,7 +161,7 @@
|
||||
data-composition-id="graphics"
|
||||
data-composition-src="compositions/graphics.html"
|
||||
data-start="0"
|
||||
data-duration="17"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="2"
|
||||
></div>
|
||||
|
||||
@@ -158,7 +171,7 @@
|
||||
data-composition-id="captions"
|
||||
data-composition-src="compositions/captions.html"
|
||||
data-start="0"
|
||||
data-duration="17"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="3"
|
||||
></div>
|
||||
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { existsSync, mkdirSync, createWriteStream, rmSync } from "node:fs";
|
||||
import { homedir, platform } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { get as httpsGet } from "node:https";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
|
||||
const MODELS_DIR = join(homedir(), ".cache", "hyperframes", "whisper", "models");
|
||||
const DEFAULT_MODEL = "base.en";
|
||||
|
||||
export type WhisperSource = "env" | "system" | "brew" | "build";
|
||||
|
||||
export interface WhisperResult {
|
||||
executablePath: string;
|
||||
source: WhisperSource;
|
||||
}
|
||||
|
||||
// --- Download helper --------------------------------------------------------
|
||||
|
||||
function downloadFile(url: string, dest: string): Promise<void> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const follow = (u: string) => {
|
||||
httpsGet(u, (res) => {
|
||||
if (res.statusCode === 301 || res.statusCode === 302) {
|
||||
const location = res.headers.location;
|
||||
if (location) {
|
||||
follow(location);
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (res.statusCode !== 200) {
|
||||
reject(new Error(`Download failed: HTTP ${res.statusCode}`));
|
||||
return;
|
||||
}
|
||||
const file = createWriteStream(dest);
|
||||
pipeline(res, file).then(resolve).catch(reject);
|
||||
}).on("error", reject);
|
||||
};
|
||||
follow(url);
|
||||
});
|
||||
}
|
||||
|
||||
function getModelUrl(model: string): string {
|
||||
return `https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-${model}.bin`;
|
||||
}
|
||||
|
||||
// --- Find helpers -----------------------------------------------------------
|
||||
|
||||
function whichBinary(name: string): string | undefined {
|
||||
try {
|
||||
const result = execFileSync("which", [name], {
|
||||
encoding: "utf-8",
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
timeout: 5000,
|
||||
}).trim();
|
||||
return result || undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function findFromEnv(): WhisperResult | undefined {
|
||||
const envPath = process.env["HYPERFRAMES_WHISPER_PATH"];
|
||||
if (envPath && existsSync(envPath)) {
|
||||
return { executablePath: envPath, source: "env" };
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function findFromSystem(): WhisperResult | undefined {
|
||||
for (const name of ["whisper-cli", "whisper"]) {
|
||||
const path = whichBinary(name);
|
||||
if (path) return { executablePath: path, source: "system" };
|
||||
}
|
||||
|
||||
// Check brew paths directly on macOS
|
||||
if (platform() === "darwin") {
|
||||
for (const p of ["/opt/homebrew/bin/whisper-cli", "/usr/local/bin/whisper-cli"]) {
|
||||
if (existsSync(p)) return { executablePath: p, source: "system" };
|
||||
}
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
// --- Build from source ------------------------------------------------------
|
||||
|
||||
const BUILD_DIR = join(homedir(), ".cache", "hyperframes", "whisper", "whisper.cpp");
|
||||
const WHISPER_REPO = "https://github.com/ggml-org/whisper.cpp.git";
|
||||
|
||||
function findBuiltBinary(): WhisperResult | undefined {
|
||||
for (const p of [
|
||||
join(BUILD_DIR, "build", "bin", "whisper-cli"),
|
||||
join(BUILD_DIR, "build", "whisper-cli"),
|
||||
]) {
|
||||
if (existsSync(p)) return { executablePath: p, source: "build" };
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function buildFromSource(onProgress?: (msg: string) => void): WhisperResult {
|
||||
// Clean stale builds — if BUILD_DIR exists but has no binary, nuke and re-clone
|
||||
if (existsSync(BUILD_DIR) && !findBuiltBinary()) {
|
||||
rmSync(BUILD_DIR, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
if (!existsSync(BUILD_DIR)) {
|
||||
onProgress?.("Downloading whisper.cpp...");
|
||||
mkdirSync(join(homedir(), ".cache", "hyperframes", "whisper"), {
|
||||
recursive: true,
|
||||
});
|
||||
execFileSync("git", ["clone", "--depth", "1", WHISPER_REPO, BUILD_DIR], {
|
||||
stdio: "ignore",
|
||||
timeout: 60_000,
|
||||
env: { ...process.env, GIT_TERMINAL_PROMPT: "0" },
|
||||
});
|
||||
}
|
||||
|
||||
onProgress?.("Building whisper.cpp (this may take a minute)...");
|
||||
try {
|
||||
execFileSync("cmake", ["-B", "build"], {
|
||||
cwd: BUILD_DIR,
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
timeout: 120_000,
|
||||
});
|
||||
execFileSync("cmake", ["--build", "build", "--config", "Release", "-j"], {
|
||||
cwd: BUILD_DIR,
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
timeout: 300_000,
|
||||
});
|
||||
} catch (err: unknown) {
|
||||
// Build failed — capture diagnostics, then clean up so next attempt starts fresh
|
||||
let detail = "";
|
||||
if (err && typeof err === "object" && "stderr" in err) {
|
||||
const stderr = String(err.stderr).trim();
|
||||
if (stderr) detail = `\n${stderr.slice(-500)}`;
|
||||
}
|
||||
rmSync(BUILD_DIR, { recursive: true, force: true });
|
||||
throw new Error(
|
||||
`whisper-cpp build failed. Ensure cmake and a C compiler are installed.${detail}`,
|
||||
);
|
||||
}
|
||||
|
||||
const result = findBuiltBinary();
|
||||
if (!result) throw new Error("Build completed but whisper-cli not found");
|
||||
return result;
|
||||
}
|
||||
|
||||
// --- Public API -------------------------------------------------------------
|
||||
|
||||
export function findWhisper(): WhisperResult | undefined {
|
||||
return findFromEnv() ?? findFromSystem() ?? findBuiltBinary();
|
||||
}
|
||||
|
||||
export function getInstallInstructions(): string {
|
||||
if (platform() === "darwin") {
|
||||
return "brew install whisper-cpp";
|
||||
}
|
||||
return "See https://github.com/ggml-org/whisper.cpp#building";
|
||||
}
|
||||
|
||||
function hasBrew(): boolean {
|
||||
return whichBinary("brew") !== undefined;
|
||||
}
|
||||
|
||||
function hasGit(): boolean {
|
||||
return whichBinary("git") !== undefined;
|
||||
}
|
||||
|
||||
function hasCmake(): boolean {
|
||||
return whichBinary("cmake") !== undefined;
|
||||
}
|
||||
|
||||
export async function ensureWhisper(options?: {
|
||||
onProgress?: (msg: string) => void;
|
||||
}): Promise<WhisperResult> {
|
||||
// 1. Already installed?
|
||||
const existing = findWhisper();
|
||||
if (existing) return existing;
|
||||
|
||||
// 2. Try brew (macOS, fastest — pre-built bottle)
|
||||
if (platform() === "darwin" && hasBrew()) {
|
||||
options?.onProgress?.("Installing whisper-cpp via Homebrew...");
|
||||
try {
|
||||
execFileSync("brew", ["install", "whisper-cpp"], {
|
||||
stdio: "ignore",
|
||||
timeout: 300_000,
|
||||
});
|
||||
const installed = findFromSystem();
|
||||
if (installed) return { ...installed, source: "brew" };
|
||||
} catch {
|
||||
// brew failed — fall through
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Build from source (needs git + cmake + C compiler)
|
||||
if (hasGit() && hasCmake()) {
|
||||
try {
|
||||
return buildFromSource(options?.onProgress);
|
||||
} catch {
|
||||
// build failed — fall through
|
||||
}
|
||||
}
|
||||
|
||||
// 4. Give up — tell the user how
|
||||
throw new Error(`whisper-cpp not found. Install: ${getInstallInstructions()}`);
|
||||
}
|
||||
|
||||
export async function ensureModel(
|
||||
model: string = DEFAULT_MODEL,
|
||||
options?: { onProgress?: (message: string) => void },
|
||||
): Promise<string> {
|
||||
const modelPath = join(MODELS_DIR, `ggml-${model}.bin`);
|
||||
if (existsSync(modelPath)) return modelPath;
|
||||
|
||||
mkdirSync(MODELS_DIR, { recursive: true });
|
||||
|
||||
options?.onProgress?.(`Downloading model ${model}...`);
|
||||
await downloadFile(getModelUrl(model), modelPath);
|
||||
|
||||
if (!existsSync(modelPath)) {
|
||||
throw new Error(`Model download failed: ${model}`);
|
||||
}
|
||||
|
||||
return modelPath;
|
||||
}
|
||||
|
||||
export function hasFFmpeg(): boolean {
|
||||
try {
|
||||
execFileSync("ffmpeg", ["-version"], { stdio: "ignore", timeout: 5000 });
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export { MODELS_DIR, DEFAULT_MODEL };
|
||||
@@ -0,0 +1,179 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { existsSync, readFileSync, mkdirSync, unlinkSync } from "node:fs";
|
||||
import { join, extname } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { ensureWhisper, ensureModel, hasFFmpeg, DEFAULT_MODEL } from "./manager.js";
|
||||
|
||||
const AUDIO_EXTENSIONS = new Set([".mp3", ".wav", ".m4a", ".aac", ".ogg", ".flac"]);
|
||||
const VIDEO_EXTENSIONS = new Set([".mp4", ".webm", ".mov", ".mkv", ".avi"]);
|
||||
|
||||
export interface TranscribeOptions {
|
||||
model?: string;
|
||||
onProgress?: (message: string) => void;
|
||||
}
|
||||
|
||||
export interface TranscribeResult {
|
||||
transcriptPath: string;
|
||||
wordCount: number;
|
||||
durationSeconds: number;
|
||||
}
|
||||
|
||||
function isAudioFile(filePath: string): boolean {
|
||||
return AUDIO_EXTENSIONS.has(extname(filePath).toLowerCase());
|
||||
}
|
||||
|
||||
function isVideoFile(filePath: string): boolean {
|
||||
return VIDEO_EXTENSIONS.has(extname(filePath).toLowerCase());
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract audio from a video file as 16kHz mono WAV (whisper requirement).
|
||||
*/
|
||||
function extractAudio(videoPath: string): string {
|
||||
const wavPath = join(tmpdir(), `hyperframes-audio-${Date.now()}.wav`);
|
||||
execFileSync(
|
||||
"ffmpeg",
|
||||
["-i", videoPath, "-vn", "-ar", "16000", "-ac", "1", "-f", "wav", "-y", wavPath],
|
||||
{ stdio: "ignore", timeout: 120_000 },
|
||||
);
|
||||
return wavPath;
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if a WAV file is already 16kHz mono via ffprobe.
|
||||
*/
|
||||
function isWav16kMono(filePath: string): boolean {
|
||||
try {
|
||||
const raw = execFileSync(
|
||||
"ffprobe",
|
||||
["-v", "quiet", "-print_format", "json", "-show_streams", filePath],
|
||||
{ encoding: "utf-8", timeout: 10_000 },
|
||||
);
|
||||
const parsed: {
|
||||
streams?: {
|
||||
codec_type?: string;
|
||||
sample_rate?: string;
|
||||
channels?: number;
|
||||
}[];
|
||||
} = JSON.parse(raw);
|
||||
const audio = parsed.streams?.find((s) => s.codec_type === "audio");
|
||||
return audio?.sample_rate === "16000" && audio?.channels === 1;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert audio file to 16kHz mono WAV if not already in that format.
|
||||
*/
|
||||
function prepareAudio(audioPath: string): string {
|
||||
if (extname(audioPath).toLowerCase() === ".wav" && isWav16kMono(audioPath)) {
|
||||
return audioPath;
|
||||
}
|
||||
|
||||
// Convert to whisper-compatible WAV
|
||||
const wavPath = join(tmpdir(), `hyperframes-audio-${Date.now()}.wav`);
|
||||
execFileSync(
|
||||
"ffmpeg",
|
||||
["-i", audioPath, "-ar", "16000", "-ac", "1", "-f", "wav", "-y", wavPath],
|
||||
{ stdio: "ignore", timeout: 120_000 },
|
||||
);
|
||||
return wavPath;
|
||||
}
|
||||
|
||||
/**
|
||||
* Transcribe an audio or video file and save transcript.json to the output directory.
|
||||
*/
|
||||
export async function transcribe(
|
||||
inputPath: string,
|
||||
outputDir: string,
|
||||
options?: TranscribeOptions,
|
||||
): Promise<TranscribeResult> {
|
||||
const model = options?.model ?? DEFAULT_MODEL;
|
||||
|
||||
// 1. Ensure whisper binary
|
||||
options?.onProgress?.("Checking whisper...");
|
||||
const whisper = await ensureWhisper({ onProgress: options?.onProgress });
|
||||
|
||||
// 2. Ensure model
|
||||
options?.onProgress?.("Checking model...");
|
||||
const modelPath = await ensureModel(model, {
|
||||
onProgress: options?.onProgress,
|
||||
});
|
||||
|
||||
// 3. Prepare audio
|
||||
let wavPath: string;
|
||||
const ext = extname(inputPath).toLowerCase();
|
||||
|
||||
if (isAudioFile(inputPath)) {
|
||||
options?.onProgress?.("Preparing audio...");
|
||||
wavPath = prepareAudio(inputPath);
|
||||
} else if (isVideoFile(inputPath)) {
|
||||
if (!hasFFmpeg()) {
|
||||
throw new Error(
|
||||
"ffmpeg is required to extract audio from video. Install: brew install ffmpeg",
|
||||
);
|
||||
}
|
||||
options?.onProgress?.("Extracting audio from video...");
|
||||
wavPath = extractAudio(inputPath);
|
||||
} else {
|
||||
throw new Error(`Unsupported file type: ${ext}`);
|
||||
}
|
||||
|
||||
// 4. Run whisper
|
||||
options?.onProgress?.("Transcribing...");
|
||||
const outputBase = join(outputDir, "transcript");
|
||||
mkdirSync(outputDir, { recursive: true });
|
||||
|
||||
execFileSync(
|
||||
whisper.executablePath,
|
||||
[
|
||||
"--model",
|
||||
modelPath,
|
||||
"--output-json-full",
|
||||
"--output-file",
|
||||
outputBase,
|
||||
"--dtw",
|
||||
model,
|
||||
"--suppress-nst",
|
||||
wavPath,
|
||||
],
|
||||
{ stdio: "ignore", timeout: 300_000 },
|
||||
);
|
||||
|
||||
// 5. Read and validate output
|
||||
const transcriptPath = `${outputBase}.json`;
|
||||
if (!existsSync(transcriptPath)) {
|
||||
throw new Error("Whisper did not produce output. Check the input file.");
|
||||
}
|
||||
|
||||
const transcript = JSON.parse(readFileSync(transcriptPath, "utf-8"));
|
||||
const segments = transcript.transcription ?? [];
|
||||
|
||||
let wordCount = 0;
|
||||
let maxEnd = 0;
|
||||
for (const seg of segments) {
|
||||
for (const token of seg.tokens ?? []) {
|
||||
const text = (token.text ?? "").trim();
|
||||
if (text && !text.startsWith("[_") && !text.startsWith("[BLANK")) wordCount++;
|
||||
if (token.offsets?.to > maxEnd) maxEnd = token.offsets.to;
|
||||
}
|
||||
}
|
||||
|
||||
// Clean up temp WAV if we created one
|
||||
if (wavPath !== inputPath) {
|
||||
try {
|
||||
unlinkSync(wavPath);
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
transcriptPath,
|
||||
wordCount,
|
||||
durationSeconds: maxEnd / 1000,
|
||||
};
|
||||
}
|
||||
|
||||
export { isAudioFile, isVideoFile };
|
||||
Reference in New Issue
Block a user