Files
hyperframes/packages/studio/vite.config.ts
Vance IngallsandClaude Opus 4.6 f4367d5726 feat(cli): add whisper transcription and template improvements (#53)
* feat(cli): add whisper transcription to init flow

New modules:
- whisper/manager.ts: download/cache whisper.cpp binary + model
  (~/.cache/hyperframes/whisper/)
- whisper/transcribe.ts: extract audio, run whisper, save transcript.json

Init flow changes:
- "Got a video or audio file?" now accepts audio-only files (mp3, wav, m4a)
- "Generate captions from audio?" prompt after file selection
- Transcription produces transcript.json in project root
- Graceful fallback if whisper/ffmpeg unavailable

Supports: macOS ARM64/x86, Linux x86_64. Downloads whisper.cpp v1.7.3
from GitHub releases and ggml-base.en model from Hugging Face.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): use brew/system whisper instead of downloading binaries

whisper.cpp doesn't ship pre-built macOS/Linux CLI binaries.
Use brew install whisper-cpp on macOS (auto-installs if brew available),
system PATH lookup otherwise. Model still downloaded from Hugging Face.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): simplify whisper install — detect or instruct, don't build

Remove build-from-source complexity. If whisper-cpp is found on PATH,
use it. If not, show install instructions instead of blocking:

  "To generate captions, install whisper-cpp: brew install whisper-cpp"

The transcription prompt only appears when whisper is available.
When it's not, the user sees the install command and can re-run init.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(cli): auto-install whisper via brew or build from source

ensureWhisper() now tries 4 strategies in order:
1. System PATH (whisper-cli or whisper already installed)
2. Homebrew (macOS: brew install whisper-cpp)
3. Build from source (git clone + cmake, ~30-60s)
4. Show install instructions as last resort

Init flow always asks "Generate captions?" — whisper is installed
automatically in the background if needed. No user intervention
required on macOS with Xcode CLI tools or any system with git+cmake.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): add window.__timelines guard to all templates

The studio bundler doesn't always initialize window.__timelines
before template scripts run, causing "Cannot set properties of
undefined" errors. Add defensive guard to every template.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(cli): patch template captions with actual transcript data

After scaffolding, if transcript.json exists, replace the hardcoded
word array in the template's captions composition with the real
transcript data. The template's caption animation and styling are
preserved — only the word data changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): show install notice when whisper needs to be installed

When whisper-cpp isn't found, show an info message before the spinner:
"whisper-cpp not found — installing automatically..."
Then the spinner shows "Installing whisper-cpp (this may take a moment)..."

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): add muted and playsinline to all template video elements

The framework requires video elements to have muted and playsinline
attributes. All four templates were missing these, causing video
to not play in the studio preview.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): flat asset structure + separate audio tracks in templates

Assets: video, images, fonts all go at project root (not assets/ or
fonts/ subdirectories). The studio preview can't resolve relative
paths from subdirectories due to the /preview URL suffix.

Audio: added <audio> elements alongside muted <video> in all 4
templates so the video's audio plays back. The framework requires
muted video + separate audio element.

Removed assets/ and fonts/ directory creation from scaffoldProject.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(studio): inject base tag for asset resolution in preview

The preview iframe serves bundled HTML from /api/projects/:id/preview
but relative asset paths (video.mp4, font.woff2) resolve to the wrong
URL without a <base> tag. Now injects <base href="/api/projects/:id/preview/">
so relative paths route through the static asset handler.

Also adds proper MIME types for video, audio, image, and font files
served from the preview asset route.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(studio): serve HyperFrames runtime in dev mode

The preview runtime script had an empty src — the framework never
loaded, so video playback and clip lifecycle didn't work.

Now auto-detects packages/cli/dist/hyperframe-runtime.js and serves
it at /api/runtime.js. No env var needed in dev mode.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): filter whisper special tokens from transcript

Use --output-json instead of --output-json-full to avoid special
tokens like [_TT_485] and [BLANK_AUDIO]. Also filter remaining
bracket tokens when building the word array for captions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): use --output-json-full for word-level timestamps

--output-json only produces segment-level timing (no tokens).
--output-json-full is required for word-level timestamps that
the captions template needs. Special tokens are filtered out
by the patchTranscript function.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(cli): patch template durations to match uploaded video

Templates now use __VIDEO_DURATION__ placeholder that gets replaced
with the actual probed video duration. All data-duration values on
the root composition, video, audio, and caption clips are updated.

Without a video, defaults to 10 seconds.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): merge punctuation tokens with preceding word

Whisper outputs punctuation (. , ! ?) as separate tokens. These
appeared as standalone words in captions, sometimes in the wrong
group. Now merged with the preceding word during transcript
normalization.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): match both TRANSCRIPT and script variable names in templates

Three templates use `const TRANSCRIPT = [...]` while warm-grain uses
`const script = [...]`. The patchTranscript function now matches both.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): security and template fixes

- Replace shell injection risk (execSync rm) with unlinkSync in transcribe.ts
- Add GIT_TERMINAL_PROMPT=0 to whisper buildFromSource git clone
- Fix hardcoded data-duration="18" in warm-grain captions template
- Add data-start="0" to root compositions in swiss-grid, vignelli, warm-grain
- Add data-start="0" to warm-grain grain-overlay composition
- Deduplicate hasFFmpeg: remove from init.ts, import from whisper/manager.ts
- Add my-video/ and packages/studio/data/ to .gitignore

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): format warm-grain captions and fix TS nullability errors

- Format warm-grain/compositions/captions.html
- Add optional chaining on token.offsets (may be undefined)
- Use intermediate variable for lastWord to satisfy TS strict checks

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(cli): add blank template option, smart defaults for video vs audio

- Blank template: minimal scaffolding (root composition, video, audio,
  GSAP timeline) with __VIDEO_SRC__ and __VIDEO_DURATION__ placeholders
- Template defaults: video uploads default to "blank" (user brings
  their own content), audio-only defaults to "warm-grain" (motion
  graphics template since there's no video to show)
- Audio-only projects now tracked with isAudioOnly flag

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): address whisper review feedback

- Clean stale builds: if BUILD_DIR exists but no binary, nuke and retry
- Build failures clean up BUILD_DIR so next attempt starts fresh
- patchTranscript regex scoped within <script> blocks to prevent
  matching across block boundaries
- Removed hardcoded model size hint (~148MB)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): add missing rmSync import to whisper manager

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: remove test project and lock file

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): address review items 7-12 — execFileSync, build diagnostics, WAV verification

- manager.ts: replace all execSync with execFileSync to prevent command injection
- manager.ts: capture cmake stderr and include in build failure error message
- transcribe.ts: verify WAV is 16kHz mono via ffprobe before passing to whisper
- init.ts: replace fragile JSON formatting with JSON.stringify(words, null, 2)
- init.ts: fix default duration from "10" to "5" matching DEFAULT_META
- init.ts: add probeAudioDuration() and --audio/--skip-transcribe flags
- init.ts: extract finalizeProject() to reduce code path duplication
- init.ts: wire transcription into non-interactive path

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 11:05:15 -07:00

423 lines
16 KiB
TypeScript

import { defineConfig, type Plugin } from "vite";
import react from "@vitejs/plugin-react";
import {
readFileSync,
readdirSync,
existsSync,
statSync,
writeFileSync,
lstatSync,
realpathSync,
} from "node:fs";
import { join, resolve, sep } from "node:path";
/** Reject paths that escape the project directory. */
function isSafePath(base: string, resolved: string): boolean {
const norm = resolve(base) + sep;
return resolved.startsWith(norm) || resolved === resolve(base);
}
// Lazy-load the bundler via Vite's SSR module loader (resolves .ts imports correctly)
let _bundler: ((dir: string) => Promise<string>) | null = null;
/** Minimal project API for standalone dev mode */
function devProjectApi(): Plugin {
const dataDir = resolve(__dirname, "data/projects");
return {
name: "studio-dev-api",
configureServer(server): void {
// Load the bundler via Vite's SSR module loader (resolves .ts imports)
const getBundler = async () => {
if (!_bundler) {
try {
const mod = await server.ssrLoadModule("@hyperframes/core/compiler");
_bundler = (dir: string) => mod.bundleToSingleHtml(dir);
} catch (err) {
console.warn("[Studio] Failed to load compiler, previews will use raw HTML:", err);
_bundler = null as never;
}
}
return _bundler;
};
server.middlewares.use(async (req, res, next) => {
if (!req.url?.startsWith("/api/")) return next();
// Render endpoints — not yet wired up in standalone studio
if (
req.url.startsWith("/api/render/") ||
(req.method === "POST" && req.url.match(/\/api\/projects\/[^/]+\/render/))
) {
res.writeHead(501, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "Render not available in standalone studio mode" }));
return;
}
// GET /api/runtime.js — serve the HyperFrames runtime
if (req.method === "GET" && req.url === "/api/runtime.js") {
const cliRuntime = resolve(__dirname, "..", "cli", "dist", "hyperframe-runtime.js");
if (existsSync(cliRuntime)) {
res.writeHead(200, { "Content-Type": "text/javascript", "Cache-Control": "no-store" });
res.end(readFileSync(cliRuntime, "utf-8"));
} else {
res.writeHead(404);
res.end("runtime not built");
}
return;
}
// GET /api/projects — list all projects with session metadata
if (req.method === "GET" && (req.url === "/api/projects" || req.url === "/api/projects/")) {
// Build session → project mapping for titles
const sessionsDir = resolve(dataDir, "../sessions");
const sessionMap = new Map<string, { sessionId: string; title: string }>();
if (existsSync(sessionsDir)) {
for (const file of readdirSync(sessionsDir).filter((f) => f.endsWith(".json"))) {
try {
const raw = JSON.parse(readFileSync(join(sessionsDir, file), "utf-8"));
if (raw.projectId) {
sessionMap.set(raw.projectId, {
sessionId: file.replace(".json", ""),
title: raw.title || "Untitled",
});
}
} catch {
/* skip corrupt */
}
}
}
const projects = readdirSync(dataDir, { withFileTypes: true })
.filter(
(d) =>
(d.isDirectory() || d.isSymbolicLink()) &&
existsSync(join(dataDir, d.name, "index.html")),
)
.map((d) => {
const session = sessionMap.get(d.name);
return { id: d.name, title: session?.title ?? d.name, sessionId: session?.sessionId };
})
.sort((a, b) => a.title.localeCompare(b.title));
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify({ projects }));
return;
}
// GET /api/resolve-session/:sessionId — resolve session ID to project ID
const sessionMatch = req.url.match(/^\/api\/resolve-session\/([^/]+)/);
if (req.method === "GET" && sessionMatch) {
const sessionsDir = resolve(dataDir, "../sessions");
const sessionFile = join(sessionsDir, `${sessionMatch[1]}.json`);
if (existsSync(sessionFile)) {
try {
const raw = JSON.parse(readFileSync(sessionFile, "utf-8"));
if (raw.projectId) {
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify({ projectId: raw.projectId, title: raw.title }));
return;
}
} catch {
/* ignore */
}
}
res.writeHead(404, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "Session not found" }));
return;
}
const match = req.url.match(/^\/api\/projects\/([^/]+)(.*)/);
if (!match) return next();
let [, projectId, rest] = match;
let projectDir = join(dataDir, projectId);
// If project ID not found, try resolving it as a session ID
if (!existsSync(projectDir)) {
const sessionsDir = resolve(dataDir, "../sessions");
const sessionFile = join(sessionsDir, `${projectId}.json`);
if (existsSync(sessionFile)) {
try {
const session = JSON.parse(readFileSync(sessionFile, "utf-8"));
if (session.projectId) {
projectId = session.projectId;
projectDir = join(dataDir, projectId);
}
} catch {
/* ignore */
}
}
}
if (!existsSync(projectDir)) {
res.writeHead(404, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "not found" }));
return;
}
// GET /api/projects/:id
if (req.method === "GET" && !rest) {
const files: string[] = [];
function walk(d: string, prefix: string) {
for (const entry of readdirSync(d, { withFileTypes: true })) {
const rel = prefix ? `${prefix}/${entry.name}` : entry.name;
if (entry.isDirectory()) walk(join(d, entry.name), rel);
else files.push(rel);
}
}
walk(projectDir, "");
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify({ id: projectId, files }));
return;
}
// GET /api/projects/:id/preview — bundle and serve the full composition
if (req.method === "GET" && rest === "/preview") {
try {
const bundler = await getBundler();
let bundled = bundler
? await bundler(projectDir)
: readFileSync(join(projectDir, "index.html"), "utf-8");
// Inject <base> so relative asset paths resolve through /preview/ route
const baseTag = `<base href="/api/projects/${projectId}/preview/">`;
if (bundled.includes("<head>")) {
bundled = bundled.replace("<head>", `<head>${baseTag}`);
} else {
bundled = baseTag + bundled;
}
// Inject runtime if available and not already set
const cliRuntime = resolve(__dirname, "..", "cli", "dist", "hyperframe-runtime.js");
if (existsSync(cliRuntime) && bundled.includes('src=""')) {
bundled = bundled.replace(
'data-hyperframes-preview-runtime="1" src=""',
'data-hyperframes-preview-runtime="1" src="/api/runtime.js"',
);
}
res.writeHead(200, {
"Content-Type": "text/html; charset=utf-8",
"Cache-Control": "no-store",
});
res.end(bundled);
} catch {
// Fallback to raw HTML if bundling fails
const file = join(projectDir, "index.html");
if (existsSync(file)) {
res.writeHead(200, { "Content-Type": "text/html; charset=utf-8" });
res.end(readFileSync(file, "utf-8"));
} else {
res.writeHead(404);
res.end("not found");
}
}
return;
}
// GET /api/projects/:id/preview/comp/* — serve sub-composition as standalone playable page
if (req.method === "GET" && rest.startsWith("/preview/comp/")) {
const compPath = decodeURIComponent(rest.replace("/preview/comp/", "").split("?")[0]);
const compFile = resolve(projectDir, compPath);
if (
!isSafePath(projectDir, compFile) ||
!existsSync(compFile) ||
!statSync(compFile).isFile()
) {
res.writeHead(404);
res.end("not found");
return;
}
let rawComp = readFileSync(compFile, "utf-8");
// Extract content from <template> if present
const templateMatch = rawComp.match(/<template>([\s\S]*)<\/template>/i);
let content = templateMatch ? templateMatch[1] : rawComp;
// Inline nested data-composition-src references (keep the attr for drill-down navigation)
content = content.replace(
/(<[^>]*?)(data-composition-src=["']([^"']+)["'])([^>]*>)/g,
(_match, before, srcAttr, src, after) => {
const nestedFile = join(projectDir, src);
if (!existsSync(nestedFile)) return before + srcAttr + after;
const nestedRaw = readFileSync(nestedFile, "utf-8");
const nestedTemplate = nestedRaw.match(/<template>([\s\S]*)<\/template>/i);
const nestedContent = nestedTemplate ? nestedTemplate[1] : nestedRaw;
// Extract styles, scripts, and body from nested content
const styles: string[] = [];
const scripts: string[] = [];
let body = nestedContent
.replace(/<style[^>]*>([\s\S]*?)<\/style>/gi, (_, css) => {
styles.push(css);
return "";
})
.replace(/<script[^>]*>([\s\S]*?)<\/script>/gi, (_, js) => {
scripts.push(js);
return "";
});
// Find the inner root with data-composition-id and use its children
const innerRootMatch = body.match(
/<([a-z][a-z0-9]*)\b[^>]*data-composition-id[^>]*>([\s\S]*)<\/\1>/i,
);
const innerHTML = innerRootMatch ? innerRootMatch[2] : body;
// Keep data-composition-src on the host for drill-down URL resolution
return (
before +
srcAttr +
after.replace(/>$/, ">") +
innerHTML +
(styles.length ? `<style>${styles.join("\n")}</style>` : "") +
(scripts.length
? `<script>${scripts.map((s) => `(function(){try{${s}}catch(e){}})();`).join("\n")}</script>`
: "")
);
},
);
// Build a standalone HTML page with GSAP + runtime
// Resolve runtime: env var → built CLI dist → empty (no runtime)
let runtimeUrl = (process.env.HYPERFRAME_RUNTIME_URL || "").trim();
if (!runtimeUrl) {
// In dev: serve the built runtime from the CLI dist
const cliRuntime = resolve(__dirname, "..", "cli", "dist", "hyperframe-runtime.js");
if (existsSync(cliRuntime)) {
runtimeUrl = `/api/runtime.js`;
}
}
const standalone = `<!DOCTYPE html>
<html>
<head>
<script src="https://cdn.jsdelivr.net/npm/gsap@3/dist/gsap.min.js"></script>
<script data-hyperframes-preview-runtime="1" src="${runtimeUrl}"></script>
</head>
<body>
${content}
</body>
</html>`;
res.writeHead(200, {
"Content-Type": "text/html; charset=utf-8",
"Cache-Control": "no-store",
});
res.end(standalone);
return;
}
// GET /api/projects/:id/preview/* — serve static assets (images, audio, etc.)
if (req.method === "GET" && rest.startsWith("/preview/")) {
const subPath = decodeURIComponent(rest.replace("/preview/", "").split("?")[0]);
const file = resolve(projectDir, subPath);
if (!isSafePath(projectDir, file) || !existsSync(file) || !statSync(file).isFile()) {
res.writeHead(404);
res.end("not found");
return;
}
const isText = /\.(html|css|js|json|svg|txt)$/i.test(subPath);
const mimeTypes: Record<string, string> = {
".html": "text/html",
".js": "text/javascript",
".css": "text/css",
".json": "application/json",
".svg": "image/svg+xml",
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".webp": "image/webp",
".gif": "image/gif",
".mp4": "video/mp4",
".webm": "video/webm",
".mp3": "audio/mpeg",
".wav": "audio/wav",
".m4a": "audio/mp4",
".ogg": "audio/ogg",
".woff2": "font/woff2",
".woff": "font/woff",
".ttf": "font/ttf",
};
const ext = "." + subPath.split(".").pop()?.toLowerCase();
const contentType = mimeTypes[ext] ?? "application/octet-stream";
res.writeHead(200, { "Content-Type": contentType });
res.end(readFileSync(file, isText ? "utf-8" : undefined));
return;
}
// GET /api/projects/:id/files/:path — returns JSON { filename, content }
if (req.method === "GET" && rest.startsWith("/files/")) {
const filePath = decodeURIComponent(rest.replace("/files/", ""));
const file = resolve(projectDir, filePath);
if (!isSafePath(projectDir, file) || !existsSync(file)) {
res.writeHead(404);
res.end("not found");
return;
}
const content = readFileSync(file, "utf-8");
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify({ filename: filePath, content }));
return;
}
// PUT /api/projects/:id/files/:path
if (req.method === "PUT" && rest.startsWith("/files/")) {
const filePath = decodeURIComponent(rest.replace("/files/", ""));
const file = resolve(projectDir, filePath);
if (!isSafePath(projectDir, file)) {
res.writeHead(403, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "forbidden" }));
return;
}
let body = "";
req.on("data", (chunk: Buffer) => {
body += chunk.toString();
});
req.on("end", () => {
writeFileSync(file, body, "utf-8");
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify({ ok: true }));
});
return;
}
next();
});
// Watch project directories for external file changes (user editing HTML outside the editor).
// Resolve symlinks so the watcher sees the real file paths.
const realProjectPaths: string[] = [];
try {
for (const entry of readdirSync(dataDir, { withFileTypes: true })) {
const full = join(dataDir, entry.name);
try {
const real = lstatSync(full).isSymbolicLink() ? realpathSync(full) : full;
realProjectPaths.push(real);
server.watcher.add(real);
} catch {
/* skip broken symlinks */
}
}
} catch {
/* dataDir doesn't exist yet */
}
// When a project file changes, send HMR event to refresh the preview
server.watcher.on("change", (filePath: string) => {
const isProjectFile = realProjectPaths.some((p) => filePath.startsWith(p));
if (
isProjectFile &&
(filePath.endsWith(".html") || filePath.endsWith(".css") || filePath.endsWith(".js"))
) {
console.log(`[Studio] File changed: ${filePath}`);
server.ws.send({ type: "custom", event: "hf:file-change", data: {} });
}
});
},
};
}
export default defineConfig({
plugins: [react(), devProjectApi()],
build: {
outDir: "dist",
emptyOutDir: true,
},
server: {
port: 5190,
},
});