* feat(cli): add whisper transcription to init flow
New modules:
- whisper/manager.ts: download/cache whisper.cpp binary + model
(~/.cache/hyperframes/whisper/)
- whisper/transcribe.ts: extract audio, run whisper, save transcript.json
Init flow changes:
- "Got a video or audio file?" now accepts audio-only files (mp3, wav, m4a)
- "Generate captions from audio?" prompt after file selection
- Transcription produces transcript.json in project root
- Graceful fallback if whisper/ffmpeg unavailable
Supports: macOS ARM64/x86, Linux x86_64. Downloads whisper.cpp v1.7.3
from GitHub releases and ggml-base.en model from Hugging Face.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): use brew/system whisper instead of downloading binaries
whisper.cpp doesn't ship pre-built macOS/Linux CLI binaries.
Use brew install whisper-cpp on macOS (auto-installs if brew available),
system PATH lookup otherwise. Model still downloaded from Hugging Face.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): simplify whisper install — detect or instruct, don't build
Remove build-from-source complexity. If whisper-cpp is found on PATH,
use it. If not, show install instructions instead of blocking:
"To generate captions, install whisper-cpp: brew install whisper-cpp"
The transcription prompt only appears when whisper is available.
When it's not, the user sees the install command and can re-run init.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): auto-install whisper via brew or build from source
ensureWhisper() now tries 4 strategies in order:
1. System PATH (whisper-cli or whisper already installed)
2. Homebrew (macOS: brew install whisper-cpp)
3. Build from source (git clone + cmake, ~30-60s)
4. Show install instructions as last resort
Init flow always asks "Generate captions?" — whisper is installed
automatically in the background if needed. No user intervention
required on macOS with Xcode CLI tools or any system with git+cmake.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): add window.__timelines guard to all templates
The studio bundler doesn't always initialize window.__timelines
before template scripts run, causing "Cannot set properties of
undefined" errors. Add defensive guard to every template.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): patch template captions with actual transcript data
After scaffolding, if transcript.json exists, replace the hardcoded
word array in the template's captions composition with the real
transcript data. The template's caption animation and styling are
preserved — only the word data changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): show install notice when whisper needs to be installed
When whisper-cpp isn't found, show an info message before the spinner:
"whisper-cpp not found — installing automatically..."
Then the spinner shows "Installing whisper-cpp (this may take a moment)..."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): add muted and playsinline to all template video elements
The framework requires video elements to have muted and playsinline
attributes. All four templates were missing these, causing video
to not play in the studio preview.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): flat asset structure + separate audio tracks in templates
Assets: video, images, fonts all go at project root (not assets/ or
fonts/ subdirectories). The studio preview can't resolve relative
paths from subdirectories due to the /preview URL suffix.
Audio: added <audio> elements alongside muted <video> in all 4
templates so the video's audio plays back. The framework requires
muted video + separate audio element.
Removed assets/ and fonts/ directory creation from scaffoldProject.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(studio): inject base tag for asset resolution in preview
The preview iframe serves bundled HTML from /api/projects/:id/preview
but relative asset paths (video.mp4, font.woff2) resolve to the wrong
URL without a <base> tag. Now injects <base href="/api/projects/:id/preview/">
so relative paths route through the static asset handler.
Also adds proper MIME types for video, audio, image, and font files
served from the preview asset route.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(studio): serve HyperFrames runtime in dev mode
The preview runtime script had an empty src — the framework never
loaded, so video playback and clip lifecycle didn't work.
Now auto-detects packages/cli/dist/hyperframe-runtime.js and serves
it at /api/runtime.js. No env var needed in dev mode.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): filter whisper special tokens from transcript
Use --output-json instead of --output-json-full to avoid special
tokens like [_TT_485] and [BLANK_AUDIO]. Also filter remaining
bracket tokens when building the word array for captions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): use --output-json-full for word-level timestamps
--output-json only produces segment-level timing (no tokens).
--output-json-full is required for word-level timestamps that
the captions template needs. Special tokens are filtered out
by the patchTranscript function.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): patch template durations to match uploaded video
Templates now use __VIDEO_DURATION__ placeholder that gets replaced
with the actual probed video duration. All data-duration values on
the root composition, video, audio, and caption clips are updated.
Without a video, defaults to 10 seconds.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): merge punctuation tokens with preceding word
Whisper outputs punctuation (. , ! ?) as separate tokens. These
appeared as standalone words in captions, sometimes in the wrong
group. Now merged with the preceding word during transcript
normalization.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): match both TRANSCRIPT and script variable names in templates
Three templates use `const TRANSCRIPT = [...]` while warm-grain uses
`const script = [...]`. The patchTranscript function now matches both.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): security and template fixes
- Replace shell injection risk (execSync rm) with unlinkSync in transcribe.ts
- Add GIT_TERMINAL_PROMPT=0 to whisper buildFromSource git clone
- Fix hardcoded data-duration="18" in warm-grain captions template
- Add data-start="0" to root compositions in swiss-grid, vignelli, warm-grain
- Add data-start="0" to warm-grain grain-overlay composition
- Deduplicate hasFFmpeg: remove from init.ts, import from whisper/manager.ts
- Add my-video/ and packages/studio/data/ to .gitignore
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): format warm-grain captions and fix TS nullability errors
- Format warm-grain/compositions/captions.html
- Add optional chaining on token.offsets (may be undefined)
- Use intermediate variable for lastWord to satisfy TS strict checks
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): add blank template option, smart defaults for video vs audio
- Blank template: minimal scaffolding (root composition, video, audio,
GSAP timeline) with __VIDEO_SRC__ and __VIDEO_DURATION__ placeholders
- Template defaults: video uploads default to "blank" (user brings
their own content), audio-only defaults to "warm-grain" (motion
graphics template since there's no video to show)
- Audio-only projects now tracked with isAudioOnly flag
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): address whisper review feedback
- Clean stale builds: if BUILD_DIR exists but no binary, nuke and retry
- Build failures clean up BUILD_DIR so next attempt starts fresh
- patchTranscript regex scoped within <script> blocks to prevent
matching across block boundaries
- Removed hardcoded model size hint (~148MB)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): add missing rmSync import to whisper manager
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: remove test project and lock file
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): address review items 7-12 — execFileSync, build diagnostics, WAV verification
- manager.ts: replace all execSync with execFileSync to prevent command injection
- manager.ts: capture cmake stderr and include in build failure error message
- transcribe.ts: verify WAV is 16kHz mono via ffprobe before passing to whisper
- init.ts: replace fragile JSON formatting with JSON.stringify(words, null, 2)
- init.ts: fix default duration from "10" to "5" matching DEFAULT_META
- init.ts: add probeAudioDuration() and --audio/--skip-transcribe flags
- init.ts: extract finalizeProject() to reduce code path duplication
- init.ts: wire transcription into non-interactive path
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Port the regression test infrastructure from the internal repo to OSS.
Runs golden-baseline visual/audio comparisons inside Docker for deterministic results.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update .gitignore to allow packages/producer/tests/*/output/
- Commit compiled.html snapshots and output.mp4 golden baselines
(MP4s tracked via Git LFS)
These were excluded by the blanket output/ gitignore rule but are
needed for regression tests to pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CODE_OF_CONDUCT.md (Contributor Covenant v2.1)
- SECURITY.md (responsible disclosure policy)
- pnpm-workspace.yaml stub for monorepo
- .gitignore: add .debug/ and *.tgz
- CONTRIBUTING.md: link to Code of Conduct
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- .gitignore: remove blanket video file ignores (may need LFS for regression test fixtures)
- CONTRIBUTING.md: strip dev setup details until packages are ported (leave TODO)
- README.md: strip packages table, comparison, requirements, docs link (leave TODO)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- README with hero section, quick start, HTML schema example, package overview, and Remotion comparison
- MIT LICENSE (copyright HeyGen)
- CONTRIBUTING.md with dev setup, commit conventions, and project structure
- GitHub issue templates (bug report, feature request) and PR template
- .gitignore for Node.js/TypeScript projects
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
