mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-04 07:19:52 +00:00
c4bcc52f3b3058fc6cef32c341210a3a2753e413
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 WangandClaude Opus 4.7
|
d1f992570a |
fix(cli): use 'where' instead of 'which' on Windows for FFmpeg and br… (#336)
* fix(cli): use 'where' instead of 'which' on Windows for FFmpeg and browser detection - findFFmpeg() now uses 'where ffmpeg' on Windows, 'which ffmpeg' on Unix - whichBinary() now uses 'where' on Windows, 'which' on Unix Fixes FFmpeg detection failure on Windows where 'which' command doesn't exist. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(cli): handle multi-line output from Windows 'where' command Windows 'where' can return multiple paths (one per line) when there are multiple matches on PATH. Take only the first non-empty line. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(cli): extend Windows 'where' fix to whisper, tts, and clipboard modules - whisper/manager.ts: whichBinary() now uses 'where' on Windows - tts/synthesize.ts: findPython() now uses 'where' on Windows - utils/clipboard.ts: detectProvider() now uses 'where' on Windows All functions handle multi-line output from 'where' command. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
James RussoandClaude Opus 4.6
|
b23b0751da |
fix(player): parent-frame media playback for mobile (#266)
* fix(player): parent-frame media playback for mobile Mobile browsers block media.play() inside iframes when the user gesture happened in the parent frame — postMessage doesn't transfer user activation (per the User Activation v2 spec). ## Problem The player renders compositions in a sandboxed iframe. When a user taps play in the parent frame, the player sends a postMessage to the iframe's runtime, which calls audio.play(). On mobile, this fails silently because the iframe has no user activation context. ## Solution The player now extracts ALL timed media elements (audio/video with data-start) from the iframe's DOM (same-origin access), creates parent-frame copies, and disables the iframe originals. On play(), parentMedia.play() runs synchronously in the gesture call stack, satisfying mobile autoplay policy. ### Generic media handling - Finds all `audio[data-start], video[data-start]` in the iframe - Creates a parent-frame copy for each (Audio or Video element) - Preserves data-start offsets for correct seek positioning - Strips data-start from iframe elements so the runtime ignores them - Falls back to iframe media for cross-origin iframes ### `audio-src` attribute Convenience for the common single-narration case. When set, the player starts preloading audio immediately — before the iframe loads. This eliminates the loading delay that caused jittery playback. ### No active sync Both parent media and the GSAP timeline are real-time systems. When started simultaneously, they naturally stay within ~10ms — no drift correction needed. Active sync with coarse granularity (50ms polling) caused MORE jitter than it prevented via repeated audio seeks. ## CI - Added unified `test` job replacing separate per-package test jobs - Added root `test` script: `bun run --filter '*' test` - New packages with test scripts are automatically included - Added happy-dom for player DOM tests ## Tests - 10 new tests for parent-frame media: preloading, play, pause, seek, muted/rate sync, cleanup, attribute changes - All 21 player tests pass Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(shader-transitions): pass CI when no test files exist Add --passWithNoTests to vitest run so the unified test job doesn't fail on packages that have a test script but no test files yet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): update tests for new id field and GSAP lint rule - normalize.test.ts: loadTranscript now assigns id fields (w0, w1, etc.) to SRT/VTT results and empty string for words-json passthrough - lintProject.test.ts: add GSAP CDN script to validHtml() fixture to satisfy the missing_gsap_script lint rule added in core Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): add missing data-start/data-duration to validHtml fixture The validHtml() test fixture was missing data-start and data-duration attributes, triggering the root_composition_missing_data_start and root_composition_missing_data_duration lint warnings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ci): fetch LFS objects for producer test job Producer regression tests compare rendered output against reference MP4 files stored in git LFS. Without lfs: true, checkout fetches pointer files instead of actual videos, causing "moov atom not found" errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: remove redundant test-producer job The regression workflow already runs the same 28 producer fixtures in a Docker container with prod-matching Chrome/fonts/ffmpeg, sharded across 8 parallel matrix jobs with 40-min timeouts. The CI test-producer job was a duplicate that ran on bare runners with worse determinism and a 15-min timeout too short for all fixtures. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
James RussoandClaude Opus 4.6
|
7389c0c89b |
feat(cli): add tts command for local text-to-speech via Kokoro-82M (#201)
* feat(cli): add `tts` command for local text-to-speech via Kokoro-82M Adds `hyperframes tts` — generate speech audio locally using Kokoro-82M (ONNX), no API key needed. Mirrors the transcribe command architecture. - New command: `hyperframes tts "text" --voice af_heart --output speech.wav` - 54 voices across 8 languages, ~5x realtime on CPU - Auto-downloads model (~311 MB) + voices (~27 MB) to ~/.cache/hyperframes/tts/ - Requires Python 3.8+ with kokoro-onnx installed - Extracted shared `downloadFile` utility from whisper/manager.ts with atomic .tmp→rename to prevent partial download corruption - Added hyperframes-tts skill with voice selection guide - Updated CLAUDE.md with TTS docs, voice table, and skill reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(tts): improve skill per skill-creator guidelines - Move trigger info from body to frontmatter description - Remove `trigger` field (not a valid frontmatter field) - Remove CLI flag docs Claude can derive from --help - Remove redundant voice tables (keep content-to-voice mapping) - Fix composition audio example to use actual <audio> element pattern - Keep non-obvious workflows: TTS+transcribe for captions, long scripts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(tts): add guidance for using external TTS sources Help users understand when to use cloud TTS (voice cloning, broader languages, higher quality) vs the built-in Kokoro model, and how external audio integrates into the same composition workflow. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(tts): prioritize HeyGen API as recommended cloud TTS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(tts): remove external TTS section for now Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tts): set required: false on input arg so --list works standalone Citty treats positional args as required by default unless explicitly set to required: false. Without this, `hyperframes tts --list` fails with "Missing required positional argument". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(tts): add --help examples and fix required:false for --list Add examples section to `tts --help` matching the pattern from other commands (transcribe, render, etc.). Fix citty positional arg requiring explicit `required: false` for --list to work standalone. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add CLI command checklist to CLAUDE.md Ensure new commands always get --help examples in help.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Vance Ingalls
|
d36c1785b9 |
feat(captions): energy-based technique selection and mandatory quality checks (#176)
## Summary - Rewrite script-to-style mapping as an energy detection table (high → low) with mandatory animation requirements: karaoke baseline, 2+ highlight techniques, kinetic exits - Replace `tl.call()` per-frame audio-reactive pattern with group-level GSAP tweens — read peak bass/treble for each group's time range and modulate entrance intensity at build time, no per-frame callbacks needed - Add transcript quality check with automatic retry rules (>20% music tokens = retry with larger model) - Add caption word structure lint rule (`.caption-group` + `<span>`) for studio editor compatibility - Add multilingual model guidance and decision tree for model selection ## Test plan - [ ] Skill files render correctly as markdown - [ ] Cross-references between SKILL.md, dynamic-techniques.md, and transcript-guide.md resolve correctly - [ ] `dynamic-techniques.md` audio-reactive section uses `tl.to()`/`tl.set()` only, no `tl.call()` loops 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Vance Ingalls
|
37404f23da |
feat(whisper+captions): language detection, audio-reactive captions, multilingual defaults (#175)
## Summary **Whisper improvements:** - Auto-detect language and switch from `.en` to multilingual model when needed - Detect speech onset in WAV to strip hallucinated words before speech begins - Merge whisper-cpp token fragments: contractions (`didn` + `'t` → `didn't`), split capitals (`C` + `aught` → `Caught`), dropped-g (`shin` + `in'` → `shinin'`) - Interpolate zero-duration word clusters for reliable karaoke timing **Captions skill updates (folded from #176):** - Rewrite script-to-style mapping as an energy detection table (high → low) with mandatory animation requirements: karaoke baseline, 2+ highlight techniques, kinetic exits - Replace `tl.call()` per-frame audio-reactive pattern with group-level GSAP tweens — read peak bass/treble for each group's time range and modulate entrance intensity at build time - Add transcript quality check with automatic retry rules (>20% music tokens = retry with larger model) - Add caption word structure lint rule (`.caption-group` + `<span>`) for studio editor compatibility **Multilingual defaults (folded from #186):** - Default whisper model changed from `small.en` to `small` to prevent silent translation of non-English audio - Added non-negotiable language rule to captions skill ## Test plan - [ ] `pnpm test` passes (contraction merging, fragment merging, zero-duration interpolation, speech onset) - [ ] Transcribe non-English audio — verify it transcribes in original language, not translates - [ ] Skill files render correctly, cross-references resolve - [ ] `dynamic-techniques.md` audio-reactive section uses `tl.to()`/`tl.set()` only, no `tl.call()` loops 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
James RussoandClaude Opus 4.6
|
2f99e33bbe |
feat(cli,core): standalone transcribe command, transcript normalization, caption lint rules (#151)
* feat(cli,core): add standalone transcribe command, transcript normalization, and caption lint rules
- Add `hyperframes transcribe` command for transcribing audio/video and importing
existing transcripts (SRT, VTT, OpenAI Whisper API JSON, whisper.cpp JSON)
- Add transcript format normalizer (normalize.ts) with auto-detection and
conversion to standard [{text, start, end}] word arrays
- Upgrade default whisper model from base.en to small.en for better accuracy
- Add --model and --language flags to both `transcribe` and `init` commands
- Extract shared patchCaptionHtml() to eliminate duplication between init.ts
and transcribe.ts (init.ts reduced by ~55 lines)
- Add 3 caption lint rules: caption_exit_missing_hard_kill,
caption_text_overflow_risk, caption_container_relative_position
- Update captions skill with model guide, format docs, music guidance,
text overflow prevention, caption exit guarantee pattern
- Expand captions skill trigger to cover lyrics, karaoke, lyric videos
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(cli): add transcribe command and --model/--language flags to CLI docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): fix blank template lint issues
- blank/index.html: remove data-start from video (was nested in timed parent),
add class="clip" for initial hidden state
- blank/captions.html: add max-width + overflow:hidden to prevent text clipping,
add tl.set hard kill after exit tween to prevent stuck captions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: add lint-after-edit rule to repo and project CLAUDE.md
Agents must run `npx hyperframes lint` after editing compositions.
Also expand captions skill description in project template.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style: format _shared/CLAUDE.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
||
|
Vance IngallsandClaude Opus 4.6
|
f4367d5726 |
feat(cli): add whisper transcription and template improvements (#53)
* feat(cli): add whisper transcription to init flow New modules: - whisper/manager.ts: download/cache whisper.cpp binary + model (~/.cache/hyperframes/whisper/) - whisper/transcribe.ts: extract audio, run whisper, save transcript.json Init flow changes: - "Got a video or audio file?" now accepts audio-only files (mp3, wav, m4a) - "Generate captions from audio?" prompt after file selection - Transcription produces transcript.json in project root - Graceful fallback if whisper/ffmpeg unavailable Supports: macOS ARM64/x86, Linux x86_64. Downloads whisper.cpp v1.7.3 from GitHub releases and ggml-base.en model from Hugging Face. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): use brew/system whisper instead of downloading binaries whisper.cpp doesn't ship pre-built macOS/Linux CLI binaries. Use brew install whisper-cpp on macOS (auto-installs if brew available), system PATH lookup otherwise. Model still downloaded from Hugging Face. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): simplify whisper install — detect or instruct, don't build Remove build-from-source complexity. If whisper-cpp is found on PATH, use it. If not, show install instructions instead of blocking: "To generate captions, install whisper-cpp: brew install whisper-cpp" The transcription prompt only appears when whisper is available. When it's not, the user sees the install command and can re-run init. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(cli): auto-install whisper via brew or build from source ensureWhisper() now tries 4 strategies in order: 1. System PATH (whisper-cli or whisper already installed) 2. Homebrew (macOS: brew install whisper-cpp) 3. Build from source (git clone + cmake, ~30-60s) 4. Show install instructions as last resort Init flow always asks "Generate captions?" — whisper is installed automatically in the background if needed. No user intervention required on macOS with Xcode CLI tools or any system with git+cmake. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): add window.__timelines guard to all templates The studio bundler doesn't always initialize window.__timelines before template scripts run, causing "Cannot set properties of undefined" errors. Add defensive guard to every template. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(cli): patch template captions with actual transcript data After scaffolding, if transcript.json exists, replace the hardcoded word array in the template's captions composition with the real transcript data. The template's caption animation and styling are preserved — only the word data changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): show install notice when whisper needs to be installed When whisper-cpp isn't found, show an info message before the spinner: "whisper-cpp not found — installing automatically..." Then the spinner shows "Installing whisper-cpp (this may take a moment)..." Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): add muted and playsinline to all template video elements The framework requires video elements to have muted and playsinline attributes. All four templates were missing these, causing video to not play in the studio preview. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): flat asset structure + separate audio tracks in templates Assets: video, images, fonts all go at project root (not assets/ or fonts/ subdirectories). The studio preview can't resolve relative paths from subdirectories due to the /preview URL suffix. Audio: added <audio> elements alongside muted <video> in all 4 templates so the video's audio plays back. The framework requires muted video + separate audio element. Removed assets/ and fonts/ directory creation from scaffoldProject. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(studio): inject base tag for asset resolution in preview The preview iframe serves bundled HTML from /api/projects/:id/preview but relative asset paths (video.mp4, font.woff2) resolve to the wrong URL without a <base> tag. Now injects <base href="/api/projects/:id/preview/"> so relative paths route through the static asset handler. Also adds proper MIME types for video, audio, image, and font files served from the preview asset route. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(studio): serve HyperFrames runtime in dev mode The preview runtime script had an empty src — the framework never loaded, so video playback and clip lifecycle didn't work. Now auto-detects packages/cli/dist/hyperframe-runtime.js and serves it at /api/runtime.js. No env var needed in dev mode. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): filter whisper special tokens from transcript Use --output-json instead of --output-json-full to avoid special tokens like [_TT_485] and [BLANK_AUDIO]. Also filter remaining bracket tokens when building the word array for captions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): use --output-json-full for word-level timestamps --output-json only produces segment-level timing (no tokens). --output-json-full is required for word-level timestamps that the captions template needs. Special tokens are filtered out by the patchTranscript function. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(cli): patch template durations to match uploaded video Templates now use __VIDEO_DURATION__ placeholder that gets replaced with the actual probed video duration. All data-duration values on the root composition, video, audio, and caption clips are updated. Without a video, defaults to 10 seconds. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): merge punctuation tokens with preceding word Whisper outputs punctuation (. , ! ?) as separate tokens. These appeared as standalone words in captions, sometimes in the wrong group. Now merged with the preceding word during transcript normalization. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): match both TRANSCRIPT and script variable names in templates Three templates use `const TRANSCRIPT = [...]` while warm-grain uses `const script = [...]`. The patchTranscript function now matches both. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): security and template fixes - Replace shell injection risk (execSync rm) with unlinkSync in transcribe.ts - Add GIT_TERMINAL_PROMPT=0 to whisper buildFromSource git clone - Fix hardcoded data-duration="18" in warm-grain captions template - Add data-start="0" to root compositions in swiss-grid, vignelli, warm-grain - Add data-start="0" to warm-grain grain-overlay composition - Deduplicate hasFFmpeg: remove from init.ts, import from whisper/manager.ts - Add my-video/ and packages/studio/data/ to .gitignore Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): format warm-grain captions and fix TS nullability errors - Format warm-grain/compositions/captions.html - Add optional chaining on token.offsets (may be undefined) - Use intermediate variable for lastWord to satisfy TS strict checks Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(cli): add blank template option, smart defaults for video vs audio - Blank template: minimal scaffolding (root composition, video, audio, GSAP timeline) with __VIDEO_SRC__ and __VIDEO_DURATION__ placeholders - Template defaults: video uploads default to "blank" (user brings their own content), audio-only defaults to "warm-grain" (motion graphics template since there's no video to show) - Audio-only projects now tracked with isAudioOnly flag Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): address whisper review feedback - Clean stale builds: if BUILD_DIR exists but no binary, nuke and retry - Build failures clean up BUILD_DIR so next attempt starts fresh - patchTranscript regex scoped within <script> blocks to prevent matching across block boundaries - Removed hardcoded model size hint (~148MB) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): add missing rmSync import to whisper manager Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove test project and lock file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): address review items 7-12 — execFileSync, build diagnostics, WAV verification - manager.ts: replace all execSync with execFileSync to prevent command injection - manager.ts: capture cmake stderr and include in build failure error message - transcribe.ts: verify WAV is 16kHz mono via ffprobe before passing to whisper - init.ts: replace fragile JSON formatting with JSON.stringify(words, null, 2) - init.ts: fix default duration from "10" to "5" matching DEFAULT_META - init.ts: add probeAudioDuration() and --audio/--skip-transcribe flags - init.ts: extract finalizeProject() to reduce code path duplication - init.ts: wire transcription into non-interactive path Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |