mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-08-31 02:41:44 +00:00
main
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ukimsanov
|
bebaf679d9 | docs: rebuild developer and rendering reference | ||
|
ukimsanov
|
52d586dd31 |
docs: fix 53 inaccuracies across all documentation
Exhaustive audit of every MDX file in docs/ against skill references and package source code. Every API signature, default value, flag, and technical claim verified against ground truth. |
||
|
Miguel Ángel
|
0e052e42d2 | fix(engine): support AMD AMF GPU encoding | ||
|
Miguel Ángel
|
215811334f |
fix(producer): accept plain integer fps in createRenderJob
The rational `Fps = { num, den }` refactor in
|
||
|
Miguel Ángel
|
395fb9c084 |
feat: add browser GPU render mode (#571)
## Problem HyperFrames already had `--gpu`, but that flag only controlled FFmpeg hardware encoding. The browser capture path still forced Chrome/WebGL through SwiftShader software GL via `--use-angle=swiftshader`, so WebGL-heavy local renders could leave the biggest bottleneck on the CPU path. That made the existing flag naming easy to misread: `--gpu` sounded like it accelerated the whole render, but it did not change the browser frame-capture backend. ## What this fixes - Enables host browser GPU acceleration automatically for local CLI renders. - Adds `--no-browser-gpu` as the local opt-out for software Chrome/WebGL capture. - Keeps `--browser-gpu` as an explicit local browser-GPU request. - Adds `browserGpuMode: "software" | "hardware"` to engine config, with `PRODUCER_BROWSER_GPU_MODE` env support for lower-level producer users. - Keeps Docker browser capture on the deterministic software path. - Maps hardware browser GPU mode to platform-native Chrome backends: - macOS: Metal-backed ANGLE - Windows: D3D11-backed ANGLE - Linux: EGL - Blocks explicit `--browser-gpu --docker` with a clear error because Docker browser GPU passthrough is not cross-platform. - Clarifies docs so `--gpu` means FFmpeg encoder GPU and browser GPU means Chrome/WebGL capture GPU. - Keeps encoder backend selection auto-detected from FFmpeg capabilities: - NVIDIA: NVENC - macOS: VideoToolbox - Linux: VAAPI - Intel: QSV ## Why two flags There are two separate GPU surfaces in the render pipeline: 1. Browser GPU controls Chrome frame capture. - Affects WebGL, canvas, CSS rendering, compositing, and screenshot capture inside the browser. - This is enabled automatically for local CLI renders. - Use `--no-browser-gpu` when you want the software browser baseline. 2. `--gpu` controls FFmpeg video encoding. - Affects the final encode step after frames have already been captured. - The concrete encoder is auto-detected from the host FFmpeg build and hardware. - It can be faster for some machines/codecs, but it is not equivalent to browser rendering acceleration. The controls stay independent because users may want: - `hyperframes render` for the fast local default with browser GPU capture. - `hyperframes render --no-browser-gpu` for the software-browser local baseline. - `hyperframes render --gpu` for browser GPU capture plus hardware FFmpeg encoding. - `hyperframes render --no-browser-gpu --gpu` for software browser capture plus hardware FFmpeg encoding. - `hyperframes render --docker` for deterministic browser capture. ## Why `--gpu` does not imply browser GPU Keeping `--gpu` scoped to FFmpeg encoding avoids a semantic break and keeps the risk profile explicit: - `--gpu` already means encoder acceleration. Expanding it to also change Chrome capture would silently alter behavior for users who only wanted hardware encoding. - Browser GPU and encoder GPU have different portability. Encoder GPU can work in Docker when the host exposes the right devices; browser GPU passthrough is not cross-platform, so this PR intentionally blocks explicit `--browser-gpu --docker`. - The Apple presentation benchmark shows why the controls should stay separate: browser GPU capture was the useful improvement, while macOS VideoToolbox via `--gpu` was slower and produced larger output for this `standard` H.264 run. If HyperFrames later wants a single umbrella acceleration control, it should be explicit, for example `--acceleration browser|encoder|all` or `--gpu=browser|encoder|all`, rather than changing the meaning of the existing boolean `--gpu`. ## Root cause `buildChromeArgs()` always injected `--use-gl=angle --use-angle=swiftshader`. `disableGpu` only appended `--disable-gpu`; it did not provide a hardware-GPU mode. That made the public `--gpu` flag look broader than it was, because render capture stayed software-backed even when encoder GPU was requested. ## Verification ### Local checks - `bun install` - `bun run build:hyperframes-runtime` - `bun run --filter @hyperframes/engine test src/config.test.ts src/services/browserManager.test.ts` - `bun run --filter @hyperframes/cli test src/utils/dockerRunArgs.test.ts src/commands/render.test.ts` - `bun run --filter @hyperframes/cli typecheck` - `bun run --filter @hyperframes/engine typecheck` - `bun run --filter @hyperframes/producer typecheck` - `cd packages/producer && bunx vitest run src/services/renderOrchestrator.test.ts` - `bunx oxlint packages/cli/src/commands/render.ts packages/cli/src/commands/render.test.ts packages/cli/src/utils/dockerRunArgs.ts packages/cli/src/utils/dockerRunArgs.test.ts packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/browserManager.ts packages/engine/src/services/browserManager.test.ts packages/producer/src/services/renderOrchestrator.test.ts` - `bunx oxfmt --check ...` on changed source/docs files - `git diff --check` - `bun packages/cli/src/cli.ts render --help | rg -n "browser-gpu|no-browser-gpu|GPU"` - `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --output /tmp/hf-auto-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict` - Render plan prints `GPU: browser GPU (auto)`. - `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --no-browser-gpu --output /tmp/hf-software-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict` - Render plan does not print browser GPU. - `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --docker --browser-gpu --output /tmp/should-not-render.mp4` - Exits 1 with `Browser GPU is local-only`. - `buildDockerRunArgs()` regression coverage asserts Docker container args include `--no-browser-gpu`, preventing nested container renders from re-enabling browser GPU through the local CLI default. - `resolveBrowserGpuForCli()` regression coverage asserts `PRODUCER_BROWSER_GPU_MODE=software` opts out when no CLI browser-GPU flag is supplied, while explicit `--browser-gpu` / `--no-browser-gpu` still win. - `ffmpeg -v error -i /tmp/hf-auto-browser-gpu-smoke.mp4 -f null -` - `ffmpeg -v error -i /tmp/hf-software-browser-gpu-smoke.mp4 -f null -` - `ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of json /tmp/hf-browser-gpu-smoke.mp4` -> H.264, 1920x1080, 24fps, 5.0s ### Apple presentation benchmark Rendered `/Users/miguel07code/Downloads/apple-presentation.zip` as supplied after extracting to `/tmp/hf-apple-profile/apple-presentation`. Fixed settings: - 1920x1080 - 30fps - `standard` quality - 4240 frames - 141.32s duration - 8-worker cap; render auto-calibration used 6 capture workers - macOS host detected FFmpeg GPU encoder: `videotoolbox` | Mode | Equivalent flags after this PR | Wall time | vs software-browser baseline | Speed | Capture | Encode | Output | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Software browser + CPU encode | `--no-browser-gpu` | 120.77s | baseline | 1.17x | 97.87s | 10.04s | 8.38MB | | Browser GPU + CPU encode | default local render | 70.10s | 42.0% faster | 2.02x | 50.72s | 9.91s | 8.39MB | | Software browser + encoder GPU | `--no-browser-gpu --gpu` | 133.16s | 10.3% slower | 1.06x | 103.58s | 18.31s | 25.43MB | | Browser GPU + encoder GPU | `--gpu` | 74.12s | 38.6% faster | 1.91x | 46.69s | 17.93s | 25.45MB | Result: browser GPU capture is the meaningful improvement for this WebGL/browser-capture-heavy presentation. VideoToolbox encoding was slower and produced larger files for this current `standard` H.264 path, so `--gpu` should stay separate and opt-in. Why `--gpu` plus browser GPU was slower than browser GPU alone: the combined run captured about 4.0s faster than browser GPU alone, but VideoToolbox encoding was about 8.0s slower than CPU x264 encoding, so the encode loss outweighed the capture gain. ### VideoToolbox flag check I also isolated the encode stage against the already-captured Apple frames to check whether macOS GPU encoding only needed special flags. `ffmpeg -h encoder=h264_videotoolbox` does not expose a CRF/CQ-style quality option like x264. It exposes bitrate-oriented and VideoToolbox-specific options such as `-b:v`, `-realtime`, `-profile`, `-coder`, `-prio_speed`, `-power_efficient`, and `-allow_sw`. That means our current `-q:v` mapping is not equivalent to x264 CRF and can produce very different bitrate/size behavior. Measured full-frame encode variants on this host: | VideoToolbox variant | Encode wall time | Output size | Bitrate | | --- | ---: | ---: | ---: | | Current `-q:v 64 -allow_sw 1` | 18.76s | 25.31MB | 1.43 Mbps | | Current without `-allow_sw 1` | 18.21s | 25.31MB | 1.43 Mbps | | `-b:v 500k -maxrate 750k -bufsize 1000k -profile high -coder cabac -realtime 1 -prio_speed 1 -power_efficient 0` | 20.58s | 7.42MB | 0.42 Mbps | | Same with `-b:v 1500k` | 20.84s | 16.70MB | 0.95 Mbps | | `-b:v 500k -profile baseline -coder cavlc -realtime 1 -prio_speed 1 -power_efficient 0` | 18.11s | 8.94MB | 0.51 Mbps | Conclusion: VideoToolbox can be made size/bitrate-predictable with explicit `--video-bitrate`, but the tested speed-oriented flags did not make it faster than CPU x264 wall time for this render. That reinforces keeping `--gpu` encoder acceleration explicit and separate from browser GPU capture. Artifacts from the local benchmark: - `/tmp/hf-apple-profile/results/cpu.mp4` - `/tmp/hf-apple-profile/results/browser-gpu.mp4` - `/tmp/hf-apple-profile/results/encoder-gpu.mp4` - `/tmp/hf-apple-profile/results/full-gpu.mp4` - `/tmp/hf-apple-profile/results/summary.json` All four benchmark MP4s completed `ffprobe` and full `ffmpeg -f null` decode checks. ### Pixel comparison Compared decoded MP4 output between software-browser and browser-GPU renders: - Apple presentation: - 4240 frames compared - 636 exact matching decoded frame hashes - 3604 different decoded frame hashes - Average PSNR: 57.79 dB - `css-spinner-render-compat` clean fixture: - 120 frames compared - 0 exact matching decoded frame hashes - Average PSNR: 61.57 dB Interpretation: browser GPU output is not strict hash/pixel-identical to the software-browser path after lossy H.264 encode, but the measured deltas are visually tiny. Above 50 dB PSNR is typically visually indistinguishable for normal video review. Use `--no-browser-gpu` or Docker when strict cross-run/cross-machine reproducibility matters more than local speed. ### Browser verification - Started HyperFrames Studio preview for `packages/producer/tests/css-spinner-render-compat/src`. - Used `agent-browser` to open `http://localhost:5191#project/src` and verify the composition loaded in Studio. - Screenshots: - `/tmp/hf-gpu-browser-proof/preview-loaded.png` - `/tmp/hf-gpu-browser-proof/preview-playing.png` - `/tmp/hf-gpu-browser-proof/preview-frame-60.png` - Agent-browser recordings: - `/tmp/hf-gpu-browser-proof/preview-playback.webm` - `/tmp/hf-gpu-browser-proof/preview-seek.webm` ## Notes - Browser GPU is enabled automatically for local CLI renders and disabled in Docker. - `--no-browser-gpu` is the opt-out for software Chrome/WebGL capture. - `--gpu` remains encoder-only and opt-in. - The Apple presentation zip has existing lint errors around unmanaged nested videos and imperative media `play()` calls. The benchmark still compares the same supplied source across modes, but it should not be treated as a clean deterministic-composition fixture. |
||
|
Vance IngallsandClaude Opus 4.6
|
8e5593b6ba |
feat(render): auto-detect HDR from media probes, add --sdr flag (#526)
* feat(render): auto-detect HDR from media probes, add --sdr flag Replace the --hdr opt-in model with automatic detection. When no flags are passed, the renderer probes all video/image sources and enables HDR output if any HDR color space is detected. Existing --hdr flag becomes a force override. New --sdr flag forces SDR output. Behavior matrix: (no flags) + HDR content → HDR output (no flags) + SDR content → SDR output --hdr → force HDR (defaults to HLG if no HDR sources) --sdr → force SDR (skips probing) --hdr --sdr → error Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: align HDR auto-detect docs and tests --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Vance Ingalls
|
00af29c169 |
fix(cli): forward --hdr through Docker render + HDR docs (#346)
## Summary This PR ended up covering the full HDR Docker/docs follow-through plus the producer/engine work needed to make HDR still images render and regress correctly in CI. The branch now does four things: - forwards `--hdr` through the Docker render path in the CLI - adds and expands HDR documentation across the docs site - adds first-class HDR still-image support to the engine/producer pipeline - adds targeted HDR regression coverage, including a CI-safe fallback for PNG HDR metadata detection when `ffprobe` does not expose PNG color tags ## What changed ### CLI and docs - `hyperframes render --docker --hdr` now preserves `--hdr` when invoking the in-container CLI - added a dedicated HDR guide and linked it from CLI, producer, engine, rendering, and common-mistakes docs - documented HDR constraints and verification flow: HDR source requirements, MP4/H.265 Main10 output, PQ/HLG handling, Docker usage, and common SDR fallback causes ### Engine and producer HDR image support - added `ImageElement` support to the engine composition model and parsing path - threaded image elements through producer compilation and orchestration - probed image sources for HDR color spaces so image-only compositions can trigger HDR output without requiring an HDR video source - included HDR image start times in stacking queries so the layered compositor can place images correctly in z-order - integrated HDR image compositing into the layered HDR render loop alongside native HDR video layers and SDR DOM overlays - forced screenshot mode for HDR layered compositing where required to keep DOM/HDR layer composition deterministic - skipped readiness waiting for natively extracted HDR videos in the engine path where it was unnecessary and could block layered HDR flows ### HDR metadata robustness - added a fallback in `extractVideoMetadata()` to read PNG `cICP` metadata directly when `ffprobe` omits color-space fields for PNGs - this specifically fixes CI/Docker detection for the `hdr-image-only` fixture, where the render was falling back to SDR because the PNG was not being recognized as BT.2020 PQ ### Regression coverage and fixture cleanup - added `hdr-image-only`, a regression fixture that validates HDR still-image rendering end to end - added `hdr-pq`, a focused HDR PQ regression fixture for the video path - updated regression CI to run an `hdr` shard with `--sequential hdr-pq hdr-image-only` - removed the older larger `hdr-regression/*` fixture set in favor of the smaller targeted regressions used by CI - added the necessary fixture generation/readme material and checked-in golden outputs for the new HDR tests ## Why The original PR description only covered the CLI flag forwarding and docs work. Since then, the branch also picked up the missing runtime support needed for HDR still images and the regression coverage to keep that path from breaking. The practical issue this closes is: - local host runs could pass while CI failed `hdr-image-only` - the failure was a full-frame visual mismatch caused by SDR fallback, not unstable rendering - root cause was PNG HDR metadata not being surfaced by `ffprobe` in the CI Docker environment - parsing the PNG `cICP` chunk directly makes HDR detection deterministic across environments ## Test plan ### Local targeted checks ```bash bunx oxlint packages/engine/src/utils/ffprobe.ts packages/engine/src/utils/ffprobe.test.ts bunx oxfmt packages/engine/src/utils/ffprobe.ts packages/engine/src/utils/ffprobe.test.ts bun --cwd packages/engine test src/utils/ffprobe.test.ts src/utils/hdr.test.ts ``` ### Producer regression runs on host ```bash bun run --cwd packages/core build:hyperframes-runtime:modular bun --cwd packages/producer test -- --sequential --exclude-tags slow,render-compat,hdr bun --cwd packages/producer test -- --sequential hdr-pq hdr-image-only ``` Observed result: - `fast` shard: 7 passed, 0 failed - `hdr` shard: 2 passed, 0 failed ### CI-equivalent Docker verification ```bash docker build -f Dockerfile.test -t hyperframes-producer:test . docker run --rm \ --security-opt seccomp=unconfined \ --shm-size=4g \ -v "$PWD/packages/producer/tests:/app/packages/producer/tests" \ hyperframes-producer:test \ --sequential hdr-pq hdr-image-only ``` Observed result: - `hdr-image-only`: passed - `hdr-pq`: passed - shard summary: 2 passed, 0 failed ### Specific regression fixed Before the PNG `cICP` fallback, the Docker/CI run failed `hdr-image-only` with: - missing `"[Render] HDR source detected — output: PQ ..."` log line - full-frame visual mismatch across all 100 checkpoints - PSNR ~17 on every frame, indicating a consistent SDR-vs-HDR pipeline mismatch After the fallback, the same Docker path recognizes the PNG as HDR and the shard passes. |
||
|
JamesandClaude Opus 4.6
|
684b103ba4 |
docs: add WebM transparency docs to engine, producer, and CLI examples
- Engine: document getEncoderPreset() for MP4/WebM, VP9 alpha flags, Opus audio in mux step - Producer: document format field in RenderConfig, WebM usage example, pipeline steps updated for WebM - CLI: add render command examples in --help (including WebM overlay) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
JamesandClaude Opus 4.6
|
db892f4e8f |
docs: audit and fix all documentation against actual codebase
Comprehensive audit of every documentation page against the actual source code, fixing incorrect APIs, wrong CLI flags, nonexistent templates, and missing public exports. Also documents the new agent-friendly CLI design. Key fixes: - Quickstart: `npx create-hyperframe` → `npx hyperframes init`, Node 20→22 - Templates: replaced nonexistent blank/title-card/video-edit with actual templates (blank, warm-grain, play-mode, swiss-grid, vignelli) - CLI: removed nonexistent short flags (-o/-f/-q/-w), added missing commands (browser, docs, telemetry, skills), documented agent-friendly non-interactive default and --human-friendly flag - Producer: replaced nonexistent `render()` API with actual `createRenderJob()`/`executeRenderJob()`, added server API docs - Engine: replaced nonexistent `createEngine()` with actual session-based API, added HfProtocol, encoding, streaming, parallel rendering docs - Core: fixed wrong type names (Composition/Clip→TimelineElement), wrong function names (parseHyperframeHtml→parseHtml), documented all 4 entry points (main, /lint, /compiler, /runtime) - Studio: added all missing exports (NLELayout, SourceEditor, PropertyPanel, FileTree, StudioApp, hooks, Tailwind preset) - All pages: --output not -o, Node 22+ not 20+ Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
JamesandClaude Opus 4.6
|
915fe2f47a |
docs: improve quality based on Remotion/Stripe/Tailwind patterns
Major improvements across all 18 pages: - Use Mintlify components: <Steps> for tutorials, <Tabs> for alternatives, <CodeGroup> for multi-platform commands, <Tree> for directory structures, <AccordionGroup> for FAQ/scannable content, <Mermaid> for diagrams - Add filename annotations to all code blocks (e.g., ```html index.html) - Add numbered comments inside multi-step code examples - Show expected terminal output after CLI commands - Add "When to use" / "When NOT to use" sections to all package pages - Add "Next Steps" CardGroup to every page (no dead-end pages) - Cross-link between pages at point of curiosity (not just "see also" dumps) - Expand thin pages (engine, studio) with architecture details and examples - Add decision guides (rendering modes, template selection) - Use <Warning> and <Note> sparingly (max 2-3 per page) Also adds DOCS_GUIDELINES.md at repo root with writing standards. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
JamesandClaude Opus 4.6
|
00bd2e5ae2 |
docs: add Mintlify documentation site
Set up /docs directory with docs.json config, HeyGen branding (logo, favicon, #7559FF purple), and 18 MDX pages covering: - Getting started (introduction, quickstart) - Concepts (compositions, data attributes, frame adapters, determinism) - Guides (GSAP animation, templates, rendering, common mistakes, troubleshooting) - Package docs (core, engine, producer, studio, CLI) - Reference (HTML schema) and contributing guide Content adapted from existing repo docs (core/docs/, cli/src/docs/, README). Validated with `mint validate` and `mint broken-links`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |