ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-03 04:38:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
595 Commits 856 Branches 404 Tags
395fb9c084df665659705cb723a3a60c2aa27c73
Commit Graph
7 Commits
Author SHA1 Message Date
Miguel Ángel 395fb9c084 feat: add browser GPU render mode (#571) 
## Problem

HyperFrames already had `--gpu`, but that flag only controlled FFmpeg hardware encoding. The browser capture path still forced Chrome/WebGL through SwiftShader software GL via `--use-angle=swiftshader`, so WebGL-heavy local renders could leave the biggest bottleneck on the CPU path.

That made the existing flag naming easy to misread: `--gpu` sounded like it accelerated the whole render, but it did not change the browser frame-capture backend.

## What this fixes

- Enables host browser GPU acceleration automatically for local CLI renders.
- Adds `--no-browser-gpu` as the local opt-out for software Chrome/WebGL capture.
- Keeps `--browser-gpu` as an explicit local browser-GPU request.
- Adds `browserGpuMode: "software" | "hardware"` to engine config, with `PRODUCER_BROWSER_GPU_MODE` env support for lower-level producer users.
- Keeps Docker browser capture on the deterministic software path.
- Maps hardware browser GPU mode to platform-native Chrome backends:
  - macOS: Metal-backed ANGLE
  - Windows: D3D11-backed ANGLE
  - Linux: EGL
- Blocks explicit `--browser-gpu --docker` with a clear error because Docker browser GPU passthrough is not cross-platform.
- Clarifies docs so `--gpu` means FFmpeg encoder GPU and browser GPU means Chrome/WebGL capture GPU.
- Keeps encoder backend selection auto-detected from FFmpeg capabilities:
  - NVIDIA: NVENC
  - macOS: VideoToolbox
  - Linux: VAAPI
  - Intel: QSV

## Why two flags

There are two separate GPU surfaces in the render pipeline:

1. Browser GPU controls Chrome frame capture.
   - Affects WebGL, canvas, CSS rendering, compositing, and screenshot capture inside the browser.
   - This is enabled automatically for local CLI renders.
   - Use `--no-browser-gpu` when you want the software browser baseline.

2. `--gpu` controls FFmpeg video encoding.
   - Affects the final encode step after frames have already been captured.
   - The concrete encoder is auto-detected from the host FFmpeg build and hardware.
   - It can be faster for some machines/codecs, but it is not equivalent to browser rendering acceleration.

The controls stay independent because users may want:

- `hyperframes render` for the fast local default with browser GPU capture.
- `hyperframes render --no-browser-gpu` for the software-browser local baseline.
- `hyperframes render --gpu` for browser GPU capture plus hardware FFmpeg encoding.
- `hyperframes render --no-browser-gpu --gpu` for software browser capture plus hardware FFmpeg encoding.
- `hyperframes render --docker` for deterministic browser capture.

## Why `--gpu` does not imply browser GPU

Keeping `--gpu` scoped to FFmpeg encoding avoids a semantic break and keeps the risk profile explicit:

- `--gpu` already means encoder acceleration. Expanding it to also change Chrome capture would silently alter behavior for users who only wanted hardware encoding.
- Browser GPU and encoder GPU have different portability. Encoder GPU can work in Docker when the host exposes the right devices; browser GPU passthrough is not cross-platform, so this PR intentionally blocks explicit `--browser-gpu --docker`.
- The Apple presentation benchmark shows why the controls should stay separate: browser GPU capture was the useful improvement, while macOS VideoToolbox via `--gpu` was slower and produced larger output for this `standard` H.264 run.

If HyperFrames later wants a single umbrella acceleration control, it should be explicit, for example `--acceleration browser|encoder|all` or `--gpu=browser|encoder|all`, rather than changing the meaning of the existing boolean `--gpu`.

## Root cause

`buildChromeArgs()` always injected `--use-gl=angle --use-angle=swiftshader`. `disableGpu` only appended `--disable-gpu`; it did not provide a hardware-GPU mode. That made the public `--gpu` flag look broader than it was, because render capture stayed software-backed even when encoder GPU was requested.

## Verification

### Local checks

- `bun install`
- `bun run build:hyperframes-runtime`
- `bun run --filter @hyperframes/engine test src/config.test.ts src/services/browserManager.test.ts`
- `bun run --filter @hyperframes/cli test src/utils/dockerRunArgs.test.ts src/commands/render.test.ts`
- `bun run --filter @hyperframes/cli typecheck`
- `bun run --filter @hyperframes/engine typecheck`
- `bun run --filter @hyperframes/producer typecheck`
- `cd packages/producer && bunx vitest run src/services/renderOrchestrator.test.ts`
- `bunx oxlint packages/cli/src/commands/render.ts packages/cli/src/commands/render.test.ts packages/cli/src/utils/dockerRunArgs.ts packages/cli/src/utils/dockerRunArgs.test.ts packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/browserManager.ts packages/engine/src/services/browserManager.test.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `bunx oxfmt --check ...` on changed source/docs files
- `git diff --check`
- `bun packages/cli/src/cli.ts render --help | rg -n "browser-gpu|no-browser-gpu|GPU"`
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --output /tmp/hf-auto-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict`
  - Render plan prints `GPU: browser GPU (auto)`.
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --no-browser-gpu --output /tmp/hf-software-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict`
  - Render plan does not print browser GPU.
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --docker --browser-gpu --output /tmp/should-not-render.mp4`
  - Exits 1 with `Browser GPU is local-only`.
- `buildDockerRunArgs()` regression coverage asserts Docker container args include `--no-browser-gpu`, preventing nested container renders from re-enabling browser GPU through the local CLI default.
- `resolveBrowserGpuForCli()` regression coverage asserts `PRODUCER_BROWSER_GPU_MODE=software` opts out when no CLI browser-GPU flag is supplied, while explicit `--browser-gpu` / `--no-browser-gpu` still win.
- `ffmpeg -v error -i /tmp/hf-auto-browser-gpu-smoke.mp4 -f null -`
- `ffmpeg -v error -i /tmp/hf-software-browser-gpu-smoke.mp4 -f null -`
- `ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of json /tmp/hf-browser-gpu-smoke.mp4` -> H.264, 1920x1080, 24fps, 5.0s

### Apple presentation benchmark

Rendered `/Users/miguel07code/Downloads/apple-presentation.zip` as supplied after extracting to `/tmp/hf-apple-profile/apple-presentation`.

Fixed settings:

- 1920x1080
- 30fps
- `standard` quality
- 4240 frames
- 141.32s duration
- 8-worker cap; render auto-calibration used 6 capture workers
- macOS host detected FFmpeg GPU encoder: `videotoolbox`

| Mode | Equivalent flags after this PR | Wall time | vs software-browser baseline | Speed | Capture | Encode | Output |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Software browser + CPU encode | `--no-browser-gpu` | 120.77s | baseline | 1.17x | 97.87s | 10.04s | 8.38MB |
| Browser GPU + CPU encode | default local render | 70.10s | 42.0% faster | 2.02x | 50.72s | 9.91s | 8.39MB |
| Software browser + encoder GPU | `--no-browser-gpu --gpu` | 133.16s | 10.3% slower | 1.06x | 103.58s | 18.31s | 25.43MB |
| Browser GPU + encoder GPU | `--gpu` | 74.12s | 38.6% faster | 1.91x | 46.69s | 17.93s | 25.45MB |

Result: browser GPU capture is the meaningful improvement for this WebGL/browser-capture-heavy presentation. VideoToolbox encoding was slower and produced larger files for this current `standard` H.264 path, so `--gpu` should stay separate and opt-in.

Why `--gpu` plus browser GPU was slower than browser GPU alone: the combined run captured about 4.0s faster than browser GPU alone, but VideoToolbox encoding was about 8.0s slower than CPU x264 encoding, so the encode loss outweighed the capture gain.

### VideoToolbox flag check

I also isolated the encode stage against the already-captured Apple frames to check whether macOS GPU encoding only needed special flags.

`ffmpeg -h encoder=h264_videotoolbox` does not expose a CRF/CQ-style quality option like x264. It exposes bitrate-oriented and VideoToolbox-specific options such as `-b:v`, `-realtime`, `-profile`, `-coder`, `-prio_speed`, `-power_efficient`, and `-allow_sw`. That means our current `-q:v` mapping is not equivalent to x264 CRF and can produce very different bitrate/size behavior.

Measured full-frame encode variants on this host:

| VideoToolbox variant | Encode wall time | Output size | Bitrate |
| --- | ---: | ---: | ---: |
| Current `-q:v 64 -allow_sw 1` | 18.76s | 25.31MB | 1.43 Mbps |
| Current without `-allow_sw 1` | 18.21s | 25.31MB | 1.43 Mbps |
| `-b:v 500k -maxrate 750k -bufsize 1000k -profile high -coder cabac -realtime 1 -prio_speed 1 -power_efficient 0` | 20.58s | 7.42MB | 0.42 Mbps |
| Same with `-b:v 1500k` | 20.84s | 16.70MB | 0.95 Mbps |
| `-b:v 500k -profile baseline -coder cavlc -realtime 1 -prio_speed 1 -power_efficient 0` | 18.11s | 8.94MB | 0.51 Mbps |

Conclusion: VideoToolbox can be made size/bitrate-predictable with explicit `--video-bitrate`, but the tested speed-oriented flags did not make it faster than CPU x264 wall time for this render. That reinforces keeping `--gpu` encoder acceleration explicit and separate from browser GPU capture.

Artifacts from the local benchmark:

- `/tmp/hf-apple-profile/results/cpu.mp4`
- `/tmp/hf-apple-profile/results/browser-gpu.mp4`
- `/tmp/hf-apple-profile/results/encoder-gpu.mp4`
- `/tmp/hf-apple-profile/results/full-gpu.mp4`
- `/tmp/hf-apple-profile/results/summary.json`

All four benchmark MP4s completed `ffprobe` and full `ffmpeg -f null` decode checks.

### Pixel comparison

Compared decoded MP4 output between software-browser and browser-GPU renders:

- Apple presentation:
  - 4240 frames compared
  - 636 exact matching decoded frame hashes
  - 3604 different decoded frame hashes
  - Average PSNR: 57.79 dB
- `css-spinner-render-compat` clean fixture:
  - 120 frames compared
  - 0 exact matching decoded frame hashes
  - Average PSNR: 61.57 dB

Interpretation: browser GPU output is not strict hash/pixel-identical to the software-browser path after lossy H.264 encode, but the measured deltas are visually tiny. Above 50 dB PSNR is typically visually indistinguishable for normal video review. Use `--no-browser-gpu` or Docker when strict cross-run/cross-machine reproducibility matters more than local speed.

### Browser verification

- Started HyperFrames Studio preview for `packages/producer/tests/css-spinner-render-compat/src`.
- Used `agent-browser` to open `http://localhost:5191#project/src` and verify the composition loaded in Studio.
- Screenshots:
  - `/tmp/hf-gpu-browser-proof/preview-loaded.png`
  - `/tmp/hf-gpu-browser-proof/preview-playing.png`
  - `/tmp/hf-gpu-browser-proof/preview-frame-60.png`
- Agent-browser recordings:
  - `/tmp/hf-gpu-browser-proof/preview-playback.webm`
  - `/tmp/hf-gpu-browser-proof/preview-seek.webm`

## Notes

- Browser GPU is enabled automatically for local CLI renders and disabled in Docker.
- `--no-browser-gpu` is the opt-out for software Chrome/WebGL capture.
- `--gpu` remains encoder-only and opt-in.
- The Apple presentation zip has existing lint errors around unmanaged nested videos and imperative media `play()` calls. The benchmark still compares the same supplied source across modes, but it should not be treated as a clean deterministic-composition fixture.
 2026-04-30 06:46:14 +02:00
James Russo 57cdf7d80e perf(engine): content-addressed extraction cache for video frames (#446) 
## What

Adds a content-addressed cache for extracted video frames, keyed on the tuple `(path, mtime, size, mediaStart, duration, fps, format)`. Repeat renders of the same composition (studio edit → re-render, preview → final) skip the ffmpeg extraction entirely.

## Why

Video frame extraction is the dominant non-capture phase for video-heavy compositions. Studio iteration workflows extract the same frames over and over — each render burns ffmpeg time that adds no value.

Validated on `/tmp/hf-fixtures/cfr-sdr-cache`:
```
Cold (miss): extractMs=69,  videoExtractMs=70,  totalElapsedMs=2052
Warm (hit):  extractMs=1,   videoExtractMs=2,   totalElapsedMs=1964
cacheHits: 0→1, cacheMisses: 1→0
```
The fixture is tiny (3s CFR SDR @ 30fps), so the wall-clock delta is small; the extraction-time delta (69→1ms, 98%) scales linearly with source length. For heavy-iteration workflows (a user rendering the same composition while tuning encoding params), extraction time goes to zero on every repeat render.

Depends on #444 (instrumentation surface) and #445 (segment-scope HDR preflight — otherwise cache keys would be unstable across renders on mixed-HDR compositions).

## How

- New `packages/engine/src/services/extractionCache.ts`:
  - SHA-256 key over a stable JSON encoding of `(path, mtime_ms, size, mediaStart, duration, fps, format)`. Infinity duration is normalized to `-1` so unresolved natural-duration sources still produce stable keys.
  - Truncates to 16 hex chars in the entry directory name — 64 bits of entropy is plenty at cache scale and keeps `ls` output short.
  - `hfcache-v2-` schema prefix — bumping it invalidates old entries (callers own gc policy; the cache owns keys).
  - `.hf-complete` dotfile sentinel. An entry dir without the sentinel is treated as a miss (covers crash-mid-extract and abandoned writes); the next render re-extracts over the partial frames with `-y`.
  - `FRAME_FILENAME_PREFIX = "frame_"` shared with the extractor — future refactors only need to touch one place to rename frames.
- `EngineConfig.extractCacheDir` (env: `HYPERFRAMES_EXTRACT_CACHE_DIR`) gates the feature. Undefined disables caching — extraction runs into the render's workDir and cleanup removes it on render end, preserving the prior behaviour exactly. No default root is chosen by the engine; the caller (CLI, app, studio) owns the location policy.
- `ExtractedFrames.ownedByLookup` flag prevents `FrameLookupTable.cleanup` from rm'ing a shared cache dir at render end. Set to `true` on both hits and misses (misses own the directory they wrote into, but hand it over to the cache rather than deleting it).
- Phase 3 extractor flow:
  1. Snapshot `(videoPath, mediaStart, start, end)` per resolved video BEFORE Phase 2a/2b preflight mutates them — so cache keys are stable across renders that use workDir-local normalized files (those files have fresh mtimes every render).
  2. Compute key, `lookupCacheEntry`.
  3. On hit: rebuild `ExtractedFrames` from the cache dir plus the Phase 2-probed `VideoMetadata` — no re-ffprobe.
  4. On miss: `ensureCacheEntryDir`, extract with `extractVideoFramesRange(..., outputDirOverride)`, then `markCacheEntryComplete` (the sentinel write is the last step so a crash leaves the dir un-sentineled).
- `extractVideoFramesRange` gains an `outputDirOverride` parameter so cache-miss writes land directly in the keyed dir (no `join(outputDir, videoId)` wrapping).

## Test plan

- [x] 19 unit tests in `extractionCache.test.ts` covering key determinism, mtime/size invalidation, format/fps/mediaStart/duration invalidation, Infinity normalization, sentinel semantics, missing-file tolerance
- [x] 2 integration tests in `videoFrameExtractor.test.ts`:
  - "reuses extracted frames on a warm cache hit" — asserts `cacheHits=1`, `extractMs<50ms` on second call against a CFR SDR fixture
  - "invalidates the cache when fps changes" — different fps on second call forces a new miss
- [x] End-to-end validation with `HYPERFRAMES_EXTRACT_CACHE_DIR` set, two runs of the same fixture
- [x] Lint + format (oxlint + oxfmt)
- [x] Typecheck (engine + producer)
 2026-04-24 00:35:31 -04:00
Vance Ingalls 5256a93b2d feat(engine): wire options.hdr through chunkEncoder + dynamic SDR→HDR transfer (#370) 
## Summary

Three independent fixes that share a common thread: HDR config flowing correctly from `EngineConfig` down through every encoder. The headline fix: disk-based HDR encodes via `chunkEncoder` were silently producing BT.709-tagged output despite `options.hdr` being set.

## Why

`Chunk 3` of `plans/hdr-followups.md`. The streaming encoder was correct but `chunkEncoder.buildEncoderArgs` hard-coded BT.709 color tags and the `bt709` VUI block in `-x265-params`, even when callers passed an HDR `EncoderOptions`. Today this is harmless because `renderOrchestrator` routes native-HDR content to `streamingEncoder` and only feeds `chunkEncoder` sRGB Chrome screenshots — but the contract was a lie, and any future caller that wired HDR through `chunkEncoder` would silently get SDR output.

## What changed

**3A — `chunkEncoder` respects `options.hdr` (BT.2020 + mastering metadata).** When `options.hdr` is set, the libx265 software path emits `bt2020nc` plus the matching transfer (`smpte2084` for PQ, `arib-std-b67` for HLG) at the codec level *and* embeds master-display + max-cll SEI in `-x265-params` via `getHdrEncoderColorParams`. libx264 still tags BT.709 inside `-x264-params` (libx264 has no HDR support) but the codec-level color flags flip so the container describes pixels truthfully. GPU H.265 (nvenc/videotoolbox/qsv/vaapi) gets the BT.2020 tags but no `-x265-params` block, so static mastering metadata is omitted — acceptable for previews, not HDR-aware delivery.

**3B — `convertSdrToHdr` accepts a target transfer.** `videoFrameExtractor.convertSdrToHdr` was hard-coded to `transfer=arib-std-b67` (HLG) regardless of the surrounding composition's dominant transfer. `extractAllVideoFrames` now calls `analyzeCompositionHdr` first, then passes the dominant transfer (`"pq"` or `"hlg"`) into `convertSdrToHdr` so an SDR clip mixed into a PQ timeline gets converted with `smpte2084`, not `arib-std-b67`.

**3C — `EngineConfig.hdr` type matches its declared shape.** The IIFE for the `hdr` field returned `undefined` when `PRODUCER_HDR_TRANSFER` wasn't `"hlg"` or `"pq"`, but the field is typed as `{ transfer: HdrTransfer } | false`. Returning `false` matches the type and avoids a downstream `undefined` check.

## Test plan

- [x] `chunkEncoder.test.ts`: replaced the previous "HDR options ignored" assertions with 8 new specs covering BT.2020 + transfer tagging, master-display/max-cll embedding, libx264 fallback behavior, GPU H.265 + HDR (tags but no x265-params), and range conversion for both SDR and HDR CPU paths.
- [x] All 313 engine unit tests pass (5 new HDR specs).
- [x] `ffprobe` an HDR composition rendered through the chunk encoder path: shows `bt2020nc` color matrix, `smpte2084` transfer, and mastering display metadata.

## Stack

Chunk 3 of `plans/hdr-followups.md`. Independent of Chunks 1/4 (touches separate code paths).
 2026-04-22 20:36:29 -07:00
Miguel Ángel b98093aa1c fix: remove hidden audio gain in renders (#362) 
## Summary

This fixes a render-time audio correctness bug where Hyperframes applied a hidden post-mix gain to every rendered output, boosting audio by about +2.6 dB and causing clipping on normally leveled sources.

It also fixes a related mute bug where `data-volume="0"` was treated as falsy and silently converted back to full volume during audio track preparation.

Additionally, this PR fixes the Studio workspace typecheck path for `@hyperframes/player`, so local pre-commit/typecheck flows no longer depend on the Player package having been built first.

## Root Cause

The issue report measured a near-constant gain increase and suspected a hidden normalization step. After tracing the engine audio path, the root cause turned out to be explicit code, not FFmpeg behavior:

- `packages/engine/src/config.ts` defaulted `audioGain` to `1.35`
- `packages/engine/src/services/audioMixer.ts` always appended a post-mix FFmpeg filter:
  - `[mixed]volume=${masterOutputGain}[out]`
- with the default config, that meant every render got multiplied by `1.35`

That exactly matches the issue reporter's measured scalar boost.

While investigating the workaround, I also found a second correctness bug:

- `processCompositionAudio()` used `element.volume || 1.0`
- that coerced `0` to `1.0`
- so `data-volume="0"` did not actually mute the track in rendered output

Separately, the repo-level Studio typecheck could fail before any build step because:

- `packages/studio/src/player/components/Player.tsx` imports `@hyperframes/player`
- `packages/player/package.json` points TypeScript at built `dist/*` outputs
- in a fresh workspace, those built outputs may not exist yet
- Studio therefore failed type resolution for `@hyperframes/player` during pre-commit/typecheck

## What Changed

1. Set the engine default `audioGain` back to unity (`1`)
2. Preserve explicit zero volumes by changing `element.volume || 1.0` to `element.volume ?? 1.0`
3. Added regression coverage for both behaviors
4. Updated the producer-side config fixture to reflect the corrected default
5. Added a Studio tsconfig path mapping for `@hyperframes/player` to the local workspace source and widened `rootDir` so workspace typecheck succeeds without requiring a prior Player build

## Why These Changes Are Needed

This is not a UX preference issue; it is a correctness and API contract issue.

- The docs describe `data-volume` as a direct 0-1 control.
- Rendered output should preserve source levels unless the author explicitly changes them.
- Hidden global gain makes output non-deterministic from the author's perspective.
- `data-volume="0"` must mean silence, not full-volume playback.
- Local workspace typecheck should not require unrelated package build artifacts to exist first.

Leaving the current behavior in place means:

- voice recordings near normal peak levels can clip during render
- authors need undocumented manual compensation (`0.75`-ish scaling) to get unity output
- mute semantics in docs and code diverge
- local pre-commit/typecheck can fail for reasons unrelated to the actual diff being committed

## Testing

### Focused regression tests

Ran:

- `packages/engine/node_modules/.bin/vitest run packages/engine/src/config.test.ts packages/engine/src/services/audioMixer.test.ts`

Result:

- `10 passed`

These tests specifically verify:

- default resolved `audioGain` is `1`
- a track with `volume: 0` stays `volume=0` in the FFmpeg filter graph
- the post-mix output filter stays at unity gain (`[mixed]volume=1[out]`)

### Broader package verification

Ran:

- `bun run --filter @hyperframes/engine test`
- `bun run --filter @hyperframes/engine build`
- `packages/engine/node_modules/.bin/vitest run packages/producer/src/services/renderOrchestrator.test.ts`
- `bun run --filter @hyperframes/producer typecheck`
- `bun run --filter @hyperframes/studio typecheck`
- `bunx oxlint packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/audioMixer.ts packages/engine/src/services/audioMixer.test.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `bunx oxfmt packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/audioMixer.ts packages/engine/src/services/audioMixer.test.ts packages/producer/src/services/renderOrchestrator.test.ts packages/studio/tsconfig.json`
- `bunx lefthook run pre-commit`

Results:

- full engine test suite passed (`309 passed`)
- engine build passed
- touched producer test file passed (`7 passed`)
- producer typecheck passed
- studio typecheck passed
- oxlint passed with `0 warnings, 0 errors`
- formatting passed
- pre-commit hook no longer hits the prior `@hyperframes/player` module-resolution blocker

## Known Verification Limitation

There is no meaningful browser UI flow for this bug: the defect is in the engine/CLI audio render pipeline rather than an interactive browser surface. Because of that, verification was done at the renderer and test level rather than through an agent-browser flow.

## User Impact

After this change:

- rendered audio matches source level by default
- authors no longer need to compensate for a hidden +2.6 dB boost
- `data-volume="0"` correctly mutes rendered audio
- the documented volume contract matches engine behavior again
- local workspace typecheck no longer depends on prebuilt `@hyperframes/player` artifacts

Closes #361.
 2026-04-21 20:33:25 +02:00
Vance Ingalls 5a3fde19d4 feat(engine): add HDR video output pipeline (#265) 
## Summary

Adds the ability to render HDR video output (H.265 10-bit, BT.2020) from HyperFrames compositions. When the renderer detects HDR source video, it automatically switches to the HDR output pipeline — no flags needed.

## What it does

- **Auto-detection** — Probes each video source with `ffprobe`. If any has bt2020/PQ/HLG color metadata, the output switches to H.265 10-bit with correct color tags. SDR-only compositions are unaffected (H.264, bt709).
- **HLG pass-through** — Native HLG pixels from FFmpeg extraction are piped directly to the encoder without conversion. This avoids brightness loss from HLG→linear→PQ conversion (which requires an OOTF system gamma we can't reliably apply).
- **Encoder HDR support** — Both chunk and streaming encoders accept HDR presets: `libx265`, `yuv420p10le`, BT.2020 color primaries, `hvc1` codec tag (required for Apple playback).
- **WebGPU HDR capture (gated)** — A complete WebGPU float16 readback pipeline is implemented and tested but gated behind headed Chrome (headless doesn't expose WebGPU). Ready for future use with WebGPU canvas content.
- **HDR utilities** — `detectTransfer()` (PQ vs HLG), `getHdrEncoderColorParams()`, `analyzeCompositionHdr()`. 15 unit tests.

## Key design decisions

| Decision | Why |
|----------|-----|
| No `--hdr` flag | SDR content encoded as HDR causes orange shift in browsers. Auto-detect eliminates this. |
| HLG pass-through (not HLG→PQ) | Conversion loses brightness without OOTF. Pass-through matches source exactly. |
| `hvc1` codec tag | Apple QuickTime requires `hvc1` (not `hev1`) for HEVC playback. |
| 1-hour streaming timeout | HDR capture at ~6fps needs more time than the default 10-minute FFmpeg timeout. |

## Files changed

| File | What changed |
|------|-------------|
| `packages/engine/src/utils/hdr.ts` | **NEW** — HDR detection, transfer types, encoder params (15 tests) |
| `packages/engine/src/services/hdrCapture.ts` | **NEW** — WebGPU readback, HLG conversion, PQ encode |
| `packages/engine/src/services/streamingEncoder.ts` | HDR presets, raw rgb48le input, color tags |
| `packages/engine/src/services/chunkEncoder.ts` | HDR presets, conditional color tags |
| `packages/producer/src/services/renderOrchestrator.ts` | Auto-detection loop, HDR pass-through capture path |

## How to test

Render a composition with an HDR video source. The output should be H.265 10-bit with HDR metadata visible in `ffprobe` (bt2020, arib-std-b67 or smpte2084). Plays correctly in QuickTime and on HDR displays.

## Stack position

**2 of 6** — Stacked on #258 (SDR/HDR normalization). Provides the encoder infrastructure that phases 1-5 build on.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-04-19 15:10:59 -07:00
Vance Ingalls 20be2ea1c2 style: apply oxfmt baseline formatting across all source files (#25) 
## Summary
- Run `oxfmt .` across the entire codebase to establish formatted baseline
- 299 files changed — mechanical formatting only, no logic changes
- Double quotes, semicolons, 2-space indent, trailing commas, 100 print width

Part 3/4 of [VA-851](https://linear.app/heygen/issue/VA-851/pre-migration-configure-eslint-prettier-and-conventional-commits)

## Test plan
- [x] `pnpm format:check` — all 426 files pass
- [x] `pnpm -r typecheck` — all packages pass
- [x] `pnpm build` — all packages build
- [x] All 348 tests pass
 2026-03-23 17:15:14 -07:00
Vance IngallsandClaude Opus 4.6 9f8e5ba5a1 initial code (#2) 
* feat: initial code port from hyperframes-internal

Port all OSS-ready packages from the internal monorepo:
- @hyperframes/core — shared types, HTML generation, GSAP utilities, runtime
- @hyperframes/cli — CLI for creating, previewing, and rendering compositions
- @hyperframes/engine — framework-agnostic rendering engine (BeginFrame + FFmpeg)
- @hyperframes/producer — video rendering pipeline (Puppeteer + FFmpeg)
- @hyperframes/ui-player — browser-based video player component
- @hyperframes/studio — composition editor (React frontend + Hono backend)

Includes regression test suite with Docker-based test harness.

All HeyGen-internal references, deployment infrastructure, and
proprietary assets have been removed. Package names migrated
from @app/* to @hyperframes/*.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: scrub internal codenames and stale references from OSS port

- Replace static.heygen.ai runtime URLs in test fixtures
- Remove internal CDN publish script (publish-hyperframe-runtime.ts)
- Replace sandbox-studio, sandbox-interceptor, __magicEditRuntime
  with neutral names (studio, hyperframe-runtime, __hyperframeRuntime)
- Fix stale Vault API / localhost references in docs
- Remove broken deprecated_studio link

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove remaining internal codenames and stale references

- Delete stale producer README.md and PIPELINE.md (referenced nonexistent files)
- Replace "Cerberus" codename with "HyperFrames" in test design reviews
- Replace magic-edit postMessage identifiers with hf-preview/hf-parent
- Rename debug-magic-edit-timeline.ts to debug-timeline.ts
- Replace "Motion Cut" with "HyperFrames" in Timeline comments
- Fix studio/CLI references to nonexistent archive package
  (use local data/projects/ dir, stub render proxy)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-21 22:43:56 -07:00