## Problem
Closes#574.
On Windows with cached headless-shell Chrome, a composition that reuses the same video file in three timeline clips can fail before frame capture starts:
```html
<video id="video1" src="1.mp4" data-start="0" muted data-duration="4" data-track-index="0" data-media-start="0"></video>
<video id="video2" src="1.mp4" data-start="4" muted data-duration="4" data-track-index="0" data-media-start="4"></video>
<video id="video3" src="1.mp4" data-start="8" muted data-duration="4" data-track-index="0" data-media-start="8"></video>
```
The reported render reaches video frame extraction, then dies at frame-capture initialization with:
```text
[FrameCapture] video metadata not ready after 45000ms. Video elements must load metadata before capture starts.
```
The important detail is that by this stage HyperFrames has already extracted video pixels through FFmpeg. Native Chromium video metadata is only being waited on for DOM layout stability, not because Chromium is the source of rendered pixels.
## Root Cause
The render pipeline has two separate media responsibilities:
- FFmpeg extracts video frames and audio from declared media.
- Chromium owns DOM layout and capture, while injected FFmpeg frames supply the video pixels before each captured frame.
Before this PR, every capture session still waited for every DOM `<video>` to reach `readyState >= 1` unless the element was a native HDR exception. That made native browser media metadata a hard render prerequisite even when the browser would not decode or provide the final video pixels.
That is why the issue fails at `25% Starting frame capture`: FFmpeg extraction has already succeeded, but capture initialization blocks on repeated native `<video src="1.mp4">` metadata loading in cached Windows headless-shell Chrome.
There was a second constraint: the readiness wait also prevents first-frame layout bugs. If a skipped `<video>` has no native metadata, Chromium can use the default `300x150` intrinsic video size, which breaks layouts such as `width: 100%; height: auto` before the first injected frame. The fix therefore must not simply skip all video readiness waits; it must provide dimensions for any skipped videos.
## What This Fixes
- Treats videos with successfully extracted FFmpeg frames and usable dimensions as out-of-band rendered video sources.
- Skips native browser metadata readiness waits for those extracted videos because Chromium is not responsible for their pixels.
- Passes FFmpeg-probed dimensions into capture as `videoMetadataHints`.
- Applies those hints before the readiness wait in both screenshot and BeginFrame initialization paths.
- Sets missing `width` / `height` attributes and an explicit `aspect-ratio` only when the element does not already provide one, preserving author styles where present.
- Keeps native HDR video IDs in the skip list, preserving the existing HEVC/HDR behavior where Chrome may not decode the source but FFmpeg/native HDR compositing can still render it.
- Uses one `buildCaptureOptions()` helper so calibration, HDR DOM capture, streaming capture, parallel capture, and sequential capture receive the same skip IDs and metadata hints.
- Adds tests for the skip-list and metadata-hint contract.
- Adds a Windows CI regression that reproduces the issue shape after the canary render warms the cached-browser path.
## Reviewer Map
Primary files:
- `packages/producer/src/services/renderOrchestrator.ts`
- `collectVideoReadinessSkipIds()` includes native HDR IDs plus extracted videos that have finite positive FFmpeg dimensions.
- `collectVideoMetadataHints()` converts extracted FFmpeg metadata into capture hints.
- `buildCaptureOptions()` threads `skipReadinessVideoIds` and `videoMetadataHints` into every capture path.
- `packages/engine/src/services/frameCapture.ts`
- `applyVideoMetadataHints()` runs in the page before video readiness polling.
- Both screenshot and BeginFrame initialization call it before checking non-skipped videos for `readyState >= 1`.
- `packages/engine/src/types.ts`
- Adds `CaptureVideoMetadataHint` and documents that readiness skips should be paired with metadata hints when layout may depend on intrinsic dimensions.
- `packages/producer/src/services/renderOrchestrator.test.ts`
- Covers that extracted videos with dimensions are skipped, invalid dimensions are not, native HDR IDs are preserved, and hints are stable/sorted.
- `.github/workflows/windows-render.yml`
- Adds the issue #574 Windows regression with the exact three-clip markup and a generated deterministic `1.mp4`.
## Why This Is Safe
The skip is intentionally gated:
- A standard video is skipped only after `extractAllVideoFrames()` succeeded for that video and returned usable dimensions.
- Videos with invalid dimensions are not skipped, so the old browser readiness guard still applies.
- DOM videos are still present for layout and element bounds; only the native metadata wait is skipped for sources whose pixels come from FFmpeg injection.
- Metadata hints are applied conservatively: existing `width`, `height`, and explicit `aspect-ratio` are not overwritten.
- Non-extracted videos, images, fonts, page readiness, and `window.__hf` readiness keep the existing waits.
- The fix is not limited to the sequential path from the issue; it is threaded through calibration, HDR DOM capture, streaming encode, parallel capture, and sequential capture.
A first local revision skipped readiness too broadly and caused `overlay-montage-prod` first-frame layout shrinkage. The current version fixes that by pairing skips with FFmpeg metadata hints; `overlay-montage-prod` now passes and is listed in verification below.
## Verification
### Root-Cause Reproduction Before Fix
The reporter did not attach the actual `1.mp4`, so the regression uses the exact issue markup and a deterministic generated 12s H.264 file named `1.mp4`.
I reproduced the failure in GitHub Actions by running this branch's new Windows workflow against unpatched `main`:
```bash
gh workflow run windows-render.yml --repo heygen-com/hyperframes --ref fix/reused-video-metadata -f ref=main
```
That means the workflow contains the new issue #574 regression, but the code under test is `main` without this fix.
Baseline failure:
- Run: https://github.com/heygen-com/hyperframes/actions/runs/25174603730
- Failed job: https://github.com/heygen-com/hyperframes/actions/runs/25174603730/job/73803179086
- Checkout proof: `ref: main`, `origin/main`, commit `8662598a3ac64018a2999d189ffb369e6d46b53a`.
- Failure proof: `Browser: cache`, `staticDuration:12`, `videoCount:3`, then `25% Starting frame capture` -> `[FrameCapture] video metadata not ready after 15000ms`.
This is the same failure class as the issue, on Windows, in cached-browser mode, before the fix.
### Fixed Windows Regression
The same regression passes on this PR branch:
- Run: https://github.com/heygen-com/hyperframes/actions/runs/25175048215
- Passing job: https://github.com/heygen-com/hyperframes/actions/runs/25175048215/job/73804781017
- Checkout proof: PR merge contains `79d6b41c9f2ba137cbfb9301678e0815b16c4f5a` merged into `8662598a3ac64018a2999d189ffb369e6d46b53a`.
- Passing proof: `Browser: cache`, `staticDuration:12`, `videoCount:3`, `25% Starting frame capture`, captures `360/360` frames, renders `issue-574.mp4`, and `ffprobe` verifies `1920x1080 @ 30/1, 12s`.
### Local Checks
- `bun run build:hyperframes-runtime`
- `bunx vitest run packages/producer/src/services/renderOrchestrator.test.ts`
- `bun run --filter @hyperframes/producer typecheck`
- `bun run --filter @hyperframes/engine typecheck`
- `bunx oxlint packages/engine/src/services/frameCapture.ts packages/engine/src/types.ts packages/engine/src/index.ts packages/producer/src/services/renderOrchestrator.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `bunx oxfmt --check .github/workflows/windows-render.yml packages/engine/src/services/frameCapture.ts packages/engine/src/types.ts packages/engine/src/index.ts packages/producer/src/services/renderOrchestrator.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `git diff --check`
- Lefthook pre-commit: lint, format, typecheck where applicable
- Lefthook commit-msg: commitlint
### Local Render Checks
- Created `/tmp/hf-issue-574-repro` with the issue shape: three clips using the same `1.mp4`, `data-media-start=0/4/8`, 12s total.
- `PRODUCER_PLAYER_READY_TIMEOUT_MS=5000 bun packages/cli/src/cli.ts render /tmp/hf-issue-574-repro --workers 1 --quality draft --fps 30 --output /tmp/hf-issue-574-h264-fixed-v2.mp4` -> completed.
- Created `/tmp/hf-issue-574-prores` with the same three-clip shape using one FFmpeg-readable ProRes `.mov`, which exercises the browser-metadata failure class because Chromium should not be needed to decode the source.
- `PRODUCER_PLAYER_READY_TIMEOUT_MS=3000 bun packages/cli/src/cli.ts render /tmp/hf-issue-574-prores --workers 1 --quality draft --fps 30 --output /tmp/hf-issue-574-prores-fixed-v2.mp4` -> completed.
- `bun run --filter @hyperframes/producer test --sequential --keep-temp overlay-montage-prod` -> passed; this guards against skipped metadata shrinking `height:auto` video layout before the first injected frame.
- `ffmpeg -v error -i /tmp/hf-issue-574-prores-fixed-v2.mp4 -f null -`
- `ffmpeg -v error -i /tmp/hf-issue-574-h264-fixed-v2.mp4 -f null -`
- `ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of json /tmp/hf-issue-574-h264-fixed-v2.mp4` -> H.264, 320x180, 30fps, 12.0s.
### Current PR Checks
- Windows render verification: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048215.
- Windows tests: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048215.
- Main CI build/lint/typecheck/test/smoke jobs: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048175.
- Regression shards observed passing include HDR, render-compat, styles A-G, and `overlay-montage-prod`. At the time this body was updated, the `fast` regression shard was still in progress in run https://github.com/heygen-com/hyperframes/actions/runs/25174515546.
### Browser Verification
- Used `agent-browser` to open `file:///tmp/hf-issue-574-h264-fixed-v2.mp4` and verify the rendered output displays in Chromium.
- Screenshot: `.debug/issue-574/h264-output-page.png`
- Agent-browser recording: `.debug/issue-574/h264-output-playback.webm`
## Notes / Caveats
- The reporter's exact `1.mp4` was not attached to #574. The committed Windows regression uses a generated deterministic H.264 file with the same filename and exact markup from the issue.
- The exact H.264 issue shape did not reproduce the timeout on this macOS/system-Chrome machine before the fix; it rendered successfully locally. The GitHub Actions baseline above reproduces it on Windows/cache without the fix.
- The Windows fixture intentionally runs after the existing canary render so the browser path is `Browser: cache`, matching the reporter's environment.
- The generated fixture emits sparse-keyframe warnings. Those warnings are expected and are not the failure being fixed; the baseline failure occurs before any frame capture because native browser video metadata never becomes ready.
- Browser proof artifacts are local-only under `.debug/issue-574/` and intentionally not committed.
## Problem
HyperFrames already had `--gpu`, but that flag only controlled FFmpeg hardware encoding. The browser capture path still forced Chrome/WebGL through SwiftShader software GL via `--use-angle=swiftshader`, so WebGL-heavy local renders could leave the biggest bottleneck on the CPU path.
That made the existing flag naming easy to misread: `--gpu` sounded like it accelerated the whole render, but it did not change the browser frame-capture backend.
## What this fixes
- Enables host browser GPU acceleration automatically for local CLI renders.
- Adds `--no-browser-gpu` as the local opt-out for software Chrome/WebGL capture.
- Keeps `--browser-gpu` as an explicit local browser-GPU request.
- Adds `browserGpuMode: "software" | "hardware"` to engine config, with `PRODUCER_BROWSER_GPU_MODE` env support for lower-level producer users.
- Keeps Docker browser capture on the deterministic software path.
- Maps hardware browser GPU mode to platform-native Chrome backends:
- macOS: Metal-backed ANGLE
- Windows: D3D11-backed ANGLE
- Linux: EGL
- Blocks explicit `--browser-gpu --docker` with a clear error because Docker browser GPU passthrough is not cross-platform.
- Clarifies docs so `--gpu` means FFmpeg encoder GPU and browser GPU means Chrome/WebGL capture GPU.
- Keeps encoder backend selection auto-detected from FFmpeg capabilities:
- NVIDIA: NVENC
- macOS: VideoToolbox
- Linux: VAAPI
- Intel: QSV
## Why two flags
There are two separate GPU surfaces in the render pipeline:
1. Browser GPU controls Chrome frame capture.
- Affects WebGL, canvas, CSS rendering, compositing, and screenshot capture inside the browser.
- This is enabled automatically for local CLI renders.
- Use `--no-browser-gpu` when you want the software browser baseline.
2. `--gpu` controls FFmpeg video encoding.
- Affects the final encode step after frames have already been captured.
- The concrete encoder is auto-detected from the host FFmpeg build and hardware.
- It can be faster for some machines/codecs, but it is not equivalent to browser rendering acceleration.
The controls stay independent because users may want:
- `hyperframes render` for the fast local default with browser GPU capture.
- `hyperframes render --no-browser-gpu` for the software-browser local baseline.
- `hyperframes render --gpu` for browser GPU capture plus hardware FFmpeg encoding.
- `hyperframes render --no-browser-gpu --gpu` for software browser capture plus hardware FFmpeg encoding.
- `hyperframes render --docker` for deterministic browser capture.
## Why `--gpu` does not imply browser GPU
Keeping `--gpu` scoped to FFmpeg encoding avoids a semantic break and keeps the risk profile explicit:
- `--gpu` already means encoder acceleration. Expanding it to also change Chrome capture would silently alter behavior for users who only wanted hardware encoding.
- Browser GPU and encoder GPU have different portability. Encoder GPU can work in Docker when the host exposes the right devices; browser GPU passthrough is not cross-platform, so this PR intentionally blocks explicit `--browser-gpu --docker`.
- The Apple presentation benchmark shows why the controls should stay separate: browser GPU capture was the useful improvement, while macOS VideoToolbox via `--gpu` was slower and produced larger output for this `standard` H.264 run.
If HyperFrames later wants a single umbrella acceleration control, it should be explicit, for example `--acceleration browser|encoder|all` or `--gpu=browser|encoder|all`, rather than changing the meaning of the existing boolean `--gpu`.
## Root cause
`buildChromeArgs()` always injected `--use-gl=angle --use-angle=swiftshader`. `disableGpu` only appended `--disable-gpu`; it did not provide a hardware-GPU mode. That made the public `--gpu` flag look broader than it was, because render capture stayed software-backed even when encoder GPU was requested.
## Verification
### Local checks
- `bun install`
- `bun run build:hyperframes-runtime`
- `bun run --filter @hyperframes/engine test src/config.test.ts src/services/browserManager.test.ts`
- `bun run --filter @hyperframes/cli test src/utils/dockerRunArgs.test.ts src/commands/render.test.ts`
- `bun run --filter @hyperframes/cli typecheck`
- `bun run --filter @hyperframes/engine typecheck`
- `bun run --filter @hyperframes/producer typecheck`
- `cd packages/producer && bunx vitest run src/services/renderOrchestrator.test.ts`
- `bunx oxlint packages/cli/src/commands/render.ts packages/cli/src/commands/render.test.ts packages/cli/src/utils/dockerRunArgs.ts packages/cli/src/utils/dockerRunArgs.test.ts packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/browserManager.ts packages/engine/src/services/browserManager.test.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `bunx oxfmt --check ...` on changed source/docs files
- `git diff --check`
- `bun packages/cli/src/cli.ts render --help | rg -n "browser-gpu|no-browser-gpu|GPU"`
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --output /tmp/hf-auto-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict`
- Render plan prints `GPU: browser GPU (auto)`.
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --no-browser-gpu --output /tmp/hf-software-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict`
- Render plan does not print browser GPU.
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --docker --browser-gpu --output /tmp/should-not-render.mp4`
- Exits 1 with `Browser GPU is local-only`.
- `buildDockerRunArgs()` regression coverage asserts Docker container args include `--no-browser-gpu`, preventing nested container renders from re-enabling browser GPU through the local CLI default.
- `resolveBrowserGpuForCli()` regression coverage asserts `PRODUCER_BROWSER_GPU_MODE=software` opts out when no CLI browser-GPU flag is supplied, while explicit `--browser-gpu` / `--no-browser-gpu` still win.
- `ffmpeg -v error -i /tmp/hf-auto-browser-gpu-smoke.mp4 -f null -`
- `ffmpeg -v error -i /tmp/hf-software-browser-gpu-smoke.mp4 -f null -`
- `ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of json /tmp/hf-browser-gpu-smoke.mp4` -> H.264, 1920x1080, 24fps, 5.0s
### Apple presentation benchmark
Rendered `/Users/miguel07code/Downloads/apple-presentation.zip` as supplied after extracting to `/tmp/hf-apple-profile/apple-presentation`.
Fixed settings:
- 1920x1080
- 30fps
- `standard` quality
- 4240 frames
- 141.32s duration
- 8-worker cap; render auto-calibration used 6 capture workers
- macOS host detected FFmpeg GPU encoder: `videotoolbox`
| Mode | Equivalent flags after this PR | Wall time | vs software-browser baseline | Speed | Capture | Encode | Output |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Software browser + CPU encode | `--no-browser-gpu` | 120.77s | baseline | 1.17x | 97.87s | 10.04s | 8.38MB |
| Browser GPU + CPU encode | default local render | 70.10s | 42.0% faster | 2.02x | 50.72s | 9.91s | 8.39MB |
| Software browser + encoder GPU | `--no-browser-gpu --gpu` | 133.16s | 10.3% slower | 1.06x | 103.58s | 18.31s | 25.43MB |
| Browser GPU + encoder GPU | `--gpu` | 74.12s | 38.6% faster | 1.91x | 46.69s | 17.93s | 25.45MB |
Result: browser GPU capture is the meaningful improvement for this WebGL/browser-capture-heavy presentation. VideoToolbox encoding was slower and produced larger files for this current `standard` H.264 path, so `--gpu` should stay separate and opt-in.
Why `--gpu` plus browser GPU was slower than browser GPU alone: the combined run captured about 4.0s faster than browser GPU alone, but VideoToolbox encoding was about 8.0s slower than CPU x264 encoding, so the encode loss outweighed the capture gain.
### VideoToolbox flag check
I also isolated the encode stage against the already-captured Apple frames to check whether macOS GPU encoding only needed special flags.
`ffmpeg -h encoder=h264_videotoolbox` does not expose a CRF/CQ-style quality option like x264. It exposes bitrate-oriented and VideoToolbox-specific options such as `-b:v`, `-realtime`, `-profile`, `-coder`, `-prio_speed`, `-power_efficient`, and `-allow_sw`. That means our current `-q:v` mapping is not equivalent to x264 CRF and can produce very different bitrate/size behavior.
Measured full-frame encode variants on this host:
| VideoToolbox variant | Encode wall time | Output size | Bitrate |
| --- | ---: | ---: | ---: |
| Current `-q:v 64 -allow_sw 1` | 18.76s | 25.31MB | 1.43 Mbps |
| Current without `-allow_sw 1` | 18.21s | 25.31MB | 1.43 Mbps |
| `-b:v 500k -maxrate 750k -bufsize 1000k -profile high -coder cabac -realtime 1 -prio_speed 1 -power_efficient 0` | 20.58s | 7.42MB | 0.42 Mbps |
| Same with `-b:v 1500k` | 20.84s | 16.70MB | 0.95 Mbps |
| `-b:v 500k -profile baseline -coder cavlc -realtime 1 -prio_speed 1 -power_efficient 0` | 18.11s | 8.94MB | 0.51 Mbps |
Conclusion: VideoToolbox can be made size/bitrate-predictable with explicit `--video-bitrate`, but the tested speed-oriented flags did not make it faster than CPU x264 wall time for this render. That reinforces keeping `--gpu` encoder acceleration explicit and separate from browser GPU capture.
Artifacts from the local benchmark:
- `/tmp/hf-apple-profile/results/cpu.mp4`
- `/tmp/hf-apple-profile/results/browser-gpu.mp4`
- `/tmp/hf-apple-profile/results/encoder-gpu.mp4`
- `/tmp/hf-apple-profile/results/full-gpu.mp4`
- `/tmp/hf-apple-profile/results/summary.json`
All four benchmark MP4s completed `ffprobe` and full `ffmpeg -f null` decode checks.
### Pixel comparison
Compared decoded MP4 output between software-browser and browser-GPU renders:
- Apple presentation:
- 4240 frames compared
- 636 exact matching decoded frame hashes
- 3604 different decoded frame hashes
- Average PSNR: 57.79 dB
- `css-spinner-render-compat` clean fixture:
- 120 frames compared
- 0 exact matching decoded frame hashes
- Average PSNR: 61.57 dB
Interpretation: browser GPU output is not strict hash/pixel-identical to the software-browser path after lossy H.264 encode, but the measured deltas are visually tiny. Above 50 dB PSNR is typically visually indistinguishable for normal video review. Use `--no-browser-gpu` or Docker when strict cross-run/cross-machine reproducibility matters more than local speed.
### Browser verification
- Started HyperFrames Studio preview for `packages/producer/tests/css-spinner-render-compat/src`.
- Used `agent-browser` to open `http://localhost:5191#project/src` and verify the composition loaded in Studio.
- Screenshots:
- `/tmp/hf-gpu-browser-proof/preview-loaded.png`
- `/tmp/hf-gpu-browser-proof/preview-playing.png`
- `/tmp/hf-gpu-browser-proof/preview-frame-60.png`
- Agent-browser recordings:
- `/tmp/hf-gpu-browser-proof/preview-playback.webm`
- `/tmp/hf-gpu-browser-proof/preview-seek.webm`
## Notes
- Browser GPU is enabled automatically for local CLI renders and disabled in Docker.
- `--no-browser-gpu` is the opt-out for software Chrome/WebGL capture.
- `--gpu` remains encoder-only and opt-in.
- The Apple presentation zip has existing lint errors around unmanaged nested videos and imperative media `play()` calls. The benchmark still compares the same supplied source across modes, but it should not be treated as a clean deterministic-composition fixture.
* fix(hdr): filter zero-opacity elements and support overflow:hidden clip rects in HDR compositor
Two bugs in the HDR render pipeline:
1. Child data-start elements inside a parent with opacity:0 were still
composited as independent layers, painting over content in later scenes.
Fix: filter elements with effective opacity 0 before groupIntoLayers().
2. CSS overflow:hidden on ancestor elements was ignored for HDR video layers,
causing videos inside clipped containers (e.g. split-screen halves) to
render full-frame. Fix: add clipRect to ElementStackingInfo, compute it
from ancestor overflow:hidden in queryElementStacking(), and crop the
source buffer to clip bounds before blitting in blitHdrVideoLayer().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(hdr): move opacity filter into blit loop to preserve hide-list correctness
The previous approach filtered zero-opacity elements before groupIntoLayers(),
which broke the DOM screenshot hide-list — invisible video elements' <img>
replacements weren't properly hidden from sibling layer screenshots, causing
the vignelli-stacking regression.
Fix: keep all elements in groupIntoLayers() for correct hide-list generation.
Skip zero-opacity HDR elements only during the actual blit step with an early
`continue` in the compositing loop.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(hdr): route identity-matrix HDR elements through region blit for clip rect support
parseTransformMatrix returns a valid matrix even for untransformed HDR
elements (Chrome reports matrix(1,0,0,1,0,0)). This made the affine blit
path always run, bypassing the region blit path which is the only one that
applies clip rects from overflow:hidden ancestors.
Fix: detect identity matrices and route them through the region path so
the cropRgb48le clip logic is reachable for split-screen layouts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(hdr): handle translation-only matrices for clip rect support
The previous isIdentity check only caught matrix(1,0,0,1,0,0). Elements
with layout translation (e.g. right-half split at left:960px reporting
matrix(1,0,0,1,960,0)) still routed through the affine path where clip
rects are not applied.
Fix: check for translation-only matrices (scale=1, rotation=0, any tx/ty)
and route those through the region blit path. el.x/el.y from
getBoundingClientRect already include the translation, so the region path
handles positioning correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(render): auto-detect HDR from media probes, add --sdr flag
Replace the --hdr opt-in model with automatic detection. When no flags
are passed, the renderer probes all video/image sources and enables HDR
output if any HDR color space is detected. Existing --hdr flag becomes
a force override. New --sdr flag forces SDR output.
Behavior matrix:
(no flags) + HDR content → HDR output
(no flags) + SDR content → SDR output
--hdr → force HDR (defaults to HLG if no HDR sources)
--sdr → force SDR (skips probing)
--hdr --sdr → error
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Revert "feat(render): auto-detect HDR from media probes, add --sdr flag"
This reverts commit 69fb52196f.
* chore(hdr): simplify review fixes — remove redundant guard, add image clip warning
- Remove redundant viewportMatrix.length >= 6 check (parseTransformMatrix
always returns 6-element array or null)
- Add clip rect warning log to blitHdrImageLayer for parity with video path
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extends RenderConfig.format with "png-sequence" and patches two correctness
gaps so the existing "webm" / "mov" values actually preserve the alpha
channel end-to-end.
Engine fixes:
- screenshotService.pageScreenshotCapture: drop optimizeForSpeed for PNG
captures. The fast path uses an alpha-unaware codec that crushes real
alpha values; kept for opaque jpeg captures where it is harmless.
- frameCapture: replace the inline setDefaultBackgroundColorOverride
block (which fired pre-navigation and was reset by page.goto) with a
proper initTransparentBackground() call inside initializeSession,
after the window.__hf readiness poll. This also injects the
html/body/[data-composition-id]{background:transparent !important}
stylesheet so compositions with custom body / #root backgrounds do not
defeat the override. Wired into both screenshot-mode and beginframe-mode
branches.
Producer:
- RenderConfig.format extended to "mp4" | "webm" | "mov" | "png-sequence"
with full JSDoc.
- Streaming encode is bypassed for png-sequence (frames go straight to
disk). FORMAT_EXT extended.
- New Stage-5 png-sequence branch: mkdir outputPath, copy captured PNGs as
frame_NNNNNN.png, copy audio.aac sidecar when audio is present.
- Stage-6 mux/faststart and the debug copy are wrapped in !isPngSequence.
- README.md: new "Transparent Video Output" section.
Tests:
- New fixture tests/transparency-regression/ tagged "transparency".
- New tsx script src/transparency-test.ts asserts pixel-level alpha for
webm + png-sequence outputs. Wired as "test:transparency".
- Default "test" / "test:update" scripts pass --exclude-tags transparency
so the golden-MP4 harness ignores the new fixture.
Verified locally on macOS arm64: typecheck clean across engine + producer,
producer renderOrchestrator vitest 10/10, transparency-test passes for
both webm and png-sequence with end-to-end pixel assertions.
* fix(engine): suppress benign AbortError spam from frame-capture pageerror
Frame capture pauses → seeks → screenshots → plays audio/video many times
per second. HTMLMediaElement.play() returns a promise that rejects with
AbortError whenever another pause() lands before it resolves — which it
does, every frame. The rejection is benign (output frames and mixed
audio are unaffected) but the frameCapture pageerror handler was logging
it to stderr, producing dozens of identical lines per render:
[Browser:PAGEERROR] AbortError: The play() request was interrupted by
a call to pause(). https://goo.gl/LdLk22
Filter out exactly this pattern before console.error — still pushed to
browserConsoleBuffer so it's available in the failure-diagnostic dump.
* fix(engine): trim play-abort filter comment and drop unnecessary regex flags
The why-it-exists explanation belongs in the commit message and PR
description, not as a 12-line comment block at the call site.
Chrome's play()/pause() AbortError message is always lowercase, so
the case-insensitive flag implies uncertainty that doesn't exist.
Replace the two /play\(\)/i and /pause\(\)/i regexes with plain
String.prototype.includes — same outcome, less ceremony.
## Summary
- preserve alpha for render-injected video frames by detecting alpha streams with ffprobe and extracting alpha video frames as PNG
- keep `<video loop>` semantics through static parsing, compiler duration resolution, browser media discovery, and render frame lookup
- fail embedded preview startup before opening a broken browser page when the Studio bundle is missing
- align snapshot frame injection with looped media timing and VP9 alpha extraction
## Why
The Studio preview and rendered MP4 could disagree for timed transparent looped videos. The Comfy funding composition exposed two separate parity bugs: render-injected frames needed alpha-preserving PNG extraction, and the compiler was clamping a looped `data-duration="4"` video down to the 3.125s source duration. After the first source cycle, render lookup treated the video as inactive, hid the native video, and produced the blank polygon/glow the user saw around the rounded `0:03` mark.
`hyperframes lint` and `hyperframes validate` did not catch this because they check syntax/load/console/accessibility, not preview-vs-render visual parity. This PR adds regression coverage for the compiler loop-duration path and frame lookup path.
## Verification
- `bun run --filter @hyperframes/core test -- src/compiler/timingCompiler.test.ts src/compiler/htmlCompiler.test.ts`
- `bun test packages/producer/src/services/htmlCompiler.test.ts`
- `bun run --filter @hyperframes/engine test -- videoFrameExtractor ffprobe`
- `bun run --filter @hyperframes/core typecheck`
- `bun run --filter @hyperframes/engine typecheck`
- `bun run --filter @hyperframes/producer typecheck`
- `bun run --filter @hyperframes/cli typecheck`
- `bun run lint`
- `bun run format:check ...` on touched files
- Comfy project: `node packages/cli/dist/cli.js validate` -> no console errors, 44 text elements pass WCAG AA
- Comfy project patched render from source: `/tmp/comfy-render-compare/fixed6-comfy.mp4`, 1920x1080, 30fps, 21.8s, 654 frames
- 3.00s-3.97s render contact sheet: `/tmp/comfy-render-compare/fixed6-window-contact.png`
- targeted fixed render capture at 3.733s: `/tmp/comfy-render-compare/probe-capture-fixed/captured/frame_000112.jpg`
- agent-browser Studio proof screenshot at 3.7s: `/tmp/comfy-render-compare/agent-browser-studio-3_7-fixed.png`
- agent-browser-driven recording of 3s seek pass: `/tmp/comfy-render-compare/agent-browser-wysiwyg-3s-fixed.webm`
Note: `bun run --filter @hyperframes/cli dev -- validate` is blocked in source mode by the existing `contrast-audit.browser.js` default-export loader issue; packaged `node packages/cli/dist/cli.js validate` passes for this project.
## Problem
Template-wrapped sub-compositions could still lose correct parent timing during render in more than one place.
In the validated repros, a host sub-composition starting after the intro (and in one follow-up repro, starting at `20s` after earlier compositions) contained scene-local media inside it. On the broken paths:
- template-wrapped media could be missed during compile and scheduled at raw scene-local time
- already-correct first-pass offsets could be clobbered during `recompileWithResolutions()`
- even after those two fixes, the browser-metadata reconcile step in `executeRenderJob()` could still overwrite a compiled global `end` with a scene-local `data-end` from the inlined DOM, clipping the tail off late-start sub-composition media
## What this fixes
### Template-wrapped media discovery
- `parseVideoElements`, `parseImageElements`, and `parseAudioElements` now unwrap a single top-level `<template>` wrapper before scraping media
- the unwrap helper is DOM-based, not regex-based, so it avoids the CodeQL backtracking warning and only unwraps the exact single-wrapper shape we want
- multiple sibling templates or other top-level content are left untouched instead of being rewritten heuristically
### Offset preservation after duration resolution
- `recompileWithResolutions()` now preserves the first-pass sub-composition media arrays when the already-inlined HTML no longer contains `[data-composition-src]` hosts
- that prevents correctly offset media metadata from being overwritten by scene-local media parsed from the merged DOM
### Browser metadata reconciliation in the compiled time origin
- browser-discovered media can still report scene-local `data-start` / `data-end` from the merged DOM after inlining
- the producer now reprojects browser `end` values into the compiled element's time origin before reconciling them back into `composition.videos` / `composition.audios`
- this prevents late-start sub-composition media from getting truncated back to a scene-local end during the probe phase
### Regression coverage
- adds focused engine tests for the template unwrap helper
- adds producer regression coverage for both the initial compile path and the post-inline `recompileWithResolutions()` path
- adds producer regression coverage for late-start host compositions (`t≈20`) with scene-local media inside them
- adds producer unit coverage for the browser-end reprojection helper used by the reconcile path
## Root cause
There were three distinct renderer failures behind the bug:
### 1. Template contents were invisible to the media scrapers
`parseSubCompositions()` reads raw sub-composition HTML and applies the host offset to discovered media. But the engine media helpers were querying the parsed document directly, and linkedom follows browser semantics here: top-level `<template>` contents live in a `DocumentFragment`, so `querySelectorAll()` never saw those `<video>` / `<audio>` / `<img>` nodes.
That meant template-wrapped sub-compositions could silently produce zero discovered media during the first pass.
### 2. The duration-resolution recompile could clobber already-correct offsets
After the browser resolves composition durations, `recompileWithResolutions()` reparses the already-inlined HTML. By that point the original `[data-composition-src]` hosts are gone, so `parseSubCompositions()` legitimately returns no nested media.
The old code still rebuilt the deduped media arrays from the merged DOM, which let scene-local media parsed from the inlined HTML overwrite the correctly offset first-pass metadata.
### 3. The browser probe reconcile path mixed two timing coordinate systems
`discoverMediaFromBrowser()` reads `data-start` / `data-end` directly from the live DOM after sub-compositions are already inlined. For nested media, those attributes can still be scene-local even though the compiled metadata has already been offset into the parent host timeline.
The old reconcile path compared those values directly and overwrote `existing.end` whenever the numbers differed. For a late-start sub-composition, that could replace a correct global end like `25.5` with a scene-local end like `5.5`, cutting the clip off during render.
## Verification
### Local checks
- `bun test packages/engine/src/utils/htmlTemplate.test.ts`
- `bun test packages/producer/src/services/htmlCompiler.test.ts`
- `bunx vitest run packages/producer/src/services/renderOrchestrator.test.ts`
- `bun run --filter @hyperframes/engine test`
- `bun run --filter @hyperframes/engine typecheck`
- `bun run --filter @hyperframes/producer typecheck`
- `bunx oxlint packages/engine/src/utils/htmlTemplate.ts packages/engine/src/utils/htmlTemplate.test.ts packages/producer/src/services/renderOrchestrator.ts packages/producer/src/services/renderOrchestrator.test.ts packages/producer/src/services/htmlCompiler.test.ts`
- `bunx oxfmt --check packages/engine/src/utils/htmlTemplate.ts packages/engine/src/utils/htmlTemplate.test.ts`
- `bun run build:producer`
### Render / browser verification
Verified against two local repros:
1. **Early offset repro**
- host starts at `2s`
- child media is scene-local `0-4s`
- compiled render summary keeps the child video/audio at `start: 2`
- browser verification via `agent-browser` confirmed the `2.2s` frame still shows the child clip active in the host timeline
2. **Late offset repro**
- earlier compositions run first, then the target host starts at `20s`
- child media starts scene-local at `1.5s` and should remain visible through `24.5s`
- compiled render summary keeps the child video/audio at `start: 21.5`, `end: 25.5`
- browser verification via `agent-browser` confirmed the `24.5s` frame still shows the late clip visible, which is the exact tail-clipping case the old reconcile path could break
## Notes
- the `/tmp/hf-pr475-repro` and `/tmp/hf-pr476-late-offset-repro` projects plus their browser-proof artifacts are verification-only and are not part of this PR
- this PR stays narrowly scoped to sub-composition media timing across compile, recompile, and browser probe reconciliation; it does not broaden into general sub-composition HTML normalization beyond the single-wrapper case
## Summary
Document why the `window.__name` polyfill in `frameCapture.ts` is necessary, expand the inline comment with the full per-runtime matrix, and add a regression test that surfaces transpiler behavior on the next failure.
Outcome of the Chunk 12 investigation: **keep the polyfill**.
## Why
`Chunk 12` of `plans/hdr-followups.md`. The polyfill had a vague comment and no test, so it was unclear whether it was still needed or could be deleted.
## Empirical findings
Probe in `/tmp/hf-name-probe`:
| Runtime / build | Injects `__name(fn, "name")` wrappers in `Function.prototype.toString()`? |
|-----------------|---------------------------------------------------------------------------|
| `bun` (TS loader) | No — verified for top-level and nested named functions / arrow expressions. |
| `tsx` (esbuild loader, `keepNames=true`) | **Yes** for nested named functions / arrows; observed crash mode in dev/test. |
| `tsc` (`noEmit` and emit) | No — does not inject the helper. |
| `tsup` for `@hyperframes/cli` (`noExternal: ["@hyperframes/engine"]`) | Polyfill *definition* is bundled, but `__name(...)` *call sites* are absent in `packages/cli/dist/cli.js` (grepped). |
**Root cause.** `@hyperframes/engine`'s `package.json` exports raw TypeScript (`main`/`exports` → `./src/index.ts`), so every consumer's transpiler decides whether to inject `__name`. Anything that runs through `tsx` (producer parity-harness, ad-hoc dev scripts, `bun run --filter @hyperframes/engine test` via Vitest's loader) will serialize wrapped function bodies into `page.evaluate(...)` and crash with `ReferenceError: __name is not defined`.
**Decision.** Keep the no-op `window.__name` shim. Cost is one `evaluateOnNewDocument` call. The alternative (rewriting every `page.evaluate(fn)` site to `page.addScriptTag({ content: "..." })`, like `packages/cli/src/commands/contrast-audit.browser.js` already does) is far more invasive and easy to regress.
## What changed
- Expanded the inline comment in `packages/engine/src/services/frameCapture.ts` to explain the per-runtime matrix above and point to the script-tag alternative.
- New `packages/engine/src/services/frameCapture-namePolyfill.test.ts` — a pure unit test (matches the rest of the engine package's no-browser-launch convention) that:
1. Asserts the polyfill is wired up via `evaluateOnNewDocument` and runs before the first awaited `browser.version()` call.
2. Probes the active Vitest transpiler for `__name(...)` injection so the next maintainer can see at a glance whether the upstream behavior has shifted.
## Test plan
- [x] `bun run --filter @hyperframes/engine test` → 408/408 pass (3 new tests in this file).
- [x] `bunx tsc --noEmit -p packages/engine` clean.
- [x] `bunx oxlint` and `bunx oxfmt --check` clean on edited files.
## Stack
Chunk 12 of `plans/hdr-followups.md`. Independent of all other chunks; closes out the investigation item.
## What
Adds a content-addressed cache for extracted video frames, keyed on the tuple `(path, mtime, size, mediaStart, duration, fps, format)`. Repeat renders of the same composition (studio edit → re-render, preview → final) skip the ffmpeg extraction entirely.
## Why
Video frame extraction is the dominant non-capture phase for video-heavy compositions. Studio iteration workflows extract the same frames over and over — each render burns ffmpeg time that adds no value.
Validated on `/tmp/hf-fixtures/cfr-sdr-cache`:
```
Cold (miss): extractMs=69, videoExtractMs=70, totalElapsedMs=2052
Warm (hit): extractMs=1, videoExtractMs=2, totalElapsedMs=1964
cacheHits: 0→1, cacheMisses: 1→0
```
The fixture is tiny (3s CFR SDR @ 30fps), so the wall-clock delta is small; the extraction-time delta (69→1ms, 98%) scales linearly with source length. For heavy-iteration workflows (a user rendering the same composition while tuning encoding params), extraction time goes to zero on every repeat render.
Depends on #444 (instrumentation surface) and #445 (segment-scope HDR preflight — otherwise cache keys would be unstable across renders on mixed-HDR compositions).
## How
- New `packages/engine/src/services/extractionCache.ts`:
- SHA-256 key over a stable JSON encoding of `(path, mtime_ms, size, mediaStart, duration, fps, format)`. Infinity duration is normalized to `-1` so unresolved natural-duration sources still produce stable keys.
- Truncates to 16 hex chars in the entry directory name — 64 bits of entropy is plenty at cache scale and keeps `ls` output short.
- `hfcache-v2-` schema prefix — bumping it invalidates old entries (callers own gc policy; the cache owns keys).
- `.hf-complete` dotfile sentinel. An entry dir without the sentinel is treated as a miss (covers crash-mid-extract and abandoned writes); the next render re-extracts over the partial frames with `-y`.
- `FRAME_FILENAME_PREFIX = "frame_"` shared with the extractor — future refactors only need to touch one place to rename frames.
- `EngineConfig.extractCacheDir` (env: `HYPERFRAMES_EXTRACT_CACHE_DIR`) gates the feature. Undefined disables caching — extraction runs into the render's workDir and cleanup removes it on render end, preserving the prior behaviour exactly. No default root is chosen by the engine; the caller (CLI, app, studio) owns the location policy.
- `ExtractedFrames.ownedByLookup` flag prevents `FrameLookupTable.cleanup` from rm'ing a shared cache dir at render end. Set to `true` on both hits and misses (misses own the directory they wrote into, but hand it over to the cache rather than deleting it).
- Phase 3 extractor flow:
1. Snapshot `(videoPath, mediaStart, start, end)` per resolved video BEFORE Phase 2a/2b preflight mutates them — so cache keys are stable across renders that use workDir-local normalized files (those files have fresh mtimes every render).
2. Compute key, `lookupCacheEntry`.
3. On hit: rebuild `ExtractedFrames` from the cache dir plus the Phase 2-probed `VideoMetadata` — no re-ffprobe.
4. On miss: `ensureCacheEntryDir`, extract with `extractVideoFramesRange(..., outputDirOverride)`, then `markCacheEntryComplete` (the sentinel write is the last step so a crash leaves the dir un-sentineled).
- `extractVideoFramesRange` gains an `outputDirOverride` parameter so cache-miss writes land directly in the keyed dir (no `join(outputDir, videoId)` wrapping).
## Test plan
- [x] 19 unit tests in `extractionCache.test.ts` covering key determinism, mtime/size invalidation, format/fps/mediaStart/duration invalidation, Infinity normalization, sentinel semantics, missing-file tolerance
- [x] 2 integration tests in `videoFrameExtractor.test.ts`:
- "reuses extracted frames on a warm cache hit" — asserts `cacheHits=1`, `extractMs<50ms` on second call against a CFR SDR fixture
- "invalidates the cache when fps changes" — different fps on second call forces a new miss
- [x] End-to-end validation with `HYPERFRAMES_EXTRACT_CACHE_DIR` set, two runs of the same fixture
- [x] Lint + format (oxlint + oxfmt)
- [x] Typecheck (engine + producer)
## What
Scopes the SDR→HDR preflight re-encode to the segment the composition actually uses, mirroring the existing VFR→CFR segment-scope fix.
## Why
`convertSdrToHdr` was re-encoding entire source files, so a 30-minute SDR screen recording contributing a 2-second clip in a mixed HDR/SDR composition ate multi-second preflight time that produced frames no one would ever read. Validated on a mixed 30s-SDR + 2s-HDR fixture: `hdrPreflightMs` drops **87%** (1162→148ms), `videoExtractMs` drops **82%** (1272→231ms), `tmpPeakBytes` drops **45%** (8.2MB→4.5MB).
Depends on #444 (phase-level instrumentation) for the measurement surface.
## How
- `convertSdrToHdr` gains `startTime` and `duration` parameters ahead of the upstream `targetTransfer` arg added by #370. New signature: `convertSdrToHdr(input, output, startTime, duration, targetTransfer, signal, config)`. `-ss $start -t $duration` is added to the ffmpeg args.
- Phase 2 now captures the full `VideoMetadata` per `resolvedVideos` entry (previously just `colorSpace`) so the caller can compute `segDuration` from `video.end - video.start` with a fallback to `metadata.durationSeconds - video.mediaStart` for unbounded (Infinity) clips — without firing another ffprobe.
- After a successful convert, `entry.video.mediaStart` is zeroed out via shallow-copy (doesn't mutate the caller's `VideoElement`) so downstream extraction seeks from 0 instead of the original offset. Mirrors what the VFR→CFR path already does.
## Test plan
Validation on `/tmp/hf-fixtures/hdr-sdr-mixed-scope`:
```
hdrPreflightMs: >1000 → 150 (gate: <300) ✓
videoExtractMs: 1272 → 237 (-82%)
tmpPeakBytes: 8.2MB → 4.5MB (-45%)
```
- [x] Unit test: new regression test synthesizes 10s SDR + 2s HDR fixture inline and asserts the converted file's duration matches the 2s used segment (pre-fix matched the 10s source)
- [x] Lint + format
- [x] Typecheck
- [x] Manual perf validation against synthesized fixture
## What
Adds per-phase timings and counters to `extractAllVideoFrames` and surfaces them on the producer's `RenderPerfSummary` as `videoExtractBreakdown` alongside a new `tmpPeakBytes` workDir size sample.
## Why
Phase 2 video extraction has five distinct sub-phases (resolve, HDR probe, HDR preflight, VFR probe, VFR preflight, per-video extract) and today they collapse into a single `videoExtractMs` stage timing. That makes every subsequent perf PR in this stack immeasurable — you can't tell whether a win came from cache hits, preflight scope reduction, or pure extraction speed.
This PR is foundational for PR #445 (segment-scope HDR preflight) and PR #446 (content-addressed extraction cache).
## How
- New `ExtractionPhaseBreakdown` type with `resolveMs`, `hdrProbeMs`, `hdrPreflightMs/Count`, `vfrProbeMs`, `vfrPreflightMs/Count`, `extractMs`, `cacheHits`, `cacheMisses`. Populated inline with `Date.now()` wrappers — overhead is sub-millisecond on every phase.
- Returned on `ExtractionResult.phaseBreakdown`.
- Producer extends `RenderPerfSummary` with `videoExtractBreakdown?: ExtractionPhaseBreakdown` and `tmpPeakBytes?: number`. `tmpPeakBytes` is sampled from the workDir right before cleanup via a new recursive-size helper that swallows errors (purely observational — a missing workDir must never fail the render).
No changes to the capture-lifecycle resource tracking — earlier versions of this instrumentation plumbed injector LRU stats through `RenderOrchestrator`, which conflicted hard with upstream #371 (`buildHdrCaptureOptions` refactor). Dropped that piece for a marginal observability loss.
## Test plan
Validation on `packages/producer/tests/vfr-screen-recording`:
```json
"videoExtractBreakdown": {
"resolveMs": 0, "hdrProbeMs": 0, "hdrPreflightMs": 0, "hdrPreflightCount": 0,
"vfrProbeMs": 0, "vfrPreflightMs": 166, "vfrPreflightCount": 1,
"extractMs": 97, "cacheHits": 0, "cacheMisses": 0
},
"tmpPeakBytes": 4578598
```
Total elapsed within noise of pre-PR baseline (2665 → 2673 → 3228ms across hosts).
- [x] Unit test: phase-breakdown assertion added to `videoFrameExtractor.test.ts`
- [x] Lint + format (oxlint + oxfmt)
- [x] Typecheck (engine + producer)
- [x] Manual perf validation against VFR fixture
## Summary
Extract the `frameDirMaxIndexCache` from a private module-scoped Map inside `renderOrchestrator.ts` into its own `frameDirCache.ts` module, then add a 11-test bun:test suite that pins the cross-job isolation contract added in Chunk 5B.
## Why
`Chunk 9E` of `plans/hdr-followups.md`. The cache lived as a private Map inside `renderOrchestrator.ts`, which made the cross-job isolation contract from Chunk 5B impossible to unit-test directly. Extracting it both makes the contract testable and reduces orchestrator complexity slightly.
## What changed
- New `packages/producer/src/services/frameDirCache.ts` exposes `getMaxFrameIndex` / `clearMaxFrameIndex` / `getMaxFrameIndexCacheSize` (plus a test-only `__resetMaxFrameIndexCacheForTests` helper). Behavior is unchanged: callers still get the same module-scoped sharing inside a job, and `renderOrchestrator`'s outer `finally` still clears every entry it registered so the cache cannot grow monotonically across renders.
- `renderOrchestrator.ts`: imports the new helpers, drops the unused `readdirSync` import, updates inline comments, and replaces two `frameDirMaxIndexCache.delete` sites with `clearMaxFrameIndex`.
- New `frameDirCache.test.ts` (bun:test, 11 tests) covering:
- Reading the max index from a populated directory.
- Ignoring filenames that don't match `frame_NNNN.png` (wrong ext, wrong prefix, wrong case, double extension, empty index group, same-named subdirectory).
- Empty- and missing-directory paths returning `0` and being cached.
- Intra-job invariant: subsequent readdir mutations not observed once cached.
- `clearMaxFrameIndex` forcing a re-read; returns `false` for paths that were never cached.
- Per-directory isolation when multiple directories are registered.
- The cross-job contract from Chunk 5B: cache empty between well-behaved jobs, doesn't grow monotonically across 20 simulated renders with 3 HDR videos each (steady-state cache size stays at 3), and a buggy job that forgets to clear leaks exactly its own entries rather than affecting unrelated jobs.
## Test plan
- [x] `frameDirCache.test.ts` 11/11 pass.
- [x] Existing producer tests unchanged.
- [x] Behavior preserved: same module-scoped sharing inside a job, same outer-`finally` eviction.
## Stack
Chunk 9E of `plans/hdr-followups.md`. Test-driven extraction; complements Chunk 5B.
## Summary
Add unit tests that mock `child_process.spawn` to drive an in-memory "ffmpeg" through the success/failure paths used by the producer's HDR encoder and by Chunk 5A's defensive `close()` in `renderOrchestrator`.
## Why
`Chunk 9D` of `plans/hdr-followups.md`. The contracts the orchestrator's try/finally cleanup (Chunk 5A) and the abort path rely on were entirely uncovered. A regression in `spawnStreamingEncoder`'s lifecycle handling would surface as a leaked ffmpeg process or a hung render, both of which are hard to diagnose after the fact.
## What changed
`packages/engine/src/services/streamingEncoder.test.ts`: 7 new specs covering:
- Successful exit after explicit `close()`.
- Non-zero exit before `close()` returns a failure result (no throw).
- `ENOENT` on spawn returns a failure result (no throw).
- Abort signal triggers `SIGTERM` and a `"cancelled"` result.
- `close()` is idempotent and never throws on a second call.
- `writeFrame` returns `false` after the encoder has exited.
- `close()` detaches the abort listener so post-close aborts don't re-kill ffmpeg.
These contracts are what the `renderOrchestrator` try/finally cleanup added in Chunk 5A relies on, and what the ffprobe-unavailable test (Chunk 9B) hinted at for the encoder side.
## Test plan
- [x] All new specs pass.
- [x] No production code changes — pure regression coverage of existing lifecycle behavior.
## Stack
Chunk 9D of `plans/hdr-followups.md`. Test-only change, complements Chunk 5A.
Guard buildHyperframesRuntimeScript() against missing entry.ts so it
returns null instead of crashing with esbuild stderr output. Add
getHyperframeRuntimeScript() that returns the pre-built IIFE as a
baked-in string constant — no esbuild, no file I/O, no import.meta.url.
Consolidate CLI runtime source resolution into a single module with
a clear priority chain: esbuild from source (dev) → inlined constant
(production) → pre-built artifact file (fallback).
Add CI smoke test that npm-packs the CLI, installs globally, runs
hyperframes preview, and asserts no stderr errors + runtime endpoint
returns JS.
Bump version to 0.4.16.
* fix(cli): resolve runtime fallback for globally-installed hyperframes
When hyperframes is installed globally via npm, the `loadRuntimeSourceFallback()`
path that dynamically imports @hyperframes/core and runs esbuild fails because
@hyperframes/core is inlined into cli.js and import.meta.url resolves to the
wrong location for the entry.ts source file.
Add a disk-based fallback that searches for the pre-built IIFE runtime artifact
in multiple locations:
- Alongside the bundled CLI (dist/hyperframe-runtime.js, dist/hyperframe.runtime.iife.js)
- Walking up from __dirname through node_modules
The esbuild path is tried first to preserve live-rebuild behavior in dev,
with the pre-built artifact search as a safety net for the bundled context.
Also adds the IIFE artifact name variant to resolveRuntimePath() in the
studio server so it checks both naming conventions.
* fix(cli): gate esbuild fallback on source availability
The previous fix still triggered esbuild's stderr output before the
catch could suppress it. Now check whether the runtime entry.ts source
file actually exists before attempting the on-the-fly build, avoiding
the noisy error in global installs entirely.
* fix(cli): remove noisy console.warn from runtime fallback
The caller already handles a null return — no need to warn about
something the user can't act on. If both paths fail, the /api/runtime.js
route returns a 404 which the studio handles gracefully.
* style(engine): fix oxfmt trailing blank line in chunkEncoder test
* fix(cli): guard against null/undefined from loadHyperframeRuntimeSource
Fall through to the pre-built artifact if the function returns a
falsy value without throwing.
* refactor(cli): consolidate runtime source resolution into single module
Replace the scattered path-probing logic with a single loadRuntimeSource()
that encodes the full priority chain: esbuild from source (dev only,
gated on entry.ts existence) → pre-built artifact alongside cli.js →
core/dist artifact → node_modules walk.
Rename loadRuntimeSourceFallback → loadRuntimeSource since it's now
the primary resolution function, not a fallback.
## Summary
Mock `node:child_process.spawn` to surface `ENOENT` and verify ffprobe's three callers behave correctly when ffprobe is missing.
## Why
`Chunk 9B` of `plans/hdr-followups.md`. The PNG cICP fallback in `extractMediaMetadata` was added to support environments without ffprobe, but no test pinned the behavior — silently regressing it would break HDR image support on any system without ffprobe installed.
## What changed
`packages/engine/src/utils/ffprobe.test.ts`: mocks `child_process.spawn` to surface `ENOENT` and asserts:
- `extractMediaMetadata` falls back to PNG cICP metadata for image inputs.
- `extractMediaMetadata` rethrows for non-image inputs lacking a still-image fallback.
- `extractAudioMetadata` + `analyzeKeyframeIntervals` propagate the install-hint error verbatim.
## Test plan
- [x] All new tests pass.
- [x] No production code changes — pure regression coverage of existing fallback behavior.
## Stack
Chunk 9B of `plans/hdr-followups.md`. Test-only change, independent of all other chunks.
## Summary
Add four midpoint (`p=0.5`) regression invariants applied via a `describe` loop over `ALL_SHADERS`, so every existing and future shader transition automatically gets coverage at the most viewer-visible point in the animation.
## Why
`Chunk 9G` of `plans/hdr-followups.md`. Existing smoke tests cover only the endpoints (`p=0 ≈ from`, `p=1 ≈ to`), which miss a class of regressions that surface specifically at the midpoint and let shaders silently rot in CI:
- A shader becomes a no-op (returns input as-is)
- A shader prematurely completes (returns target at midpoint)
- A shader doesn't write to the output buffer at all
- A shader loses determinism (`Math.random` / `Date.now` / leaked state)
## What changed
`packages/engine/src/utils/shaderTransitions.test.ts`: a single `describe` loop over `ALL_SHADERS` that asserts at `p=0.5`:
1. `output ≠ from` — catches no-ops
2. `output ≠ to` — catches premature completion
3. `output` is non-zero — catches blank output
4. `output` is deterministic — catches accidental non-determinism
Uses two distinct uniform input colors (40000/30000/20000 vs 10000/10000/10000) so equality checks have distinct byte patterns to compare against. Even shaders that warp UVs (which would be no-ops on uniform input alone) produce `mix16(from, to, 0.5)` at every pixel, distinct from both inputs.
## Test plan
- [x] 60 new tests (4 invariants × 15 shaders), all passing.
- [x] Any new transition added to the registry automatically picks up the same coverage.
## Stack
Chunk 9G of `plans/hdr-followups.md`. Test-only change, independent of all other chunks.
## Summary
Add a 12-row reference table covering the full sRGB range with byte-exact 16-bit HLG and PQ signal values, plus three guard tests, locking down the `buildSrgbToHdrLut()` math.
## Why
`Chunk 9F` of `plans/hdr-followups.md`. The matrix-free fast path through `blitRgba8OverRgb48le` runs every DOM pixel through `buildSrgbToHdrLut()` (sRGB EOTF → linear → HDR OETF → 16-bit). Any drift in the EOTF/OETF math — constant changes, branch swaps, rounding-mode regressions — would silently corrupt every text / UI / overlay pixel composited onto an HDR frame.
Existing tests covered structural invariants (transparent passthrough, opaque overwrite, alpha blending, channel symmetry, HLG ≠ PQ) but no byte-exact reference values, so a uniform scale or constant tweak could pass everything.
## What changed
- 12-row reference table in `alphaBlit.test.ts` covering black, shadow, mid-grays, highlight, near-white, and white with exact 16-bit HLG and PQ signal values.
- Three guard tests:
- **Asymmetric R/G/B (HLG):** each channel hits the LUT independently.
- **Asymmetric R/G/B (PQ):** same, on the PQ path.
- **BT.2408 SDR-white invariant:** PQ caps sRGB 255 at 38055 (~203 nits), well below HLG's 65535. This is the load-bearing detail that makes PQ headroom work — locking the exact value prevents a future "fix" that would re-scale PQ to peak-at-SDR-white and clip every real HDR pixel.
Reference values mirror `buildSrgbToHdrLut()` exactly and were verified against the existing HLG mid-gray comment in the file.
## Test plan
- [x] All new tests pass against the current LUT.
- [x] Existing `alphaBlit.test.ts` invariants unchanged.
## Stack
Chunk 9F of `plans/hdr-followups.md`. Test-only change, independent of all other chunks.
## Summary
Track `tests/*/src/*.png` via Git LFS to mirror the existing policy for golden videos and `.mp4` fixtures.
## Why
`Chunk 11C` of `plans/hdr-followups.md`. Without this rule, regression suites that grow PNG fixtures over time would bloat the working-tree history and slow shallow clones.
## What changed
- `.gitattributes`: add `tests/*/src/*.png` to the LFS-tracked patterns.
- Migrates the six existing PNG fixtures (1.6 MB combined: `hdr-photo-pq.png` plus `heygen-promo-preview-assets/` screenshots) onto LFS in the same commit so the rule applies retroactively.
## Test plan
- [x] `git lfs ls-files` includes the HDR PNG fixtures after commit.
- [x] Working tree size for these files goes from 1.6 MB to 6 × ~130 B LFS pointers.
## Stack
Chunk 11C of `plans/hdr-followups.md`. Independent of all code changes.
hevc_nvenc uses the same p1..p7 preset vocabulary as h264_nvenc, so the
mapping in `mapPresetForGpuEncoder` applies to both codecs. The initial
regression suite only covered `codec: "h264"`, which left a gap: a
future refactor that split the H.264 and H.265 NVENC paths could
silently regress one codec without any test catching it.
Add three-case loops (ultrafast → p1, medium → p4, veryslow → p7) under
`codec: "h265"` to both `buildEncoderArgs` and `buildStreamingArgs`
test blocks. Each case also asserts that `-c:v hevc_nvenc` is selected
so the test fails loudly if the codec plumbing is broken, not just the
preset translation.
Follow-up to #442 per review comment from @jrusso1020.
Co-authored-by: roi32 <75878108+roi32@users.noreply.github.com>
NVENC rejects the libx264 preset vocabulary (ultrafast / medium / slow /
...) with AVERROR(EINVAL) ("Error applying encoder options: Invalid
argument"), which surfaces as a bare `FFmpeg exited with code -22` from
spawn(). Because ENCODER_PRESETS passes these names straight through to
h264_nvenc / hevc_nvenc, every `--gpu` render using the `draft` tier
failed; `standard` (medium) and `high` (slow) only worked coincidentally
on ffmpeg builds that happened to accept those aliases. QSV has the same
problem on a narrower set (ultrafast / superfast / placebo).
Add `mapPresetForGpuEncoder` in utils/gpuEncoder.ts that translates the
libx264 vocabulary to each encoder's native names:
- nvenc: libx264 -> p1..p7 (already-native pN values pass through);
unknown values fall back to p4 (medium)
- qsv: ultrafast / superfast -> veryfast; placebo -> veryslow;
everything else passes through
- videotoolbox / vaapi / null: unchanged
Both buildEncoderArgs (chunkEncoder.ts) and buildStreamingArgs
(streamingEncoder.ts) now route through the helper before pushing
`-preset` to the ffmpeg arg vector.
To make the next encoder-options failure diagnosable without re-running
ffmpeg by hand, \`formatFfmpegError\` in utils/runFfmpeg.ts now appends
the last 15 non-empty stderr lines to the error string. The four call
sites that previously swallowed stderr (encodeFramesFromDir,
muxVideoWithAudio, applyFaststart, and the streaming encoder exit
handler) have been updated.
Tested end-to-end on an RTX 4080 with ffmpeg 8.1 NVENC across
\`--quality draft|standard|high\` plus \`--video-bitrate\` and \`--crf\`
overrides; the 6 renders were visually equivalent to the CPU baseline.
Co-authored-by: roi32 <75878108+roi32@users.noreply.github.com>
## Summary
Two correctness fixes in the HDR transform & clipping pipeline: `parseTransformMatrix` now handles `matrix3d(...)` (GSAP's default `force3D: true`), and shader-transitions sets every non-first scene to `opacity: 0` at `t=0` so the engine doesn't over-composite at the start.
## Why
`Chunk 4` of `plans/hdr-followups.md`. Transform extraction and border-radius computation existed but were dead — an HDR video with `rotation: 45` rendered un-rotated, and 3-scene compositions ghosted at `t=0` because every scene defaulted to CSS `opacity: 1` and contributed to the first frame.
## What changed
**Matrix3d support in `parseTransformMatrix`.** `DOMMatrix.toString()` emits `matrix3d` whenever any ancestor in the chain has used a 3D transform — most importantly GSAP's default `force3D: true`, which converts `translate(...)` into `translate3d(..., 0)`. Without this, every GSAP-driven transform was silently dropped during HDR compositing because `videoFrameInjector.getViewportMatrix()` would return `matrix3d(...)` and the blit path would parse it as `null` and fall back to identity. The 16-value column-major form is converted to its 2D affine projection (indices 0, 1, 4, 5, 12, 13 → m11, m12, m21, m22, m41, m42); Z, perspective, and out-of-plane rotation components are dropped.
**Initial-state opacity in `initEngineMode`.** The browser preview branch uses a GL canvas overlay during transitions, so scene opacity at `t=0` doesn't matter visually. The engine branch reads scene opacity directly via `queryElementStacking()` to decide which layers to composite. Without an explicit initial-state tween, every scene defaulted to CSS `opacity: 1` and contributed to the very first frame, causing ghosting/overlap until the first transition fired. `tl.set()` at position 0 anchors the initial state in the timeline graph so reverse seeks from inside a later transition restore it correctly.
These two fixes together make `el.transform` and `el.borderRadius` (already wired in Chunk 7A's `compositeHdrFrame`) actually flow through the GSAP-animated case, and keep the engine's per-frame compositing aligned with what the user sees in browser preview.
## Test plan
- [x] 6 new `alphaBlit.test.ts` cases (identity matrix3d, translate3d, scale + translate3d, rotateZ, malformed arg count, non-finite values).
- [x] Existing `hdr-regression` Window H already CSS-sets `#scene-b { opacity: 0 }` as a fallback; the new `tl.set` is redundant for that case but harmless and removes the need for compositions to remember the CSS workaround.
- [x] Manual: rotated HDR video (`rotation: 45`) appears rotated; `border-radius: 50%` clips to circle; 3-scene composition has no overlap at `t=0`.
## Stack
Chunk 4 of `plans/hdr-followups.md`. Window F of the regression suite documents the bug; the next PR in the stack tightens the `maxFrameFailures` budget to 0.
## Summary
Four behavior-preserving refactors that reduce complexity in `renderOrchestrator.ts` and clarify the engine ffprobe utility surface. Lands after the correctness fixes (Chunks 1–5) so the refactored code is already correct.
## Why
`Chunk 7` of `plans/hdr-followups.md`. The HDR composite block had grown a ~200 LOC inline closure with 14 captured deps, a repeated capture-options spread, a `extractVideoMetadata` name that now also handles still images, and per-frame re-creation of debug helpers.
## What changed
**7A — Hoist `compositeToBuffer` into a module-scoped helper.** Extract the inline HDR closure into a top-level `compositeHdrFrame()` that takes an `HdrCompositeContext` struct. Construct the context once at the top of the HDR render block and pass it through. Removes a deeply-nested closure from the middle of the orchestrator.
**7B — `buildHdrCaptureOptions()` helper.** Factor the repeated `{ ...captureOptions, skipReadinessVideoIds: ... }` spread into a named helper at the call site.
**7C — Rename `extractVideoMetadata` → `extractMediaMetadata`.** Reflects that the helper handles still images (PNG/JPEG/WebP) in addition to video. Update all callers in engine + producer (`videoFrameExtractor`, `htmlCompiler`, regression-harness, producer ffprobe re-export, tests). Re-export the old name as a deprecated alias from `@hyperframes/engine` for backward compatibility, plus the producer re-export shim.
**7D — Hoist debug counters to module scope.** `countNonZeroAlpha` and `countNonZeroRgb48` are now module-scoped so they aren't re-created per frame and so the closure has fewer captures.
Also touches the `hdr-regression` and `hdr-hlg-regression` README + `meta.json` files reviewed during this refactor.
## Test plan
- [x] `bunx tsc --noEmit -p packages/producer && bunx tsc --noEmit -p packages/engine` clean.
- [x] Engine tests: 313 pass, 0 fail (1218 expect calls).
- [x] `bunx oxlint` + `bunx oxfmt --check` clean on 8 changed source files.
- [x] Diff is structural only — no behavioral changes.
## Stack
Chunk 7 of `plans/hdr-followups.md`. Lands after the correctness fixes (Chunks 1–5) per the suggested merge order.
## What
Replace the `choco install ffmpeg` step in `windows-render.yml` with a direct download of the upstream Windows GPL build from [`BtbN/FFmpeg-Builds`](https://github.com/BtbN/FFmpeg-Builds/releases/latest) on GitHub Releases.
## Why
The `Render on windows-latest` canary started failing on every PR with:
```
[NuGet] Response status code does not indicate success: 504 (Gateway Timeout).
[NuGet] Response status code does not indicate success: 503 (Service Unavailable).
```
The Chocolatey community feed (`community.chocolatey.org/api/v2/package/ffmpeg/8.1.0`) is degraded for the `ffmpeg` package right now. The earlier 3-attempt retry I added wasn't enough — every attempt across multiple runs failed with 503/504, so retrying does nothing.
The Chocolatey path is also a bit indirect for what this job actually validates. The real point of the canary is the [PR #336](https://github.com/heygen-com/hyperframes/pull/336) fix where `findFFmpeg()` / `where ffmpeg` discovery has to work on a fresh Windows runner. As long as `ffmpeg.exe` ends up on `PATH`, the underlying thing under test (the harness can find ffmpeg, capture frames, mux to MP4) is exercised exactly the same.
BtbN/FFmpeg-Builds is the canonical upstream nightly Windows GPL build (Chocolatey itself rebundles essentially the same artifact), so this is closer to the source, not further from it.
## How
- Download `ffmpeg-master-latest-win64-gpl.zip` from the BtbN release with `Invoke-WebRequest` (3-attempt retry with backoff).
- Extract to `$env:RUNNER_TEMP/ffmpeg` and locate `ffmpeg.exe` recursively.
- Add the bin directory to `$env:GITHUB_PATH` so all subsequent steps in the job (the Bun-driven harness, `findFFmpeg()`, etc.) see ffmpeg on `PATH` exactly the same way as before.
- Print `ffmpeg -version` as a sanity check.
## Test plan
- [ ] CI: `Render on windows-latest` job goes green on this PR.
- [ ] Subsequent PRs no longer get blocked on `choco install ffmpeg` 503s.
Miguel Ángel