Commit Graph
11 Commits
Author SHA1 Message Date
andybpanandClaude Opus 5.5 758ad7f585 fix(hdr): bound HDR image decode and resample sizes (#7917)
* fix(hdr): bound HDR image decode and resample sizes

The native HDR image path sized its host buffers from the PNG header and
the element's page-measured layout box, with no ceiling.

- PNG decode checks IHDR dimensions and bit depth before allocating,
  inflates with maxOutputLength = height * (stride + 1), and rejects image
  data whose inflated length does not match the header.
- HDR images are limited to 8192x8192 pixels; resampleRgb48leObjectFit
  refuses targets above that.
- A layout box larger than the output frame is shrunk uniformly to fit it.
- Decoded and resampled HDR image buffers count against the HDR extraction
  budget (HDR_EXTRACTION_MAX_BYTES / cgroup limit) for the render's lifetime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(engine): raise the packed size budget to 510000 bytes

The PNG header checks and bounded inflate add about 500 bytes to the
packed engine, which was already within a few hundred bytes of its
1% tolerance over the 500000-byte budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 03:15:55 +00:00
Miguel Ángel d796846d83 fix(engine): SDR content in HLG exports sits at reference white, not peak (#5749)
* fix(engine): SDR content in HLG exports sits at reference white, not peak

The sRGB to HLG conversion for DOM layers fed SDR-relative light straight into the HLG OETF, whose input is light relative to the HLG peak, so SDR white landed at 100% signal. Graded HLG footage (graded as SDR, then converted back) and every white overlay were pushed to peak and blew out. SDR white now sits at 75% HLG signal (BT.2408), the same model the PQ branch uses with 203 nits.

* docs(engine): shorten the SDR-white note in the HDR LUT

* docs(engine): cut the LUT generator's notes to its usage and drift rule
2026-10-19 14:30:48 +00:00
Abdullah M. Nazif 95d5494baa fix(engine): preserve out-of-range HDR object positions (#5266) 2026-10-08 20:00:37 +00:00
miga-heygenandMiguel Ángel 0a1c6f5a45 feat(engine): sub-frame multi-sample motion blur, capture-level accumulation (#4019)
Deliverable one of issue #4010: average K sub-frame captures into each output
frame so rotation, scale and opacity blur the way After Effects does, which the
translation-only catalog component (#4008) cannot do. Opt in per render with
CaptureOptions.motionBlur; absent means the capture path is unchanged.

The blocker was quantizeTimeToFrame flooring every seek onto the output frame
grid, which collapsed all K samples onto one instant and made the averaged frame
byte-identical to the unblurred one. quantizeSeekTime now owns that decision at
all three quantize sites (runtime player, its no-timeline fallback, and the init
transport) and refines the grid by an integer subdivision count, which can only
contain the output grid, never move a frame-time instant off it. The finer grid
rounds rather than floors: a sample time round-trips through the page bridge as
a float, and past a few million ticks that error exceeds the floor epsilon.

Accumulation sits inside captureFrameCore AFTER the static-dedup early return,
so a static frame still costs one reused buffer and only moving frames pay K. It
is not a before-capture hook, which would have disabled dedup outright.

Callback invariant: one eventful seek per output frame at the frame time, then K
suppressed sample seeks, then a suppressed restore to the frame time. The
restore matters because a suppressed seek still moves the playhead, so without
it the next frame's eventful seek would swallow the callbacks inside the shutter
window.

Scope of this deliverable: screenshot capture mode and PNG output. beginframe,
drawelement and JPEG encoding are rejected with a named error rather than
silently rendering unblurred. Video content is rejected for the same reason it
is a documented limitation: injected frames are extracted per output frame and
cannot follow a sub-frame seek.

Blend defaults to sRGB, matching AE's non-linearised 8-bpc working space and
the catalog component. Linear light is an explicit opt-in.

Samples are folded into a running average as they are captured rather than held
until the end, so peak memory is one accumulator plus one decoded sample instead
of all K: at 1080p that is a constant 41 MB rather than 141 MB at 16 samples and
540 MB at the 64 maximum, on every frame of the render.

PNG chunk CRC32 now has one owner in utils/crc32.ts, shared by the new encoder
and ffprobe's metadata reader. The encoder previously imported zlib's crc32 by
name, which throws at module evaluation on Node 22.0 and 22.1; ffprobe already
carried the guarded version and the comment explaining why. The sRGB transfer
function is likewise exported once and shared.

Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
2026-09-16 20:10:06 -04:00
James Russo e5d1d9bf0e fix(engine): avoid transform regex backtracking (#3704) 2026-09-05 11:53:43 -04:00
Vance Ingalls b0fb664873 fix: render shader transitions for SDR compositions (#640)
* feat: cache shader transition preview frames

* fix: move shader transition loading to player

* fix: render shader transitions for sdr compositions
2026-05-06 01:29:44 -07:00
Vance Ingalls ad44c3133a perf(hdr): reduce layered composite overhead (#538) 2026-04-28 15:30:35 -07:00
Vance Ingalls 2e1a1d91a2 fix(engine,shader): handle matrix3d transforms and hide non-first scenes (#374)
## Summary

Two correctness fixes in the HDR transform & clipping pipeline: `parseTransformMatrix` now handles `matrix3d(...)` (GSAP's default `force3D: true`), and shader-transitions sets every non-first scene to `opacity: 0` at `t=0` so the engine doesn't over-composite at the start.

## Why

`Chunk 4` of `plans/hdr-followups.md`. Transform extraction and border-radius computation existed but were dead — an HDR video with `rotation: 45` rendered un-rotated, and 3-scene compositions ghosted at `t=0` because every scene defaulted to CSS `opacity: 1` and contributed to the first frame.

## What changed

**Matrix3d support in `parseTransformMatrix`.** `DOMMatrix.toString()` emits `matrix3d` whenever any ancestor in the chain has used a 3D transform — most importantly GSAP's default `force3D: true`, which converts `translate(...)` into `translate3d(..., 0)`. Without this, every GSAP-driven transform was silently dropped during HDR compositing because `videoFrameInjector.getViewportMatrix()` would return `matrix3d(...)` and the blit path would parse it as `null` and fall back to identity. The 16-value column-major form is converted to its 2D affine projection (indices 0, 1, 4, 5, 12, 13 → m11, m12, m21, m22, m41, m42); Z, perspective, and out-of-plane rotation components are dropped.

**Initial-state opacity in `initEngineMode`.** The browser preview branch uses a GL canvas overlay during transitions, so scene opacity at `t=0` doesn't matter visually. The engine branch reads scene opacity directly via `queryElementStacking()` to decide which layers to composite. Without an explicit initial-state tween, every scene defaulted to CSS `opacity: 1` and contributed to the very first frame, causing ghosting/overlap until the first transition fired. `tl.set()` at position 0 anchors the initial state in the timeline graph so reverse seeks from inside a later transition restore it correctly.

These two fixes together make `el.transform` and `el.borderRadius` (already wired in Chunk 7A's `compositeHdrFrame`) actually flow through the GSAP-animated case, and keep the engine's per-frame compositing aligned with what the user sees in browser preview.

## Test plan

- [x] 6 new `alphaBlit.test.ts` cases (identity matrix3d, translate3d, scale + translate3d, rotateZ, malformed arg count, non-finite values).
- [x] Existing `hdr-regression` Window H already CSS-sets `#scene-b { opacity: 0 }` as a fallback; the new `tl.set` is redundant for that case but harmless and removes the need for compositions to remember the CSS workaround.
- [x] Manual: rotated HDR video (`rotation: 45`) appears rotated; `border-radius: 50%` clips to circle; 3-scene composition has no overlap at `t=0`.

## Stack

Chunk 4 of `plans/hdr-followups.md`. Window F of the regression suite documents the bug; the next PR in the stack tightens the `maxFrameFailures` budget to 0.
2026-04-22 23:55:40 -07:00
Vance Ingalls 5de5df7fbb refactor(types): tighten type safety, dedupe HfTransitionMeta, prune dead LUT export (#366)
## Summary

Four small, mechanical type-safety cleanups across `engine`, `producer`, and `shader-transitions`. Zero behavior change — pure pre-cleanup so the rest of the stack ships against a tighter baseline.

## Why

`Chunk 6` of `plans/hdr-followups.md`. Several non-null assertions and a duplicate interface had accumulated as rebase artifacts and leftover work-in-progress; lands first because it touches files later chunks edit and removes friction during review.

## What changed

- `renderOrchestrator.ts`: replace `layers[layerIdx]!` with a `for (const [layerIdx, layer] of layers.entries())` so both index and element come from the iterator.
- `engine/types.ts`: drop the duplicate `HfTransitionMeta` interface (rebase artifact); the original definition above it is the documented one. The orphaned doc comment now precedes `HfProtocol`.
- `shader-transitions/hyper-shader.ts`: keep the local `HfTransitionMeta` declaration (the package ships as a standalone CDN bundle and must not depend on `@hyperframes/engine`), but add a sync comment pointing at the source of truth in `engine/src/types.ts`.
- `alphaBlit.ts` + `engine/index.ts`: drop `export` from `getSrgbToHdrLut` and remove its re-export. It was only ever called by the internal `blitRgba8OverRgb48le`; the public surface was dead code.

## Test plan

- [x] `bun run --filter @hyperframes/engine typecheck`
- [x] `bun run --filter @hyperframes/producer typecheck`
- [x] `bun run --filter @hyperframes/shader-transitions typecheck`
- [x] `bun run --filter @hyperframes/engine test` — 308/308 pass (no test changes; assertions removed in code only).

## Stack

Chunk 6 of `plans/hdr-followups.md`. Mechanical cleanup landed early per the suggested merge order.
2026-04-22 16:56:17 -07:00
Vance Ingalls 00af29c169 fix(cli): forward --hdr through Docker render + HDR docs (#346)
## Summary

This PR ended up covering the full HDR Docker/docs follow-through plus the producer/engine work needed to make HDR still images render and regress correctly in CI.

The branch now does four things:

- forwards `--hdr` through the Docker render path in the CLI
- adds and expands HDR documentation across the docs site
- adds first-class HDR still-image support to the engine/producer pipeline
- adds targeted HDR regression coverage, including a CI-safe fallback for PNG HDR metadata detection when `ffprobe` does not expose PNG color tags

## What changed

### CLI and docs

- `hyperframes render --docker --hdr` now preserves `--hdr` when invoking the in-container CLI
- added a dedicated HDR guide and linked it from CLI, producer, engine, rendering, and common-mistakes docs
- documented HDR constraints and verification flow: HDR source requirements, MP4/H.265 Main10 output, PQ/HLG handling, Docker usage, and common SDR fallback causes

### Engine and producer HDR image support

- added `ImageElement` support to the engine composition model and parsing path
- threaded image elements through producer compilation and orchestration
- probed image sources for HDR color spaces so image-only compositions can trigger HDR output without requiring an HDR video source
- included HDR image start times in stacking queries so the layered compositor can place images correctly in z-order
- integrated HDR image compositing into the layered HDR render loop alongside native HDR video layers and SDR DOM overlays
- forced screenshot mode for HDR layered compositing where required to keep DOM/HDR layer composition deterministic
- skipped readiness waiting for natively extracted HDR videos in the engine path where it was unnecessary and could block layered HDR flows

### HDR metadata robustness

- added a fallback in `extractVideoMetadata()` to read PNG `cICP` metadata directly when `ffprobe` omits color-space fields for PNGs
- this specifically fixes CI/Docker detection for the `hdr-image-only` fixture, where the render was falling back to SDR because the PNG was not being recognized as BT.2020 PQ

### Regression coverage and fixture cleanup

- added `hdr-image-only`, a regression fixture that validates HDR still-image rendering end to end
- added `hdr-pq`, a focused HDR PQ regression fixture for the video path
- updated regression CI to run an `hdr` shard with `--sequential hdr-pq hdr-image-only`
- removed the older larger `hdr-regression/*` fixture set in favor of the smaller targeted regressions used by CI
- added the necessary fixture generation/readme material and checked-in golden outputs for the new HDR tests

## Why

The original PR description only covered the CLI flag forwarding and docs work. Since then, the branch also picked up the missing runtime support needed for HDR still images and the regression coverage to keep that path from breaking.

The practical issue this closes is:

- local host runs could pass while CI failed `hdr-image-only`
- the failure was a full-frame visual mismatch caused by SDR fallback, not unstable rendering
- root cause was PNG HDR metadata not being surfaced by `ffprobe` in the CI Docker environment
- parsing the PNG `cICP` chunk directly makes HDR detection deterministic across environments

## Test plan

### Local targeted checks

```bash
bunx oxlint packages/engine/src/utils/ffprobe.ts packages/engine/src/utils/ffprobe.test.ts
bunx oxfmt packages/engine/src/utils/ffprobe.ts packages/engine/src/utils/ffprobe.test.ts
bun --cwd packages/engine test src/utils/ffprobe.test.ts src/utils/hdr.test.ts
```

### Producer regression runs on host

```bash
bun run --cwd packages/core build:hyperframes-runtime:modular
bun --cwd packages/producer test -- --sequential --exclude-tags slow,render-compat,hdr
bun --cwd packages/producer test -- --sequential hdr-pq hdr-image-only
```

Observed result:
- `fast` shard: 7 passed, 0 failed
- `hdr` shard: 2 passed, 0 failed

### CI-equivalent Docker verification

```bash
docker build -f Dockerfile.test -t hyperframes-producer:test .

docker run --rm \
  --security-opt seccomp=unconfined \
  --shm-size=4g \
  -v "$PWD/packages/producer/tests:/app/packages/producer/tests" \
  hyperframes-producer:test \
  --sequential hdr-pq hdr-image-only
```

Observed result:
- `hdr-image-only`: passed
- `hdr-pq`: passed
- shard summary: 2 passed, 0 failed

### Specific regression fixed

Before the PNG `cICP` fallback, the Docker/CI run failed `hdr-image-only` with:

- missing `"[Render] HDR source detected — output: PQ ..."` log line
- full-frame visual mismatch across all 100 checkpoints
- PSNR ~17 on every frame, indicating a consistent SDR-vs-HDR pipeline mismatch

After the fallback, the same Docker path recognizes the PNG as HDR and the shard passes.
2026-04-20 12:16:24 -07:00
Vance Ingalls a21a62b574 feat(engine): add HDR two-pass compositing — DOM layer + native HLG video (#288)
## Summary

Compositions with HDR video AND DOM overlays (text, graphics, SDR video) couldn't render both correctly — either HDR data was lost (Chrome captures sRGB only) or DOM overlays were missing (FFmpeg pass-through skips Chrome). This PR adds in-memory alpha compositing that combines both.

## What it does

**Per-frame two-pass capture:**
1. **DOM pass** — Chrome screenshots the page with a transparent background (CDP alpha). HDR videos are hidden, leaving transparent holes where they go.
2. **HDR pass** — Pre-extracted native HLG/PQ frames (16-bit PNG from FFmpeg) are read from disk.
3. **Composite** — DOM pixels (sRGB RGBA8) are alpha-composited over HDR pixels (rgb48le) in Node.js memory, with sRGB→HLG/PQ conversion via a 256-entry lookup table.

**Key components:**
- `decodePng()` / `decodePngToRgb48le()` — Pure Node.js PNG decoders (no native dependencies). Support all 5 PNG filter types.
- `blitRgba8OverRgb48le()` — Alpha composite with per-pixel sRGB→HDR LUT conversion. Fast paths for alpha=0 (skip) and alpha=255 (overwrite).
- `initTransparentBackground()` + `captureAlphaPng()` — Split CDP transparent background setup (once) from per-frame screenshot capture (eliminates 2 CDP round-trips per frame).
- Single-pass FFmpeg extraction — All HDR frames extracted in one sequential FFmpeg run (avoids duplicate frames from per-frame `-ss` fast seek).

## Key design decisions

| Decision | Why |
|----------|-----|
| In-memory compositing (not FFmpeg overlay) | Eliminates ~2400 process spawns + temp files per render. Pure pixel math is 10x faster. |
| 16-bit PNG intermediate | Raw `-f rawvideo` loses color metadata, causing moiré artifacts. PNG is self-describing. |
| sRGB→HLG LUT (256 entries) | DOM content is sRGB. Without conversion, it appears orange-shifted in HLG stream. |
| Native HDR detection before extraction | `extractAllVideoFrames` converts SDR→HDR. Pre-extraction probe identifies original HDR sources so only truly-HDR videos get native extraction. |

## Files changed

| File | What changed |
|------|-------------|
| `packages/engine/src/utils/alphaBlit.ts` | **NEW** — PNG decode, sRGB→HDR LUT, alpha compositing (14 tests) |
| `packages/engine/src/services/screenshotService.ts` | Transparent background CDP, `captureAlphaPng()` |
| `packages/engine/src/services/videoFrameInjector.ts` | `hideVideoElements()` / `showVideoElements()` |
| `packages/engine/src/services/streamingEncoder.ts` | Input color space tags for rgb48le |
| `packages/producer/src/services/renderOrchestrator.ts` | Two-pass HDR capture loop, native HDR detection |

## How to test

Render a composition with an HDR video background and text overlays. Both should be visible — HDR video at full quality, text crisp with correct colors (not orange-shifted).

## Stack position

**3 of 6** — Stacked on #265 (HDR output pipeline). This is the foundation for all layered compositing that follows.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-04-19 16:26:54 -07:00