ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-03 04:38:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,033 Commits 856 Branches 404 Tags
ed5747a723b7c9296b5668b5911831f7437cbc8e
Commit Graph
16 Commits
Author SHA1 Message Date
Vance IngallsClaude Opus 4.6Vai
 		f84cc492de perf(engine): faster shader transitions via page-side WebGL compositing (#832) 
* fix(cli): prefer puppeteer cache + numeric version sort (staff review)

Two correctness fixes from PR #821 self-review:

1. Cache priority order. Previous order was hyperframes-managed cache →
   puppeteer cache. HF cache is pinned to CHROME_VERSION (131-era) which
   lags 17+ releases behind upstream; if a user separately installed a
   newer chrome-headless-shell via @puppeteer/browsers install, the CLI
   would silently hand engine the older HF-cache binary while engine's
   own resolveHeadlessShellPath would have picked the newer one. Flip
   the priority so puppeteer cache wins, matching engine semantics.

2. Numeric (not lexicographic) version sort. `readdirSync.sort().reverse()`
   over names like `linux-148.0.7778.97` and `linux-99.0.6533.123` would
   return `linux-99...` first because character '9' outranks '1'. Parse
   each name into integer segments and compare them numerically.

Tests: add both-caches-populated and linux-148-beats-linux-99 cases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf(engine): page-side compositing for shader transitions (opt-in spike)

Add an opt-in `--page-side-compositing` flag (CLI) backed by a new engine
config field `enablePageSideCompositing` and env var `HF_PAGE_SIDE_COMPOSITING`.
When set, SDR shader-transition compositions skip the Node-side layered blend
(the hf#677 chain) and instead run the shader inside Chrome via a page-side
WebGL canvas; the engine then captures ONE opaque RGB frame per output frame
via the existing streaming capture path.

This is the strongest non-beginFrame perf lever for Mac users, who cannot
take the beginFrame `~5×` path (Chromium structural limit, crbug.com/40656275).
Stacks on top of the hf#677 1.95× baseline.

Default OFF — existing fixture pins (byte-exact MP4 output) are preserved.
Opt-in path is intentionally PSNR-pinned, not byte-equal (WebGL is f32; Node
is f64). HDR content forces the existing layered path regardless.

Implementation:
- engine: new `EngineConfig.enablePageSideCompositing` (default false).
- producer/fileServer: new `HF_PAGE_SIDE_COMPOSITING_STUB` early-page script
  injected into the served HTML head when the flag is on.
- producer/renderOrchestrator: when the flag + no HDR + no png-sequence,
  route SDR transitions through the streaming path instead of the layered
  HDR stage.
- shader-transitions: new `engineModePageComposite.ts` installs a fullscreen
  WebGL compositor overlay and wraps `window.__hf.seek` so each seek inside
  a transition window captures both scenes via the Chromium
  `drawElementImage` API to GL textures, runs the fragment shader, and
  displays the composited result on the overlay canvas. The engine takes
  one screenshot per frame and sees the composited overlay.
- cli: new `--page-side-compositing` flag sets `HF_PAGE_SIDE_COMPOSITING=true`
  before producer load.
- scripts/page-side-compositing-smoke: bundled-CLI smoke that renders a
  representative fixture with and without the flag, validates the canary
  strings are in the shipped bundles, and writes a wall-time pair.

Determinism trade documented in the engine config doc-comment. The smoke
script enforces the bundled-CLI validation discipline from prior perf work
(see internal feedback note `validate_bundled_cli_not_dev_path`).

Runtime requirement: Chromium's `CanvasDrawElement` feature (already
enabled by the engine's `--enable-features=CanvasDrawElement` launch flag).
When the runtime feature is unavailable, the page-side installer logs a
warning and falls back to opacity-flip mode — the engine still takes the
streaming path; the transition window degrades to a hard scene swap. Vance
will validate on Mac Chrome where the feature is supported.

Co-Authored-By: Vai <vai@heygen.com>

* fix(shader-transitions): use html2canvas for page-side compositor capture

The original drawElementImage approach fails in engine render mode because
the virtual-time shim prevents Chromium from generating paint records for
cloned elements. drawElementImage requires a cached paint record from the
browser's compositor — clones created at capture time never receive one
because (a) shimmed rAFs deadlock inside the seek wrapper, (b) original
rAFs don't produce real paints under virtual-time control, and
(c) layoutsubtree canvases don't apply CSS stylesheet rules to children.

Switch scene capture to html2canvas (foreignObjectRendering: false), the
same JS-based renderer already used by the preview-mode fallback path in
capture.ts. html2canvas reads computed styles and renders via its own
canvas drawing pipeline with no dependency on the browser paint cycle.

Also fixes:
- Engine seek must return the result so Puppeteer awaits async seek
  promises (frameCapture.ts).
- GSAP opacity cache: compositor must restore scene opacity before seek,
  not after — GSAP caches inline values and skips re-writes.
- Support check gates on WebGL availability, not drawElementImage.

Perf: 15-scene shader-perf fixture (28s, 14 transitions, 30fps)
  Baseline (Node-side layered): 137s
  Page-side (html2canvas+WebGL): 33s → 4.1× speedup

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(shader-transitions): simplify review fixes for page-side compositor

- Use uploadTexture (zeroes canvas backing store after upload) to prevent
  ~2.2GB transient memory pressure across 280 html2canvas calls per render
- Add ignoreElements + stabilizeTransformedBoxShadows to html2canvas call,
  matching the preview-path capture.ts behavior
- Parallelize from/to scene captures with Promise.all
- Wrap post-capture render in try/finally so opacity is always restored
- Fix WebGL context leak in isPageSideCompositingSupported probe
- Remove dead ResolvedTransition.index field
- Export stabilizeTransformedBoxShadows from capture.ts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(producer): unify page-side compositing gating and Docker forwarding

Addresses three issues from staff review:

1. ignoreElements filter stripped all in-scene canvases (Chart.js, D3,
   p5.js) — narrowed to data-no-capture only since the compositor canvas
   is a body sibling never in the scene subtree.

2. Docker mode silently dropped --page-side-compositing — thread
   pageSideCompositing through DockerRenderOptions/buildDockerRunArgs
   with regression tests.

3. Fragmented gating across 4 independent sites could disagree:
   - Stub injection gated only on cfg flag (leaked into HDR/alpha)
   - Probe-created fileServer never got the stub
   - needsAlpha (WebM/MOV) not excluded from the gate
   - WebGL-unavailable fallback claimed layered path would run but
     orchestrator had already disabled it

   Fix: compute stub injection at the same site as the layered-bypass
   decision (after hasHdrContent is known), using addPreHeadScript on
   the already-running fileServer. Single predicate now gates both
   decisions, including !needsAlpha.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf(engine): two-phase drawElementImage capture for page-side compositing

Replace html2canvas with native drawElementImage for scene capture in
the page-side compositor. drawElementImage reads from the browser's own
paint cache, giving pixel-identical output to the preview path.

The blocker was that cloned elements inside layoutsubtree canvases have
no cached paint record under virtual time — the compositor only paints
when explicitly triggered. Fix: split the seek+composite into two phases
with an engine-forced paint between them.

Phase 1 (seek wrapper, page-side):
  - GSAP seek positions the timeline
  - Clone FROM/TO scenes into visible layoutsubtree staging canvases
  - Set window.__hf_page_composite_pending flag

Engine paint force (frameCapture.ts):
  - Detect pending flag after seek returns
  - Fire micro Page.captureScreenshot (1x1 clip) via CDP to force the
    browser compositor to paint all visible elements including staging
    canvas children

Phase 2 (page.evaluate, page-side):
  - drawElementImage reads the now-valid paint records
  - Upload textures to WebGL, run shader, show GL overlay

Key insight: staging canvases must be visible (not opacity:0) for the
browser to paint their children. They sit at z-index:-9998, behind
the main DOM and covered by the GL overlay during transitions.

Perf: 15-scene fixture (28s, 14 transitions, 30fps):
  Baseline (Node-side layered): 137s
  html2canvas + WebGL:           33s (3.7×)
  drawElementImage + WebGL:      21s (6.6×)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf(engine): optimize two-phase compositor hot path

- uploadTextureSource instead of uploadTexture: eliminates ~2.3GB of
  canvas buffer alloc/dealloc churn (persistent staging canvases don't
  need the one-shot zeroing behavior)
- Fold hasPending check into seek page.evaluate: eliminates one CDP
  round-trip per frame (~700 unnecessary IPC calls on non-transition
  frames)
- Fix renderShader error handling: on failure, leave source scenes
  visible as fallback instead of hiding both scenes + GL overlay
  (which produced black frames)
- Move mutable state declarations above resolveComposite to prevent
  TDZ risk on refactor

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(engine): staff review — staging cleanup, pending flag, beginFrame guard

- Clear staging canvas children when leaving transition window (prevents
  visible clone bleed-through on transparent compositions)
- Clear __hf_page_composite_pending on all resolveComposite exit paths
- Guard micro-screenshot paint force against beginFrame mode (CDP
  Page.captureScreenshot conflicts with beginFrame compositor control)
- Update CLI flag description: document video/canvas limitation, remove
  stale PSNR claim

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(engine): default-on page-side compositing for SDR shader transitions

Page-side compositing is now enabled by default for SDR shader-transition
renders without video content. The 6.6× speedup applies automatically —
no flag needed.

Auto-disables when:
- HDR content detected
- Alpha output (WebM/MOV/PNG-sequence)
- Composition contains <video> elements (cloneNode loses playback state)
- beginFrame capture mode (Linux headless)

Use --no-page-side-compositing to force the Node-side layered path.

Changes:
- Engine config: enablePageSideCompositing defaults to true
- CLI: flag default flipped to true; --no-page-side-compositing disables
- Orchestrator: added composition.videos.length === 0 gate
- Docker: forwards --no-page-side-compositing when explicitly disabled
- Config tests updated for new default

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(engine): support video elements on page-side compositing fast path

Three-phase capture protocol lets shader transitions render video scenes
without falling back to the slow Node-side layered pipeline:

1. Seek → compositor records transition metadata, sets pending flag
2. onBeforeCapture → video frame injector updates <img> replacements
3. prepare → cloneNode picks up current video frames, img.decode() awaits
4. micro-screenshot → forces browser to paint cloned elements
5. resolve → drawElementImage reads paint records, shader composites

Key changes:
- Remove `composition.videos.length === 0` gate from orchestrator
- Split compositor resolve into prepare (clone) + resolve (shader)
- Move onBeforeCapture before compositor prepare in frameCapture.ts
- Await img.decode() on cloned data-URI images to prevent stale frames
- Stop manipulating scene opacity in compositor (GL canvas overlay suffices)
- Add gsap.set declaration for shader-transitions ambient types
- Add video_missing_timing_attrs lint rule for <video> without id/data-start/data-end

Performance: compositions with video now render at 7.5s (6 workers) instead
of 2m38s on the layered path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(core): auto-inject data-start on video/audio so frame extraction works without explicit attrs

The timing compiler now injects data-start="0" on <video> and <audio>
elements that lack it. This makes discoverMediaFromBrowser() find the
element (it queries video[data-start]), so the frame extraction pipeline
activates automatically. Videos "just work" without requiring authors to
add data-start, data-end, or id attributes.

Also removes the video_missing_timing_attrs lint rule — the compiler
handles the missing attributes automatically, so the lint rule would
only false-positive.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(core): add data-hf-auto-start sentinel on auto-injected video timing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(producer): add discoverVideoVisibilityFromTimeline for runtime video discovery

Seeks the GSAP timeline in Puppeteer to discover when each video's parent
scene is visible (opacity > 0). Uses coarse sampling at 100ms steps followed
by binary search refinement to frame-level precision (1/60s). Only processes
videos with the data-hf-auto-start sentinel so author-specified timing is
never overridden.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(producer): integrate runtime video visibility discovery into probe stage

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(producer): trigger browser probe for auto-start videos, remove debug logging

The probe stage was skipping browser launch when composition duration was
already known, which meant discoverVideoVisibilityFromTimeline never ran.
Now needsBrowser also checks for data-hf-auto-start sentinel in compiled HTML.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(scripts): use mkdtempSync for smoke test work directory

Replaces hardcoded /tmp/hf-page-side-smoke with a unique temp directory
via mkdtempSync to resolve CodeQL "insecure temporary file" alert.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: format smoke test script

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Vai <vai@heygen.com>
 2026-05-15 16:23:12 -07:00
Miguel Ángel 0f3b207b23 fix: format JSON files, remove unused function, fix capture test mocks 2026-05-06 08:22:55 -07:00
Miguel Ángel 6ef52972fd fix: resolve merge conflict in shader-transitions capture.ts 
Merges main's refactored capture (CaptureSceneOptions, forceVisible,
stabilizeTransformedBoxShadows, foreignObjectRendering fallback) with
our HTML-in-Canvas drawElementImage capture path. The native capture
tries first and falls back to html2canvas on failure.
 2026-05-06 07:49:31 -07:00
Vance Ingalls a7b308b667 feat: cache shader transition preview frames (#634) 
* feat: cache shader transition preview frames

* fix: move shader transition loading to player
 2026-05-06 01:25:09 -07:00
Miguel Ángel 6d2bfe7aaa feat(engine): enable CanvasDrawElement in renderer Chrome args 
Cherry-picked from feat/html-in-canvas-launch (PR #611):

- Enable --enable-features=CanvasDrawElement in Chrome browser args
  so HTML-in-canvas compositions render correctly
- Add native drawElementImage() capture path for shader transitions
  with existing fallback preserved
- Reuse renderer Chrome args in hyperframes validate for consistent
  WebGL/CanvasDrawElement environment
- Add capture.test.ts for the new shader transition capture path
 2026-05-06 00:40:24 -07:00
ukimsanovandClaude Opus 4.6 a8f0752bf6 fix(shader-transitions): address review — NaN guard, canvas sync, shared defaults 
- Validate data-width/data-height: fall back to defaults if NaN or <= 0
- Sync existing #gl-canvas dimensions on reuse (if init called twice)
- Import DEFAULT_WIDTH/DEFAULT_HEIGHT in capture.ts instead of
  hardcoding 1920/1080 in parameter defaults

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-23 21:41:42 -04:00
ukimsanovandClaude Opus 4.6 6b84971f6b feat(shader-transitions): support any aspect ratio (vertical, square) 
Read data-width/data-height from the composition root instead of
hardcoding 1920x1080. Enables shader transitions on vertical
(1080x1920) and square (1080x1080) compositions.

Changes across 3 files:
- webgl.ts: WIDTH/HEIGHT constants → DEFAULT_WIDTH/DEFAULT_HEIGHT,
  createContext and renderShader accept width/height params
- capture.ts: captureScene and captureIncomingScene accept
  width/height params for html2canvas
- hyper-shader.ts: reads data-width/data-height from root element,
  passes dimensions to all webgl and capture calls

GLSL shaders unchanged — they already use u_resolution uniform for
all coordinate math and work at any aspect ratio.

Backwards compatible: all params default to 1920x1080 when not
provided or when data-width/data-height are missing from the DOM.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-23 21:19:49 -04:00
ukimsanovandClaude Opus 4.6 18136d4eb5 fix(shader-transitions): add inWindow guard to .catch fallback handler 
Address Vance's review on #456: the .catch handler was missing the
same tl.time() window check that .then has. Late-rejecting captures
(Safari + SVG-filter compositions) could fire gsap.to/fromTo on
scenes the playhead already left, causing flash-to-black mid-scene.

Wraps the CSS crossfade fallback in the same inWindow guard so
stale catch handlers are no-ops, matching the .then behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-23 20:17:13 -04:00
ukimsanov 41b4017273 docs(shader-transitions): clarify allowTaint caveat in capture comment 
Address Copilot review comment on #456: the old `allowTaint` doc comment
said the resulting canvas is "still usable as a WebGL texture via
gl.texImage2D (no pixel read-back required)", which is wrong. A tainted
canvas CANNOT be uploaded to WebGL — the spec requires SecurityError on
non-origin-clean sources with no opt-out. That's exactly what we observe
in Safari + SVG-filter compositions, and what hyper-shader.ts's catch
handler now handles via CSS crossfade.

Update the comment to correctly describe the flag's effect: it only
moves the failure point from html2canvas to texImage2D; the end-user
UX is the same (smooth CSS fade in either case). The flag remains
defensively correct for the non-taint branches where it genuinely
helps (cross-origin images with `Access-Control-Allow-Origin`).

No code change — comment only.

Made-with: Cursor
 2026-04-23 16:20:32 -04:00
ukimsanov b3b458fda4 fix(shader-transitions): harden capture against visibility, scrub, Safari taint 
Three interrelated fixes for the live-player path in
@hyperframes/shader-transitions (Studio preview, <hyperframes-player>
embeds, Claude Design in-pane iframe). Zero changes to engine mode —
initEngineMode is byte-identical; producer render pipeline and CLI
hyperframes render produce byte-identical output.

1. captureIncomingScene now forces visibility:visible during capture.
   The HF runtime sets visibility:hidden on [data-start] elements outside
   their playback window. With centered shader timing (transition.time =
   boundary - duration/2), html2canvas captures the incoming scene while
   it's still hidden → blank texture → visible blink mid-transition.
   Fix saves, overrides, captures, and restores visibility only for the
   capture window. Empirically validated via a direct html2canvas probe:
   captures of visibility:hidden elements return blank; with override,
   they return real content.

2. post-capture.dom guards on tl.time() window before mutating DOM.
   On scrub across multiple shader transitions, tl.call() fires several
   transitions' callbacks in rapid succession; each launches async
   html2canvas; each .then() unconditionally set all .scene opacities to
   0, enabled shader canvas, and pointed state at that transition. The
   last to resolve won — often for a transition the playhead had left.
   Result: scenes stuck opacity:0 mid-scene; blank screen until the next
   transition's end.call ran. Fix: check tl.time() is still inside
   [T, T+dur] before applying state; otherwise skip.

3. .catch fallback does CSS crossfade instead of hard cut.
   When capture fails (Safari canvas taint from SVG data URLs, CORS
   errors, extreme DOM complexity) the old catch snapped all scenes to
   opacity:0 then set incoming to opacity:1 — jarring instant jump. Fix
   uses gsap.to/fromTo on opacity over the intended transition duration;
   smooth 0.5s fade is strictly better UX. Hard cut preserved as
   last-resort if elements are missing.

Also adds defensive useCORS: true and allowTaint: true to the
html2canvas call. No behavior change in Chrome (capture normally
succeeds); adds resilience for cross-origin images with CORS headers
and SVG-tainted canvases respectively.

Known limitations (out of scope, follow-up tracked):

- Safari + cross-origin iframe: html2canvas is 10-12x slower than Chrome
  due to WebKit's DocumentCloner.cloneNode perf (html2canvas#3108),
  causing perceptible per-transition freezes (1.5-2s each) in Claude
  Design's in-pane preview. Needs pre-capture architecture (cache
  incoming-scene textures at init) to eliminate per-transition cost.
- SVG filter data URLs fundamentally taint html2canvas output in Safari;
  WebGL's texImage2D has no framework opt-out (WebGL spec). Addressed
  at the composition level via the Claude Design skill's anti-pattern 4
  in a parallel PR.

Made-with: Cursor
 2026-04-23 15:15:58 -04:00
Vance Ingalls 2e1a1d91a2 fix(engine,shader): handle matrix3d transforms and hide non-first scenes (#374) 
## Summary

Two correctness fixes in the HDR transform & clipping pipeline: `parseTransformMatrix` now handles `matrix3d(...)` (GSAP's default `force3D: true`), and shader-transitions sets every non-first scene to `opacity: 0` at `t=0` so the engine doesn't over-composite at the start.

## Why

`Chunk 4` of `plans/hdr-followups.md`. Transform extraction and border-radius computation existed but were dead — an HDR video with `rotation: 45` rendered un-rotated, and 3-scene compositions ghosted at `t=0` because every scene defaulted to CSS `opacity: 1` and contributed to the first frame.

## What changed

**Matrix3d support in `parseTransformMatrix`.** `DOMMatrix.toString()` emits `matrix3d` whenever any ancestor in the chain has used a 3D transform — most importantly GSAP's default `force3D: true`, which converts `translate(...)` into `translate3d(..., 0)`. Without this, every GSAP-driven transform was silently dropped during HDR compositing because `videoFrameInjector.getViewportMatrix()` would return `matrix3d(...)` and the blit path would parse it as `null` and fall back to identity. The 16-value column-major form is converted to its 2D affine projection (indices 0, 1, 4, 5, 12, 13 → m11, m12, m21, m22, m41, m42); Z, perspective, and out-of-plane rotation components are dropped.

**Initial-state opacity in `initEngineMode`.** The browser preview branch uses a GL canvas overlay during transitions, so scene opacity at `t=0` doesn't matter visually. The engine branch reads scene opacity directly via `queryElementStacking()` to decide which layers to composite. Without an explicit initial-state tween, every scene defaulted to CSS `opacity: 1` and contributed to the very first frame, causing ghosting/overlap until the first transition fired. `tl.set()` at position 0 anchors the initial state in the timeline graph so reverse seeks from inside a later transition restore it correctly.

These two fixes together make `el.transform` and `el.borderRadius` (already wired in Chunk 7A's `compositeHdrFrame`) actually flow through the GSAP-animated case, and keep the engine's per-frame compositing aligned with what the user sees in browser preview.

## Test plan

- [x] 6 new `alphaBlit.test.ts` cases (identity matrix3d, translate3d, scale + translate3d, rotateZ, malformed arg count, non-finite values).
- [x] Existing `hdr-regression` Window H already CSS-sets `#scene-b { opacity: 0 }` as a fallback; the new `tl.set` is redundant for that case but harmless and removes the need for compositions to remember the CSS workaround.
- [x] Manual: rotated HDR video (`rotation: 45`) appears rotated; `border-radius: 50%` clips to circle; 3-scene composition has no overlap at `t=0`.

## Stack

Chunk 4 of `plans/hdr-followups.md`. Window F of the regression suite documents the bug; the next PR in the stack tightens the `maxFrameFailures` budget to 0.
 2026-04-22 23:55:40 -07:00
Vance Ingalls 9512744c2e refactor(shader-transitions): extract DEFAULT_DURATION and DEFAULT_EASE constants (#367) 
## Summary

Extract `DEFAULT_DURATION = 0.7` and `DEFAULT_EASE = "power2.inOut"` as shared constants in `hyper-shader.ts` and apply them at all three fallback sites (metadata write, browser/render mode, engine mode).

## Why

`Chunk 2` of `plans/hdr-followups.md`. The three fallback sites had drifted apart: the metadata path used `1s` / `"none"` while the actual rendering used `0.7s` / `"power2.inOut"`. A transition that omitted `duration`/`ease` would render at 0.7 s but tell the producer it was 1 s, throwing off the producer's compositing window planning and producing a visible ~0.3 s brightness dropout.

This is a small, high-value correctness fix that runs before the larger Chunk 1 / Chunk 4 work.

## What changed

- New module-level `DEFAULT_DURATION` and `DEFAULT_EASE` constants in `packages/shader-transitions/src/hyper-shader.ts`.
- All three fallback call sites (metadata, browser, engine) now use the constants.
- Explicit `ease: "none"` on the timeline-length anchor tweens elsewhere in the file is intentional (those are linear interpolators driving the shader's progress uniform) and is left unchanged.

## Test plan

- [x] Render a composition with a transition that omits `duration` and `ease` — no brightness dip in the last ~0.3 s of the transition.
- [x] Preview (browser mode) and render (engine mode) produce matching blending curves.
- [x] Render with explicit `duration: 1.5` still works (constants are fallbacks only).

## Stack

Chunk 2 of `plans/hdr-followups.md`. Lands ahead of Chunk 1 (opacity) per the suggested merge order.
 2026-04-22 17:02:42 -07:00
Vance Ingalls 5de5df7fbb refactor(types): tighten type safety, dedupe HfTransitionMeta, prune dead LUT export (#366) 
## Summary

Four small, mechanical type-safety cleanups across `engine`, `producer`, and `shader-transitions`. Zero behavior change — pure pre-cleanup so the rest of the stack ships against a tighter baseline.

## Why

`Chunk 6` of `plans/hdr-followups.md`. Several non-null assertions and a duplicate interface had accumulated as rebase artifacts and leftover work-in-progress; lands first because it touches files later chunks edit and removes friction during review.

## What changed

- `renderOrchestrator.ts`: replace `layers[layerIdx]!` with a `for (const [layerIdx, layer] of layers.entries())` so both index and element come from the iterator.
- `engine/types.ts`: drop the duplicate `HfTransitionMeta` interface (rebase artifact); the original definition above it is the documented one. The orphaned doc comment now precedes `HfProtocol`.
- `shader-transitions/hyper-shader.ts`: keep the local `HfTransitionMeta` declaration (the package ships as a standalone CDN bundle and must not depend on `@hyperframes/engine`), but add a sync comment pointing at the source of truth in `engine/src/types.ts`.
- `alphaBlit.ts` + `engine/index.ts`: drop `export` from `getSrgbToHdrLut` and remove its re-export. It was only ever called by the internal `blitRgba8OverRgb48le`; the public surface was dead code.

## Test plan

- [x] `bun run --filter @hyperframes/engine typecheck`
- [x] `bun run --filter @hyperframes/producer typecheck`
- [x] `bun run --filter @hyperframes/shader-transitions typecheck`
- [x] `bun run --filter @hyperframes/engine test` — 308/308 pass (no test changes; assertions removed in code only).

## Stack

Chunk 6 of `plans/hdr-followups.md`. Mechanical cleanup landed early per the suggested merge order.
 2026-04-22 16:56:17 -07:00
Vance IngallsandClaude Opus 4.6 99a903be2f feat(hdr): layered HDR compositing, shader transitions, and HDR image support (#268) 
* feat(hdr): shader transitions, --hdr flag, and SDR rendering fixes

- 15 GLSL→TypeScript shader transitions on rgb48le buffers
- Dual-scene compositing with scene detection via window.__hf.transitions
- --hdr flag gates ffprobe probing (zero overhead on SDR compositions)
- Cross-transfer conversion (PQ↔HLG) via OOTF-corrected composite LUT
- Buffer.from() copy in writeFrame() fixes streaming encoder race condition
- SDR rendering fixes (three stacked bugs)
- Object.assign fix for window.__hf preservation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: tighten shader smoke thresholds + assert .scene contract

- Tighten the all-transitions smoke test thresholds: at progress=0 we now
  require the center pixel R-channel > 35000 (was > 25000) and at
  progress=1 < 15000 (was < 25000). The old midpoint of 25000 sat exactly
  halfway between the test from-pixel (40000) and to-pixel (10000), so a
  half-blended transition would silently pass.
- Add a runtime assertion in HyperShader.init() that every scene id
  resolves to a DOM element with the .scene class. Without this, missing
  ids silently no-op when textures + querySelectorAll(.scene) run later.

Addresses deferred review feedback from PR #268.

* fix(hdr): restore VIRTUAL_TIME_SHIM and applyRenderModeHints in renderOrchestrator

Commit c6b4619c ("feat(hdr): shader transitions, --hdr flag, and SDR
rendering fixes") accidentally removed two pieces of the deterministic
rendering pipeline:

1. The `VIRTUAL_TIME_SHIM` injected via `createFileServer.preHeadScripts`,
   which freezes `Date.now()` and `requestAnimationFrame` so RAF-driven
   animations advance only when `window.__hf.seek(t)` is called.
2. The `applyRenderModeHints` function and its post-`compileForRender`
   call site, which auto-forces screenshot capture mode for compositions
   the compiler flagged as needing it (RAF, iframes, etc.).

Without (1), RAF animations advanced by wall-clock between the main-loop
seek and the per-DOM-layer seek inside `compositeToBuffer`, producing the
sawtooth PSNR pattern on `raf-ball-render-compat` (high PSNR at integer
seconds, ~24 dB everywhere else). Without (2), `iframe-render-compat`
lost its automatic fallback to screenshot mode and the child-document
motion stopped being captured.

Both helpers are still produced by `htmlCompiler` and exercised by
`renderOrchestrator.test.ts` — the orchestrator just stopped calling
them. Restored:

- Re-import `VIRTUAL_TIME_SHIM` from `./fileServer.js`
- Pass `preHeadScripts: [VIRTUAL_TIME_SHIM]` to both `createFileServer`
  call sites (probe + main render)
- Re-add `applyRenderModeHints` (matching the test expectations) and
  call it immediately after `compileForRender`
- Persist `renderModeHints` in `summary.json` and the
  "Compiled composition metadata" log line

Fixes the `iframe-render-compat` and `raf-ball-render-compat` regression
failures on `feat/hdr-layered-compositing`.

Made-with: Cursor

* test(engine): expand sampleRgb48le coverage + audit Uint16Array alignment

Adds:
- 8 new sampleRgb48le bilinear-interpolation tests covering boundary
  pixels, sub-pixel weights, edge clamping, and odd-byte-offset Buffers.
- uint16-alignment-audit.test.ts documenting the alignment requirement
  for Uint16Array views over Buffer slices vs. readUInt16LE/writeUInt16LE.

Background: ~105 hot-loop sites in shader transitions still use
readUInt16LE/writeUInt16LE. Switching to Uint16Array views would cut
overhead but requires guaranteed even byteOffsets — these tests document
the contract before any future refactor lands.

* fix(engine,producer): mask DOM layers during HDR layered compositing

The HDR layered compositor blits z-ordered layers over a shared canvas. DOM
layers used a full-page screenshot from `captureAlphaPng`, which captures
*every* painted pixel on the page — root background, sibling-scene content,
overlay UI elements that aren't part of the current layer. Those opaque
pixels were then blitted over the canvas, overwriting any HDR content
composited beneath in earlier layers.

The previous workaround toggled `display:none` on hide ids via
`hideVideoElements`/`showVideoElements`. That correctly hid native videos
but did nothing about the root composition's background or about overlay
elements that the layer grouping considered part of a different layer.

This commit replaces the workaround with a precise CSS mask installed
before each DOM screenshot:

1. `applyDomLayerMask` injects a stylesheet that hides every `body *` and
   re-shows the layer's elements (and their descendants and their injected
   `__render_frame_*` siblings) with `visibility: visible !important`. CSS
   visibility is *not* multiplicative through descendants — a child with
   `visibility: visible` overrides an ancestor's `visibility: hidden`, so
   deeply nested layer content still paints even though every intermediate
   ancestor is hidden by the mass-hide rule.
2. Non-layer data-start ids are inline-hidden with
   `visibility: hidden !important`. Inline `!important` beats stylesheet
   `!important`, so this overrides the show rule for elements that fall
   under a show selector but should NOT paint — most importantly HDR
   videos and other-layer SDR videos that live as descendants of `#root`.
3. `removeDomLayerMask` tears the stylesheet down and clears the inline
   `visibility`/`opacity` properties so subsequent video frame injection
   gets a clean slate.

Crucially the mask only sets `visibility`, never `opacity`. CSS opacity
*is* multiplicative — `opacity: 0` on `#root` would zero out every
descendant including layer videos, even with `visibility: visible`. We
also extend `initTransparentBackground` to force the composition root
(`[data-composition-id]`) transparent in addition to `html`/`body`,
because compositions almost always set `#root { background: ... }` and
that background paints across the whole viewport otherwise.

Both compositing paths use the new helpers:
- The per-layer DOM branch (`compositeToBuffer`) for normal frames.
- The transition path (single DOM screenshot per scene) so transition
  frames also get a clean per-scene capture.

Adds extensive `KEEP_TEMP=1`-gated diagnostics to `compositeToBuffer`:
per-layer pixel-add accounting, dumps of every captured DOM PNG, and a
periodic raw `rgb48le` snapshot of the composite buffer. These were
essential to diagnosing the root-overwrite bug and stay zero-cost in
normal renders. Also stops the workDir / per-video frame-dir cleanup
when `KEEP_TEMP=1` so the dumps survive past frame N.

Made-with: Cursor

* fix(engine): preserve GSAP-applied opacity across DOM-layer captures

SDR clips inside an HDR composition were rendering at full opacity even
when the user had animated their wrapper opacity (e.g. fade-in or
yoyo). Two bugs in the per-layer screenshot path conspired to drop the
GSAP-applied opacity on the floor:

1. removeDomLayerMask was unconditionally calling
   `el.style.removeProperty("opacity")` on every wrapper after each
   layer capture. applyDomLayerMask only ever sets `visibility`, so the
   only inline opacity present is the value GSAP wrote. Stripping it
   between layer captures means that on the next capture (at the same
   timestamp), GSAP's `totalTime(t, false)` no-ops because the timeline
   is already at that time — the opacity is never restored, and the
   wrapper renders fully opaque.

2. injectVideoFramesBatch was reading the source <video>'s computed
   opacity via `parseFloat(computedStyle.opacity) || 1` and copying it
   onto the injected <img>. Because syncVideoFrameVisibility forces the
   <video> to `opacity: 0 !important` to hide it during capture, the
   computed value is always 0, which `|| 1` then silently flips to
   full opacity. The <img> is a sibling of the <video> inside the same
   wrapper, so it should inherit opacity from the wrapper directly
   instead of having a value hard-set on it.

Fix both: drop the opacity removal in removeDomLayerMask, skip opacity
when copying visual properties from <video> to <img>, and explicitly
clear any stale inline opacity on the <img> so it inherits from the
wrapper that GSAP is animating.

Made-with: Cursor

* fix(producer): correct hdrLayerStartTimes typo to hdrVideoStartTimes

The diagnostic logging block in executeRenderJob's HDR layer composite
path referenced an undeclared `hdrLayerStartTimes` map. The correct
variable, declared and populated earlier in the same function, is
`hdrVideoStartTimes`. The typo was introduced alongside the DOM-layer
masking work and broke the producer build/typecheck on CI.

Made-with: Cursor

* fix(engine): restore video opacity copy to injected frame img

Commit 188ebcca removed the opacity copy from `injectVideoFramesBatch` on
the assumption that the <img> sibling would inherit GSAP's opacity from
a shared wrapper. That breaks any composition where GSAP animates opacity
directly on the <video> element itself: the <img> has no animated
ancestor and renders at full opacity throughout any fade, even when the
user's intent is partial or zero opacity.

The CI `style-7-prod` and `style-8-prod` regressions caught this:
the <video id="aroll"> fade-in from 3.0-3.5s rendered as a hard cut
because the <img> inherited opacity 1 regardless of GSAP's tween.

Restore the old explicit copy from `computedStyle.opacity` to the
<img>'s inline opacity, with the `|| 1` fallback intentionally
preserved. The fallback is load-bearing: GSAP's seek does not re-apply
tweens that have already completed, so post-fade frames read opacity 0
from the stale `opacity: 0 !important` we apply to hide the native
<video>. The `|| 1` recovers the tween's end-state opacity 1 for
those frames, matching the final on-screen intent and the existing
baseline renders.

Handles both DOM shapes:
- GSAP on wrapper: video's own computed opacity is 1, img set to 1,
  wrapper's opacity applies via stacking as before.
- GSAP on <video>: video's computed opacity is the tween value, copied
  to img directly since they are siblings.

Fixes:
- style-7-prod: 0 failed frames (was 2 @ t=3.17, 3.33)
- style-8-prod: 0 failed frames (was 2 @ t=3.05, 3.24)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-19 19:00:58 -07:00
Vance Ingalls 8548a17771 feat(hdr): GSAP transforms and border-radius masks on HDR video (#290) 
## Summary

HDR video elements with GSAP animations (position, scale, rotation, opacity) and CSS border-radius rendered without any transforms applied — the video just sat at (0,0) full-size. This PR adds affine transform support and rounded-corner masking for natively-composited HDR video.

## What it does

**Affine blit with bilinear interpolation:**
- `blitRgb48leAffine()` — Takes a 4x4 DOMMatrix and maps each destination pixel back to source coordinates via the inverse transform. Bilinear interpolation between the 4 nearest source pixels produces smooth edges under rotation and non-integer scaling. Optional opacity and border-radius parameters.
- `parseTransformMatrix()` — Parses CSS `matrix(a,b,c,d,e,f)` strings into `[a,b,c,d,e,f]` tuples.

**Accumulated viewport matrix:**
- `getViewportMatrix()` — Walks the `offsetParent` chain from element to viewport, accumulating position offsets and CSS transforms at each level. Correctly handles `transform-origin` using the CSS sandwich: `translate(origin) × M × translate(-origin)`. This is critical because GSAP animates transforms on wrapper divs, not directly on the video element.

**Effective opacity:**
- `getEffectiveOpacity()` — Multiplies opacity values walking up the ancestor chain. Uses `Number.isNaN()` (not `|| 1`) so opacity:0 isn't incorrectly treated as 1.

**Border-radius masks:**
- `roundedRectAlpha()` — Per-pixel anti-aliased rounded-rectangle mask with support for independent corner radii.
- `getEffectiveBorderRadius()` — Walks ancestors for `overflow:hidden` + border-radius. Resolves percentage values (e.g., `50%` for circles) via `offsetWidth`/`offsetHeight`.

**Layout dimensions for extraction:**
- Uses `offsetWidth`/`offsetHeight` (unaffected by CSS transforms) instead of `getBoundingClientRect()` (which returns the transformed bounding box and wobbles under rotation).

## Files changed

| File | What changed |
|------|-------------|
| `packages/engine/src/utils/alphaBlit.ts` | `blitRgb48leAffine()`, `parseTransformMatrix()`, `roundedRectAlpha()`, `cornerAlpha()` |
| `packages/engine/src/services/videoFrameInjector.ts` | `getViewportMatrix()`, `getEffectiveOpacity()`, `getEffectiveBorderRadius()`, `layoutWidth`/`layoutHeight` on `ElementStackingInfo` |
| `packages/producer/src/services/renderOrchestrator.ts` | Affine blit path, extraction at layout dimensions, border-radius parameter passing |

## How to test

Render a composition with an HDR video that has GSAP scale + rotation animation and a `border-radius: 50%` wrapper (circle mask). The video should rotate smoothly with round edges — no wobble, no sharp corners.

## Stack position

**5 of 6** — Stacked on #289 (z-ordered layers). Adds transform and masking support to the HDR blit that the layer compositor uses.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-04-19 17:03:29 -07:00
Vance Ingalls cb3d94c2a5 feat: add @hyperframes/shader-transitions package (#251) 
## Summary

New `@hyperframes/shader-transitions` package that encapsulates WebGL shader transitions into a single `HyperShader.init()` call. Replaces ~200 lines of per-composition boilerplate that LLMs failed to wire correctly 60% of the time.

### API

```js
var tl = HyperShader.init({
  bgColor: "#0a0a1a",
  accentColor: "#6366f1",
  scenes: ["scene1", "scene2", "scene3", "scene4", "scene5"],
  transitions: [
    { time: 7.2, shader: "cross-warp-morph", duration: 0.7 },
    { time: 15.2, shader: "domain-warp", duration: 0.7 },
  ]
});
tl.from("#s1-title", { y: 50, opacity: 0, duration: 0.7 }, 0.3);
```

### What the library handles

- **13 shader programs**: domain-warp, ridged-burn, whip-pan, sdf-iris, ripple-waves, gravitational-lens, cinematic-zoom, chromatic-split, glitch, swirl-vortex, thermal-distortion, cross-warp-morph, light-leak
- **html2canvas** bundled as dependency (not CDN) — single script tag for CLI users
- **DOM-during-holds**: canvas hidden between transitions, GSAP animations play on live DOM
- **Async capture with pause/resume**: timeline pauses during capture, resumes after textures uploaded — prevents progress tween from running ahead
- **Accent color theming**: `accentColor` derives dark/mid/bright uniforms. Burns, glows, leaks match the composition palette
- **Graceful degradation**: falls back silently when WebGL unavailable

### Code quality (from 3 review agents)

- No `!` non-null assertions — all WebGL creation calls throw on failure
- Vertex shader compiled once, cached across all programs
- Uniform/attribute locations cached per program via WeakMap (not looked up every frame)
- Captured canvases freed after texture upload (8MB each)
- Single timeline creation (was creating two, discarding one)
- Shared `tickShader()` render callback (was copy-pasted)
- `.finally()` for DOM restore in capture (was duplicated in `.then`/`.catch`)
- `parseHex` validates input (was silently producing NaN on invalid hex)
- Dead `ND`/`CP` shader library exports removed

### Shader-compatible CSS rules (transitions.md)

6 rules for compositions using shader transitions:
1. No `transparent` in gradients (canvas interpolates through black)
2. No gradient backgrounds on elements < 4px
3. No CSS variables on captured elements
4. `data-no-capture` for uncapturable decoratives
5. No gradient opacity < 0.15
6. Every `.scene` must have explicit `background-color` matching `bgColor`

### Build output

- IIFE (~214KB with html2canvas bundled, ~65KB gzipped) — `window.HyperShader`
- ESM + CJS + TypeScript declarations
- tsup build following `@hyperframes/player` conventions

## Test plan
- [ ] `bun run build` succeeds (includes shader-transitions)
- [ ] `bunx oxlint packages/shader-transitions/src/` — 0 errors
- [ ] Create a composition using `HyperShader.init()` — verify transitions fire, DOM animations play, accent colors match
- [ ] Test graceful degradation: composition works without WebGL (no transitions, no crash)
- [ ] Verify pause/resume: scrub to transition boundary — no jump in progress

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-04-13 18:44:00 -07:00