Files
f84cc492de perf(engine): faster shader transitions via page-side WebGL compositing (#832)
* fix(cli): prefer puppeteer cache + numeric version sort (staff review)

Two correctness fixes from PR #821 self-review:

1. Cache priority order. Previous order was hyperframes-managed cache →
   puppeteer cache. HF cache is pinned to CHROME_VERSION (131-era) which
   lags 17+ releases behind upstream; if a user separately installed a
   newer chrome-headless-shell via @puppeteer/browsers install, the CLI
   would silently hand engine the older HF-cache binary while engine's
   own resolveHeadlessShellPath would have picked the newer one. Flip
   the priority so puppeteer cache wins, matching engine semantics.

2. Numeric (not lexicographic) version sort. `readdirSync.sort().reverse()`
   over names like `linux-148.0.7778.97` and `linux-99.0.6533.123` would
   return `linux-99...` first because character '9' outranks '1'. Parse
   each name into integer segments and compare them numerically.

Tests: add both-caches-populated and linux-148-beats-linux-99 cases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf(engine): page-side compositing for shader transitions (opt-in spike)

Add an opt-in `--page-side-compositing` flag (CLI) backed by a new engine
config field `enablePageSideCompositing` and env var `HF_PAGE_SIDE_COMPOSITING`.
When set, SDR shader-transition compositions skip the Node-side layered blend
(the hf#677 chain) and instead run the shader inside Chrome via a page-side
WebGL canvas; the engine then captures ONE opaque RGB frame per output frame
via the existing streaming capture path.

This is the strongest non-beginFrame perf lever for Mac users, who cannot
take the beginFrame `~5×` path (Chromium structural limit, crbug.com/40656275).
Stacks on top of the hf#677 1.95× baseline.

Default OFF — existing fixture pins (byte-exact MP4 output) are preserved.
Opt-in path is intentionally PSNR-pinned, not byte-equal (WebGL is f32; Node
is f64). HDR content forces the existing layered path regardless.

Implementation:
- engine: new `EngineConfig.enablePageSideCompositing` (default false).
- producer/fileServer: new `HF_PAGE_SIDE_COMPOSITING_STUB` early-page script
  injected into the served HTML head when the flag is on.
- producer/renderOrchestrator: when the flag + no HDR + no png-sequence,
  route SDR transitions through the streaming path instead of the layered
  HDR stage.
- shader-transitions: new `engineModePageComposite.ts` installs a fullscreen
  WebGL compositor overlay and wraps `window.__hf.seek` so each seek inside
  a transition window captures both scenes via the Chromium
  `drawElementImage` API to GL textures, runs the fragment shader, and
  displays the composited result on the overlay canvas. The engine takes
  one screenshot per frame and sees the composited overlay.
- cli: new `--page-side-compositing` flag sets `HF_PAGE_SIDE_COMPOSITING=true`
  before producer load.
- scripts/page-side-compositing-smoke: bundled-CLI smoke that renders a
  representative fixture with and without the flag, validates the canary
  strings are in the shipped bundles, and writes a wall-time pair.

Determinism trade documented in the engine config doc-comment. The smoke
script enforces the bundled-CLI validation discipline from prior perf work
(see internal feedback note `validate_bundled_cli_not_dev_path`).

Runtime requirement: Chromium's `CanvasDrawElement` feature (already
enabled by the engine's `--enable-features=CanvasDrawElement` launch flag).
When the runtime feature is unavailable, the page-side installer logs a
warning and falls back to opacity-flip mode — the engine still takes the
streaming path; the transition window degrades to a hard scene swap. Vance
will validate on Mac Chrome where the feature is supported.

Co-Authored-By: Vai <vai@heygen.com>

* fix(shader-transitions): use html2canvas for page-side compositor capture

The original drawElementImage approach fails in engine render mode because
the virtual-time shim prevents Chromium from generating paint records for
cloned elements. drawElementImage requires a cached paint record from the
browser's compositor — clones created at capture time never receive one
because (a) shimmed rAFs deadlock inside the seek wrapper, (b) original
rAFs don't produce real paints under virtual-time control, and
(c) layoutsubtree canvases don't apply CSS stylesheet rules to children.

Switch scene capture to html2canvas (foreignObjectRendering: false), the
same JS-based renderer already used by the preview-mode fallback path in
capture.ts. html2canvas reads computed styles and renders via its own
canvas drawing pipeline with no dependency on the browser paint cycle.

Also fixes:
- Engine seek must return the result so Puppeteer awaits async seek
  promises (frameCapture.ts).
- GSAP opacity cache: compositor must restore scene opacity before seek,
  not after — GSAP caches inline values and skips re-writes.
- Support check gates on WebGL availability, not drawElementImage.

Perf: 15-scene shader-perf fixture (28s, 14 transitions, 30fps)
  Baseline (Node-side layered): 137s
  Page-side (html2canvas+WebGL): 33s → 4.1× speedup

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(shader-transitions): simplify review fixes for page-side compositor

- Use uploadTexture (zeroes canvas backing store after upload) to prevent
  ~2.2GB transient memory pressure across 280 html2canvas calls per render
- Add ignoreElements + stabilizeTransformedBoxShadows to html2canvas call,
  matching the preview-path capture.ts behavior
- Parallelize from/to scene captures with Promise.all
- Wrap post-capture render in try/finally so opacity is always restored
- Fix WebGL context leak in isPageSideCompositingSupported probe
- Remove dead ResolvedTransition.index field
- Export stabilizeTransformedBoxShadows from capture.ts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(producer): unify page-side compositing gating and Docker forwarding

Addresses three issues from staff review:

1. ignoreElements filter stripped all in-scene canvases (Chart.js, D3,
   p5.js) — narrowed to data-no-capture only since the compositor canvas
   is a body sibling never in the scene subtree.

2. Docker mode silently dropped --page-side-compositing — thread
   pageSideCompositing through DockerRenderOptions/buildDockerRunArgs
   with regression tests.

3. Fragmented gating across 4 independent sites could disagree:
   - Stub injection gated only on cfg flag (leaked into HDR/alpha)
   - Probe-created fileServer never got the stub
   - needsAlpha (WebM/MOV) not excluded from the gate
   - WebGL-unavailable fallback claimed layered path would run but
     orchestrator had already disabled it

   Fix: compute stub injection at the same site as the layered-bypass
   decision (after hasHdrContent is known), using addPreHeadScript on
   the already-running fileServer. Single predicate now gates both
   decisions, including !needsAlpha.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf(engine): two-phase drawElementImage capture for page-side compositing

Replace html2canvas with native drawElementImage for scene capture in
the page-side compositor. drawElementImage reads from the browser's own
paint cache, giving pixel-identical output to the preview path.

The blocker was that cloned elements inside layoutsubtree canvases have
no cached paint record under virtual time — the compositor only paints
when explicitly triggered. Fix: split the seek+composite into two phases
with an engine-forced paint between them.

Phase 1 (seek wrapper, page-side):
  - GSAP seek positions the timeline
  - Clone FROM/TO scenes into visible layoutsubtree staging canvases
  - Set window.__hf_page_composite_pending flag

Engine paint force (frameCapture.ts):
  - Detect pending flag after seek returns
  - Fire micro Page.captureScreenshot (1x1 clip) via CDP to force the
    browser compositor to paint all visible elements including staging
    canvas children

Phase 2 (page.evaluate, page-side):
  - drawElementImage reads the now-valid paint records
  - Upload textures to WebGL, run shader, show GL overlay

Key insight: staging canvases must be visible (not opacity:0) for the
browser to paint their children. They sit at z-index:-9998, behind
the main DOM and covered by the GL overlay during transitions.

Perf: 15-scene fixture (28s, 14 transitions, 30fps):
  Baseline (Node-side layered): 137s
  html2canvas + WebGL:           33s (3.7×)
  drawElementImage + WebGL:      21s (6.6×)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf(engine): optimize two-phase compositor hot path

- uploadTextureSource instead of uploadTexture: eliminates ~2.3GB of
  canvas buffer alloc/dealloc churn (persistent staging canvases don't
  need the one-shot zeroing behavior)
- Fold hasPending check into seek page.evaluate: eliminates one CDP
  round-trip per frame (~700 unnecessary IPC calls on non-transition
  frames)
- Fix renderShader error handling: on failure, leave source scenes
  visible as fallback instead of hiding both scenes + GL overlay
  (which produced black frames)
- Move mutable state declarations above resolveComposite to prevent
  TDZ risk on refactor

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(engine): staff review — staging cleanup, pending flag, beginFrame guard

- Clear staging canvas children when leaving transition window (prevents
  visible clone bleed-through on transparent compositions)
- Clear __hf_page_composite_pending on all resolveComposite exit paths
- Guard micro-screenshot paint force against beginFrame mode (CDP
  Page.captureScreenshot conflicts with beginFrame compositor control)
- Update CLI flag description: document video/canvas limitation, remove
  stale PSNR claim

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(engine): default-on page-side compositing for SDR shader transitions

Page-side compositing is now enabled by default for SDR shader-transition
renders without video content. The 6.6× speedup applies automatically —
no flag needed.

Auto-disables when:
- HDR content detected
- Alpha output (WebM/MOV/PNG-sequence)
- Composition contains <video> elements (cloneNode loses playback state)
- beginFrame capture mode (Linux headless)

Use --no-page-side-compositing to force the Node-side layered path.

Changes:
- Engine config: enablePageSideCompositing defaults to true
- CLI: flag default flipped to true; --no-page-side-compositing disables
- Orchestrator: added composition.videos.length === 0 gate
- Docker: forwards --no-page-side-compositing when explicitly disabled
- Config tests updated for new default

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(engine): support video elements on page-side compositing fast path

Three-phase capture protocol lets shader transitions render video scenes
without falling back to the slow Node-side layered pipeline:

1. Seek → compositor records transition metadata, sets pending flag
2. onBeforeCapture → video frame injector updates <img> replacements
3. prepare → cloneNode picks up current video frames, img.decode() awaits
4. micro-screenshot → forces browser to paint cloned elements
5. resolve → drawElementImage reads paint records, shader composites

Key changes:
- Remove `composition.videos.length === 0` gate from orchestrator
- Split compositor resolve into prepare (clone) + resolve (shader)
- Move onBeforeCapture before compositor prepare in frameCapture.ts
- Await img.decode() on cloned data-URI images to prevent stale frames
- Stop manipulating scene opacity in compositor (GL canvas overlay suffices)
- Add gsap.set declaration for shader-transitions ambient types
- Add video_missing_timing_attrs lint rule for <video> without id/data-start/data-end

Performance: compositions with video now render at 7.5s (6 workers) instead
of 2m38s on the layered path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(core): auto-inject data-start on video/audio so frame extraction works without explicit attrs

The timing compiler now injects data-start="0" on <video> and <audio>
elements that lack it. This makes discoverMediaFromBrowser() find the
element (it queries video[data-start]), so the frame extraction pipeline
activates automatically. Videos "just work" without requiring authors to
add data-start, data-end, or id attributes.

Also removes the video_missing_timing_attrs lint rule — the compiler
handles the missing attributes automatically, so the lint rule would
only false-positive.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(core): add data-hf-auto-start sentinel on auto-injected video timing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(producer): add discoverVideoVisibilityFromTimeline for runtime video discovery

Seeks the GSAP timeline in Puppeteer to discover when each video's parent
scene is visible (opacity > 0). Uses coarse sampling at 100ms steps followed
by binary search refinement to frame-level precision (1/60s). Only processes
videos with the data-hf-auto-start sentinel so author-specified timing is
never overridden.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(producer): integrate runtime video visibility discovery into probe stage

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(producer): trigger browser probe for auto-start videos, remove debug logging

The probe stage was skipping browser launch when composition duration was
already known, which meant discoverVideoVisibilityFromTimeline never ran.
Now needsBrowser also checks for data-hf-auto-start sentinel in compiled HTML.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(scripts): use mkdtempSync for smoke test work directory

Replaces hardcoded /tmp/hf-page-side-smoke with a unique temp directory
via mkdtempSync to resolve CodeQL "insecure temporary file" alert.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: format smoke test script

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Vai <vai@heygen.com>
2026-05-15 16:23:12 -07:00

177 lines
6.3 KiB
JavaScript
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/usr/bin/env node
/**
* Bundled-CLI smoke for `--page-side-compositing` (opt-in spike).
*
* Validates that:
* 1. The bundled CLI accepts the new flag.
* 2. The local `@hyperframes/shader-transitions` IIFE bundle carries the
* page-side compositor canary string (build is wired correctly).
* 3. Rendering the fixture WITH and WITHOUT the flag both produce valid
* MP4s with the same duration. (Pixel-equality is NOT a correctness
* property here — see the determinism note in the PR body.)
* 4. Wall-time pair is captured for the PR body.
*
* Per `feedback_validate_bundled_cli_not_dev_path.md`: the canonical
* execution path is the BUNDLED CLI at `packages/cli/dist/cli.js`, never
* `bun run` against raw TS sources. Do not "improve" this script to use
* `bun run` — bundle-specific bugs (path resolvers, env bootstrap, lazy
* modules) are invisible to the dev path.
*
* Usage from the repo root:
*
* node scripts/page-side-compositing-smoke/run.mjs
*
* Outputs:
*
* <tmpdir>/raw.mp4 (baseline path, flag off)
* <tmpdir>/page-side.mp4 (page-side path, flag on)
* <tmpdir>/wall-times.json (wall-time pair for the PR)
*/
import { execFileSync, spawnSync } from "node:child_process";
import {
copyFileSync,
existsSync,
mkdirSync,
mkdtempSync,
rmSync,
statSync,
writeFileSync,
} from "node:fs";
import { dirname, join, resolve } from "node:path";
import { tmpdir } from "node:os";
import { fileURLToPath } from "node:url";
const HERE = dirname(fileURLToPath(import.meta.url));
const REPO_ROOT = resolve(HERE, "..", "..");
const FIXTURE_SRC = join(HERE, "fixture");
const WORK_DIR = mkdtempSync(join(tmpdir(), "hf-page-side-smoke-"));
const FIXTURE_RUN_DIR = join(WORK_DIR, "fixture");
const CLI_PATH = join(REPO_ROOT, "packages", "cli", "dist", "cli.js");
const SHADER_BUNDLE = join(REPO_ROOT, "packages", "shader-transitions", "dist", "index.global.js");
// Canary string defined in
// `packages/shader-transitions/src/engineModePageComposite.ts` —
// kept identical here on purpose. Test broken → engineModePageComposite.test
// fails first.
const PAGE_COMPOSITOR_CANARY = "__hf_page_compositor_v1__";
function note(line) {
process.stdout.write("[smoke] " + line + "\n");
}
function fail(msg) {
process.stderr.write("[smoke] FAIL: " + msg + "\n");
process.exit(1);
}
function assertExists(path, label) {
if (!existsSync(path)) fail(label + " missing at " + path);
}
function assertCanary() {
assertExists(SHADER_BUNDLE, "shader-transitions IIFE bundle");
const buf = execFileSync("grep", ["-c", PAGE_COMPOSITOR_CANARY, SHADER_BUNDLE]);
const count = Number(buf.toString().trim());
if (count < 1) {
fail(
"shader-transitions bundle is missing the page-side compositor canary " +
`("${PAGE_COMPOSITOR_CANARY}"). Rebuild @hyperframes/shader-transitions and re-run.`,
);
}
note(`canary present in ${SHADER_BUNDLE} (${count}× hit)`);
}
function assertCliCanary() {
assertExists(CLI_PATH, "bundled CLI");
// The CLI bundle should carry the env-var key for HF_PAGE_SIDE_COMPOSITING
// and the page-side flag name. Either confirms the engine + producer +
// CLI side of the change is wired through tsup.
const cliCanaries = ["HF_PAGE_SIDE_COMPOSITING", "page-side-compositing"];
for (const needle of cliCanaries) {
const buf = execFileSync("grep", ["-c", needle, CLI_PATH]);
const count = Number(buf.toString().trim());
if (count < 1) {
fail(`bundled CLI is missing canary "${needle}". Rebuild @hyperframes/cli and re-run.`);
}
note(`CLI bundle carries "${needle}" (${count}× hit)`);
}
}
function setupFixture() {
if (existsSync(WORK_DIR)) rmSync(WORK_DIR, { recursive: true, force: true });
mkdirSync(FIXTURE_RUN_DIR, { recursive: true });
copyFileSync(join(FIXTURE_SRC, "index.html"), join(FIXTURE_RUN_DIR, "index.html"));
copyFileSync(SHADER_BUNDLE, join(FIXTURE_RUN_DIR, "shader-transitions.global.js"));
note("fixture staged at " + FIXTURE_RUN_DIR);
}
function runRender(label, outputPath, extraArgs) {
note(`render: ${label}${outputPath}`);
const argv = [
CLI_PATH,
"render",
FIXTURE_RUN_DIR,
"-o",
outputPath,
"--fps",
"30",
"--workers",
"1",
"--quality",
"draft",
"--quiet",
...extraArgs,
];
const t0 = Date.now();
const res = spawnSync("node", argv, { stdio: "inherit", cwd: REPO_ROOT });
const wallMs = Date.now() - t0;
if (res.status !== 0) {
fail(`render "${label}" exited with status ${res.status}`);
}
if (!existsSync(outputPath)) {
fail(`render "${label}" did not produce ${outputPath}`);
}
const size = statSync(outputPath).size;
if (size < 1024) {
fail(`render "${label}" produced suspiciously small output (${size} bytes)`);
}
note(` wall=${(wallMs / 1000).toFixed(2)}s size=${(size / 1024).toFixed(1)}KB`);
return { label, wallMs, sizeBytes: size, outputPath };
}
function main() {
note(`repo root: ${REPO_ROOT}`);
assertExists(CLI_PATH, "bundled CLI (packages/cli/dist/cli.js)");
assertExists(SHADER_BUNDLE, "shader-transitions IIFE bundle");
assertCliCanary();
assertCanary();
setupFixture();
const baseline = runRender("baseline (flag OFF)", join(WORK_DIR, "raw.mp4"), []);
const pageSide = runRender("page-side (flag ON)", join(WORK_DIR, "page-side.mp4"), [
"--page-side-compositing",
]);
const summary = {
baseline: { wallMs: baseline.wallMs, sizeBytes: baseline.sizeBytes },
pageSide: { wallMs: pageSide.wallMs, sizeBytes: pageSide.sizeBytes },
walltimeRatio: baseline.wallMs / Math.max(1, pageSide.wallMs),
notes:
"fixture: 2s @ 30fps, 1280x720, single cross-warp-morph transition. " +
"Wall times include CLI startup + browser launch — for a perf signal, " +
"amortize over a longer fixture on the target host (Vance's Mac).",
};
writeFileSync(join(WORK_DIR, "wall-times.json"), JSON.stringify(summary, null, 2));
note("wrote " + join(WORK_DIR, "wall-times.json"));
note("baseline: " + (baseline.wallMs / 1000).toFixed(2) + "s");
note("page-side: " + (pageSide.wallMs / 1000).toFixed(2) + "s");
note(
`ratio (baseline/page-side): ${summary.walltimeRatio.toFixed(2)}× ` +
"(>1 means page-side faster, <1 means slower — sandbox is software " +
"WebGL, so a ratio near 1 here is expected and NOT predictive of Mac).",
);
note("OK");
}
main();