perf(engine): faster shader transitions via page-side WebGL compositing (#832)

* fix(cli): prefer puppeteer cache + numeric version sort (staff review) Two correctness fixes from PR #821 self-review: 1. Cache priority order. Previous order was hyperframes-managed cache → puppeteer cache. HF cache is pinned to CHROME_VERSION (131-era) which lags 17+ releases behind upstream; if a user separately installed a newer chrome-headless-shell via @puppeteer/browsers install, the CLI would silently hand engine the older HF-cache binary while engine's own resolveHeadlessShellPath would have picked the newer one. Flip the priority so puppeteer cache wins, matching engine semantics. 2. Numeric (not lexicographic) version sort. `readdirSync.sort().reverse()` over names like `linux-148.0.7778.97` and `linux-99.0.6533.123` would return `linux-99...` first because character '9' outranks '1'. Parse each name into integer segments and compare them numerically. Tests: add both-caches-populated and linux-148-beats-linux-99 cases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(engine): page-side compositing for shader transitions (opt-in spike) Add an opt-in `--page-side-compositing` flag (CLI) backed by a new engine config field `enablePageSideCompositing` and env var `HF_PAGE_SIDE_COMPOSITING`. When set, SDR shader-transition compositions skip the Node-side layered blend (the hf#677 chain) and instead run the shader inside Chrome via a page-side WebGL canvas; the engine then captures ONE opaque RGB frame per output frame via the existing streaming capture path. This is the strongest non-beginFrame perf lever for Mac users, who cannot take the beginFrame `~5×` path (Chromium structural limit, crbug.com/40656275). Stacks on top of the hf#677 1.95× baseline. Default OFF — existing fixture pins (byte-exact MP4 output) are preserved. Opt-in path is intentionally PSNR-pinned, not byte-equal (WebGL is f32; Node is f64). HDR content forces the existing layered path regardless. Implementation: - engine: new `EngineConfig.enablePageSideCompositing` (default false). - producer/fileServer: new `HF_PAGE_SIDE_COMPOSITING_STUB` early-page script injected into the served HTML head when the flag is on. - producer/renderOrchestrator: when the flag + no HDR + no png-sequence, route SDR transitions through the streaming path instead of the layered HDR stage. - shader-transitions: new `engineModePageComposite.ts` installs a fullscreen WebGL compositor overlay and wraps `window.__hf.seek` so each seek inside a transition window captures both scenes via the Chromium `drawElementImage` API to GL textures, runs the fragment shader, and displays the composited result on the overlay canvas. The engine takes one screenshot per frame and sees the composited overlay. - cli: new `--page-side-compositing` flag sets `HF_PAGE_SIDE_COMPOSITING=true` before producer load. - scripts/page-side-compositing-smoke: bundled-CLI smoke that renders a representative fixture with and without the flag, validates the canary strings are in the shipped bundles, and writes a wall-time pair. Determinism trade documented in the engine config doc-comment. The smoke script enforces the bundled-CLI validation discipline from prior perf work (see internal feedback note `validate_bundled_cli_not_dev_path`). Runtime requirement: Chromium's `CanvasDrawElement` feature (already enabled by the engine's `--enable-features=CanvasDrawElement` launch flag). When the runtime feature is unavailable, the page-side installer logs a warning and falls back to opacity-flip mode — the engine still takes the streaming path; the transition window degrades to a hard scene swap. Vance will validate on Mac Chrome where the feature is supported. Co-Authored-By: Vai <vai@heygen.com> * fix(shader-transitions): use html2canvas for page-side compositor capture The original drawElementImage approach fails in engine render mode because the virtual-time shim prevents Chromium from generating paint records for cloned elements. drawElementImage requires a cached paint record from the browser's compositor — clones created at capture time never receive one because (a) shimmed rAFs deadlock inside the seek wrapper, (b) original rAFs don't produce real paints under virtual-time control, and (c) layoutsubtree canvases don't apply CSS stylesheet rules to children. Switch scene capture to html2canvas (foreignObjectRendering: false), the same JS-based renderer already used by the preview-mode fallback path in capture.ts. html2canvas reads computed styles and renders via its own canvas drawing pipeline with no dependency on the browser paint cycle. Also fixes: - Engine seek must return the result so Puppeteer awaits async seek promises (frameCapture.ts). - GSAP opacity cache: compositor must restore scene opacity before seek, not after — GSAP caches inline values and skips re-writes. - Support check gates on WebGL availability, not drawElementImage. Perf: 15-scene shader-perf fixture (28s, 14 transitions, 30fps) Baseline (Node-side layered): 137s Page-side (html2canvas+WebGL): 33s → 4.1× speedup Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(shader-transitions): simplify review fixes for page-side compositor - Use uploadTexture (zeroes canvas backing store after upload) to prevent ~2.2GB transient memory pressure across 280 html2canvas calls per render - Add ignoreElements + stabilizeTransformedBoxShadows to html2canvas call, matching the preview-path capture.ts behavior - Parallelize from/to scene captures with Promise.all - Wrap post-capture render in try/finally so opacity is always restored - Fix WebGL context leak in isPageSideCompositingSupported probe - Remove dead ResolvedTransition.index field - Export stabilizeTransformedBoxShadows from capture.ts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(producer): unify page-side compositing gating and Docker forwarding Addresses three issues from staff review: 1. ignoreElements filter stripped all in-scene canvases (Chart.js, D3, p5.js) — narrowed to data-no-capture only since the compositor canvas is a body sibling never in the scene subtree. 2. Docker mode silently dropped --page-side-compositing — thread pageSideCompositing through DockerRenderOptions/buildDockerRunArgs with regression tests. 3. Fragmented gating across 4 independent sites could disagree: - Stub injection gated only on cfg flag (leaked into HDR/alpha) - Probe-created fileServer never got the stub - needsAlpha (WebM/MOV) not excluded from the gate - WebGL-unavailable fallback claimed layered path would run but orchestrator had already disabled it Fix: compute stub injection at the same site as the layered-bypass decision (after hasHdrContent is known), using addPreHeadScript on the already-running fileServer. Single predicate now gates both decisions, including !needsAlpha. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf(engine): two-phase drawElementImage capture for page-side compositing Replace html2canvas with native drawElementImage for scene capture in the page-side compositor. drawElementImage reads from the browser's own paint cache, giving pixel-identical output to the preview path. The blocker was that cloned elements inside layoutsubtree canvases have no cached paint record under virtual time — the compositor only paints when explicitly triggered. Fix: split the seek+composite into two phases with an engine-forced paint between them. Phase 1 (seek wrapper, page-side): - GSAP seek positions the timeline - Clone FROM/TO scenes into visible layoutsubtree staging canvases - Set window.__hf_page_composite_pending flag Engine paint force (frameCapture.ts): - Detect pending flag after seek returns - Fire micro Page.captureScreenshot (1x1 clip) via CDP to force the browser compositor to paint all visible elements including staging canvas children Phase 2 (page.evaluate, page-side): - drawElementImage reads the now-valid paint records - Upload textures to WebGL, run shader, show GL overlay Key insight: staging canvases must be visible (not opacity:0) for the browser to paint their children. They sit at z-index:-9998, behind the main DOM and covered by the GL overlay during transitions. Perf: 15-scene fixture (28s, 14 transitions, 30fps): Baseline (Node-side layered): 137s html2canvas + WebGL: 33s (3.7×) drawElementImage + WebGL: 21s (6.6×) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf(engine): optimize two-phase compositor hot path - uploadTextureSource instead of uploadTexture: eliminates ~2.3GB of canvas buffer alloc/dealloc churn (persistent staging canvases don't need the one-shot zeroing behavior) - Fold hasPending check into seek page.evaluate: eliminates one CDP round-trip per frame (~700 unnecessary IPC calls on non-transition frames) - Fix renderShader error handling: on failure, leave source scenes visible as fallback instead of hiding both scenes + GL overlay (which produced black frames) - Move mutable state declarations above resolveComposite to prevent TDZ risk on refactor Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(engine): staff review — staging cleanup, pending flag, beginFrame guard - Clear staging canvas children when leaving transition window (prevents visible clone bleed-through on transparent compositions) - Clear __hf_page_composite_pending on all resolveComposite exit paths - Guard micro-screenshot paint force against beginFrame mode (CDP Page.captureScreenshot conflicts with beginFrame compositor control) - Update CLI flag description: document video/canvas limitation, remove stale PSNR claim Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(engine): default-on page-side compositing for SDR shader transitions Page-side compositing is now enabled by default for SDR shader-transition renders without video content. The 6.6× speedup applies automatically — no flag needed. Auto-disables when: - HDR content detected - Alpha output (WebM/MOV/PNG-sequence) - Composition contains <video> elements (cloneNode loses playback state) - beginFrame capture mode (Linux headless) Use --no-page-side-compositing to force the Node-side layered path. Changes: - Engine config: enablePageSideCompositing defaults to true - CLI: flag default flipped to true; --no-page-side-compositing disables - Orchestrator: added composition.videos.length === 0 gate - Docker: forwards --no-page-side-compositing when explicitly disabled - Config tests updated for new default Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(engine): support video elements on page-side compositing fast path Three-phase capture protocol lets shader transitions render video scenes without falling back to the slow Node-side layered pipeline: 1. Seek → compositor records transition metadata, sets pending flag 2. onBeforeCapture → video frame injector updates <img> replacements 3. prepare → cloneNode picks up current video frames, img.decode() awaits 4. micro-screenshot → forces browser to paint cloned elements 5. resolve → drawElementImage reads paint records, shader composites Key changes: - Remove `composition.videos.length === 0` gate from orchestrator - Split compositor resolve into prepare (clone) + resolve (shader) - Move onBeforeCapture before compositor prepare in frameCapture.ts - Await img.decode() on cloned data-URI images to prevent stale frames - Stop manipulating scene opacity in compositor (GL canvas overlay suffices) - Add gsap.set declaration for shader-transitions ambient types - Add video_missing_timing_attrs lint rule for <video> without id/data-start/data-end Performance: compositions with video now render at 7.5s (6 workers) instead of 2m38s on the layered path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(core): auto-inject data-start on video/audio so frame extraction works without explicit attrs The timing compiler now injects data-start="0" on <video> and <audio> elements that lack it. This makes discoverMediaFromBrowser() find the element (it queries video[data-start]), so the frame extraction pipeline activates automatically. Videos "just work" without requiring authors to add data-start, data-end, or id attributes. Also removes the video_missing_timing_attrs lint rule — the compiler handles the missing attributes automatically, so the lint rule would only false-positive. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(core): add data-hf-auto-start sentinel on auto-injected video timing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(producer): add discoverVideoVisibilityFromTimeline for runtime video discovery Seeks the GSAP timeline in Puppeteer to discover when each video's parent scene is visible (opacity > 0). Uses coarse sampling at 100ms steps followed by binary search refinement to frame-level precision (1/60s). Only processes videos with the data-hf-auto-start sentinel so author-specified timing is never overridden. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(producer): integrate runtime video visibility discovery into probe stage Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(producer): trigger browser probe for auto-start videos, remove debug logging The probe stage was skipping browser launch when composition duration was already known, which meant discoverVideoVisibilityFromTimeline never ran. Now needsBrowser also checks for data-hf-auto-start sentinel in compiled HTML. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(scripts): use mkdtempSync for smoke test work directory Replaces hardcoded /tmp/hf-page-side-smoke with a unique temp directory via mkdtempSync to resolve CodeQL "insecure temporary file" alert. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: format smoke test script Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Vai <vai@heygen.com>
2026-09-10 22:20:14 +00:00 · 2026-05-15 16:23:12 -07:00
parent fb9002598d
commit f84cc492de
21 changed files with 1197 additions and 31 deletions
@@ -428,6 +428,29 @@ const HF_EARLY_STUB = `(function() {
  if (!window.__hf) window.__hf = {};
 })();`;

+/**
+ * Page-side compositing opt-in flag stub.
+ *
+ * When the engine is launched with `enablePageSideCompositing: true`, the
+ * orchestrator injects this stub into the very top of every served HTML
+ * page. The flag is read by `@hyperframes/shader-transitions`' engine-mode
+ * `init()` to switch from the default opacity-flip mode (which leaves
+ * shader blending to the Node side via the hf#677 layered pipeline) to a
+ * page-side WebGL compositor that runs the shader inside Chrome and
+ * exposes a single opaque RGB frame for the engine to capture.
+ *
+ * Sentinel ONLY — no logic here. The compositor itself ships inside
+ * `@hyperframes/shader-transitions` and is loaded by the composition's
+ * regular script bundle.
+ *
+ * Default OFF: when the flag is not set, behavior is byte-identical to
+ * the existing layered path.
+ */
+export const HF_PAGE_SIDE_COMPOSITING_STUB = `(function() {
+  if (typeof window === "undefined") return;
+  window.__HF_PAGE_SIDE_COMPOSITING__ = true;
+})();`;
+
 /**
 * Bridge script: maps window.__player (Hyperframe runtime) → window.__hf (engine protocol).
 * Injected after RENDER_MODE_SCRIPT so the engine's frameCapture can find window.__hf.
@@ -525,6 +548,7 @@ export interface FileServerHandle {
  url: string;
  port: number;
  close: () => void;
+  addPreHeadScript: (script: string) => void;
 }

 export function createFileServer(options: FileServerOptions): Promise<FileServerHandle> {
@@ -631,6 +655,9 @@ export function createFileServer(options: FileServerOptions): Promise<FileServer
      resolve({
        url: `http://localhost:${info.port}`,
        port: info.port,
+        addPreHeadScript: (script: string) => {
+          preHeadScripts.push(script);
+        },
        close: () => {
          for (const socket of connections) socket.destroy();
          connections.clear();
@@ -1151,6 +1151,109 @@ export async function discoverMediaFromBrowser(page: Page): Promise<BrowserMedia
  return elements as BrowserMediaElement[];
 }

+export interface VideoVisibilityWindow {
+  videoId: string;
+  visibleStart: number;
+  visibleEnd: number;
+}
+
+/**
+ * Seek the GSAP timeline to discover when each video's parent scene is visible.
+ * Only processes videos with the data-hf-auto-start sentinel (auto-injected timing).
+ */
+export async function discoverVideoVisibilityFromTimeline(
+  page: Page,
+  compositionDuration: number,
+): Promise<VideoVisibilityWindow[]> {
+  if (compositionDuration <= 0) return [];
+
+  return page.evaluate((duration: number) => {
+    const results: { videoId: string; visibleStart: number; visibleEnd: number }[] = [];
+    const videos = document.querySelectorAll("video[data-hf-auto-start]");
+    if (videos.length === 0) return results;
+
+    const timelines = (window as unknown as { __timelines?: Record<string, unknown> }).__timelines;
+    if (!timelines) return results;
+
+    const rootEl = document.querySelector("[data-composition-id]");
+    const compId = rootEl?.getAttribute("data-composition-id");
+    if (!compId) return results;
+
+    const tl = timelines[compId] as
+      | {
+          totalTime?: (t: number, suppressEvents?: boolean) => unknown;
+          seek?: (t: number, suppressEvents?: boolean) => unknown;
+        }
+      | undefined;
+    if (!tl) return results;
+
+    const seekTl = (t: number) => {
+      if (typeof tl.totalTime === "function") {
+        tl.totalTime(t, true);
+      } else if (typeof tl.seek === "function") {
+        tl.seek(t, true);
+      }
+    };
+
+    const SAMPLE_STEP = 0.1;
+    const BINARY_PRECISION = 1 / 60;
+
+    for (const videoEl of videos) {
+      const id = videoEl.id;
+      if (!id) continue;
+
+      const sceneEl = videoEl.closest(".scene") || videoEl;
+
+      let firstVisible: number | null = null;
+      let lastVisible: number | null = null;
+
+      for (let t = 0; t <= duration; t += SAMPLE_STEP) {
+        seekTl(t);
+        const opacity = parseFloat(window.getComputedStyle(sceneEl).opacity);
+        if (opacity > 0) {
+          if (firstVisible === null) firstVisible = t;
+          lastVisible = t;
+        }
+      }
+
+      if (firstVisible === null || lastVisible === null) continue;
+
+      // Binary search left boundary
+      let lo = Math.max(0, firstVisible - SAMPLE_STEP);
+      let hi = firstVisible;
+      while (hi - lo > BINARY_PRECISION) {
+        const mid = (lo + hi) / 2;
+        seekTl(mid);
+        const opacity = parseFloat(window.getComputedStyle(sceneEl).opacity);
+        if (opacity > 0) hi = mid;
+        else lo = mid;
+      }
+      const exactStart = hi;
+
+      // Binary search right boundary
+      lo = lastVisible;
+      hi = Math.min(duration, lastVisible + SAMPLE_STEP);
+      while (hi - lo > BINARY_PRECISION) {
+        const mid = (lo + hi) / 2;
+        seekTl(mid);
+        const opacity = parseFloat(window.getComputedStyle(sceneEl).opacity);
+        if (opacity > 0) lo = mid;
+        else hi = mid;
+      }
+      const exactEnd = lo;
+
+      results.push({
+        videoId: id,
+        visibleStart: Math.max(0, exactStart),
+        visibleEnd: Math.min(duration, exactEnd),
+      });
+    }
+
+    seekTl(0);
+    return results;
+  }, compositionDuration);
+}
+
 /**
 * Resolve composition durations via Puppeteer by querying window.__timelines.
 * The page must already have the interceptor loaded and timelines registered.
@@ -38,6 +38,7 @@ import { fpsToNumber } from "@hyperframes/core";
 import type { CompiledComposition } from "../../htmlCompiler.js";
 import {
  discoverMediaFromBrowser,
+  discoverVideoVisibilityFromTimeline,
  recompileWithResolutions,
  resolveCompositionDurations,
 } from "../../htmlCompiler.js";
@@ -104,7 +105,9 @@ export async function runProbeStage(input: ProbeStageInput): Promise<ProbeStageR
  let lastBrowserConsole: string[] = [];

  const probeStart = Date.now();
-  const needsBrowser = composition.duration <= 0 || compiled.unresolvedCompositions.length > 0;
+  const hasAutoStartVideos = compiled.html.includes("data-hf-auto-start");
+  const needsBrowser =
+    composition.duration <= 0 || compiled.unresolvedCompositions.length > 0 || hasAutoStartVideos;

  if (needsBrowser) {
    const reasons = [];
@@ -287,6 +290,28 @@ export async function runProbeStage(input: ProbeStageInput): Promise<ProbeStageR
        }
      }
    }
+
+    // Runtime video discovery: for videos with auto-injected timing (data-hf-auto-start),
+    // seek the GSAP timeline to find actual scene visibility windows and override start/end.
+    if (composition.videos.length > 0) {
+      const visibilityWindows = await discoverVideoVisibilityFromTimeline(
+        probeSession.page,
+        composition.duration,
+      );
+      assertNotAborted();
+
+      for (const win of visibilityWindows) {
+        const video = composition.videos.find((v) => v.id === win.videoId);
+        if (!video) continue;
+        if (win.visibleStart >= 0 && win.visibleEnd > win.visibleStart) {
+          video.start = win.visibleStart;
+          video.end = win.visibleEnd;
+          log.info(
+            `[Probe] Runtime video discovery: ${video.id} visible ${win.visibleStart.toFixed(2)}s–${win.visibleEnd.toFixed(2)}s`,
+          );
+        }
+      }
+    }
  }
  const browserProbeMs = Date.now() - probeStart;

@@ -78,7 +78,12 @@ import {
 import { join, dirname, resolve } from "path";
 import { randomUUID } from "crypto";
 import { fileURLToPath } from "url";
-import { createFileServer, type FileServerHandle, VIRTUAL_TIME_SHIM } from "./fileServer.js";
+import {
+  createFileServer,
+  type FileServerHandle,
+  HF_PAGE_SIDE_COMPOSITING_STUB,
+  VIRTUAL_TIME_SHIM,
+} from "./fileServer.js";
 import { defaultLogger, type ProducerLogger } from "../logger.js";
 import { type HdrImageTransferCache } from "./hdrImageTransferCache.js";
 import {
@@ -1618,7 +1623,9 @@ export async function executeRenderJob(
    const stage4Start = Date.now();
    updateJobStatus(job, "rendering", "Starting frame capture", 25, onProgress);

-    // Start file server (may already be running from duration discovery)
+    // Start file server (may already be running from duration discovery).
+    // The page-side compositing stub is injected later (after hasHdrContent
+    // is known) via addPreHeadScript — see usePageSideCompositingForTransitions.
    if (!fileServer) {
      fileServer = await createFileServer({
        projectDir,
@@ -1742,11 +1749,34 @@ export async function executeRenderJob(
    // issues (orange shift) with no quality benefit.
    const nativeHdrIds = new Set([...nativeHdrVideoIds, ...nativeHdrImageIds]);
    const hasHdrContent = Boolean(effectiveHdr && nativeHdrIds.size > 0);
-    const useLayeredComposite = shouldUseLayeredComposite({
-      hasHdrContent,
-      hasShaderTransitions: compiled.hasShaderTransitions,
-      isPngSequence,
-    });
+    // Page-side compositing opt-in: when the engine is configured to run the
+    // shader blend inside Chrome via a page-side WebGL canvas, the layered
+    // Node-side composite path is unnecessary for SDR shader transitions.
+    // The streaming path takes ONE opaque RGB screenshot per output frame —
+    // exactly the single capture the page-side compositor produces. HDR
+    // content still forces the layered path (HDR layers need per-layer
+    // alpha + native HDR raw frame compositing in Node; that's out of scope
+    // for this opt-in).
+    const usePageSideCompositingForTransitions =
+      cfg.enablePageSideCompositing &&
+      compiled.hasShaderTransitions &&
+      !hasHdrContent &&
+      !isPngSequence &&
+      !needsAlpha;
+    if (usePageSideCompositingForTransitions) {
+      fileServer.addPreHeadScript(HF_PAGE_SIDE_COMPOSITING_STUB);
+      log.info(
+        "[Render] Page-side compositing enabled — bypassing Node-side layered " +
+          "shader-blend path. Engine will capture one opaque RGB frame per output frame.",
+      );
+    }
+    const useLayeredComposite =
+      !usePageSideCompositingForTransitions &&
+      shouldUseLayeredComposite({
+        hasHdrContent,
+        hasShaderTransitions: compiled.hasShaderTransitions,
+        isPngSequence,
+      });
    const encoderHdr = hasHdrContent ? effectiveHdr : undefined;
    // png-sequence has no encoder, but the rest of the orchestrator still
    // reads `preset.quality` for `effectiveQuality` and `preset.codec` for