mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
* fix(cli): prefer puppeteer cache + numeric version sort (staff review) Two correctness fixes from PR #821 self-review: 1. Cache priority order. Previous order was hyperframes-managed cache → puppeteer cache. HF cache is pinned to CHROME_VERSION (131-era) which lags 17+ releases behind upstream; if a user separately installed a newer chrome-headless-shell via @puppeteer/browsers install, the CLI would silently hand engine the older HF-cache binary while engine's own resolveHeadlessShellPath would have picked the newer one. Flip the priority so puppeteer cache wins, matching engine semantics. 2. Numeric (not lexicographic) version sort. `readdirSync.sort().reverse()` over names like `linux-148.0.7778.97` and `linux-99.0.6533.123` would return `linux-99...` first because character '9' outranks '1'. Parse each name into integer segments and compare them numerically. Tests: add both-caches-populated and linux-148-beats-linux-99 cases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(engine): page-side compositing for shader transitions (opt-in spike) Add an opt-in `--page-side-compositing` flag (CLI) backed by a new engine config field `enablePageSideCompositing` and env var `HF_PAGE_SIDE_COMPOSITING`. When set, SDR shader-transition compositions skip the Node-side layered blend (the hf#677 chain) and instead run the shader inside Chrome via a page-side WebGL canvas; the engine then captures ONE opaque RGB frame per output frame via the existing streaming capture path. This is the strongest non-beginFrame perf lever for Mac users, who cannot take the beginFrame `~5×` path (Chromium structural limit, crbug.com/40656275). Stacks on top of the hf#677 1.95× baseline. Default OFF — existing fixture pins (byte-exact MP4 output) are preserved. Opt-in path is intentionally PSNR-pinned, not byte-equal (WebGL is f32; Node is f64). HDR content forces the existing layered path regardless. Implementation: - engine: new `EngineConfig.enablePageSideCompositing` (default false). - producer/fileServer: new `HF_PAGE_SIDE_COMPOSITING_STUB` early-page script injected into the served HTML head when the flag is on. - producer/renderOrchestrator: when the flag + no HDR + no png-sequence, route SDR transitions through the streaming path instead of the layered HDR stage. - shader-transitions: new `engineModePageComposite.ts` installs a fullscreen WebGL compositor overlay and wraps `window.__hf.seek` so each seek inside a transition window captures both scenes via the Chromium `drawElementImage` API to GL textures, runs the fragment shader, and displays the composited result on the overlay canvas. The engine takes one screenshot per frame and sees the composited overlay. - cli: new `--page-side-compositing` flag sets `HF_PAGE_SIDE_COMPOSITING=true` before producer load. - scripts/page-side-compositing-smoke: bundled-CLI smoke that renders a representative fixture with and without the flag, validates the canary strings are in the shipped bundles, and writes a wall-time pair. Determinism trade documented in the engine config doc-comment. The smoke script enforces the bundled-CLI validation discipline from prior perf work (see internal feedback note `validate_bundled_cli_not_dev_path`). Runtime requirement: Chromium's `CanvasDrawElement` feature (already enabled by the engine's `--enable-features=CanvasDrawElement` launch flag). When the runtime feature is unavailable, the page-side installer logs a warning and falls back to opacity-flip mode — the engine still takes the streaming path; the transition window degrades to a hard scene swap. Vance will validate on Mac Chrome where the feature is supported. Co-Authored-By: Vai <vai@heygen.com> * fix(shader-transitions): use html2canvas for page-side compositor capture The original drawElementImage approach fails in engine render mode because the virtual-time shim prevents Chromium from generating paint records for cloned elements. drawElementImage requires a cached paint record from the browser's compositor — clones created at capture time never receive one because (a) shimmed rAFs deadlock inside the seek wrapper, (b) original rAFs don't produce real paints under virtual-time control, and (c) layoutsubtree canvases don't apply CSS stylesheet rules to children. Switch scene capture to html2canvas (foreignObjectRendering: false), the same JS-based renderer already used by the preview-mode fallback path in capture.ts. html2canvas reads computed styles and renders via its own canvas drawing pipeline with no dependency on the browser paint cycle. Also fixes: - Engine seek must return the result so Puppeteer awaits async seek promises (frameCapture.ts). - GSAP opacity cache: compositor must restore scene opacity before seek, not after — GSAP caches inline values and skips re-writes. - Support check gates on WebGL availability, not drawElementImage. Perf: 15-scene shader-perf fixture (28s, 14 transitions, 30fps) Baseline (Node-side layered): 137s Page-side (html2canvas+WebGL): 33s → 4.1× speedup Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(shader-transitions): simplify review fixes for page-side compositor - Use uploadTexture (zeroes canvas backing store after upload) to prevent ~2.2GB transient memory pressure across 280 html2canvas calls per render - Add ignoreElements + stabilizeTransformedBoxShadows to html2canvas call, matching the preview-path capture.ts behavior - Parallelize from/to scene captures with Promise.all - Wrap post-capture render in try/finally so opacity is always restored - Fix WebGL context leak in isPageSideCompositingSupported probe - Remove dead ResolvedTransition.index field - Export stabilizeTransformedBoxShadows from capture.ts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(producer): unify page-side compositing gating and Docker forwarding Addresses three issues from staff review: 1. ignoreElements filter stripped all in-scene canvases (Chart.js, D3, p5.js) — narrowed to data-no-capture only since the compositor canvas is a body sibling never in the scene subtree. 2. Docker mode silently dropped --page-side-compositing — thread pageSideCompositing through DockerRenderOptions/buildDockerRunArgs with regression tests. 3. Fragmented gating across 4 independent sites could disagree: - Stub injection gated only on cfg flag (leaked into HDR/alpha) - Probe-created fileServer never got the stub - needsAlpha (WebM/MOV) not excluded from the gate - WebGL-unavailable fallback claimed layered path would run but orchestrator had already disabled it Fix: compute stub injection at the same site as the layered-bypass decision (after hasHdrContent is known), using addPreHeadScript on the already-running fileServer. Single predicate now gates both decisions, including !needsAlpha. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf(engine): two-phase drawElementImage capture for page-side compositing Replace html2canvas with native drawElementImage for scene capture in the page-side compositor. drawElementImage reads from the browser's own paint cache, giving pixel-identical output to the preview path. The blocker was that cloned elements inside layoutsubtree canvases have no cached paint record under virtual time — the compositor only paints when explicitly triggered. Fix: split the seek+composite into two phases with an engine-forced paint between them. Phase 1 (seek wrapper, page-side): - GSAP seek positions the timeline - Clone FROM/TO scenes into visible layoutsubtree staging canvases - Set window.__hf_page_composite_pending flag Engine paint force (frameCapture.ts): - Detect pending flag after seek returns - Fire micro Page.captureScreenshot (1x1 clip) via CDP to force the browser compositor to paint all visible elements including staging canvas children Phase 2 (page.evaluate, page-side): - drawElementImage reads the now-valid paint records - Upload textures to WebGL, run shader, show GL overlay Key insight: staging canvases must be visible (not opacity:0) for the browser to paint their children. They sit at z-index:-9998, behind the main DOM and covered by the GL overlay during transitions. Perf: 15-scene fixture (28s, 14 transitions, 30fps): Baseline (Node-side layered): 137s html2canvas + WebGL: 33s (3.7×) drawElementImage + WebGL: 21s (6.6×) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf(engine): optimize two-phase compositor hot path - uploadTextureSource instead of uploadTexture: eliminates ~2.3GB of canvas buffer alloc/dealloc churn (persistent staging canvases don't need the one-shot zeroing behavior) - Fold hasPending check into seek page.evaluate: eliminates one CDP round-trip per frame (~700 unnecessary IPC calls on non-transition frames) - Fix renderShader error handling: on failure, leave source scenes visible as fallback instead of hiding both scenes + GL overlay (which produced black frames) - Move mutable state declarations above resolveComposite to prevent TDZ risk on refactor Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(engine): staff review — staging cleanup, pending flag, beginFrame guard - Clear staging canvas children when leaving transition window (prevents visible clone bleed-through on transparent compositions) - Clear __hf_page_composite_pending on all resolveComposite exit paths - Guard micro-screenshot paint force against beginFrame mode (CDP Page.captureScreenshot conflicts with beginFrame compositor control) - Update CLI flag description: document video/canvas limitation, remove stale PSNR claim Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(engine): default-on page-side compositing for SDR shader transitions Page-side compositing is now enabled by default for SDR shader-transition renders without video content. The 6.6× speedup applies automatically — no flag needed. Auto-disables when: - HDR content detected - Alpha output (WebM/MOV/PNG-sequence) - Composition contains <video> elements (cloneNode loses playback state) - beginFrame capture mode (Linux headless) Use --no-page-side-compositing to force the Node-side layered path. Changes: - Engine config: enablePageSideCompositing defaults to true - CLI: flag default flipped to true; --no-page-side-compositing disables - Orchestrator: added composition.videos.length === 0 gate - Docker: forwards --no-page-side-compositing when explicitly disabled - Config tests updated for new default Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(engine): support video elements on page-side compositing fast path Three-phase capture protocol lets shader transitions render video scenes without falling back to the slow Node-side layered pipeline: 1. Seek → compositor records transition metadata, sets pending flag 2. onBeforeCapture → video frame injector updates <img> replacements 3. prepare → cloneNode picks up current video frames, img.decode() awaits 4. micro-screenshot → forces browser to paint cloned elements 5. resolve → drawElementImage reads paint records, shader composites Key changes: - Remove `composition.videos.length === 0` gate from orchestrator - Split compositor resolve into prepare (clone) + resolve (shader) - Move onBeforeCapture before compositor prepare in frameCapture.ts - Await img.decode() on cloned data-URI images to prevent stale frames - Stop manipulating scene opacity in compositor (GL canvas overlay suffices) - Add gsap.set declaration for shader-transitions ambient types - Add video_missing_timing_attrs lint rule for <video> without id/data-start/data-end Performance: compositions with video now render at 7.5s (6 workers) instead of 2m38s on the layered path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(core): auto-inject data-start on video/audio so frame extraction works without explicit attrs The timing compiler now injects data-start="0" on <video> and <audio> elements that lack it. This makes discoverMediaFromBrowser() find the element (it queries video[data-start]), so the frame extraction pipeline activates automatically. Videos "just work" without requiring authors to add data-start, data-end, or id attributes. Also removes the video_missing_timing_attrs lint rule — the compiler handles the missing attributes automatically, so the lint rule would only false-positive. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(core): add data-hf-auto-start sentinel on auto-injected video timing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(producer): add discoverVideoVisibilityFromTimeline for runtime video discovery Seeks the GSAP timeline in Puppeteer to discover when each video's parent scene is visible (opacity > 0). Uses coarse sampling at 100ms steps followed by binary search refinement to frame-level precision (1/60s). Only processes videos with the data-hf-auto-start sentinel so author-specified timing is never overridden. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(producer): integrate runtime video visibility discovery into probe stage Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(producer): trigger browser probe for auto-start videos, remove debug logging The probe stage was skipping browser launch when composition duration was already known, which meant discoverVideoVisibilityFromTimeline never ran. Now needsBrowser also checks for data-hf-auto-start sentinel in compiled HTML. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(scripts): use mkdtempSync for smoke test work directory Replaces hardcoded /tmp/hf-page-side-smoke with a unique temp directory via mkdtempSync to resolve CodeQL "insecure temporary file" alert. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: format smoke test script Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Vai <vai@heygen.com>
188 lines
7.0 KiB
TypeScript
188 lines
7.0 KiB
TypeScript
import { describe, it, expect } from "vitest";
|
|
import {
|
|
compileTimingAttrs,
|
|
injectDurations,
|
|
extractResolvedMedia,
|
|
clampDurations,
|
|
} from "./timingCompiler.js";
|
|
|
|
describe("compileTimingAttrs", () => {
|
|
it("adds data-end when data-start and data-duration are present on a video", () => {
|
|
const html = '<video id="v1" src="a.mp4" data-start="2" data-duration="5">';
|
|
const { html: compiled, unresolved } = compileTimingAttrs(html);
|
|
|
|
expect(compiled).toContain('data-end="7"');
|
|
expect(compiled).toContain('data-has-audio="true"');
|
|
expect(unresolved).toHaveLength(0);
|
|
});
|
|
|
|
it("leaves data-end unchanged when already present", () => {
|
|
const html = '<video id="v1" src="a.mp4" data-start="0" data-end="3">';
|
|
const { html: compiled, unresolved } = compileTimingAttrs(html);
|
|
|
|
expect(compiled).toContain('data-end="3"');
|
|
expect(compiled).not.toContain("data-duration");
|
|
expect(unresolved).toHaveLength(0);
|
|
});
|
|
|
|
it("marks muted videos as visual-only audio sources", () => {
|
|
const html = '<video id="v1" src="a.mp4" data-start="0" data-duration="3" muted playsinline>';
|
|
const { html: compiled } = compileTimingAttrs(html);
|
|
|
|
expect(compiled).toContain('data-has-audio="false"');
|
|
expect(compiled).not.toContain('data-has-audio="true"');
|
|
});
|
|
|
|
it("marks video as unresolved when data-duration and data-end are missing", () => {
|
|
const html = '<video id="v1" src="a.mp4" data-start="1">';
|
|
const { unresolved } = compileTimingAttrs(html);
|
|
|
|
expect(unresolved).toHaveLength(1);
|
|
expect(unresolved[0].id).toBe("v1");
|
|
expect(unresolved[0].tagName).toBe("video");
|
|
expect(unresolved[0].start).toBe(1);
|
|
});
|
|
|
|
it("auto-assigns ids to id-less videos so unresolved duration resolution can target them", () => {
|
|
const html = '<video src="a.mp4" data-start="1">';
|
|
const { html: compiled, unresolved } = compileTimingAttrs(html);
|
|
|
|
expect(compiled).toContain('id="hf-video-0"');
|
|
expect(compiled).toContain('data-has-audio="true"');
|
|
expect(unresolved).toHaveLength(1);
|
|
expect(unresolved[0].id).toBe("hf-video-0");
|
|
expect(unresolved[0].tagName).toBe("video");
|
|
expect(unresolved[0].start).toBe(1);
|
|
});
|
|
|
|
it("auto-injects data-start='0' when missing so video is discoverable", () => {
|
|
const html = '<video src="clip.mp4" muted>';
|
|
const { html: compiled, unresolved } = compileTimingAttrs(html);
|
|
|
|
expect(compiled).toContain('data-start="0"');
|
|
expect(compiled).toContain('id="hf-video-0"');
|
|
expect(unresolved).toHaveLength(1);
|
|
expect(unresolved[0].start).toBe(0);
|
|
});
|
|
|
|
it("marks auto-injected data-start with data-hf-auto-start sentinel", () => {
|
|
const html = '<video src="clip.mp4" muted>';
|
|
const { html: compiled } = compileTimingAttrs(html);
|
|
|
|
expect(compiled).toContain('data-start="0"');
|
|
expect(compiled).toContain("data-hf-auto-start");
|
|
});
|
|
|
|
it("does not add data-hf-auto-start when author provides data-start", () => {
|
|
const html = '<video id="v1" src="clip.mp4" data-start="5" muted>';
|
|
const { html: compiled } = compileTimingAttrs(html);
|
|
|
|
expect(compiled).toContain('data-start="5"');
|
|
expect(compiled).not.toContain("data-hf-auto-start");
|
|
});
|
|
|
|
it("compiles audio tags the same as video (minus data-has-audio)", () => {
|
|
const html = '<audio id="a1" src="music.mp3" data-start="0" data-duration="10">';
|
|
const { html: compiled } = compileTimingAttrs(html);
|
|
|
|
expect(compiled).toContain('data-end="10"');
|
|
expect(compiled).not.toContain("data-has-audio");
|
|
});
|
|
|
|
it("detects unresolved div/section elements with data-start but no data-end", () => {
|
|
const html = '<div id="comp1" data-start="0" data-composition-src="comp.html">';
|
|
const { unresolved } = compileTimingAttrs(html);
|
|
|
|
expect(unresolved).toHaveLength(1);
|
|
expect(unresolved[0].id).toBe("comp1");
|
|
expect(unresolved[0].tagName).toBe("div");
|
|
expect(unresolved[0].compositionSrc).toBe("comp.html");
|
|
});
|
|
|
|
it("does not report div as unresolved when data-end is present", () => {
|
|
const html = '<div id="comp1" data-start="0" data-end="5">';
|
|
const { unresolved } = compileTimingAttrs(html);
|
|
|
|
expect(unresolved).toHaveLength(0);
|
|
});
|
|
});
|
|
|
|
describe("injectDurations", () => {
|
|
it("adds data-duration and data-end for resolved elements", () => {
|
|
const html = '<video id="v1" src="a.mp4" data-start="2">';
|
|
const result = injectDurations(html, [{ id: "v1", duration: 4 }]);
|
|
|
|
expect(result).toContain('data-duration="4"');
|
|
expect(result).toContain('data-end="6"');
|
|
});
|
|
|
|
it("injects durations for auto-assigned media ids", () => {
|
|
const { html, unresolved } = compileTimingAttrs('<video src="a.mp4" data-start="1">');
|
|
const result = injectDurations(html, [{ id: unresolved[0]!.id, duration: 4 }]);
|
|
|
|
expect(result).toContain('id="hf-video-0"');
|
|
expect(result).toContain('data-duration="4"');
|
|
expect(result).toContain('data-end="5"');
|
|
});
|
|
|
|
it("does not overwrite existing data-duration", () => {
|
|
const html = '<video id="v1" src="a.mp4" data-start="0" data-duration="3">';
|
|
const result = injectDurations(html, [{ id: "v1", duration: 10 }]);
|
|
|
|
// data-duration already present, should not be duplicated
|
|
expect(result).toContain('data-duration="3"');
|
|
});
|
|
});
|
|
|
|
describe("extractResolvedMedia", () => {
|
|
it("extracts video and audio elements with data-duration set", () => {
|
|
const html = [
|
|
'<video id="v1" src="vid.mp4" data-start="1" data-duration="5" data-media-start="0">',
|
|
'<audio id="a1" src="song.mp3" data-start="0" data-duration="10">',
|
|
'<video id="v2" src="other.mp4" data-start="0">', // no duration
|
|
].join("\n");
|
|
|
|
const resolved = extractResolvedMedia(html);
|
|
|
|
expect(resolved).toHaveLength(2);
|
|
expect(resolved[0].id).toBe("v1");
|
|
expect(resolved[0].tagName).toBe("video");
|
|
expect(resolved[0].duration).toBe(5);
|
|
expect(resolved[0].start).toBe(1);
|
|
expect(resolved[0].loop).toBe(false);
|
|
expect(resolved[1].id).toBe("a1");
|
|
expect(resolved[1].tagName).toBe("audio");
|
|
expect(resolved[1].duration).toBe(10);
|
|
});
|
|
|
|
it("marks looped media so render compilation can preserve display duration", () => {
|
|
const html = '<video id="v1" src="vid.webm" data-start="0" data-duration="4" loop>';
|
|
|
|
const resolved = extractResolvedMedia(html);
|
|
|
|
expect(resolved).toHaveLength(1);
|
|
expect(resolved[0]).toMatchObject({
|
|
id: "v1",
|
|
tagName: "video",
|
|
duration: 4,
|
|
loop: true,
|
|
});
|
|
});
|
|
|
|
it("skips elements with invalid durations", () => {
|
|
const html = '<video id="v1" src="a.mp4" data-start="0" data-duration="NaN">';
|
|
const resolved = extractResolvedMedia(html);
|
|
expect(resolved).toHaveLength(0);
|
|
});
|
|
});
|
|
|
|
describe("clampDurations", () => {
|
|
it("replaces data-duration and recomputes data-end", () => {
|
|
const html = '<video id="v1" src="a.mp4" data-start="2" data-duration="10" data-end="12">';
|
|
const result = clampDurations(html, [{ id: "v1", duration: 5 }]);
|
|
|
|
expect(result).toContain('data-duration="5"');
|
|
expect(result).toContain('data-end="7"');
|
|
});
|
|
});
|