mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 23:03:09 +00:00
perf(producer): hybrid layered/parallel path for SDR shader-transition renders (hf#732 PR 4/5) (#759)
## Summary PR 4 of 5 in the hf#732 decomposition stack. **This is where the bulk of the shader-transition speedup lives** (`~2×` verified — see Empirical validation below). Spreads per-frame DOM capture work across N DOM worker sessions and offloads the per-pixel shader-blend onto a `worker_threads` pool (the pool added in #758). ### Gating The hybrid path is gated by `shouldUseHybridLayeredPath`: - SDR content only — HDR raw-frame sources are fd-bound to one worker (per-worker `dup(fd)` is out of scope here). - `workerCount >= 2`. - Not every frame inside a transition window. When the gate trips, the hybrid loop spawns `workerCount - 1` extra DOM sessions, allocates per-worker scratch buffers, and partitions the frame range into contiguous slices via `distributeLayeredHybridFrameRanges`. Each worker walks its slice; transitions dispatch through the shader-blend pool (with inline fallback). A frame-reorder buffer fences the encoder. Pool teardown is guaranteed via try/finally on both the success and error paths. ### Structural change (heads-up to reviewers) `captureHdrStage.ts` on main was already 921 lines (over the project's 500-line ceiling). Adding the hybrid path on top would push it past 1100 and the local pre-commit hook refuses to stage files past 500. **PR 4 splits `captureHdrStage.ts` into 5 files**: - `captureHdrStage.ts` (orchestrator + cleanup invariants, 469 lines) - `captureHdrResources.ts` (HDR video extraction + image decode + dim probing) - `captureHdrFrameShared.ts` (gating predicates, partitioning, per-scene capture) - `captureHdrSequentialLoop.ts` (legacy single-session loop) - `captureHdrHybridLoop.ts` (new multi-worker path) No behavior change in any pre-existing code path: the sequential loop is byte-equivalent to the previous inline implementation (both consume `captureSceneIntoBuffer` from the shared module, so behavior parity is enforced structurally rather than by comment-keeping). `renderOrchestrator.ts` is intentionally unchanged — the stage computes its own worker budget via `calculateOptimalWorkers` rather than receiving it through the call signature. ## Stack Stacked on top of #758 (PR 3: shaderTransition pool). ## Test plan - [x] 14 new vitest tests in `captureHdrFrameShared.test.ts` pinning the hybrid gating predicate and the contiguous-chunking partitioner — all pass - [x] Producer typecheck clean - [x] oxlint clean ### Empirical validation Mark Witt fixture (Mac, Apple Silicon, hardware GPU, no beginframe): - Published CLI (pre-stack): 2m 12.2s - Cascade CLI (this stack): 1m 07.7s - **Measured speedup: 1.95× on Mac.** (Earlier "2.22×" wording was a projection from per-component micro-benchmarks; the empirical end-to-end number is 1.95× on the validated fixture.) Linux CI confirmation pending top-of-stack regression run. — Vai
This commit is contained in:
Vance Ingalls
@@ -1,5 +1,35 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// ── Worker entry path bootstrap (must run before any producer/engine load) ──
|
||||
// The hf#677 worker_threads pools (`pngDecodeBlitWorkerPool`,
|
||||
// `shaderTransitionWorkerPool`) live in the producer package and try to
|
||||
// resolve their worker entry by probing for sibling `.js` files next to
|
||||
// `import.meta.url`. When this CLI is bundled by tsup, the producer code is
|
||||
// inlined into `cli.js`, but `import.meta.url` resolves to the producer's
|
||||
// own dist path (NOT cli.js) on some module-graph layouts — so the sibling
|
||||
// probe lands in a directory that does not contain the bundled workers.
|
||||
// We emit the worker entries next to cli.js (see tsup.config.ts) and tell
|
||||
// the pools where to find them via the published env-var overrides. The
|
||||
// pools have an explicit `workerEntryPath` factory option as the canonical
|
||||
// API, but setting the env vars here covers every call site without having
|
||||
// to thread the path through the renderOrchestrator → captureHdrStage →
|
||||
// captureHdrHybridLoop chain.
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { existsSync } from "node:fs";
|
||||
|
||||
(() => {
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
const shader = join(here, "shaderTransitionWorker.js");
|
||||
const png = join(here, "pngDecodeBlitWorker.js");
|
||||
if (!process.env.HF_SHADER_WORKER_ENTRY && existsSync(shader)) {
|
||||
process.env.HF_SHADER_WORKER_ENTRY = shader;
|
||||
}
|
||||
if (!process.env.HF_PNG_DECODE_BLIT_WORKER_ENTRY && existsSync(png)) {
|
||||
process.env.HF_PNG_DECODE_BLIT_WORKER_ENTRY = png;
|
||||
}
|
||||
})();
|
||||
|
||||
// ── Fast-path exits ─────────────────────────────────────────────────────────
|
||||
// Check --version before importing anything heavy. This makes
|
||||
// `hyperframes --version` near-instant (~10ms vs ~80ms).
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
/**
|
||||
* Tests for the hf#732 hybrid layered-path gating + partitioning predicates.
|
||||
* These pin the contracts that the dispatcher in `captureHdrStage.ts`
|
||||
* depends on; both helpers are pure so the tests are cheap to maintain.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
distributeLayeredHybridFrameRanges,
|
||||
partitionTransitionFrames,
|
||||
shouldUseHybridLayeredPath,
|
||||
} from "./captureHdrFrameShared.js";
|
||||
|
||||
describe("shouldUseHybridLayeredPath", () => {
|
||||
it("returns false for HDR content (HDR raw-frame sources are fd-bound)", () => {
|
||||
expect(
|
||||
shouldUseHybridLayeredPath({
|
||||
hasHdrContent: true,
|
||||
transitionFramesCount: 30,
|
||||
totalFrames: 300,
|
||||
workerCount: 6,
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("returns false for single-worker budgets (sequential loop is already optimal)", () => {
|
||||
expect(
|
||||
shouldUseHybridLayeredPath({
|
||||
hasHdrContent: false,
|
||||
transitionFramesCount: 30,
|
||||
totalFrames: 300,
|
||||
workerCount: 1,
|
||||
}),
|
||||
).toBe(false);
|
||||
expect(
|
||||
shouldUseHybridLayeredPath({
|
||||
hasHdrContent: false,
|
||||
transitionFramesCount: 30,
|
||||
totalFrames: 300,
|
||||
workerCount: 0,
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("returns false when every frame is inside a transition window", () => {
|
||||
expect(
|
||||
shouldUseHybridLayeredPath({
|
||||
hasHdrContent: false,
|
||||
transitionFramesCount: 120,
|
||||
totalFrames: 120,
|
||||
workerCount: 6,
|
||||
}),
|
||||
).toBe(false);
|
||||
// Transition-frame count strictly greater than total is degenerate and
|
||||
// should still be rejected (parallel workers can't help).
|
||||
expect(
|
||||
shouldUseHybridLayeredPath({
|
||||
hasHdrContent: false,
|
||||
transitionFramesCount: 200,
|
||||
totalFrames: 120,
|
||||
workerCount: 6,
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("returns false for empty timelines", () => {
|
||||
expect(
|
||||
shouldUseHybridLayeredPath({
|
||||
hasHdrContent: false,
|
||||
transitionFramesCount: 0,
|
||||
totalFrames: 0,
|
||||
workerCount: 6,
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("returns true for SDR multi-worker with mixed transition/normal frames", () => {
|
||||
expect(
|
||||
shouldUseHybridLayeredPath({
|
||||
hasHdrContent: false,
|
||||
transitionFramesCount: 30,
|
||||
totalFrames: 300,
|
||||
workerCount: 6,
|
||||
}),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("returns true when there are no transitions at all (pure normal-frame parallelism)", () => {
|
||||
expect(
|
||||
shouldUseHybridLayeredPath({
|
||||
hasHdrContent: false,
|
||||
transitionFramesCount: 0,
|
||||
totalFrames: 300,
|
||||
workerCount: 6,
|
||||
}),
|
||||
).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("distributeLayeredHybridFrameRanges", () => {
|
||||
it("partitions [0, n) into contiguous slices that cover exactly the range", () => {
|
||||
const ranges = distributeLayeredHybridFrameRanges(300, 6);
|
||||
expect(ranges.length).toBe(6);
|
||||
expect(ranges[0]!.start).toBe(0);
|
||||
let prevEnd = 0;
|
||||
for (const r of ranges) {
|
||||
expect(r.start).toBe(prevEnd);
|
||||
expect(r.end).toBeGreaterThanOrEqual(r.start);
|
||||
expect(r.end).toBeLessThanOrEqual(300);
|
||||
prevEnd = r.end;
|
||||
}
|
||||
expect(prevEnd).toBe(300);
|
||||
});
|
||||
|
||||
it("does NOT pin all transition frames to one worker (contiguous chunking spreads them)", () => {
|
||||
// 300 frames, 6 workers → ~50 per worker. Transition window 60-69
|
||||
// (10 frames) falls in worker 1's slice [50, 100). The transition
|
||||
// frames are not all assigned to worker 0.
|
||||
const ranges = distributeLayeredHybridFrameRanges(300, 6);
|
||||
const worker0 = ranges[0]!;
|
||||
const transitionInWorker0 = [];
|
||||
for (let i = 60; i <= 69; i++) {
|
||||
if (i >= worker0.start && i < worker0.end) transitionInWorker0.push(i);
|
||||
}
|
||||
expect(transitionInWorker0.length).toBe(0);
|
||||
});
|
||||
|
||||
it("clamps non-positive workerCount to 1", () => {
|
||||
const ranges = distributeLayeredHybridFrameRanges(100, 0);
|
||||
expect(ranges.length).toBe(1);
|
||||
expect(ranges[0]).toEqual({ start: 0, end: 100 });
|
||||
const negative = distributeLayeredHybridFrameRanges(100, -5);
|
||||
expect(negative.length).toBe(1);
|
||||
});
|
||||
|
||||
it("assigns zero-width ranges to workers past the frame budget", () => {
|
||||
const ranges = distributeLayeredHybridFrameRanges(5, 10);
|
||||
expect(ranges.length).toBe(10);
|
||||
expect(ranges.slice(5).every((r) => r.start === r.end)).toBe(true);
|
||||
});
|
||||
|
||||
it("handles zero-frame inputs", () => {
|
||||
const ranges = distributeLayeredHybridFrameRanges(0, 4);
|
||||
expect(ranges.length).toBe(4);
|
||||
expect(ranges.every((r) => r.start === 0 && r.end === 0)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("partitionTransitionFrames", () => {
|
||||
it("returns a Set of frame indices that fall inside any transition window", () => {
|
||||
const ranges = [
|
||||
{ startFrame: 30, endFrame: 39 },
|
||||
{ startFrame: 120, endFrame: 125 },
|
||||
];
|
||||
const set = partitionTransitionFrames(ranges, 200);
|
||||
expect(set.size).toBe(10 + 6);
|
||||
expect(set.has(30)).toBe(true);
|
||||
expect(set.has(39)).toBe(true);
|
||||
expect(set.has(40)).toBe(false);
|
||||
expect(set.has(120)).toBe(true);
|
||||
expect(set.has(125)).toBe(true);
|
||||
expect(set.has(126)).toBe(false);
|
||||
});
|
||||
|
||||
it("clamps range endpoints to [0, totalFrames - 1]", () => {
|
||||
const ranges = [{ startFrame: -5, endFrame: 5 }];
|
||||
const set = partitionTransitionFrames(ranges, 3);
|
||||
expect(set.has(-1)).toBe(false);
|
||||
expect(set.has(0)).toBe(true);
|
||||
expect(set.has(2)).toBe(true);
|
||||
expect(set.has(3)).toBe(false);
|
||||
});
|
||||
|
||||
it("returns empty set for non-positive totalFrames", () => {
|
||||
expect(partitionTransitionFrames([{ startFrame: 0, endFrame: 5 }], 0).size).toBe(0);
|
||||
expect(partitionTransitionFrames([{ startFrame: 0, endFrame: 5 }], -1).size).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,351 @@
|
||||
/**
|
||||
* captureHdrFrameShared — shared helpers and types for the HDR
|
||||
* layered-composite frame loop (sequential + hybrid).
|
||||
*
|
||||
* Extracted from `captureHdrStage.ts` so the per-frame logic can live
|
||||
* under the project's 500-line file ceiling. The hybrid parallel path
|
||||
* (hf#732) adds a multi-DOM-worker dispatcher on top of the same per-
|
||||
* frame primitives the sequential loop uses, so the primitives are
|
||||
* centralized here.
|
||||
*/
|
||||
|
||||
import { rmSync } from "node:fs";
|
||||
import {
|
||||
type CaptureSession,
|
||||
type ElementStackingInfo,
|
||||
applyDomLayerMask,
|
||||
blitRgba8OverRgb48le,
|
||||
captureAlphaPng,
|
||||
decodePng,
|
||||
queryElementStacking,
|
||||
removeDomLayerMask,
|
||||
} from "@hyperframes/engine";
|
||||
import type { ProducerLogger } from "../../../logger.js";
|
||||
import {
|
||||
type HdrCompositeContext,
|
||||
type HdrPerfCollector,
|
||||
type HdrVideoFrameSource,
|
||||
type TransitionRange,
|
||||
addHdrTiming,
|
||||
blitHdrImageLayer,
|
||||
blitHdrVideoLayer,
|
||||
closeHdrVideoFrameSource,
|
||||
} from "../../renderOrchestrator.js";
|
||||
|
||||
// ─── Hybrid path gating + partitioning ─────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Decide whether the hybrid parallel layered path is safe to use. Returns
|
||||
* `false` (legacy sequential path) when:
|
||||
* - HDR content is present (HDR video raw-frame sources are fd-bound to a
|
||||
* single worker; sharing across workers is out of scope for hf#732).
|
||||
* - Every frame is inside a transition window (parallel workers buy
|
||||
* nothing; sequential loop is fine).
|
||||
* - workerCount <= 1.
|
||||
*
|
||||
* Exported so tests can pin the predicate without spinning up a render.
|
||||
*/
|
||||
export function shouldUseHybridLayeredPath(args: {
|
||||
hasHdrContent: boolean;
|
||||
transitionFramesCount: number;
|
||||
totalFrames: number;
|
||||
workerCount: number;
|
||||
}): boolean {
|
||||
if (args.hasHdrContent) return false;
|
||||
if (args.workerCount <= 1) return false;
|
||||
if (args.totalFrames <= 0) return false;
|
||||
if (args.transitionFramesCount >= args.totalFrames) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Distribute [0, totalFrames) across `workerCount` workers as roughly
|
||||
* equal contiguous slices. Transition-frame boundaries are NOT respected —
|
||||
* each worker runs both flavors of compositing on its own session.
|
||||
*/
|
||||
export function distributeLayeredHybridFrameRanges(
|
||||
totalFrames: number,
|
||||
workerCount: number,
|
||||
): Array<{ start: number; end: number }> {
|
||||
const safeWorkers = Math.max(1, workerCount);
|
||||
const safeFrames = Math.max(0, totalFrames);
|
||||
const framesPerWorker = Math.max(1, Math.ceil(safeFrames / safeWorkers));
|
||||
const ranges: Array<{ start: number; end: number }> = [];
|
||||
for (let w = 0; w < safeWorkers; w++) {
|
||||
const start = Math.min(safeFrames, w * framesPerWorker);
|
||||
const end = Math.min(safeFrames, start + framesPerWorker);
|
||||
ranges.push({ start, end });
|
||||
}
|
||||
return ranges;
|
||||
}
|
||||
|
||||
/** Build a Set of frame indices that fall inside any transition window. */
|
||||
export function partitionTransitionFrames(
|
||||
transitionRanges: ReadonlyArray<Pick<TransitionRange, "startFrame" | "endFrame">>,
|
||||
totalFrames: number,
|
||||
): Set<number> {
|
||||
const frames = new Set<number>();
|
||||
if (totalFrames <= 0) return frames;
|
||||
for (const range of transitionRanges) {
|
||||
const start = Math.max(0, range.startFrame);
|
||||
const end = Math.min(totalFrames - 1, range.endFrame);
|
||||
for (let i = start; i <= end; i++) frames.add(i);
|
||||
}
|
||||
return frames;
|
||||
}
|
||||
|
||||
// ─── Types ─────────────────────────────────────────────────────────────────
|
||||
|
||||
export interface LayeredTransitionBuffers {
|
||||
bufferA: Buffer;
|
||||
bufferB: Buffer;
|
||||
output: Buffer;
|
||||
}
|
||||
|
||||
// ─── Per-scene capture (shared by sequential transition + hybrid worker) ──
|
||||
|
||||
export interface CaptureSceneArgs {
|
||||
session: CaptureSession;
|
||||
sceneBuf: Buffer;
|
||||
sceneIds: Set<string>;
|
||||
stackingInfo: ElementStackingInfo[];
|
||||
time: number;
|
||||
width: number;
|
||||
height: number;
|
||||
nativeHdrIds: Set<string>;
|
||||
nativeHdrImageIds: Set<string>;
|
||||
beforeCaptureHook: CaptureSession["onBeforeCapture"];
|
||||
hdrCompositeCtx: HdrCompositeContext;
|
||||
compositeTransfer: "srgb" | "pq" | "hlg";
|
||||
hdrTargetTransfer: "pq" | "hlg" | undefined;
|
||||
hdrPerf: HdrPerfCollector | undefined;
|
||||
log: ProducerLogger;
|
||||
frameIdx: number;
|
||||
}
|
||||
|
||||
export async function captureSceneIntoBuffer(a: CaptureSceneArgs): Promise<void> {
|
||||
const {
|
||||
session,
|
||||
sceneBuf,
|
||||
sceneIds,
|
||||
stackingInfo,
|
||||
time,
|
||||
width,
|
||||
height,
|
||||
nativeHdrIds,
|
||||
nativeHdrImageIds,
|
||||
beforeCaptureHook,
|
||||
hdrCompositeCtx,
|
||||
compositeTransfer,
|
||||
hdrTargetTransfer,
|
||||
hdrPerf,
|
||||
log,
|
||||
frameIdx,
|
||||
} = a;
|
||||
let timingStart = Date.now();
|
||||
await session.page.evaluate((t: number) => {
|
||||
if (window.__hf && typeof window.__hf.seek === "function") window.__hf.seek(t);
|
||||
}, time);
|
||||
addHdrTiming(hdrPerf, "domLayerSeekMs", timingStart);
|
||||
if (beforeCaptureHook) {
|
||||
timingStart = Date.now();
|
||||
await beforeCaptureHook(session.page, time);
|
||||
addHdrTiming(hdrPerf, "domLayerInjectMs", timingStart);
|
||||
}
|
||||
for (const el of stackingInfo) {
|
||||
if (!el.isHdr || !sceneIds.has(el.id)) continue;
|
||||
if (nativeHdrImageIds.has(el.id)) {
|
||||
blitHdrImageLayer(
|
||||
sceneBuf,
|
||||
el,
|
||||
hdrCompositeCtx.hdrImageBuffers,
|
||||
hdrCompositeCtx.hdrImageTransferCache,
|
||||
width,
|
||||
height,
|
||||
log,
|
||||
hdrCompositeCtx.imageTransfers.get(el.id),
|
||||
hdrTargetTransfer,
|
||||
hdrPerf,
|
||||
);
|
||||
} else {
|
||||
blitHdrVideoLayer(
|
||||
sceneBuf,
|
||||
el,
|
||||
time,
|
||||
hdrCompositeCtx.fps,
|
||||
hdrCompositeCtx.hdrVideoFrameSources,
|
||||
hdrCompositeCtx.hdrVideoStartTimes,
|
||||
width,
|
||||
height,
|
||||
log,
|
||||
hdrCompositeCtx.videoTransfers.get(el.id),
|
||||
hdrTargetTransfer,
|
||||
hdrPerf,
|
||||
);
|
||||
}
|
||||
}
|
||||
const showIds = Array.from(sceneIds);
|
||||
const hideIds = stackingInfo
|
||||
.map((e) => e.id)
|
||||
.filter((id) => !sceneIds.has(id) || nativeHdrIds.has(id));
|
||||
if (hdrPerf) hdrPerf.domLayerCaptures += 1;
|
||||
timingStart = Date.now();
|
||||
await applyDomLayerMask(session.page, showIds, hideIds);
|
||||
addHdrTiming(hdrPerf, "domMaskApplyMs", timingStart);
|
||||
timingStart = Date.now();
|
||||
const domPng = await captureAlphaPng(session.page, width, height);
|
||||
addHdrTiming(hdrPerf, "domScreenshotMs", timingStart);
|
||||
timingStart = Date.now();
|
||||
await removeDomLayerMask(session.page, hideIds);
|
||||
addHdrTiming(hdrPerf, "domMaskRemoveMs", timingStart);
|
||||
try {
|
||||
timingStart = Date.now();
|
||||
const { data: domRgba } = decodePng(domPng);
|
||||
addHdrTiming(hdrPerf, "domPngDecodeMs", timingStart);
|
||||
timingStart = Date.now();
|
||||
blitRgba8OverRgb48le(domRgba, sceneBuf, width, height, compositeTransfer);
|
||||
addHdrTiming(hdrPerf, "domBlitMs", timingStart);
|
||||
} catch (err) {
|
||||
log.warn("DOM layer decode/blit failed; skipping overlay for transition scene", {
|
||||
frameIndex: frameIdx,
|
||||
sceneIds: Array.from(sceneIds),
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Per-frame transition capture (hybrid worker path) ─────────────────────
|
||||
|
||||
export interface CaptureTransitionOnWorkerArgs {
|
||||
session: CaptureSession;
|
||||
frameIdx: number;
|
||||
time: number;
|
||||
transition: TransitionRange;
|
||||
buffers: LayeredTransitionBuffers;
|
||||
nativeHdrIds: Set<string>;
|
||||
nativeHdrImageIds: Set<string>;
|
||||
sceneElements: Record<string, string[]>;
|
||||
hdrCompositeCtx: HdrCompositeContext;
|
||||
width: number;
|
||||
height: number;
|
||||
compositeTransfer: "srgb" | "pq" | "hlg";
|
||||
hdrTargetTransfer: "pq" | "hlg" | undefined;
|
||||
hdrPerf: HdrPerfCollector | undefined;
|
||||
log: ProducerLogger;
|
||||
}
|
||||
|
||||
export async function captureTransitionFrameOnWorker(
|
||||
a: CaptureTransitionOnWorkerArgs,
|
||||
): Promise<void> {
|
||||
const {
|
||||
session,
|
||||
frameIdx,
|
||||
time,
|
||||
transition,
|
||||
buffers,
|
||||
nativeHdrIds,
|
||||
nativeHdrImageIds,
|
||||
sceneElements,
|
||||
hdrCompositeCtx,
|
||||
width,
|
||||
height,
|
||||
compositeTransfer,
|
||||
hdrTargetTransfer,
|
||||
hdrPerf,
|
||||
log,
|
||||
} = a;
|
||||
const beforeCaptureHook = session.onBeforeCapture;
|
||||
if (hdrPerf) {
|
||||
hdrPerf.frames += 1;
|
||||
hdrPerf.transitionFrames += 1;
|
||||
}
|
||||
let timingStart = Date.now();
|
||||
await session.page.evaluate((t: number) => {
|
||||
if (window.__hf && typeof window.__hf.seek === "function") window.__hf.seek(t);
|
||||
}, time);
|
||||
addHdrTiming(hdrPerf, "frameSeekMs", timingStart);
|
||||
if (beforeCaptureHook) {
|
||||
timingStart = Date.now();
|
||||
await beforeCaptureHook(session.page, time);
|
||||
addHdrTiming(hdrPerf, "frameInjectMs", timingStart);
|
||||
}
|
||||
timingStart = Date.now();
|
||||
const stackingInfo = await queryElementStacking(session.page, nativeHdrIds);
|
||||
addHdrTiming(hdrPerf, "stackingQueryMs", timingStart);
|
||||
const sceneAIds = new Set(sceneElements[transition.fromScene] ?? []);
|
||||
const sceneBIds = new Set(sceneElements[transition.toScene] ?? []);
|
||||
buffers.bufferA.fill(0);
|
||||
buffers.bufferB.fill(0);
|
||||
for (const [sceneBuf, sceneIds] of [
|
||||
[buffers.bufferA, sceneAIds],
|
||||
[buffers.bufferB, sceneBIds],
|
||||
] as const) {
|
||||
await captureSceneIntoBuffer({
|
||||
session,
|
||||
sceneBuf: sceneBuf as Buffer,
|
||||
sceneIds,
|
||||
stackingInfo,
|
||||
time,
|
||||
width,
|
||||
height,
|
||||
nativeHdrIds,
|
||||
nativeHdrImageIds,
|
||||
beforeCaptureHook,
|
||||
hdrCompositeCtx,
|
||||
compositeTransfer,
|
||||
hdrTargetTransfer,
|
||||
hdrPerf,
|
||||
log,
|
||||
frameIdx,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// ─── HDR video raw-frame cleanup (sequential path only) ────────────────────
|
||||
|
||||
export function cleanupEndedHdrVideos(args: {
|
||||
time: number;
|
||||
activeTransition: TransitionRange | undefined;
|
||||
hdrVideoEndTimes: Map<string, number>;
|
||||
cleanedUpVideos: Set<string>;
|
||||
hdrVideoFrameSources: Map<string, HdrVideoFrameSource>;
|
||||
sceneElements: Record<string, string[]>;
|
||||
log: ProducerLogger;
|
||||
}): void {
|
||||
if (process.env.KEEP_TEMP === "1") return;
|
||||
const {
|
||||
time,
|
||||
activeTransition,
|
||||
hdrVideoEndTimes,
|
||||
cleanedUpVideos,
|
||||
hdrVideoFrameSources,
|
||||
sceneElements,
|
||||
log,
|
||||
} = args;
|
||||
for (const [videoId, endTime] of hdrVideoEndTimes) {
|
||||
if (time > endTime && !cleanedUpVideos.has(videoId)) {
|
||||
const stillNeeded =
|
||||
activeTransition &&
|
||||
(sceneElements[activeTransition.fromScene]?.includes(videoId) ||
|
||||
sceneElements[activeTransition.toScene]?.includes(videoId));
|
||||
if (!stillNeeded) {
|
||||
const frameSource = hdrVideoFrameSources.get(videoId);
|
||||
if (frameSource) {
|
||||
closeHdrVideoFrameSource(frameSource, log);
|
||||
try {
|
||||
rmSync(frameSource.dir, { recursive: true, force: true });
|
||||
} catch (err) {
|
||||
log.warn("Failed to clean up HDR raw frame directory", {
|
||||
videoId,
|
||||
frameDir: frameSource.dir,
|
||||
rawPath: frameSource.rawPath,
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
}
|
||||
hdrVideoFrameSources.delete(videoId);
|
||||
}
|
||||
cleanedUpVideos.add(videoId);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,290 @@
|
||||
/**
|
||||
* captureHdrHybridLoop — the hf#732 hybrid parallel layered path.
|
||||
*
|
||||
* Spreads per-frame DOM capture work across N DOM worker sessions (one
|
||||
* Chrome session per worker) and offloads the per-pixel shader-blend onto
|
||||
* a `worker_threads` pool. The encoder is fed via a frame-reorder buffer
|
||||
* so out-of-order worker completions still hit the muxer in ascending
|
||||
* index order.
|
||||
*
|
||||
* Restrictions enforced by `shouldUseHybridLayeredPath`:
|
||||
* - SDR only (HDR raw-frame sources are fd-bound to one worker).
|
||||
* - workerCount >= 2.
|
||||
* - Not every frame inside a transition window.
|
||||
*
|
||||
* Pool teardown is guaranteed in the outer `finally` regardless of which
|
||||
* path threw — see `runHybridLayeredFrameLoop`. The shader-blend pool is
|
||||
* spawned lazily (only when the composition has transitions); the DOM
|
||||
* worker sessions are always spawned.
|
||||
*/
|
||||
|
||||
import { writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
type CaptureOptions,
|
||||
type CaptureSession,
|
||||
type EngineConfig,
|
||||
type StreamingEncoder,
|
||||
type TransitionFn,
|
||||
TRANSITIONS,
|
||||
closeCaptureSession,
|
||||
createCaptureSession,
|
||||
createFrameReorderBuffer,
|
||||
crossfade,
|
||||
initTransparentBackground,
|
||||
initializeSession,
|
||||
queryElementStacking,
|
||||
} from "@hyperframes/engine";
|
||||
import type { FileServerHandle } from "../../fileServer.js";
|
||||
import type { ProducerLogger } from "../../../logger.js";
|
||||
import {
|
||||
type HdrCompositeContext,
|
||||
type HdrPerfCollector,
|
||||
type ProgressCallback,
|
||||
type RenderJob,
|
||||
type TransitionRange,
|
||||
addHdrTiming,
|
||||
compositeHdrFrame,
|
||||
} from "../../renderOrchestrator.js";
|
||||
import {
|
||||
type ShaderTransitionWorkerPool,
|
||||
createShaderTransitionWorkerPool,
|
||||
} from "../../shaderTransitionWorkerPool.js";
|
||||
import {
|
||||
type LayeredTransitionBuffers,
|
||||
captureTransitionFrameOnWorker,
|
||||
distributeLayeredHybridFrameRanges,
|
||||
partitionTransitionFrames,
|
||||
} from "./captureHdrFrameShared.js";
|
||||
import { updateJobStatus } from "../shared.js";
|
||||
|
||||
export interface HybridLoopInput {
|
||||
job: RenderJob;
|
||||
cfg: EngineConfig;
|
||||
log: ProducerLogger;
|
||||
framesDir: string;
|
||||
width: number;
|
||||
height: number;
|
||||
totalFrames: number;
|
||||
nativeHdrIds: Set<string>;
|
||||
nativeHdrImageIds: Set<string>;
|
||||
hdrCompositeCtx: HdrCompositeContext;
|
||||
hdrPerf: HdrPerfCollector | undefined;
|
||||
hdrEncoder: StreamingEncoder;
|
||||
domSession: CaptureSession;
|
||||
fileServer: FileServerHandle;
|
||||
buildCaptureOptions: () => CaptureOptions;
|
||||
createRenderVideoFrameInjector: () => Parameters<typeof createCaptureSession>[3];
|
||||
transitionRanges: TransitionRange[];
|
||||
sceneElements: Record<string, string[]>;
|
||||
compositeTransfer: "srgb" | "pq" | "hlg";
|
||||
hdrTargetTransfer: "pq" | "hlg" | undefined;
|
||||
workerCount: number;
|
||||
debugDumpEnabled: boolean;
|
||||
debugDumpDir: string | null;
|
||||
assertNotAborted: () => void;
|
||||
onProgress?: ProgressCallback;
|
||||
}
|
||||
|
||||
export async function runHybridLayeredFrameLoop(input: HybridLoopInput): Promise<void> {
|
||||
const {
|
||||
job,
|
||||
cfg,
|
||||
log,
|
||||
width,
|
||||
height,
|
||||
totalFrames,
|
||||
nativeHdrIds,
|
||||
nativeHdrImageIds,
|
||||
hdrCompositeCtx,
|
||||
hdrPerf,
|
||||
hdrEncoder,
|
||||
domSession,
|
||||
fileServer,
|
||||
buildCaptureOptions,
|
||||
createRenderVideoFrameInjector,
|
||||
transitionRanges,
|
||||
sceneElements,
|
||||
compositeTransfer,
|
||||
hdrTargetTransfer,
|
||||
workerCount,
|
||||
debugDumpEnabled,
|
||||
debugDumpDir,
|
||||
assertNotAborted,
|
||||
onProgress,
|
||||
} = input;
|
||||
const transitionFramesSet = partitionTransitionFrames(transitionRanges, totalFrames);
|
||||
const hasTransitions = transitionRanges.length > 0;
|
||||
const bufSize = width * height * 6;
|
||||
|
||||
const workerSessions: CaptureSession[] = [];
|
||||
let shaderPool: ShaderTransitionWorkerPool | null = null;
|
||||
try {
|
||||
for (let w = 0; w < workerCount - 1; w++) {
|
||||
const s = await createCaptureSession(
|
||||
fileServer.url,
|
||||
input.framesDir,
|
||||
buildCaptureOptions(),
|
||||
createRenderVideoFrameInjector(),
|
||||
cfg,
|
||||
);
|
||||
await initializeSession(s);
|
||||
await initTransparentBackground(s.page);
|
||||
workerSessions.push(s);
|
||||
}
|
||||
const sessions: CaptureSession[] = [domSession, ...workerSessions];
|
||||
const activeWorkerCount = sessions.length;
|
||||
if (hasTransitions) {
|
||||
try {
|
||||
shaderPool = await createShaderTransitionWorkerPool({ size: activeWorkerCount, log });
|
||||
} catch (err) {
|
||||
log.warn(
|
||||
"[Render] Failed to spawn shader-blend worker pool; falling back to inline shader blend",
|
||||
{ error: err instanceof Error ? err.message : String(err) },
|
||||
);
|
||||
shaderPool = null;
|
||||
}
|
||||
}
|
||||
|
||||
const workerCanvases: Buffer[] = sessions.map(() => Buffer.alloc(bufSize));
|
||||
const workerTransitionBuffers: Array<LayeredTransitionBuffers | null> = sessions.map(() =>
|
||||
hasTransitions
|
||||
? {
|
||||
bufferA: Buffer.alloc(bufSize),
|
||||
bufferB: Buffer.alloc(bufSize),
|
||||
output: Buffer.alloc(bufSize),
|
||||
}
|
||||
: null,
|
||||
);
|
||||
const workerRanges = distributeLayeredHybridFrameRanges(totalFrames, activeWorkerCount);
|
||||
let framesWritten = 0;
|
||||
const reorderBuffer = createFrameReorderBuffer(0, totalFrames);
|
||||
|
||||
const writeEncoded = async (frameIdx: number, buf: Buffer): Promise<void> => {
|
||||
await reorderBuffer.waitForFrame(frameIdx);
|
||||
const writeStart = Date.now();
|
||||
hdrEncoder.writeFrame(buf);
|
||||
addHdrTiming(hdrPerf, "encoderWriteMs", writeStart);
|
||||
reorderBuffer.advanceTo(frameIdx + 1);
|
||||
framesWritten += 1;
|
||||
job.framesRendered = framesWritten;
|
||||
if (framesWritten % 10 === 0 || framesWritten === totalFrames) {
|
||||
const frameProgress = framesWritten / totalFrames;
|
||||
updateJobStatus(
|
||||
job,
|
||||
"rendering",
|
||||
`Layered composite frame ${framesWritten}/${job.totalFrames}`,
|
||||
Math.round(25 + frameProgress * 55),
|
||||
onProgress,
|
||||
);
|
||||
}
|
||||
};
|
||||
const poolRef = shaderPool;
|
||||
|
||||
const workerTaskOf = async (w: number): Promise<void> => {
|
||||
const session = sessions[w];
|
||||
const canvas = workerCanvases[w];
|
||||
const range = workerRanges[w];
|
||||
const buffers = workerTransitionBuffers[w];
|
||||
if (!session || !canvas || !range) return;
|
||||
for (let i = range.start; i < range.end; i++) {
|
||||
assertNotAborted();
|
||||
const time = (i * job.config.fps.den) / job.config.fps.num;
|
||||
const activeTransition = transitionFramesSet.has(i)
|
||||
? transitionRanges.find((t) => i >= t.startFrame && i <= t.endFrame)
|
||||
: undefined;
|
||||
if (activeTransition && buffers) {
|
||||
await captureTransitionFrameOnWorker({
|
||||
session,
|
||||
frameIdx: i,
|
||||
time,
|
||||
transition: activeTransition,
|
||||
buffers,
|
||||
nativeHdrIds,
|
||||
nativeHdrImageIds,
|
||||
sceneElements,
|
||||
hdrCompositeCtx,
|
||||
width,
|
||||
height,
|
||||
compositeTransfer,
|
||||
hdrTargetTransfer,
|
||||
hdrPerf,
|
||||
log,
|
||||
});
|
||||
const progress =
|
||||
activeTransition.endFrame === activeTransition.startFrame
|
||||
? 1
|
||||
: (i - activeTransition.startFrame) /
|
||||
(activeTransition.endFrame - activeTransition.startFrame);
|
||||
if (poolRef) {
|
||||
const blendStart = Date.now();
|
||||
const result = await poolRef.run({
|
||||
shader: activeTransition.shader,
|
||||
bufferA: buffers.bufferA,
|
||||
bufferB: buffers.bufferB,
|
||||
output: buffers.output,
|
||||
width,
|
||||
height,
|
||||
progress,
|
||||
});
|
||||
buffers.bufferA = result.bufferA;
|
||||
buffers.bufferB = result.bufferB;
|
||||
buffers.output = result.output;
|
||||
addHdrTiming(hdrPerf, "transitionCompositeMs", blendStart);
|
||||
} else {
|
||||
const transitionFn: TransitionFn = TRANSITIONS[activeTransition.shader] ?? crossfade;
|
||||
const blendStart = Date.now();
|
||||
transitionFn(buffers.bufferA, buffers.bufferB, buffers.output, width, height, progress);
|
||||
addHdrTiming(hdrPerf, "transitionCompositeMs", blendStart);
|
||||
}
|
||||
await writeEncoded(i, buffers.output);
|
||||
} else {
|
||||
const beforeCaptureHook = session.onBeforeCapture;
|
||||
let timingStart = Date.now();
|
||||
await session.page.evaluate((t: number) => {
|
||||
if (window.__hf && typeof window.__hf.seek === "function") window.__hf.seek(t);
|
||||
}, time);
|
||||
addHdrTiming(hdrPerf, "frameSeekMs", timingStart);
|
||||
if (beforeCaptureHook) {
|
||||
timingStart = Date.now();
|
||||
await beforeCaptureHook(session.page, time);
|
||||
addHdrTiming(hdrPerf, "frameInjectMs", timingStart);
|
||||
}
|
||||
timingStart = Date.now();
|
||||
const stackingInfo = await queryElementStacking(session.page, nativeHdrIds);
|
||||
addHdrTiming(hdrPerf, "stackingQueryMs", timingStart);
|
||||
canvas.fill(0);
|
||||
// Rebind ctx to this worker's session for per-layer captures
|
||||
const wctx: HdrCompositeContext = { ...hdrCompositeCtx, domSession: session };
|
||||
timingStart = Date.now();
|
||||
await compositeHdrFrame(wctx, canvas, time, stackingInfo, undefined, i);
|
||||
addHdrTiming(hdrPerf, "normalCompositeMs", timingStart);
|
||||
if (debugDumpEnabled && debugDumpDir && i % 30 === 0) {
|
||||
writeFileSync(
|
||||
join(debugDumpDir, `frame_${String(i).padStart(4, "0")}_final_rgb48le.bin`),
|
||||
canvas,
|
||||
);
|
||||
}
|
||||
await writeEncoded(i, canvas);
|
||||
}
|
||||
}
|
||||
};
|
||||
await Promise.all(sessions.map((_, w) => workerTaskOf(w)));
|
||||
await reorderBuffer.waitForAllDone();
|
||||
} finally {
|
||||
for (const s of workerSessions) {
|
||||
await closeCaptureSession(s).catch((err) => {
|
||||
log.warn("Hybrid worker session close failed", {
|
||||
err: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
});
|
||||
}
|
||||
if (shaderPool) {
|
||||
await shaderPool.terminate().catch((err) => {
|
||||
log.warn("Shader-blend worker pool terminate failed", {
|
||||
err: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,289 @@
|
||||
/**
|
||||
* captureHdrResources — HDR resource setup helpers for the HDR layered
|
||||
* composite stage. Extracted from `captureHdrStage.ts` so the orchestrator
|
||||
* stays under the project's 500-line ceiling.
|
||||
*
|
||||
* Responsibilities (in order, called by `captureHdrStage.ts`):
|
||||
* 1. Probe per-element HDR extraction dimensions at the elements' own
|
||||
* start times (so GSAP-driven `data-start > 0` images don't fall out).
|
||||
* 2. Pre-extract every HDR video into a raw rgb48le frame file via a
|
||||
* single FFmpeg pass per video.
|
||||
* 3. Pre-decode every HDR image into rgb48le buffers, resampled to the
|
||||
* element's layout box using CSS `object-fit` / `object-position`
|
||||
* semantics.
|
||||
*
|
||||
* All helpers are SDR-content-safe: they no-op cleanly when no HDR layers
|
||||
* exist, leaving the caller with empty maps that the hot loop tolerates.
|
||||
*/
|
||||
|
||||
import { mkdirSync, openSync, readFileSync, statSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
type CaptureSession,
|
||||
decodePngToRgb48le,
|
||||
normalizeObjectFit,
|
||||
queryElementStacking,
|
||||
resampleRgb48leObjectFit,
|
||||
runFfmpeg,
|
||||
} from "@hyperframes/engine";
|
||||
import { fpsToFfmpegArg } from "@hyperframes/core";
|
||||
import type { ProducerLogger } from "../../../logger.js";
|
||||
import type {
|
||||
HdrDiagnostics,
|
||||
HdrImageBuffer,
|
||||
HdrVideoFrameSource,
|
||||
RenderJob,
|
||||
} from "../../renderOrchestrator.js";
|
||||
import type { CompositionMetadata } from "../shared.js";
|
||||
|
||||
export interface HdrResourcePrep {
|
||||
hdrVideoIds: string[];
|
||||
hdrVideoSrcPaths: Map<string, string>;
|
||||
hdrVideoStartTimes: Map<string, number>;
|
||||
hdrImageStartTimes: Map<string, number>;
|
||||
hdrExtractionDims: Map<string, { width: number; height: number }>;
|
||||
hdrImageFitInfo: Map<string, { fit: string; position: string }>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the maps the resource-extraction helpers below need. Pure data
|
||||
* transformation against `composition` + the native-HDR ID sets.
|
||||
*/
|
||||
export function planHdrResources(args: {
|
||||
composition: CompositionMetadata;
|
||||
nativeHdrVideoIds: Set<string>;
|
||||
nativeHdrImageIds: Set<string>;
|
||||
projectDir: string;
|
||||
compiledDir: string;
|
||||
existsSync: (p: string) => boolean;
|
||||
}): HdrResourcePrep {
|
||||
const { composition, nativeHdrVideoIds, nativeHdrImageIds, projectDir, compiledDir } = args;
|
||||
const hdrVideoIds = composition.videos
|
||||
.filter((v) => nativeHdrVideoIds.has(v.id))
|
||||
.map((v) => v.id);
|
||||
const hdrVideoSrcPaths = new Map<string, string>();
|
||||
for (const v of composition.videos) {
|
||||
if (!hdrVideoIds.includes(v.id)) continue;
|
||||
let srcPath = v.src;
|
||||
if (!srcPath.startsWith("/")) {
|
||||
const fromCompiled = join(compiledDir, srcPath);
|
||||
srcPath = args.existsSync(fromCompiled) ? fromCompiled : join(projectDir, srcPath);
|
||||
}
|
||||
hdrVideoSrcPaths.set(v.id, srcPath);
|
||||
}
|
||||
const hdrVideoStartTimes = new Map<string, number>();
|
||||
for (const v of composition.videos) {
|
||||
if (hdrVideoIds.includes(v.id)) hdrVideoStartTimes.set(v.id, v.start);
|
||||
}
|
||||
const hdrImageStartTimes = new Map<string, number>();
|
||||
for (const img of composition.images) {
|
||||
if (nativeHdrImageIds.has(img.id)) hdrImageStartTimes.set(img.id, img.start);
|
||||
}
|
||||
return {
|
||||
hdrVideoIds,
|
||||
hdrVideoSrcPaths,
|
||||
hdrVideoStartTimes,
|
||||
hdrImageStartTimes,
|
||||
hdrExtractionDims: new Map(),
|
||||
hdrImageFitInfo: new Map(),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Probe per-element layout dimensions at each unique start time, populating
|
||||
* `hdrExtractionDims` and `hdrImageFitInfo` in place. Also runs a fallback
|
||||
* probe for HDR images whose `data-start` instant reports zero dims (GSAP
|
||||
* `from` tweens animate the element in slightly later).
|
||||
*/
|
||||
export async function probeHdrExtractionDims(args: {
|
||||
domSession: CaptureSession;
|
||||
nativeHdrIds: Set<string>;
|
||||
nativeHdrImageIds: Set<string>;
|
||||
composition: CompositionMetadata;
|
||||
prep: HdrResourcePrep;
|
||||
}): Promise<void> {
|
||||
const { domSession, nativeHdrIds, nativeHdrImageIds, composition, prep } = args;
|
||||
const uniqueStartTimes = [
|
||||
...new Set([...prep.hdrVideoStartTimes.values(), ...prep.hdrImageStartTimes.values()]),
|
||||
].sort((a, b) => a - b);
|
||||
for (const seekTime of uniqueStartTimes) {
|
||||
await domSession.page.evaluate((t: number) => {
|
||||
if (window.__hf && typeof window.__hf.seek === "function") window.__hf.seek(t);
|
||||
}, seekTime);
|
||||
if (domSession.onBeforeCapture) {
|
||||
await domSession.onBeforeCapture(domSession.page, seekTime);
|
||||
}
|
||||
const stacking = await queryElementStacking(domSession.page, nativeHdrIds);
|
||||
for (const el of stacking) {
|
||||
if (
|
||||
el.isHdr &&
|
||||
el.layoutWidth > 0 &&
|
||||
el.layoutHeight > 0 &&
|
||||
!prep.hdrExtractionDims.has(el.id)
|
||||
) {
|
||||
prep.hdrExtractionDims.set(el.id, { width: el.layoutWidth, height: el.layoutHeight });
|
||||
}
|
||||
if (el.isHdr && nativeHdrImageIds.has(el.id) && !prep.hdrImageFitInfo.has(el.id)) {
|
||||
prep.hdrImageFitInfo.set(el.id, { fit: el.objectFit, position: el.objectPosition });
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const [imageId, startTime] of prep.hdrImageStartTimes) {
|
||||
if (prep.hdrExtractionDims.has(imageId)) continue;
|
||||
const img = composition.images.find((i) => i.id === imageId);
|
||||
if (!img) continue;
|
||||
const duration = img.end - img.start;
|
||||
const retryTime = startTime + Math.min(0.5, duration * 0.1);
|
||||
await domSession.page.evaluate((t: number) => {
|
||||
if (window.__hf && typeof window.__hf.seek === "function") window.__hf.seek(t);
|
||||
}, retryTime);
|
||||
if (domSession.onBeforeCapture) {
|
||||
await domSession.onBeforeCapture(domSession.page, retryTime);
|
||||
}
|
||||
const retryStacking = await queryElementStacking(domSession.page, nativeHdrIds);
|
||||
for (const el of retryStacking) {
|
||||
if (el.id === imageId && el.isHdr && el.layoutWidth > 0 && el.layoutHeight > 0) {
|
||||
prep.hdrExtractionDims.set(el.id, { width: el.layoutWidth, height: el.layoutHeight });
|
||||
if (!prep.hdrImageFitInfo.has(el.id)) {
|
||||
prep.hdrImageFitInfo.set(el.id, { fit: el.objectFit, position: el.objectPosition });
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract each HDR video into a raw rgb48le frame file via a single FFmpeg
|
||||
* pass per video, and open a file descriptor for each. Returns a map keyed
|
||||
* by video id. Caller owns lifecycle teardown (closing fds + rm-rf).
|
||||
*/
|
||||
export async function extractHdrVideoFrames(args: {
|
||||
job: RenderJob;
|
||||
log: ProducerLogger;
|
||||
framesDir: string;
|
||||
composition: CompositionMetadata;
|
||||
prep: HdrResourcePrep;
|
||||
width: number;
|
||||
height: number;
|
||||
abortSignal: AbortSignal | undefined;
|
||||
hdrDiagnostics: HdrDiagnostics;
|
||||
}): Promise<Map<string, HdrVideoFrameSource>> {
|
||||
const { job, log, framesDir, composition, prep, width, height, abortSignal, hdrDiagnostics } =
|
||||
args;
|
||||
const out = new Map<string, HdrVideoFrameSource>();
|
||||
for (const [videoId, srcPath] of prep.hdrVideoSrcPaths) {
|
||||
const video = composition.videos.find((v) => v.id === videoId);
|
||||
if (!video) continue;
|
||||
const frameDir = join(framesDir, `hdr_${videoId}`);
|
||||
mkdirSync(frameDir, { recursive: true });
|
||||
const duration = video.end - video.start;
|
||||
const dims = prep.hdrExtractionDims.get(videoId) ?? { width, height };
|
||||
const rawPath = join(frameDir, "frames.rgb48le");
|
||||
const ffmpegArgs = [
|
||||
"-ss",
|
||||
String(video.mediaStart),
|
||||
"-i",
|
||||
srcPath,
|
||||
"-t",
|
||||
String(duration),
|
||||
"-r",
|
||||
fpsToFfmpegArg(job.config.fps),
|
||||
"-vf",
|
||||
`scale=${dims.width}:${dims.height}:force_original_aspect_ratio=increase,crop=${dims.width}:${dims.height}`,
|
||||
"-pix_fmt",
|
||||
"rgb48le",
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-y",
|
||||
rawPath,
|
||||
];
|
||||
const result = await runFfmpeg(ffmpegArgs, { signal: abortSignal });
|
||||
if (!result.success) {
|
||||
hdrDiagnostics.videoExtractionFailures += 1;
|
||||
log.error("HDR frame pre-extraction failed; aborting render", {
|
||||
videoId,
|
||||
srcPath,
|
||||
stderr: result.stderr.slice(-400),
|
||||
});
|
||||
throw new Error(
|
||||
`HDR frame extraction failed for video "${videoId}". ` +
|
||||
`Aborting render to avoid shipping black HDR layers.`,
|
||||
);
|
||||
}
|
||||
const frameSize = dims.width * dims.height * 6;
|
||||
const frameCount = Math.floor(statSync(rawPath).size / frameSize);
|
||||
if (frameCount < 1) {
|
||||
hdrDiagnostics.videoExtractionFailures += 1;
|
||||
throw new Error(
|
||||
`HDR frame extraction produced no frames for video "${videoId}". ` +
|
||||
`Aborting render to avoid shipping black HDR layers.`,
|
||||
);
|
||||
}
|
||||
out.set(videoId, {
|
||||
dir: frameDir,
|
||||
rawPath,
|
||||
fd: openSync(rawPath, "r"),
|
||||
width: dims.width,
|
||||
height: dims.height,
|
||||
frameSize,
|
||||
frameCount,
|
||||
scratch: Buffer.allocUnsafe(frameSize),
|
||||
});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode each HDR image into an rgb48le buffer, resampling to the element's
|
||||
* layout box if known. Failures abort the render to avoid shipping missing
|
||||
* layers (the hot loop has no fallback for a missing HDR layer that the
|
||||
* composition expects to see).
|
||||
*/
|
||||
export function decodeHdrImageBuffers(args: {
|
||||
log: ProducerLogger;
|
||||
hdrImageSrcPaths: Map<string, string>;
|
||||
prep: HdrResourcePrep;
|
||||
hdrDiagnostics: HdrDiagnostics;
|
||||
}): Map<string, HdrImageBuffer> {
|
||||
const { log, hdrImageSrcPaths, prep, hdrDiagnostics } = args;
|
||||
const out = new Map<string, HdrImageBuffer>();
|
||||
for (const [imageId, srcPath] of hdrImageSrcPaths) {
|
||||
try {
|
||||
const decoded = decodePngToRgb48le(readFileSync(srcPath));
|
||||
const layout = prep.hdrExtractionDims.get(imageId);
|
||||
const fitInfo = prep.hdrImageFitInfo.get(imageId);
|
||||
if (layout && (layout.width !== decoded.width || layout.height !== decoded.height)) {
|
||||
const fit = normalizeObjectFit(fitInfo?.fit);
|
||||
const resampled = resampleRgb48leObjectFit(
|
||||
decoded.data,
|
||||
decoded.width,
|
||||
decoded.height,
|
||||
layout.width,
|
||||
layout.height,
|
||||
fit,
|
||||
fitInfo?.position,
|
||||
);
|
||||
out.set(imageId, { data: resampled, width: layout.width, height: layout.height });
|
||||
} else {
|
||||
out.set(imageId, {
|
||||
data: Buffer.from(decoded.data),
|
||||
width: decoded.width,
|
||||
height: decoded.height,
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
hdrDiagnostics.imageDecodeFailures += 1;
|
||||
log.error("HDR image decode failed; aborting render", {
|
||||
imageId,
|
||||
srcPath,
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
throw new Error(
|
||||
`HDR image decode failed for image "${imageId}". ` +
|
||||
`Aborting render to avoid shipping missing HDR image layers.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
@@ -0,0 +1,228 @@
|
||||
/**
|
||||
* captureHdrSequentialLoop — the legacy sequential HDR / shader-transition
|
||||
* frame loop. Single DOM session, single-threaded per-frame work. Used by:
|
||||
*
|
||||
* - HDR renders (HDR video raw-frame sources are fd-bound to one worker)
|
||||
* - single-worker SDR renders
|
||||
* - the all-transition edge case (parallel workers buy nothing there)
|
||||
*
|
||||
* Sister of `captureHdrHybridLoop.ts`. Both consume the same per-frame
|
||||
* primitives from `captureHdrFrameShared.ts` so behavior parity is enforced
|
||||
* by reusing the helpers rather than by careful comment-keeping.
|
||||
*/
|
||||
|
||||
import { writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
type CaptureSession,
|
||||
type StreamingEncoder,
|
||||
type TransitionFn,
|
||||
TRANSITIONS,
|
||||
crossfade,
|
||||
queryElementStacking,
|
||||
} from "@hyperframes/engine";
|
||||
import type { ProducerLogger } from "../../../logger.js";
|
||||
import {
|
||||
type HdrCompositeContext,
|
||||
type HdrPerfCollector,
|
||||
type ProgressCallback,
|
||||
type RenderJob,
|
||||
type TransitionRange,
|
||||
addHdrTiming,
|
||||
compositeHdrFrame,
|
||||
} from "../../renderOrchestrator.js";
|
||||
import {
|
||||
captureSceneIntoBuffer,
|
||||
cleanupEndedHdrVideos,
|
||||
type LayeredTransitionBuffers,
|
||||
} from "./captureHdrFrameShared.js";
|
||||
import { updateJobStatus } from "../shared.js";
|
||||
|
||||
export interface SequentialLoopInput {
|
||||
job: RenderJob;
|
||||
log: ProducerLogger;
|
||||
width: number;
|
||||
height: number;
|
||||
totalFrames: number;
|
||||
nativeHdrIds: Set<string>;
|
||||
nativeHdrImageIds: Set<string>;
|
||||
hdrCompositeCtx: HdrCompositeContext;
|
||||
hdrPerf: HdrPerfCollector | undefined;
|
||||
hdrEncoder: StreamingEncoder;
|
||||
domSession: CaptureSession;
|
||||
transitionRanges: TransitionRange[];
|
||||
sceneElements: Record<string, string[]>;
|
||||
compositeTransfer: "srgb" | "pq" | "hlg";
|
||||
hdrTargetTransfer: "pq" | "hlg" | undefined;
|
||||
hdrVideoEndTimes: Map<string, number>;
|
||||
cleanedUpVideos: Set<string>;
|
||||
hdrVideoFrameSources: Map<string, import("../../renderOrchestrator.js").HdrVideoFrameSource>;
|
||||
debugDumpEnabled: boolean;
|
||||
debugDumpDir: string | null;
|
||||
assertNotAborted: () => void;
|
||||
onProgress?: ProgressCallback;
|
||||
}
|
||||
|
||||
export async function runSequentialLayeredFrameLoop(input: SequentialLoopInput): Promise<void> {
|
||||
const {
|
||||
job,
|
||||
log,
|
||||
width,
|
||||
height,
|
||||
totalFrames,
|
||||
nativeHdrIds,
|
||||
nativeHdrImageIds,
|
||||
hdrCompositeCtx,
|
||||
hdrPerf,
|
||||
hdrEncoder,
|
||||
domSession,
|
||||
transitionRanges,
|
||||
sceneElements,
|
||||
compositeTransfer,
|
||||
hdrTargetTransfer,
|
||||
hdrVideoEndTimes,
|
||||
cleanedUpVideos,
|
||||
hdrVideoFrameSources,
|
||||
debugDumpEnabled,
|
||||
debugDumpDir,
|
||||
assertNotAborted,
|
||||
onProgress,
|
||||
} = input;
|
||||
const beforeCaptureHook = domSession.onBeforeCapture;
|
||||
const bufSize = width * height * 6;
|
||||
const hasTransitions = transitionRanges.length > 0;
|
||||
const transitionBuffers: LayeredTransitionBuffers | null = hasTransitions
|
||||
? {
|
||||
bufferA: Buffer.alloc(bufSize),
|
||||
bufferB: Buffer.alloc(bufSize),
|
||||
output: Buffer.alloc(bufSize),
|
||||
}
|
||||
: null;
|
||||
const normalCanvas = Buffer.alloc(bufSize);
|
||||
|
||||
for (let i = 0; i < totalFrames; i++) {
|
||||
assertNotAborted();
|
||||
const time = (i * job.config.fps.den) / job.config.fps.num;
|
||||
if (hdrPerf) hdrPerf.frames += 1;
|
||||
|
||||
let timingStart = Date.now();
|
||||
await domSession.page.evaluate((t: number) => {
|
||||
if (window.__hf && typeof window.__hf.seek === "function") window.__hf.seek(t);
|
||||
}, time);
|
||||
addHdrTiming(hdrPerf, "frameSeekMs", timingStart);
|
||||
|
||||
if (beforeCaptureHook) {
|
||||
timingStart = Date.now();
|
||||
await beforeCaptureHook(domSession.page, time);
|
||||
addHdrTiming(hdrPerf, "frameInjectMs", timingStart);
|
||||
}
|
||||
timingStart = Date.now();
|
||||
const stackingInfo = await queryElementStacking(domSession.page, nativeHdrIds);
|
||||
addHdrTiming(hdrPerf, "stackingQueryMs", timingStart);
|
||||
const activeTransition = transitionRanges.find((t) => i >= t.startFrame && i <= t.endFrame);
|
||||
|
||||
if (i % 30 === 0 && (log.isLevelEnabled?.("debug") ?? true)) {
|
||||
const hdrEl = stackingInfo.find((e) => e.isHdr);
|
||||
log.debug("[Render] HDR layer composite frame", {
|
||||
frame: i,
|
||||
time: time.toFixed(2),
|
||||
hdrElement: hdrEl ? { z: hdrEl.zIndex, visible: hdrEl.visible, width: hdrEl.width } : null,
|
||||
stackingCount: stackingInfo.length,
|
||||
activeTransition: activeTransition?.shader,
|
||||
});
|
||||
}
|
||||
|
||||
if (activeTransition && transitionBuffers) {
|
||||
if (hdrPerf) hdrPerf.transitionFrames += 1;
|
||||
const transitionTimingStart = Date.now();
|
||||
const progress =
|
||||
activeTransition.endFrame === activeTransition.startFrame
|
||||
? 1
|
||||
: (i - activeTransition.startFrame) /
|
||||
(activeTransition.endFrame - activeTransition.startFrame);
|
||||
const sceneAIds = new Set(sceneElements[activeTransition.fromScene] ?? []);
|
||||
const sceneBIds = new Set(sceneElements[activeTransition.toScene] ?? []);
|
||||
timingStart = Date.now();
|
||||
transitionBuffers.bufferA.fill(0);
|
||||
transitionBuffers.bufferB.fill(0);
|
||||
addHdrTiming(hdrPerf, "canvasClearMs", timingStart);
|
||||
|
||||
for (const [sceneBuf, sceneIds] of [
|
||||
[transitionBuffers.bufferA, sceneAIds],
|
||||
[transitionBuffers.bufferB, sceneBIds],
|
||||
] as const) {
|
||||
assertNotAborted();
|
||||
await captureSceneIntoBuffer({
|
||||
session: domSession,
|
||||
sceneBuf: sceneBuf as Buffer,
|
||||
sceneIds,
|
||||
stackingInfo,
|
||||
time,
|
||||
width,
|
||||
height,
|
||||
nativeHdrIds,
|
||||
nativeHdrImageIds,
|
||||
beforeCaptureHook,
|
||||
hdrCompositeCtx,
|
||||
compositeTransfer,
|
||||
hdrTargetTransfer,
|
||||
hdrPerf,
|
||||
log,
|
||||
frameIdx: i,
|
||||
});
|
||||
}
|
||||
|
||||
const transitionFn: TransitionFn = TRANSITIONS[activeTransition.shader] ?? crossfade;
|
||||
transitionFn(
|
||||
transitionBuffers.bufferA,
|
||||
transitionBuffers.bufferB,
|
||||
transitionBuffers.output,
|
||||
width,
|
||||
height,
|
||||
progress,
|
||||
);
|
||||
addHdrTiming(hdrPerf, "transitionCompositeMs", transitionTimingStart);
|
||||
timingStart = Date.now();
|
||||
hdrEncoder.writeFrame(transitionBuffers.output);
|
||||
addHdrTiming(hdrPerf, "encoderWriteMs", timingStart);
|
||||
} else {
|
||||
if (hdrPerf) hdrPerf.normalFrames += 1;
|
||||
timingStart = Date.now();
|
||||
normalCanvas.fill(0);
|
||||
addHdrTiming(hdrPerf, "canvasClearMs", timingStart);
|
||||
timingStart = Date.now();
|
||||
await compositeHdrFrame(hdrCompositeCtx, normalCanvas, time, stackingInfo, undefined, i);
|
||||
addHdrTiming(hdrPerf, "normalCompositeMs", timingStart);
|
||||
if (debugDumpEnabled && debugDumpDir && i % 30 === 0) {
|
||||
writeFileSync(
|
||||
join(debugDumpDir, `frame_${String(i).padStart(4, "0")}_final_rgb48le.bin`),
|
||||
normalCanvas,
|
||||
);
|
||||
}
|
||||
timingStart = Date.now();
|
||||
hdrEncoder.writeFrame(normalCanvas);
|
||||
addHdrTiming(hdrPerf, "encoderWriteMs", timingStart);
|
||||
}
|
||||
|
||||
cleanupEndedHdrVideos({
|
||||
time,
|
||||
activeTransition,
|
||||
hdrVideoEndTimes,
|
||||
cleanedUpVideos,
|
||||
hdrVideoFrameSources,
|
||||
sceneElements,
|
||||
log,
|
||||
});
|
||||
job.framesRendered = i + 1;
|
||||
if ((i + 1) % 10 === 0 || i + 1 === totalFrames) {
|
||||
const frameProgress = (i + 1) / totalFrames;
|
||||
updateJobStatus(
|
||||
job,
|
||||
"rendering",
|
||||
`Layered composite frame ${i + 1}/${job.totalFrames}`,
|
||||
Math.round(25 + frameProgress * 55),
|
||||
onProgress,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -10,8 +10,10 @@
|
||||
* - Decodes 16-bit HDR PNGs once and blits them as image layers.
|
||||
* - Queries Chrome z-order at layout-change boundaries and groups
|
||||
* elements into DOM / HDR video / HDR image layers.
|
||||
* - Composites bottom-to-top in Node memory, writing rgb48le buffers
|
||||
* to the encoder's stdin.
|
||||
* - Dispatches per-frame work to either the sequential layered loop
|
||||
* (HDR-content, single-worker, all-transition edge cases) or the
|
||||
* hybrid parallel loop introduced in hf#732 (multi-worker SDR with
|
||||
* `worker_threads`-pool shader blend).
|
||||
*
|
||||
* Cleanup invariants the design doc explicitly flags as risky —
|
||||
* preserved verbatim from the in-process renderer:
|
||||
@@ -27,21 +29,13 @@
|
||||
* pass `forceScreenshot: true` for the layered branch as a contract
|
||||
* check.
|
||||
*
|
||||
* Known follow-up: same runtime import cycle pattern as the other
|
||||
* capture stages — the stage imports HDR helpers from
|
||||
* `renderOrchestrator.ts` (runtime), which imports the stage back.
|
||||
* Safe at runtime; a future PR will consolidate these helpers.
|
||||
* Resource setup (HDR video extraction, image decode, dim probing) lives
|
||||
* in `captureHdrResources.ts`; per-frame work lives in
|
||||
* `captureHdrSequentialLoop.ts` and `captureHdrHybridLoop.ts`. Shared
|
||||
* primitives across both loops live in `captureHdrFrameShared.ts`.
|
||||
*/
|
||||
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
openSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
statSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { existsSync, mkdirSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
type BeforeCaptureHook,
|
||||
@@ -49,49 +43,41 @@ import {
|
||||
type EngineConfig,
|
||||
type HdrTransfer,
|
||||
type StreamingEncoder,
|
||||
type TransitionFn,
|
||||
TRANSITIONS,
|
||||
applyDomLayerMask,
|
||||
blitRgba8OverRgb48le,
|
||||
captureAlphaPng,
|
||||
calculateOptimalWorkers,
|
||||
closeCaptureSession,
|
||||
createCaptureSession,
|
||||
crossfade,
|
||||
decodePng,
|
||||
decodePngToRgb48le,
|
||||
getEncoderPreset,
|
||||
initTransparentBackground,
|
||||
initializeSession,
|
||||
normalizeObjectFit,
|
||||
queryElementStacking,
|
||||
removeDomLayerMask,
|
||||
resampleRgb48leObjectFit,
|
||||
runFfmpeg,
|
||||
spawnStreamingEncoder,
|
||||
} from "@hyperframes/engine";
|
||||
import { fpsToFfmpegArg, fpsToNumber } from "@hyperframes/core";
|
||||
import { fpsToNumber } from "@hyperframes/core";
|
||||
import type { FileServerHandle } from "../../fileServer.js";
|
||||
import type { ProducerLogger } from "../../../logger.js";
|
||||
import { createHdrImageTransferCache } from "../../hdrImageTransferCache.js";
|
||||
import {
|
||||
type HdrCompositeContext,
|
||||
type HdrDiagnostics,
|
||||
type HdrImageBuffer,
|
||||
type HdrPerfCollector,
|
||||
type HdrTransitionMeta,
|
||||
type HdrVideoFrameSource,
|
||||
type ProgressCallback,
|
||||
type RenderJob,
|
||||
type TransitionRange,
|
||||
addHdrTiming,
|
||||
blitHdrImageLayer,
|
||||
blitHdrVideoLayer,
|
||||
closeHdrVideoFrameSource,
|
||||
compositeHdrFrame,
|
||||
createHdrPerfCollector,
|
||||
resolveCompositeTransfer,
|
||||
} from "../../renderOrchestrator.js";
|
||||
import { updateJobStatus, type CompositionMetadata } from "../shared.js";
|
||||
import type { CompositionMetadata } from "../shared.js";
|
||||
import {
|
||||
decodeHdrImageBuffers,
|
||||
extractHdrVideoFrames,
|
||||
planHdrResources,
|
||||
probeHdrExtractionDims,
|
||||
} from "./captureHdrResources.js";
|
||||
import { partitionTransitionFrames, shouldUseHybridLayeredPath } from "./captureHdrFrameShared.js";
|
||||
import { runSequentialLayeredFrameLoop } from "./captureHdrSequentialLoop.js";
|
||||
import { runHybridLayeredFrameLoop } from "./captureHdrHybridLoop.js";
|
||||
|
||||
export interface CaptureHdrStageInput {
|
||||
job: RenderJob;
|
||||
@@ -135,6 +121,13 @@ export interface CaptureHdrStageInput {
|
||||
/** Mutated in place (counters incremented). */
|
||||
hdrDiagnostics: HdrDiagnostics;
|
||||
|
||||
/**
|
||||
* Worker budget for the hybrid layered path. Only consulted when the
|
||||
* gating predicate (`shouldUseHybridLayeredPath`) returns true. The
|
||||
* sequential loop always runs on a single DOM session.
|
||||
*/
|
||||
workerCount?: number;
|
||||
|
||||
abortSignal: AbortSignal | undefined;
|
||||
assertNotAborted: () => void;
|
||||
onProgress?: ProgressCallback;
|
||||
@@ -179,6 +172,7 @@ export async function runCaptureHdrStage(
|
||||
buildCaptureOptions,
|
||||
createRenderVideoFrameInjector,
|
||||
hdrDiagnostics,
|
||||
workerCount,
|
||||
abortSignal,
|
||||
assertNotAborted,
|
||||
onProgress,
|
||||
@@ -192,12 +186,8 @@ export async function runCaptureHdrStage(
|
||||
|
||||
const stageStart = Date.now();
|
||||
let lastBrowserConsole: string[] = [];
|
||||
let hdrPerf: HdrPerfCollector | undefined;
|
||||
let captureDurationMs = 0;
|
||||
let encodeMs = 0;
|
||||
// Recomputed here so the stage owns its own scope; matches the sequencer's
|
||||
// `nativeHdrIds = new Set([...nativeHdrVideoIds, ...nativeHdrImageIds])`
|
||||
// before the `if (useLayeredComposite)` branch.
|
||||
const nativeHdrIds = new Set([...nativeHdrVideoIds, ...nativeHdrImageIds]);
|
||||
|
||||
log.info(
|
||||
@@ -205,7 +195,7 @@ export async function runCaptureHdrStage(
|
||||
? "[Render] HDR layered composite: z-ordered DOM + native HDR video/image layers"
|
||||
: "[Render] Shader transition composite: z-ordered SDR DOM layers",
|
||||
);
|
||||
hdrPerf = createHdrPerfCollector();
|
||||
const hdrPerf: HdrPerfCollector = createHdrPerfCollector();
|
||||
|
||||
// Layered compositing relies on captureAlphaPng (Page.captureScreenshot
|
||||
// with a transparent background) for DOM layers. That CDP call hangs
|
||||
@@ -220,41 +210,19 @@ export async function runCaptureHdrStage(
|
||||
// caller config.)
|
||||
const hdrCfg: EngineConfig = { ...cfg, forceScreenshot: true };
|
||||
|
||||
// Use NATIVE HDR IDs (probed before SDR→HDR conversion) so only originally-HDR
|
||||
// videos are hidden + extracted natively. SDR videos stay in the DOM screenshot
|
||||
// (injected via the frame injector) and get sRGB→HLG conversion in the blit.
|
||||
// HDR images don't need an equivalent array — they're keyed off
|
||||
// `nativeHdrImageIds` directly (decoded once into `hdrImageBuffers` and blitted
|
||||
// by `blitHdrImageLayer`, with the DOM mask hiding them via `nativeHdrIds`).
|
||||
const hdrVideoIds = composition.videos
|
||||
.filter((v) => nativeHdrVideoIds.has(v.id))
|
||||
.map((v) => v.id);
|
||||
|
||||
// Resolve HDR video source paths
|
||||
const hdrVideoSrcPaths = new Map<string, string>();
|
||||
for (const v of composition.videos) {
|
||||
if (!hdrVideoIds.includes(v.id)) continue;
|
||||
let srcPath = v.src;
|
||||
if (!srcPath.startsWith("/")) {
|
||||
const fromCompiled = join(compiledDir, srcPath);
|
||||
srcPath = existsSync(fromCompiled) ? fromCompiled : join(projectDir, srcPath);
|
||||
}
|
||||
hdrVideoSrcPaths.set(v.id, srcPath);
|
||||
}
|
||||
|
||||
// Launch headless Chrome for DOM capture.
|
||||
// Pass the video frame injector so SDR videos are rendered correctly in Chrome.
|
||||
// HDR videos get injected too but are masked out via applyDomLayerMask
|
||||
// before each DOM screenshot — only the native FFmpeg-extracted HLG
|
||||
// frames are used for HDR pixels.
|
||||
if (!fileServer) throw new Error("fileServer must be initialized before HDR compositing");
|
||||
// Native HDR videos (e.g. HEVC) may be undecodable by Chrome on the
|
||||
// current platform — Linux headless-shell ships without HEVC support.
|
||||
// Their pixels come from out-of-band ffmpeg extraction, so the DOM
|
||||
// `<video>` element is only kept around for layout. Skip the per-page
|
||||
// readiness wait for these IDs; otherwise the render hangs 45s and
|
||||
// throws "video metadata not ready" even though we never asked the
|
||||
// browser to decode the video.
|
||||
|
||||
// Plan HDR resources (videos to extract, images to decode, layout-probe
|
||||
// start times) up-front — pure data transformation, no IO yet.
|
||||
const prep = planHdrResources({
|
||||
composition,
|
||||
nativeHdrVideoIds,
|
||||
nativeHdrImageIds,
|
||||
projectDir,
|
||||
compiledDir,
|
||||
existsSync,
|
||||
});
|
||||
|
||||
const domSession = await createCaptureSession(
|
||||
fileServer.url,
|
||||
framesDir,
|
||||
@@ -262,36 +230,21 @@ export async function runCaptureHdrStage(
|
||||
createRenderVideoFrameInjector(),
|
||||
hdrCfg,
|
||||
);
|
||||
// Track lifecycle of resources spawned during HDR rendering so the
|
||||
// outer finally block can defensively reclaim anything that wasn't
|
||||
// cleaned up via the success path. Both closeCaptureSession and
|
||||
// StreamingEncoder.close() are idempotent, but the flags let us avoid
|
||||
// redundant work and make the intent explicit.
|
||||
|
||||
let hdrEncoder: StreamingEncoder | null = null;
|
||||
let hdrEncoderClosed = false;
|
||||
let domSessionClosed = false;
|
||||
// Open raw HDR frame files at this scope so cleanup can close descriptors
|
||||
// on both success and early failure paths.
|
||||
const hdrVideoFrameSources = new Map<string, HdrVideoFrameSource>();
|
||||
try {
|
||||
await initializeSession(domSession);
|
||||
assertNotAborted();
|
||||
lastBrowserConsole = domSession.browserConsoleBuffer;
|
||||
|
||||
// Set transparent background once for this dedicated DOM session.
|
||||
// captureAlphaPng() per frame skips the per-frame CDP set/reset overhead.
|
||||
await initTransparentBackground(domSession.page);
|
||||
|
||||
// ── Scene detection for shader transitions ──────────────────────────
|
||||
// Query the browser for transition metadata written by @hyperframes/shader-transitions
|
||||
// (window.__hf.transitions) and discover which elements belong to each scene.
|
||||
const transitionMeta: HdrTransitionMeta[] = await domSession.page.evaluate(() => {
|
||||
return window.__hf?.transitions ?? [];
|
||||
});
|
||||
|
||||
// Contract: compositions using window.__hf.transitions must wrap each
|
||||
// scene's elements in a <div class="scene" id="sceneName"> where the id
|
||||
// matches the fromScene/toScene values declared in the transition metadata.
|
||||
const sceneElements: Record<string, string[]> = await domSession.page.evaluate(() => {
|
||||
const scenes = document.querySelectorAll(".scene");
|
||||
const map: Record<string, string[]> = {};
|
||||
@@ -306,14 +259,12 @@ export async function runCaptureHdrStage(
|
||||
}
|
||||
return map;
|
||||
});
|
||||
|
||||
const fpsDecimal = fpsToNumber(job.config.fps);
|
||||
const transitionRanges: TransitionRange[] = transitionMeta.map((t) => ({
|
||||
...t,
|
||||
startFrame: Math.floor(t.time * fpsDecimal),
|
||||
endFrame: Math.ceil((t.time + t.duration) * fpsDecimal),
|
||||
}));
|
||||
|
||||
if (transitionRanges.length > 0) {
|
||||
log.info("[Render] Detected shader transitions for layered compositing", {
|
||||
count: transitionRanges.length,
|
||||
@@ -326,9 +277,6 @@ export async function runCaptureHdrStage(
|
||||
});
|
||||
}
|
||||
|
||||
// Spawn HDR streaming encoder accepting raw rgb48le composited frames.
|
||||
// Assigned to the let declared above so the outer finally can close it
|
||||
// if any of the work between here and hdrEncoder.close() throws.
|
||||
hdrEncoder = await spawnStreamingEncoder(
|
||||
videoOnlyPath,
|
||||
{
|
||||
@@ -348,242 +296,42 @@ export async function runCaptureHdrStage(
|
||||
);
|
||||
assertNotAborted();
|
||||
|
||||
// ── Query element bounds for HDR extraction dimensions ────────────
|
||||
// Extract at each HDR video's display dimensions (not composition dimensions)
|
||||
// so the source stride matches the blit dimensions. Elements that aren't
|
||||
// visible at t=0 (e.g., data-start > 0) need to be queried at their own
|
||||
// start time so their layout dimensions are available.
|
||||
const hdrExtractionDims = new Map<string, { width: number; height: number }>();
|
||||
// CSS `object-fit` / `object-position` for HDR <img> elements. Captured
|
||||
// alongside `hdrExtractionDims` so the static-image decoder can resample
|
||||
// the rgb48le buffer into the element's layout box the same way the
|
||||
// browser would, instead of blitting the source PNG at native size.
|
||||
const hdrImageFitInfo = new Map<string, { fit: string; position: string }>();
|
||||
const hdrVideoStartTimes = new Map<string, number>();
|
||||
for (const v of composition.videos) {
|
||||
if (hdrVideoIds.includes(v.id)) {
|
||||
hdrVideoStartTimes.set(v.id, v.start);
|
||||
}
|
||||
}
|
||||
const hdrImageStartTimes = new Map<string, number>();
|
||||
for (const img of composition.images) {
|
||||
if (nativeHdrImageIds.has(img.id)) {
|
||||
hdrImageStartTimes.set(img.id, img.start);
|
||||
}
|
||||
}
|
||||
|
||||
// Collect unique start times to minimize seek operations. Merge HDR
|
||||
// video AND image start times so an HDR image with `data-start > 0`
|
||||
// also gets a stacking-query pass at its appearance moment.
|
||||
const uniqueStartTimes = [
|
||||
...new Set([...hdrVideoStartTimes.values(), ...hdrImageStartTimes.values()]),
|
||||
].sort((a, b) => a - b);
|
||||
for (const seekTime of uniqueStartTimes) {
|
||||
await domSession.page.evaluate((t: number) => {
|
||||
if (window.__hf && typeof window.__hf.seek === "function") window.__hf.seek(t);
|
||||
}, seekTime);
|
||||
if (domSession.onBeforeCapture) {
|
||||
await domSession.onBeforeCapture(domSession.page, seekTime);
|
||||
}
|
||||
const stacking = await queryElementStacking(domSession.page, nativeHdrIds);
|
||||
for (const el of stacking) {
|
||||
// Use layout dimensions (offsetWidth/offsetHeight) for extraction — these
|
||||
// are unaffected by CSS transforms (GSAP scale/rotation). getBoundingClientRect
|
||||
// returns the transformed bounding box which can be wrong for extraction.
|
||||
if (
|
||||
el.isHdr &&
|
||||
el.layoutWidth > 0 &&
|
||||
el.layoutHeight > 0 &&
|
||||
!hdrExtractionDims.has(el.id)
|
||||
) {
|
||||
hdrExtractionDims.set(el.id, { width: el.layoutWidth, height: el.layoutHeight });
|
||||
}
|
||||
// Record `object-fit` / `object-position` for HDR images so the
|
||||
// static-image decode pass can resample to layout dimensions with
|
||||
// the same semantics the browser would apply.
|
||||
if (el.isHdr && nativeHdrImageIds.has(el.id) && !hdrImageFitInfo.has(el.id)) {
|
||||
hdrImageFitInfo.set(el.id, {
|
||||
fit: el.objectFit,
|
||||
position: el.objectPosition,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback probe for HDR images that weren't captured above.
|
||||
// When an image's `data-start` aligns with the exact visibility
|
||||
// boundary (or precedes a GSAP `from` tween that animates it in
|
||||
// later), Chrome reports 0 layout dimensions at that instant.
|
||||
// Re-probe slightly into the element's visible range so the
|
||||
// resample path gets real layout dims.
|
||||
for (const [imageId, startTime] of hdrImageStartTimes) {
|
||||
if (hdrExtractionDims.has(imageId)) continue;
|
||||
const img = composition.images.find((i) => i.id === imageId);
|
||||
if (!img) continue;
|
||||
const duration = img.end - img.start;
|
||||
const retryTime = startTime + Math.min(0.5, duration * 0.1);
|
||||
await domSession.page.evaluate((t: number) => {
|
||||
if (window.__hf && typeof window.__hf.seek === "function") window.__hf.seek(t);
|
||||
}, retryTime);
|
||||
if (domSession.onBeforeCapture) {
|
||||
await domSession.onBeforeCapture(domSession.page, retryTime);
|
||||
}
|
||||
const retryStacking = await queryElementStacking(domSession.page, nativeHdrIds);
|
||||
for (const el of retryStacking) {
|
||||
if (el.id === imageId && el.isHdr && el.layoutWidth > 0 && el.layoutHeight > 0) {
|
||||
hdrExtractionDims.set(el.id, { width: el.layoutWidth, height: el.layoutHeight });
|
||||
if (!hdrImageFitInfo.has(el.id)) {
|
||||
hdrImageFitInfo.set(el.id, { fit: el.objectFit, position: el.objectPosition });
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Pre-extract all HDR video frames in a single FFmpeg pass ──────
|
||||
// Use raw rgb48le instead of PNG sequences so the hot loop can read a
|
||||
// fixed byte range per frame and skip PNG decode entirely.
|
||||
for (const [videoId, srcPath] of hdrVideoSrcPaths) {
|
||||
const video = composition.videos.find((v) => v.id === videoId);
|
||||
if (!video) continue;
|
||||
const frameDir = join(framesDir, `hdr_${videoId}`);
|
||||
mkdirSync(frameDir, { recursive: true });
|
||||
const duration = video.end - video.start;
|
||||
const dims = hdrExtractionDims.get(videoId) ?? { width, height };
|
||||
const rawPath = join(frameDir, "frames.rgb48le");
|
||||
const ffmpegArgs = [
|
||||
"-ss",
|
||||
String(video.mediaStart),
|
||||
"-i",
|
||||
srcPath,
|
||||
"-t",
|
||||
String(duration),
|
||||
"-r",
|
||||
// Pass the rational form to FFmpeg so NTSC stays exact end-to-end.
|
||||
fpsToFfmpegArg(job.config.fps),
|
||||
"-vf",
|
||||
`scale=${dims.width}:${dims.height}:force_original_aspect_ratio=increase,crop=${dims.width}:${dims.height}`,
|
||||
"-pix_fmt",
|
||||
"rgb48le",
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-y",
|
||||
rawPath,
|
||||
];
|
||||
const result = await runFfmpeg(ffmpegArgs, { signal: abortSignal });
|
||||
if (!result.success) {
|
||||
hdrDiagnostics.videoExtractionFailures += 1;
|
||||
log.error("HDR frame pre-extraction failed; aborting render", {
|
||||
videoId,
|
||||
srcPath,
|
||||
stderr: result.stderr.slice(-400),
|
||||
});
|
||||
throw new Error(
|
||||
`HDR frame extraction failed for video "${videoId}". ` +
|
||||
`Aborting render to avoid shipping black HDR layers.`,
|
||||
);
|
||||
}
|
||||
const frameSize = dims.width * dims.height * 6;
|
||||
const frameCount = Math.floor(statSync(rawPath).size / frameSize);
|
||||
if (frameCount < 1) {
|
||||
hdrDiagnostics.videoExtractionFailures += 1;
|
||||
throw new Error(
|
||||
`HDR frame extraction produced no frames for video "${videoId}". ` +
|
||||
`Aborting render to avoid shipping black HDR layers.`,
|
||||
);
|
||||
}
|
||||
hdrVideoFrameSources.set(videoId, {
|
||||
dir: frameDir,
|
||||
rawPath,
|
||||
fd: openSync(rawPath, "r"),
|
||||
width: dims.width,
|
||||
height: dims.height,
|
||||
frameSize,
|
||||
frameCount,
|
||||
scratch: Buffer.allocUnsafe(frameSize),
|
||||
});
|
||||
}
|
||||
|
||||
// ── Pre-decode all HDR image buffers once ────────────────────────
|
||||
// Static images decode exactly once, then the resulting rgb48le buffer
|
||||
// is blitted on every visible frame. Caching the decode here keeps the
|
||||
// per-frame cost to a memcpy + blit. Failures are logged and skipped so
|
||||
// a single broken file doesn't kill the render.
|
||||
//
|
||||
// We resample the decoded buffer to the element's *layout* dimensions
|
||||
// here (using CSS `object-fit` / `object-position` semantics), so the
|
||||
// affine blit downstream can treat the buffer as if the source was
|
||||
// sized to the element's box. Without this step, an `<img>` element
|
||||
// styled `object-fit: cover` would render its source PNG at native
|
||||
// pixel size inside the layout box — visually a small image floating
|
||||
// in the top-left corner of its container instead of filling it.
|
||||
const hdrImageBuffers = new Map<string, HdrImageBuffer>();
|
||||
for (const [imageId, srcPath] of hdrImageSrcPaths) {
|
||||
try {
|
||||
const decoded = decodePngToRgb48le(readFileSync(srcPath));
|
||||
const layout = hdrExtractionDims.get(imageId);
|
||||
const fitInfo = hdrImageFitInfo.get(imageId);
|
||||
if (layout && (layout.width !== decoded.width || layout.height !== decoded.height)) {
|
||||
const fit = normalizeObjectFit(fitInfo?.fit);
|
||||
const resampled = resampleRgb48leObjectFit(
|
||||
decoded.data,
|
||||
decoded.width,
|
||||
decoded.height,
|
||||
layout.width,
|
||||
layout.height,
|
||||
fit,
|
||||
fitInfo?.position,
|
||||
);
|
||||
hdrImageBuffers.set(imageId, {
|
||||
data: resampled,
|
||||
width: layout.width,
|
||||
height: layout.height,
|
||||
});
|
||||
} else {
|
||||
hdrImageBuffers.set(imageId, {
|
||||
data: Buffer.from(decoded.data),
|
||||
width: decoded.width,
|
||||
height: decoded.height,
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
hdrDiagnostics.imageDecodeFailures += 1;
|
||||
log.error("HDR image decode failed; aborting render", {
|
||||
imageId,
|
||||
srcPath,
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
throw new Error(
|
||||
`HDR image decode failed for image "${imageId}". ` +
|
||||
`Aborting render to avoid shipping missing HDR image layers.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
// ── HDR resource probing + extraction ──────────────────────────────
|
||||
await probeHdrExtractionDims({
|
||||
domSession,
|
||||
nativeHdrIds,
|
||||
nativeHdrImageIds,
|
||||
composition,
|
||||
prep,
|
||||
});
|
||||
const extracted = await extractHdrVideoFrames({
|
||||
job,
|
||||
log,
|
||||
framesDir,
|
||||
composition,
|
||||
prep,
|
||||
width,
|
||||
height,
|
||||
abortSignal,
|
||||
hdrDiagnostics,
|
||||
});
|
||||
for (const [id, source] of extracted) hdrVideoFrameSources.set(id, source);
|
||||
const hdrImageBuffers = decodeHdrImageBuffers({
|
||||
log,
|
||||
hdrImageSrcPaths,
|
||||
prep,
|
||||
hdrDiagnostics,
|
||||
});
|
||||
|
||||
assertNotAborted();
|
||||
|
||||
try {
|
||||
// The beforeCaptureHook injects SDR video frames into the DOM.
|
||||
// We call it manually since the HDR loop doesn't use captureFrame().
|
||||
const beforeCaptureHook = domSession.onBeforeCapture;
|
||||
|
||||
// Track which HDR video raw frame sources have been cleaned up.
|
||||
// Once a video's last frame has been used (time > video.end), its
|
||||
// extraction directory is deleted to free disk space. This prevents
|
||||
// disk exhaustion on compositions with many HDR videos.
|
||||
const cleanedUpVideos = new Set<string>();
|
||||
// Build a map of video end times for quick lookup
|
||||
const hdrVideoEndTimes = new Map<string, number>();
|
||||
for (const v of composition.videos) {
|
||||
if (hdrVideoFrameSources.has(v.id)) {
|
||||
hdrVideoEndTimes.set(v.id, v.end);
|
||||
}
|
||||
if (hdrVideoFrameSources.has(v.id)) hdrVideoEndTimes.set(v.id, v.end);
|
||||
}
|
||||
|
||||
// ── HDR composite helper context ───────────────────────────────────
|
||||
// The actual layer-compositing logic lives at module scope in
|
||||
// `compositeHdrFrame`; we just pre-bind its long-lived dependencies
|
||||
// here so call sites stay short.
|
||||
const debugDumpEnabled = process.env.KEEP_TEMP === "1";
|
||||
const debugDumpDir = debugDumpEnabled ? join(framesDir, "debug-composite") : null;
|
||||
if (debugDumpDir && !existsSync(debugDumpDir)) {
|
||||
@@ -591,12 +339,6 @@ export async function runCaptureHdrStage(
|
||||
}
|
||||
const compositeTransfer = resolveCompositeTransfer(hasHdrContent, effectiveHdr);
|
||||
const hdrTargetTransfer = compositeTransfer === "srgb" ? undefined : compositeTransfer;
|
||||
// Per-job LRU cache for transfer-converted HDR image buffers. Static HDR
|
||||
// images that need PQ↔HLG conversion are converted exactly once per
|
||||
// (imageId, targetTransfer) and then reused for every subsequent frame
|
||||
// instead of paying a fresh `Buffer.from` + `convertTransfer` on every
|
||||
// composite. The cache is local to this render job so concurrent renders
|
||||
// do not share state.
|
||||
const hdrCacheMaxBytes = process.env.HDR_TRANSFER_CACHE_MAX_BYTES
|
||||
? Number(process.env.HDR_TRANSFER_CACHE_MAX_BYTES)
|
||||
: undefined;
|
||||
@@ -606,7 +348,7 @@ export async function runCaptureHdrStage(
|
||||
const hdrCompositeCtx: HdrCompositeContext = {
|
||||
log,
|
||||
domSession,
|
||||
beforeCaptureHook,
|
||||
beforeCaptureHook: domSession.onBeforeCapture,
|
||||
width,
|
||||
height,
|
||||
fps: fpsToNumber(job.config.fps),
|
||||
@@ -615,7 +357,7 @@ export async function runCaptureHdrStage(
|
||||
hdrImageBuffers,
|
||||
hdrImageTransferCache,
|
||||
hdrVideoFrameSources,
|
||||
hdrVideoStartTimes,
|
||||
hdrVideoStartTimes: prep.hdrVideoStartTimes,
|
||||
imageTransfers,
|
||||
videoTransfers,
|
||||
debugDumpEnabled,
|
||||
@@ -623,272 +365,87 @@ export async function runCaptureHdrStage(
|
||||
hdrPerf,
|
||||
};
|
||||
|
||||
// ── Pre-allocate transition buffers ─────────────────────────────────
|
||||
// Each buffer is width * height * 6 bytes (~37 MB at 1080p). Reused
|
||||
// across frames to avoid per-frame allocation in the hot loop.
|
||||
const bufSize = width * height * 6;
|
||||
const hasTransitions = transitionRanges.length > 0;
|
||||
const transBufferA = hasTransitions ? Buffer.alloc(bufSize) : null;
|
||||
const transBufferB = hasTransitions ? Buffer.alloc(bufSize) : null;
|
||||
const transOutput = hasTransitions ? Buffer.alloc(bufSize) : null;
|
||||
// Pre-allocate the normal-frame canvas too — reused via .fill(0) each iteration
|
||||
// to avoid ~37 MB allocation per frame in the hot loop.
|
||||
const normalCanvas = Buffer.alloc(bufSize);
|
||||
// ── Dispatch to sequential or hybrid frame loop ────────────────────
|
||||
// Resolve the worker budget here rather than threading it through the
|
||||
// renderOrchestrator call: keeps the renderOrchestrator diff zero
|
||||
// (hf#732 PR 4 is intentionally a producer-stage-local change), at the
|
||||
// cost of recomputing the same number the orchestrator already knows.
|
||||
// The cost is negligible (one cpus() call) and the two values stay in
|
||||
// lockstep because `calculateOptimalWorkers` is pure.
|
||||
const effectiveWorkerCount =
|
||||
workerCount !== undefined
|
||||
? Math.max(1, workerCount)
|
||||
: calculateOptimalWorkers(totalFrames, job.config.workers, hdrCfg);
|
||||
const transitionFrameCount = partitionTransitionFrames(transitionRanges, totalFrames).size;
|
||||
const useHybrid = shouldUseHybridLayeredPath({
|
||||
hasHdrContent,
|
||||
transitionFramesCount: transitionFrameCount,
|
||||
totalFrames,
|
||||
workerCount: effectiveWorkerCount,
|
||||
});
|
||||
if (transitionRanges.length > 0) {
|
||||
log.info("[Render] Layered hybrid dispatch decision", {
|
||||
hybridEnabled: useHybrid,
|
||||
hasHdrContent,
|
||||
workerCount: effectiveWorkerCount,
|
||||
transitionFrameCount,
|
||||
totalFrames,
|
||||
});
|
||||
}
|
||||
|
||||
for (let i = 0; i < totalFrames; i++) {
|
||||
assertNotAborted();
|
||||
const time = (i * job.config.fps.den) / job.config.fps.num;
|
||||
if (hdrPerf) hdrPerf.frames += 1;
|
||||
|
||||
// Seek timeline
|
||||
let timingStart = Date.now();
|
||||
await domSession.page.evaluate((t: number) => {
|
||||
if (window.__hf && typeof window.__hf.seek === "function") window.__hf.seek(t);
|
||||
}, time);
|
||||
addHdrTiming(hdrPerf, "frameSeekMs", timingStart);
|
||||
|
||||
// Inject SDR video frames into the DOM
|
||||
if (beforeCaptureHook) {
|
||||
timingStart = Date.now();
|
||||
await beforeCaptureHook(domSession.page, time);
|
||||
addHdrTiming(hdrPerf, "frameInjectMs", timingStart);
|
||||
}
|
||||
|
||||
// Query ALL timed elements for z-order analysis
|
||||
timingStart = Date.now();
|
||||
const stackingInfo = await queryElementStacking(domSession.page, nativeHdrIds);
|
||||
addHdrTiming(hdrPerf, "stackingQueryMs", timingStart);
|
||||
|
||||
// Find active transition for this frame (if any)
|
||||
const activeTransition = transitionRanges.find((t) => i >= t.startFrame && i <= t.endFrame);
|
||||
|
||||
// Per-frame debug snapshot (every 30 frames). The meta object
|
||||
// requires `Array.find` over `stackingInfo` plus a number-format
|
||||
// and conditional struct allocation — non-trivial work to do
|
||||
// every 30 frames in the encode hot loop. Gate the entire block
|
||||
// on the logger's level check so production runs (level=info)
|
||||
// pay nothing.
|
||||
//
|
||||
// Audit note (PR #383 review): this is the only per-frame log
|
||||
// site in the streaming HDR encode loop that constructs
|
||||
// non-trivial metadata. The `[diag]` log.info calls inside
|
||||
// compositeToBuffer (compositeToBuffer plan, hdr layer blit,
|
||||
// dom layer blit, compositeToBuffer end) are already gated by
|
||||
// `shouldLog = debugDumpEnabled && debugFrameIndex >= 0`, where
|
||||
// debugDumpEnabled is driven by KEEP_TEMP=1 — strictly stricter
|
||||
// than an isLevelEnabled check. The HDR blit error-path
|
||||
// log.debugs only fire on caught failures, not on the happy
|
||||
// path. Any new per-frame log site that builds meta should
|
||||
// follow the same `if (log.isLevelEnabled?.("level") ?? true)`
|
||||
// pattern (or stay behind `shouldLog`) so production stays
|
||||
// allocation-free in the hot loop.
|
||||
if (i % 30 === 0 && (log.isLevelEnabled?.("debug") ?? true)) {
|
||||
const hdrEl = stackingInfo.find((e) => e.isHdr);
|
||||
log.debug("[Render] HDR layer composite frame", {
|
||||
frame: i,
|
||||
time: time.toFixed(2),
|
||||
hdrElement: hdrEl
|
||||
? { z: hdrEl.zIndex, visible: hdrEl.visible, width: hdrEl.width }
|
||||
: null,
|
||||
stackingCount: stackingInfo.length,
|
||||
activeTransition: activeTransition?.shader,
|
||||
});
|
||||
}
|
||||
|
||||
if (activeTransition && transBufferA && transBufferB && transOutput) {
|
||||
if (hdrPerf) hdrPerf.transitionFrames += 1;
|
||||
const transitionTimingStart = Date.now();
|
||||
// ── Transition frame: dual-scene compositing ──────────────────
|
||||
const progress =
|
||||
activeTransition.endFrame === activeTransition.startFrame
|
||||
? 1
|
||||
: (i - activeTransition.startFrame) /
|
||||
(activeTransition.endFrame - activeTransition.startFrame);
|
||||
|
||||
// Resolve scene element IDs
|
||||
const sceneAIds = new Set(sceneElements[activeTransition.fromScene] ?? []);
|
||||
const sceneBIds = new Set(sceneElements[activeTransition.toScene] ?? []);
|
||||
|
||||
// Zero-fill scene buffers (transition function writes every output pixel)
|
||||
timingStart = Date.now();
|
||||
transBufferA.fill(0);
|
||||
transBufferB.fill(0);
|
||||
addHdrTiming(hdrPerf, "canvasClearMs", timingStart);
|
||||
|
||||
for (const [sceneBuf, sceneIds] of [
|
||||
[transBufferA, sceneAIds],
|
||||
[transBufferB, sceneBIds],
|
||||
] as const) {
|
||||
// Re-check abort between scene A and scene B. Each scene
|
||||
// capture below performs a DOM seek, optional hook,
|
||||
// per-layer HDR blits, and a full-page screenshot — easily
|
||||
// hundreds of ms. Without this, an abort that arrives
|
||||
// during scene A's capture won't fire until the next outer
|
||||
// frame, after scene B has already been fully composited
|
||||
// and discarded.
|
||||
assertNotAborted();
|
||||
// Fresh state: seek + inject
|
||||
timingStart = Date.now();
|
||||
await domSession.page.evaluate((t: number) => {
|
||||
if (window.__hf && typeof window.__hf.seek === "function") window.__hf.seek(t);
|
||||
}, time);
|
||||
addHdrTiming(hdrPerf, "domLayerSeekMs", timingStart);
|
||||
if (beforeCaptureHook) {
|
||||
timingStart = Date.now();
|
||||
await beforeCaptureHook(domSession.page, time);
|
||||
addHdrTiming(hdrPerf, "domLayerInjectMs", timingStart);
|
||||
}
|
||||
|
||||
// Blit all HDR videos/images for this scene
|
||||
for (const el of stackingInfo) {
|
||||
if (!el.isHdr || !sceneIds.has(el.id)) continue;
|
||||
if (nativeHdrImageIds.has(el.id)) {
|
||||
blitHdrImageLayer(
|
||||
sceneBuf as Buffer,
|
||||
el,
|
||||
hdrImageBuffers,
|
||||
hdrImageTransferCache,
|
||||
width,
|
||||
height,
|
||||
log,
|
||||
imageTransfers.get(el.id),
|
||||
hdrTargetTransfer,
|
||||
hdrPerf,
|
||||
);
|
||||
} else {
|
||||
blitHdrVideoLayer(
|
||||
sceneBuf as Buffer,
|
||||
el,
|
||||
time,
|
||||
fpsToNumber(job.config.fps),
|
||||
hdrVideoFrameSources,
|
||||
hdrVideoStartTimes,
|
||||
width,
|
||||
height,
|
||||
log,
|
||||
videoTransfers.get(el.id),
|
||||
hdrTargetTransfer,
|
||||
hdrPerf,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Single DOM screenshot: mask the page so only this scene's DOM
|
||||
// elements paint. Same masking strategy as the per-layer DOM
|
||||
// branch — see applyDomLayerMask for details. Native HDR videos
|
||||
// and images are always inline-hidden so their fallback poster /
|
||||
// SDR thumbnail doesn't bleed into the DOM overlay (HDR pixels
|
||||
// are blitted separately by blitHdrVideoLayer / blitHdrImageLayer
|
||||
// above).
|
||||
const showIds = Array.from(sceneIds);
|
||||
const hideIds = stackingInfo
|
||||
.map((e) => e.id)
|
||||
.filter((id) => !sceneIds.has(id) || nativeHdrIds.has(id));
|
||||
if (hdrPerf) hdrPerf.domLayerCaptures += 1;
|
||||
timingStart = Date.now();
|
||||
await applyDomLayerMask(domSession.page, showIds, hideIds);
|
||||
addHdrTiming(hdrPerf, "domMaskApplyMs", timingStart);
|
||||
timingStart = Date.now();
|
||||
const domPng = await captureAlphaPng(domSession.page, width, height);
|
||||
addHdrTiming(hdrPerf, "domScreenshotMs", timingStart);
|
||||
timingStart = Date.now();
|
||||
await removeDomLayerMask(domSession.page, hideIds);
|
||||
addHdrTiming(hdrPerf, "domMaskRemoveMs", timingStart);
|
||||
|
||||
try {
|
||||
timingStart = Date.now();
|
||||
const { data: domRgba } = decodePng(domPng);
|
||||
addHdrTiming(hdrPerf, "domPngDecodeMs", timingStart);
|
||||
timingStart = Date.now();
|
||||
blitRgba8OverRgb48le(domRgba, sceneBuf as Buffer, width, height, compositeTransfer);
|
||||
addHdrTiming(hdrPerf, "domBlitMs", timingStart);
|
||||
} catch (err) {
|
||||
log.warn("DOM layer decode/blit failed; skipping overlay for transition scene", {
|
||||
frameIndex: i,
|
||||
sceneIds: Array.from(sceneIds),
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Apply shader transition blend directly in the active rgb48le
|
||||
// signal space. Linearizing HDR was attempted but destroys dark
|
||||
// PQ content — values below PQ ~5000 quantize to zero in 16-bit
|
||||
// linear, wiping out the bottom portion of dark video content.
|
||||
// SDR compositions use 16-bit-expanded sRGB, which matches the
|
||||
// shader design space.
|
||||
const transitionFn: TransitionFn = TRANSITIONS[activeTransition.shader] ?? crossfade;
|
||||
transitionFn(transBufferA, transBufferB, transOutput, width, height, progress);
|
||||
addHdrTiming(hdrPerf, "transitionCompositeMs", transitionTimingStart);
|
||||
|
||||
timingStart = Date.now();
|
||||
hdrEncoder.writeFrame(transOutput);
|
||||
addHdrTiming(hdrPerf, "encoderWriteMs", timingStart);
|
||||
} else {
|
||||
if (hdrPerf) hdrPerf.normalFrames += 1;
|
||||
// ── Normal frame: full layer composite (no transition) ─────────
|
||||
timingStart = Date.now();
|
||||
normalCanvas.fill(0);
|
||||
addHdrTiming(hdrPerf, "canvasClearMs", timingStart);
|
||||
timingStart = Date.now();
|
||||
await compositeHdrFrame(hdrCompositeCtx, normalCanvas, time, stackingInfo, undefined, i);
|
||||
addHdrTiming(hdrPerf, "normalCompositeMs", timingStart);
|
||||
if (debugDumpEnabled && debugDumpDir && i % 30 === 0) {
|
||||
const previewPath = join(
|
||||
debugDumpDir,
|
||||
`frame_${String(i).padStart(4, "0")}_final_rgb48le.bin`,
|
||||
);
|
||||
writeFileSync(previewPath, normalCanvas);
|
||||
}
|
||||
timingStart = Date.now();
|
||||
hdrEncoder.writeFrame(normalCanvas);
|
||||
addHdrTiming(hdrPerf, "encoderWriteMs", timingStart);
|
||||
}
|
||||
|
||||
// Clean up HDR raw frame sources for videos that have ended.
|
||||
// Frees disk space during long renders with many HDR videos.
|
||||
// Skip when KEEP_TEMP=1 so we can inspect intermediate state.
|
||||
if (process.env.KEEP_TEMP !== "1") {
|
||||
for (const [videoId, endTime] of hdrVideoEndTimes) {
|
||||
if (time > endTime && !cleanedUpVideos.has(videoId)) {
|
||||
// Also check no active transition references this video's scene
|
||||
const stillNeeded =
|
||||
activeTransition &&
|
||||
(sceneElements[activeTransition.fromScene]?.includes(videoId) ||
|
||||
sceneElements[activeTransition.toScene]?.includes(videoId));
|
||||
if (!stillNeeded) {
|
||||
const frameSource = hdrVideoFrameSources.get(videoId);
|
||||
if (frameSource) {
|
||||
closeHdrVideoFrameSource(frameSource, log);
|
||||
try {
|
||||
rmSync(frameSource.dir, { recursive: true, force: true });
|
||||
} catch (err) {
|
||||
log.warn("Failed to clean up HDR raw frame directory", {
|
||||
videoId,
|
||||
frameDir: frameSource.dir,
|
||||
rawPath: frameSource.rawPath,
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
}
|
||||
hdrVideoFrameSources.delete(videoId);
|
||||
}
|
||||
cleanedUpVideos.add(videoId);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
job.framesRendered = i + 1;
|
||||
if ((i + 1) % 10 === 0 || i + 1 === totalFrames) {
|
||||
const frameProgress = (i + 1) / totalFrames;
|
||||
updateJobStatus(
|
||||
job,
|
||||
"rendering",
|
||||
`Layered composite frame ${i + 1}/${job.totalFrames}`,
|
||||
Math.round(25 + frameProgress * 55),
|
||||
onProgress,
|
||||
);
|
||||
}
|
||||
if (useHybrid) {
|
||||
await runHybridLayeredFrameLoop({
|
||||
job,
|
||||
cfg: hdrCfg,
|
||||
log,
|
||||
framesDir,
|
||||
width,
|
||||
height,
|
||||
totalFrames,
|
||||
nativeHdrIds,
|
||||
nativeHdrImageIds,
|
||||
hdrCompositeCtx,
|
||||
hdrPerf,
|
||||
hdrEncoder,
|
||||
domSession,
|
||||
fileServer,
|
||||
buildCaptureOptions,
|
||||
createRenderVideoFrameInjector,
|
||||
transitionRanges,
|
||||
sceneElements,
|
||||
compositeTransfer,
|
||||
hdrTargetTransfer,
|
||||
workerCount: effectiveWorkerCount,
|
||||
debugDumpEnabled,
|
||||
debugDumpDir,
|
||||
assertNotAborted,
|
||||
onProgress,
|
||||
});
|
||||
} else {
|
||||
await runSequentialLayeredFrameLoop({
|
||||
job,
|
||||
log,
|
||||
width,
|
||||
height,
|
||||
totalFrames,
|
||||
nativeHdrIds,
|
||||
nativeHdrImageIds,
|
||||
hdrCompositeCtx,
|
||||
hdrPerf,
|
||||
hdrEncoder,
|
||||
domSession,
|
||||
transitionRanges,
|
||||
sceneElements,
|
||||
compositeTransfer,
|
||||
hdrTargetTransfer,
|
||||
hdrVideoEndTimes,
|
||||
cleanedUpVideos,
|
||||
hdrVideoFrameSources,
|
||||
debugDumpEnabled,
|
||||
debugDumpDir,
|
||||
assertNotAborted,
|
||||
onProgress,
|
||||
});
|
||||
}
|
||||
} finally {
|
||||
lastBrowserConsole = domSession.browserConsoleBuffer;
|
||||
@@ -902,15 +459,9 @@ export async function runCaptureHdrStage(
|
||||
if (!hdrEncodeResult.success) {
|
||||
throw new Error(`HDR encode failed: ${hdrEncodeResult.error}`);
|
||||
}
|
||||
|
||||
captureDurationMs = Date.now() - stageStart;
|
||||
encodeMs = hdrEncodeResult.durationMs;
|
||||
} finally {
|
||||
// Defensive cleanup: if anything between domSession creation and the
|
||||
// success-path closes threw, the encoder ffmpeg subprocess and the
|
||||
// browser would otherwise be leaked. Both close() methods are
|
||||
// idempotent so it's safe to call them when the flags are already set,
|
||||
// but we skip the redundant work to keep logs clean.
|
||||
if (hdrEncoder && !hdrEncoderClosed) {
|
||||
try {
|
||||
await hdrEncoder.close();
|
||||
@@ -927,9 +478,6 @@ export async function runCaptureHdrStage(
|
||||
});
|
||||
});
|
||||
}
|
||||
// Close any raw frame files that survived in-loop cleanup (early
|
||||
// failures, KEEP_TEMP=1, videos still active when the render exits).
|
||||
// The on-disk frames themselves are torn down with workDir.
|
||||
for (const frameSource of hdrVideoFrameSources.values()) {
|
||||
closeHdrVideoFrameSource(frameSource, log);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user