Merge pull request #4115 from heygen-com/lfr/08-phase2d-multi-worker-segments

feat(producer): multi-worker segmented capture and the single router predicate (Phase 2d)
This commit is contained in:
Vance Ingalls
2026-09-18 21:25:15 -07:00
committed by GitHub
10 changed files with 643 additions and 165 deletions
+30
View File
@@ -955,6 +955,36 @@ frame storage, the message names which of those applied; the usual fixes are
| `PRODUCER_STREAMING_ENCODE_DURATION_CAP_ENABLED` | `false` | `true` restores the duration ceiling below |
| `PRODUCER_STREAMING_ENCODE_MAX_DURATION_SECONDS` | `240` | The ceiling's value; inert unless the flag above is `true`, including the `0` ("never") setting |
#### Segmented capture
For long renders, capture can be split into fixed-size segments, each encoded
by its own ffmpeg and stream-copied together at the end. It costs some speed
against a single-encoder streaming render, and buys three things:
- Scratch is encoded video rather than raw frames, at any duration.
- The browser restarts at segment boundaries, so a session never accumulates
the tens of thousands of frames where Chrome starts losing its target.
- A crash costs one segment. `--resume` re-runs the render and reuses the
segments a previous attempt finished, after re-validating each one; a
resumed render is byte-identical to an uninterrupted one.
Segments live in `renders/.hf-segments/<hash>/`, where the hash covers
everything that determines segment bytes, so changing the composition, fps,
resolution, quality or segment size starts a fresh set instead of splicing
mismatched encodes. The directory is deleted after a successful render unless
`--keep-segments`, and deliberately kept after a failure, because that is what
`--resume` reads next time.
mp4 and mov only; png-sequence, gif, webm, HDR and shader-transition renders
keep their existing routes.
| Environment variable | Default | Effect |
| ------------------------------ | ---------- | ------------------------------------------------------------------------------------------ |
| `HF_SEGMENTED_CAPTURE` | unset | `true` segments any eligible render; `false` is the kill switch |
| `HF_SEGMENTED_MIN_SECONDS` | off | Segment renders at least this long. Unset means duration never routes on its own |
| `HF_SEGMENT_FRAMES` | `3000` | Frames per segment (~100 s at 30 fps); minimum 30 |
| `HF_SEGMENT_BROWSER_RECYCLE` | `3` | Restart the browser every N segments; `0` keeps one session for the whole render |
#### Render the same composition with different content
Declare the variables on the composition root, read them inside it, then
@@ -389,7 +389,7 @@ describe("sdr_segmented capture plan", () => {
useSegmentedCapture: true,
};
it("selects segmented capture only for single-worker streaming-eligible renders", () => {
it("selects segmented capture for streaming-eligible renders at any worker count", () => {
const plan = createCapturePlan(segmented);
expect(plan).toMatchObject({
kind: "sdr_segmented",
@@ -397,11 +397,22 @@ describe("sdr_segmented capture plan", () => {
forceParallelStream: false,
});
expect(Object.isFrozen(plan)).toBe(true);
expect(createCapturePlan({ ...segmented, workerCount: 3 }).kind).toBe("sdr_streaming");
// Phase 2d: several workers each own a segment and an encoder.
expect(createCapturePlan({ ...segmented, workerCount: 3 })).toMatchObject({
kind: "sdr_segmented",
workerCount: 3,
forceParallelStream: false,
});
});
it("does not segment when streaming is off or the route is layered", () => {
expect(createCapturePlan({ ...segmented, useStreamingEncode: false }).kind).toBe("sdr_disk");
it("loses to the layered route but not to the single-encoder streaming flag", () => {
// useStreamingEncode answers "one encoder for the whole render", which a
// segmented render never wants — it goes false for multi-worker, and
// requiring it here would silently drop those renders onto the disk path.
// Viability is decided by shouldSegmentCapture before this is set.
expect(createCapturePlan({ ...segmented, useStreamingEncode: false }).kind).toBe(
"sdr_segmented",
);
expect(createCapturePlan({ ...segmented, useLayeredComposite: true }).kind).toBe("hdr_layered");
});
@@ -196,7 +196,14 @@ export function createCapturePlan(input: CreateCapturePlanInput): CapturePlan {
forceParallelStream: false,
});
}
if (input.useStreamingEncode && input.useSegmentedCapture === true && input.workerCount === 1) {
// Checked before `useStreamingEncode`, which answers a question segmented
// capture does not ask: that flag is about ONE encoder for the whole render
// and so goes false for multi-worker, while a segmented render gives every
// worker its own. Whether streaming is viable at all is settled by the
// caller's predicate (`shouldSegmentCapture`) before this is ever true.
// forceParallelStream stays false for the same reason: the interleaved
// single-encoder writer has nothing to do here.
if (input.useSegmentedCapture === true) {
return Object.freeze({ ...base, kind: "sdr_segmented", forceParallelStream: false });
}
if (input.useStreamingEncode) {
@@ -0,0 +1,47 @@
import { describe, expect, it } from "vitest";
import { createSegmentQueue } from "./segmentQueue.js";
import { planSegments } from "./segmentPlan.js";
describe("createSegmentQueue", () => {
it("hands out every slice exactly once, in index order", () => {
const slices = planSegments(7, 3);
const queue = createSegmentQueue(slices);
const taken = [queue.next(), queue.next(), queue.next()];
expect(taken.map((s) => s?.index)).toEqual([0, 1, 2]);
expect(queue.next()).toBeUndefined();
});
it("reports what is left and never goes negative past exhaustion", () => {
const queue = createSegmentQueue(planSegments(4, 2));
expect(queue.remaining()).toBe(2);
queue.next();
expect(queue.remaining()).toBe(1);
queue.next();
queue.next();
queue.next();
expect(queue.remaining()).toBe(0);
});
it("gives no slice to two callers", () => {
// Interleaved pulls stand in for concurrent workers: the JS event loop
// cannot preempt next(), so claiming is atomic by construction.
const queue = createSegmentQueue(planSegments(6, 2));
const a: number[] = [];
const b: number[] = [];
for (;;) {
const first = queue.next();
if (first) a.push(first.index);
const second = queue.next();
if (second) b.push(second.index);
if (!first && !second) break;
}
expect([...a, ...b].sort((x, y) => x - y)).toEqual([0, 1, 2]);
expect(new Set([...a, ...b]).size).toBe(3);
});
it("is empty for an empty slice list", () => {
const queue = createSegmentQueue([]);
expect(queue.next()).toBeUndefined();
expect(queue.remaining()).toBe(0);
});
});
@@ -0,0 +1,22 @@
/**
* Hands segments to parallel capture workers (spec §5 Phase 2 items 1-2).
*
* Workers pull in index order but finish out of order, which is fine: the
* concat list is built from the plan, not from completion order, so ordering
* never reaches the encoders.
*/
import type { SegmentSlice } from "./segmentPlan.js";
export interface SegmentQueue {
/** The next unclaimed slice, or undefined when the queue is drained. */
next(): SegmentSlice | undefined;
remaining(): number;
}
export function createSegmentQueue(slices: readonly SegmentSlice[]): SegmentQueue {
let cursor = 0;
return {
next: () => slices[cursor++],
remaining: () => Math.max(0, slices.length - cursor),
};
}
@@ -599,4 +599,171 @@ describe("runCaptureSegmentedStage", () => {
// Exactly one retry: initial session + one recycle, not a loop.
expect(next).toBe(2);
});
it("distributes 7 segments over 3 workers, each frame once, concat in index order", async () => {
const perWorkerSessions = new Map<number, number>();
const sessionFactoryForWorker = (workerId: number) => ({
create: async () => {
perWorkerSessions.set(workerId, (perWorkerSessions.get(workerId) ?? 0) + 1);
return fakeSession(workerId);
},
});
const captured: number[] = [];
const captureFrame = mock(async (_s: unknown, i: number) => {
captured.push(i);
// Yield so the workers actually interleave rather than running serially.
await new Promise((r) => setTimeout(r, 1));
return { buffer: Buffer.alloc(1) };
});
const concat = mock(async () => ({ success: true as const }));
const stableDir = join(fixtureRoot, "workers");
const result = await runCaptureSegmentedStage({
...fakeStageInput({ totalFrames: 20 }),
probeSession: null,
segmentFrames: 3,
segmentDir: stableDir,
workerCount: 3,
sessionFactoryForWorker,
deps: {
spawnEncoder: mock(async () => okEncoder()),
captureFrame,
concat,
closeSession: mock(async () => {}),
removeFile: () => {},
},
});
expect(result.success).toBe(true);
if (!result.success) throw new Error("unreachable");
expect(result.workerCount).toBe(3);
expect(result.segments).toBe(7);
// Every frame exactly once across all workers, none duplicated.
expect([...captured].sort((a, b) => a - b)).toEqual(Array.from({ length: 20 }, (_, i) => i));
expect(new Set(captured).size).toBe(20);
expect(perWorkerSessions.size).toBe(3);
const concatInputs = concat.mock.calls[0]?.[0];
expect(concatInputs).toEqual(
Array.from({ length: 7 }, (_, i) => segmentOutputPath(stableDir, i)),
);
});
it("closes every worker session when one worker fails hard", async () => {
const closed: number[] = [];
let id = 0;
const result = runCaptureSegmentedStage({
...fakeStageInput({ totalFrames: 9 }),
probeSession: null,
segmentFrames: 3,
segmentDir: join(fixtureRoot, "workerfail"),
workerCount: 3,
sessionFactoryForWorker: () => ({ create: async () => fakeSession(id++) }),
deps: {
spawnEncoder: mock(async () => okEncoder()),
captureFrame: mock(async (_s: unknown, i: number) => {
if (i === 4) throw new Error("media_start_out_of_range");
return { buffer: Buffer.alloc(1) };
}),
concat: mock(async () => ({ success: true as const })),
closeSession: mock(async (s: { id: number }) => {
closed.push(s.id);
}),
removeFile: () => {},
},
});
await expect(result).rejects.toThrow(/media_start_out_of_range/);
// A leaked Chrome per worker is the failure mode this guards.
expect(closed.length).toBe(3);
});
it("does not fall back to streaming when several workers are in flight", async () => {
// Only the first segment's encoder fails; the others spawn fine. With one
// worker that is the documented fallback, but here other workers are
// already encoding, so discarding the render would throw away their work.
let id = 0;
let spawns = 0;
await expect(
runCaptureSegmentedStage({
...fakeStageInput({ totalFrames: 9 }),
probeSession: null,
segmentFrames: 3,
segmentDir: join(fixtureRoot, "nofallback"),
workerCount: 3,
sessionFactoryForWorker: () => ({ create: async () => fakeSession(id++) }),
deps: {
spawnEncoder: mock(async () => {
if (spawns++ === 0) throw new Error("ffmpeg missing");
return okEncoder();
}),
captureFrame: mock(async () => ({ buffer: Buffer.alloc(1) })),
concat: mock(async () => ({ success: true as const })),
closeSession: mock(async () => {}),
removeFile: () => {},
},
}),
).rejects.toThrow(/ffmpeg missing/);
});
it("still falls back when a single worker cannot spawn its first encoder", async () => {
let id = 0;
let spawns = 0;
const result = await runCaptureSegmentedStage({
...fakeStageInput({ totalFrames: 9 }),
probeSession: null,
segmentFrames: 3,
segmentDir: join(fixtureRoot, "singlefallback"),
workerCount: 1,
sessionFactoryForWorker: () => ({ create: async () => fakeSession(id++) }),
deps: {
spawnEncoder: mock(async () => {
if (spawns++ === 0) throw new Error("ffmpeg missing");
return okEncoder();
}),
captureFrame: mock(async () => ({ buffer: Buffer.alloc(1) })),
concat: mock(async () => ({ success: true as const })),
closeSession: mock(async () => {}),
removeFile: () => {},
},
});
expect(result).toEqual({ success: false });
});
it("lets every worker stop before closing any session", async () => {
// Closing a session out from under a worker that is still capturing
// orphans its ffmpeg and races the CDP connection, so the stage waits for
// all of them to settle even though one has already failed.
let id = 0;
let closedAny = false;
let capturedAfterClose = 0;
await expect(
runCaptureSegmentedStage({
...fakeStageInput({ totalFrames: 9 }),
probeSession: null,
segmentFrames: 3,
segmentDir: join(fixtureRoot, "settle"),
workerCount: 3,
sessionFactoryForWorker: () => ({ create: async () => fakeSession(id++) }),
deps: {
spawnEncoder: mock(async () => okEncoder()),
captureFrame: mock(async (_s: unknown, i: number) => {
if (closedAny) capturedAfterClose += 1;
// Segment 0's first frame fails immediately; the other workers are
// mid-flight on slow frames when it does.
if (i === 0) throw new Error("media_start_out_of_range");
await new Promise((r) => setTimeout(r, 5));
return { buffer: Buffer.alloc(1) };
}),
concat: mock(async () => ({ success: true as const })),
closeSession: mock(async () => {
closedAny = true;
}),
removeFile: () => {},
},
}),
).rejects.toThrow(/media_start_out_of_range/);
// Give any worker the stage abandoned time to resume and capture again;
// without the wait this assertion passes before they would have.
await new Promise((r) => setTimeout(r, 40));
expect(capturedAfterClose).toBe(0);
});
});
@@ -40,6 +40,7 @@ import { updateJobStatus } from "../shared.js";
import { encoderFailureError } from "../encoderInterruption.js";
import type { SdrSegmentedCapturePlan } from "../capturePlan.js";
import { planSegments, type SegmentSlice } from "../segmentPlan.js";
import { createSegmentQueue } from "../segmentQueue.js";
import { isTargetLossError } from "../segmentRecycle.js";
import {
raceAgainstStall,
@@ -95,7 +96,17 @@ export interface CaptureSegmentedStageInput {
path: string;
bytes: number;
}) => void;
/** Opens each session; defaults to reusing the probe session, then fresh ones. */
/** Parallel capture sessions pulling from one segment queue. Default 1. */
workerCount?: number;
/** Per-worker factory; takes precedence over sessionFactory when set. */
sessionFactoryForWorker?: (workerId: number) => SessionFactory;
/**
* Opens each session; defaults to reusing the probe session, then fresh
* ones. Without sessionFactoryForWorker every worker shares this one
* factory, so it must hand out a distinct session per create() call, as
* the default does; a factory that returns one shared session would have N
* workers driving one browser.
*/
sessionFactory?: SessionFactory;
/** Recycle the browser every N segments; 0 or absent keeps one session. */
browserRecycleEverySegments?: number;
@@ -114,7 +125,7 @@ export type CaptureSegmentedStageResult =
encodeMs: number;
probeSession: null;
lastBrowserConsole: string[];
workerCount: 1;
workerCount: number;
segments: number;
segmentPaths: string[];
/** Segments re-captured after a Chrome target loss. */
@@ -337,168 +348,241 @@ async function concatSegments(
});
}
interface SegmentWorker {
id: number;
ctx: SegmentCaptureContext;
sessionSegments: number;
factory: SessionFactory;
}
/** Counters shared by every worker; JS is single-threaded, so += is safe. */
interface SegmentTotals {
encodeMs: number;
segmentRetries: number;
browserRecycles: number;
}
/** Everything the per-worker helpers need, resolved once. */
interface SegmentRun {
input: CaptureSegmentedStageInput;
deps: SegmentedStageDeps;
totals: SegmentTotals;
segments: readonly SegmentSlice[];
pending: readonly SegmentSlice[];
segmentDir: string;
skipSize: number;
workerCount: number;
recycleEvery: number;
}
async function createSegmentWorker(run: SegmentRun, id: number): Promise<SegmentWorker> {
const { input, deps } = run;
const factory =
input.sessionFactoryForWorker?.(id) ?? input.sessionFactory ?? defaultSessionFactory(input);
return {
id,
factory,
sessionSegments: 0,
ctx: {
session: await factory.create(),
job: input.job,
cfg: input.cfg,
totalFrames: input.totalFrames,
segmentCount: run.segments.length,
segmentDir: run.segmentDir,
skipped: run.skipSize,
streamingEncoderOptions: input.streamingEncoderOptions,
spawnEncoder: deps.spawnEncoder,
captureFrame: deps.captureFrame,
stallTimeoutMs: resolveCaptureStallTimeoutMs(),
abortSignal: input.abortSignal,
assertNotAborted: input.assertNotAborted,
onProgress: input.onProgress,
onSegmentComplete: input.onSegmentComplete,
},
};
}
/** Close this worker's browser and open a fresh one at a segment boundary. */
async function recycleWorkerSession(
run: SegmentRun,
worker: SegmentWorker,
why: "cadence" | "retry",
): Promise<void> {
// Counters are only valid while the session is live, so harvest before close.
run.input.dedupPerfs.push(getCapturePerfSummary(worker.ctx.session));
const memory = worker.ctx.session.chromeMemory?.stats();
await run.deps.closeSession(worker.ctx.session);
worker.ctx.session = await worker.factory.create();
worker.sessionSegments = 0;
if (why === "cadence") run.totals.browserRecycles += 1;
run.input.log.info(`[Render] segment browser recycled (${why})`, {
worker: worker.id,
rendererRssPeakMb: memory?.rendererRssPeakMb,
rssLastMb: memory?.rssLastMb,
samples: memory?.samples,
});
}
/**
* The encoder never started. Only a single-worker render can still fall back:
* with several workers in flight, other segments are already being encoded
* and a fallback would throw their work away.
*/
function onSegmentSpawnFailure(
run: SegmentRun,
err: SegmentEncoderSpawnError,
segment: SegmentSlice,
): "fallback" {
if (run.workerCount > 1 || segment.index !== run.pending[0]?.index) throw err.reason;
run.input.log.warn(
"[Render] Segment encoder spawn failed; falling back to single-encoder streaming.",
{
error: err.message,
outputFormat: run.input.outputFormat,
segments: run.segments.length,
durationSeconds: run.input.job.duration,
},
);
return "fallback";
}
/**
* Chrome losing its target mid-capture is the one failure a fresh browser
* fixes; everything else reproduces, so retrying it only doubles the time to
* the same error. Once, then it propagates.
*/
async function retryAfterTargetLoss(
run: SegmentRun,
worker: SegmentWorker,
err: unknown,
segment: SegmentSlice,
segmentPath: string,
): Promise<void> {
if (!isTargetLossError(err) || run.input.abortSignal?.aborted) throw err;
run.totals.segmentRetries += 1;
run.input.updateCaptureObservability?.({ segmentRetries: run.totals.segmentRetries });
run.input.log.warn(
`[Render] segment ${segment.index}: browser target lost; retrying once on a fresh session`,
{ worker: worker.id, error: err instanceof Error ? err.message : String(err) },
);
run.deps.removeFile(segmentPath);
await recycleWorkerSession(run, worker, "retry");
run.totals.encodeMs += await captureSegment(worker.ctx, segment);
}
async function runOneSegment(
run: SegmentRun,
worker: SegmentWorker,
segment: SegmentSlice,
): Promise<"done" | "fallback"> {
const segmentPath = segmentOutputPath(run.segmentDir, segment.index);
if (run.recycleEvery > 0 && worker.sessionSegments >= run.recycleEvery) {
await recycleWorkerSession(run, worker, "cadence");
}
run.input.updateCaptureObservability?.({
capturePath: "segmented",
segmentIndex: segment.index,
});
try {
run.totals.encodeMs += await captureSegment(worker.ctx, segment);
} catch (err) {
if (err instanceof SegmentEncoderSpawnError) return onSegmentSpawnFailure(run, err, segment);
await retryAfterTargetLoss(run, worker, err, segment, segmentPath);
}
worker.sessionSegments += 1;
return "done";
}
export async function runCaptureSegmentedStage(
input: CaptureSegmentedStageInput,
): Promise<CaptureSegmentedStageResult> {
const {
workDir,
videoOnlyPath,
job,
totalFrames,
cfg,
log,
outputFormat,
streamingEncoderOptions,
abortSignal,
assertNotAborted,
onProgress,
dedupPerfs,
segmentFrames,
updateCaptureObservability,
} = input;
const { log, dedupPerfs, assertNotAborted } = input;
const deps: SegmentedStageDeps = { ...DEFAULT_DEPS, ...input.deps };
const segments = planSegments(totalFrames, segmentFrames);
const segmentDir = input.segmentDir ?? join(workDir, "segments");
const segments = planSegments(input.totalFrames, input.segmentFrames);
const segmentDir = input.segmentDir ?? join(input.workDir, "segments");
mkdirSync(segmentDir, { recursive: true });
const skip = input.completedSegments ?? new Set<number>();
// The first segment that still has to be captured. Only it can fall back to
// the plain streaming path on a spawn failure; by any later one, encoded
// segments exist that a fallback would throw away.
const firstPending = segments.find((s) => !skip.has(s.index));
const factory = input.sessionFactory ?? defaultSessionFactory(input);
const recycleEvery = input.browserRecycleEverySegments ?? 0;
let lastBrowserConsole: string[] = [];
let encodeMs = 0;
let segmentRetries = 0;
let browserRecycles = 0;
let sessionSegments = 0;
const segmentPaths: string[] = [];
const ctx: SegmentCaptureContext = {
session: await factory.create(),
job,
cfg,
totalFrames,
segmentCount: segments.length,
const pending = segments.filter((s) => !skip.has(s.index));
for (const segment of segments) {
if (skip.has(segment.index)) {
log.info("[Render] segment skipped (resume)", { index: segment.index });
}
}
const run: SegmentRun = {
input,
deps,
totals: { encodeMs: 0, segmentRetries: 0, browserRecycles: 0 },
segments,
pending,
segmentDir,
skipped: skip.size,
streamingEncoderOptions,
spawnEncoder: deps.spawnEncoder,
captureFrame: deps.captureFrame,
stallTimeoutMs: resolveCaptureStallTimeoutMs(),
abortSignal,
assertNotAborted,
onProgress,
onSegmentComplete: input.onSegmentComplete,
};
/** Close the current browser and open a fresh one at a segment boundary. */
const recycleSession = async (why: "cadence" | "retry"): Promise<void> => {
lastBrowserConsole = ctx.session.browserConsoleBuffer;
// Counters are only valid while the session is live, so harvest before close.
dedupPerfs.push(getCapturePerfSummary(ctx.session));
const memory = ctx.session.chromeMemory?.stats();
await deps.closeSession(ctx.session);
ctx.session = await factory.create();
sessionSegments = 0;
if (why === "cadence") browserRecycles += 1;
log.info(`[Render] segment browser recycled (${why})`, {
rendererRssPeakMb: memory?.rendererRssPeakMb,
rssLastMb: memory?.rssLastMb,
samples: memory?.samples,
});
skipSize: skip.size,
workerCount: Math.max(1, input.workerCount ?? 1),
recycleEvery: input.browserRecycleEverySegments ?? 0,
};
const queue = createSegmentQueue(pending);
// Built from the plan, not from completion order: workers finish out of
// order, so the concat list must not depend on who finished when.
const segmentPaths = segments.map((s) => segmentOutputPath(segmentDir, s.index));
let fellBackToStreaming = false;
/**
* One segment, with its cadence recycle and its single target-loss retry.
* Returns "fallback" only for a first-pending spawn failure, which is the
* one case the caller can still replan onto the plain streaming path.
*/
/** The encoder never started. Only the first pending segment can fall back. */
const onSpawnFailure = (err: SegmentEncoderSpawnError, segment: SegmentSlice): "fallback" => {
if (segment.index !== firstPending?.index) throw err.reason;
log.warn("[Render] Segment encoder spawn failed; falling back to single-encoder streaming.", {
error: err.message,
outputFormat,
segments: segments.length,
durationSeconds: job.duration,
});
return "fallback";
};
/**
* Chrome losing its target mid-capture is the one failure a fresh browser
* fixes; everything else reproduces, so retrying it only doubles the time to
* the same error. Once, then it propagates.
*/
const retryAfterTargetLoss = async (
err: unknown,
segment: SegmentSlice,
segmentPath: string,
): Promise<void> => {
if (!isTargetLossError(err) || abortSignal?.aborted) throw err;
segmentRetries += 1;
updateCaptureObservability?.({ segmentRetries });
log.warn(
`[Render] segment ${segment.index}: browser target lost; retrying once on a fresh session`,
{ error: err instanceof Error ? err.message : String(err) },
);
deps.removeFile(segmentPath);
await recycleSession("retry");
encodeMs += await captureSegment(ctx, segment);
};
const workers: SegmentWorker[] = [];
for (let id = 0; id < run.workerCount; id++) workers.push(await createSegmentWorker(run, id));
const consoleOf = () => workers[0]?.ctx.session.browserConsoleBuffer ?? [];
let lastBrowserConsole: string[] = consoleOf();
const runSegment = async (segment: SegmentSlice): Promise<"done" | "fallback"> => {
const segmentPath = segmentOutputPath(segmentDir, segment.index);
if (recycleEvery > 0 && sessionSegments >= recycleEvery) await recycleSession("cadence");
updateCaptureObservability?.({ capturePath: "segmented", segmentIndex: segment.index });
try {
encodeMs += await captureSegment(ctx, segment);
} catch (err) {
if (err instanceof SegmentEncoderSpawnError) return onSpawnFailure(err, segment);
await retryAfterTargetLoss(err, segment, segmentPath);
/** Pull segments until the queue drains, another worker failed, or we fell back. */
const runWorkerLoop = async (worker: SegmentWorker): Promise<void> => {
for (;;) {
if (fellBackToStreaming) return;
assertNotAborted();
const segment = queue.next();
if (!segment) return;
if ((await runOneSegment(run, worker, segment)) === "fallback") {
fellBackToStreaming = true;
return;
}
}
sessionSegments += 1;
segmentPaths.push(segmentPath);
return "done";
};
try {
assertNotAborted();
lastBrowserConsole = ctx.session.browserConsoleBuffer;
for (const segment of segments) {
assertNotAborted();
if (skip.has(segment.index)) {
segmentPaths.push(segmentOutputPath(segmentDir, segment.index));
log.info("[Render] segment skipped (resume)", { index: segment.index });
continue;
}
if ((await runSegment(segment)) === "fallback") return { success: false };
}
dedupPerfs.push(getCapturePerfSummary(ctx.session));
// allSettled, not all: closing a session out from under a worker that is
// still capturing orphans its ffmpeg and races the CDP connection, so
// every worker has to stop before the finally runs. The first error is
// rethrown below.
const outcomes = await Promise.allSettled(workers.map((w) => runWorkerLoop(w)));
for (const worker of workers) dedupPerfs.push(getCapturePerfSummary(worker.ctx.session));
const failure = outcomes.find((o) => o.status === "rejected");
if (failure && failure.status === "rejected") throw failure.reason;
} catch (error) {
lastBrowserConsole = ctx.session.browserConsoleBuffer;
lastBrowserConsole = consoleOf();
throw wrapCaptureStageError(error, lastBrowserConsole);
} finally {
lastBrowserConsole = ctx.session.browserConsoleBuffer;
await deps.closeSession(ctx.session);
lastBrowserConsole = consoleOf();
for (const worker of workers) await deps.closeSession(worker.ctx.session);
}
await concatSegments(deps.concat, segmentPaths, videoOnlyPath, abortSignal, cfg);
if (fellBackToStreaming) return { success: false };
await concatSegments(
deps.concat,
segmentPaths,
input.videoOnlyPath,
input.abortSignal,
input.cfg,
);
return {
success: true,
encodeMs,
encodeMs: run.totals.encodeMs,
probeSession: null,
lastBrowserConsole,
workerCount: 1,
workerCount: run.workerCount,
segments: segments.length,
segmentPaths,
segmentRetries,
browserRecycles,
segmentRetries: run.totals.segmentRetries,
browserRecycles: run.totals.browserRecycles,
};
}
@@ -54,7 +54,8 @@ import {
shouldPreferParallelDrawElement,
shouldPreferSingleWorkerDrawElement,
isCaptureParallelStreamRouterEnabled,
isSegmentedCaptureRequested,
shouldSegmentCapture,
SEGMENTED_MIN_SECONDS_AFTER_SOAK,
resolveParallelCaptureMode,
shouldStreamParallelCapture,
shouldUseStreamingEncode,
@@ -3349,16 +3350,68 @@ describe("resolveParallelCaptureMode", () => {
});
});
describe("isSegmentedCaptureRequested", () => {
it("is opt-in via HF_SEGMENTED_CAPTURE=true", () => {
expect(isSegmentedCaptureRequested({})).toBe(false);
expect(isSegmentedCaptureRequested({ HF_SEGMENTED_CAPTURE: "true" })).toBe(true);
expect(isSegmentedCaptureRequested({ HF_SEGMENTED_CAPTURE: " TRUE " })).toBe(true);
expect(isSegmentedCaptureRequested({ HF_SEGMENTED_CAPTURE: "false" })).toBe(false);
// Phase 2a is opt-in only: anything that is not an explicit "true" is off,
// including the values Phase 2d will later treat as a kill switch.
expect(isSegmentedCaptureRequested({ HF_SEGMENTED_CAPTURE: "1" })).toBe(false);
expect(isSegmentedCaptureRequested({ HF_SEGMENTED_CAPTURE: "" })).toBe(false);
describe("shouldSegmentCapture", () => {
const base = {
env: {} as Record<string, string | undefined>,
durationSeconds: 900,
outputFormat: "mp4",
layeredOrEffectRoute: false,
streamingOk: true,
};
it("does not route on duration until the soak flip", () => {
// The threshold ships disabled: the release that introduces the route
// must not also change which route every long render takes.
expect(shouldSegmentCapture(base)).toBe(false);
expect(shouldSegmentCapture({ ...base, durationSeconds: 36_000 })).toBe(false);
});
it("routes long mp4/mov renders once a threshold is set", () => {
const env = { HF_SEGMENTED_MIN_SECONDS: String(SEGMENTED_MIN_SECONDS_AFTER_SOAK) };
expect(shouldSegmentCapture({ ...base, env })).toBe(true);
expect(shouldSegmentCapture({ ...base, env, outputFormat: "mov" })).toBe(true);
expect(shouldSegmentCapture({ ...base, env, durationSeconds: 599 })).toBe(false);
expect(shouldSegmentCapture({ ...base, env, durationSeconds: 600 })).toBe(true);
});
it("is forced at any duration by the explicit opt-in", () => {
expect(
shouldSegmentCapture({
...base,
durationSeconds: 5,
env: { HF_SEGMENTED_CAPTURE: "true" },
}),
).toBe(true);
});
it("honours the kill switch and the exclusions", () => {
const env = { HF_SEGMENTED_MIN_SECONDS: "60" };
expect(shouldSegmentCapture({ ...base, env: { ...env, HF_SEGMENTED_CAPTURE: "false" } })).toBe(
false,
);
// The kill switch beats the force flag's other spellings too.
expect(shouldSegmentCapture({ ...base, env: { HF_SEGMENTED_CAPTURE: "off" } })).toBe(false);
expect(shouldSegmentCapture({ ...base, env, outputFormat: "webm" })).toBe(false);
expect(shouldSegmentCapture({ ...base, env, outputFormat: "png-sequence" })).toBe(false);
expect(shouldSegmentCapture({ ...base, env, outputFormat: "hls" })).toBe(false);
expect(shouldSegmentCapture({ ...base, env, layeredOrEffectRoute: true })).toBe(false);
expect(shouldSegmentCapture({ ...base, env, streamingOk: false })).toBe(false);
// Even the explicit force cannot route an excluded output.
expect(
shouldSegmentCapture({
...base,
env: { HF_SEGMENTED_CAPTURE: "true" },
outputFormat: "webm",
}),
).toBe(false);
});
it("ignores an unparseable threshold rather than routing on it", () => {
expect(shouldSegmentCapture({ ...base, env: { HF_SEGMENTED_MIN_SECONDS: "soon" } })).toBe(
false,
);
expect(shouldSegmentCapture({ ...base, env: { HF_SEGMENTED_MIN_SECONDS: "-5" } })).toBe(false);
expect(shouldSegmentCapture({ ...base, env: { HF_SEGMENTED_MIN_SECONDS: "0" } })).toBe(true);
});
});
@@ -2501,11 +2501,52 @@ export function isCaptureParallelStreamRouterEnabled(
return true;
}
/** Segmented capture opt-in (spec §5 Phase 2). Phase 2d makes it the default for long renders. */
export function isSegmentedCaptureRequested(
env: Readonly<Record<string, string | undefined>>,
): boolean {
return env.HF_SEGMENTED_CAPTURE?.trim().toLowerCase() === "true";
/**
* The duration at which segmented capture is meant to become the default
* (spec §5 Phase 2d). Not the shipped default: that flip is its own release
* step, gated on the 40-minute soak, and shipping it alongside the rest of
* this work would change the route for every long render in the same release
* that introduced the route. Set `HF_SEGMENTED_MIN_SECONDS=600` to opt a
* fleet in ahead of the flip; flipping the constant below is the whole
* change when the soak passes.
*/
export const SEGMENTED_MIN_SECONDS_AFTER_SOAK = 600;
/** No duration routes to segmented capture until the soak gate is cleared. */
const DEFAULT_SEGMENTED_MIN_SECONDS = Number.POSITIVE_INFINITY;
function resolveSegmentedMinSeconds(env: Readonly<Record<string, string | undefined>>): number {
const raw = env.HF_SEGMENTED_MIN_SECONDS;
if (raw === undefined || raw.trim() === "") return DEFAULT_SEGMENTED_MIN_SECONDS;
const parsed = Number(raw);
return Number.isFinite(parsed) && parsed >= 0 ? parsed : DEFAULT_SEGMENTED_MIN_SECONDS;
}
/**
* Whether this render captures in segments. The worker count is not an
* input: any count segments (Phase 2d), so nothing here consults it.
* `HF_SEGMENTED_CAPTURE=true`
* forces it at any duration and `=false` is the kill switch; otherwise it is
* the duration threshold above. The exclusions are the routes whose
* concat-copy or capture loop the segment contract does not cover: webm's VP9
* concat is fragile across ffmpeg versions, png-sequence and gif have no
* encoded video output, and HDR/shader-transition renders run their own
* compositor.
*/
export function shouldSegmentCapture(args: {
env: Readonly<Record<string, string | undefined>>;
durationSeconds: number;
outputFormat: string;
layeredOrEffectRoute: boolean;
/** shouldUseStreamingEncode(cfg, format, 1, duration) at the call site. */
streamingOk: boolean;
}): boolean {
const raw = args.env.HF_SEGMENTED_CAPTURE?.trim().toLowerCase();
if (raw === "false" || raw === "0" || raw === "off" || raw === "no") return false;
if (!args.streamingOk || args.layeredOrEffectRoute) return false;
if (args.outputFormat !== "mp4" && args.outputFormat !== "mov") return false;
if (raw === "true") return true;
return args.durationSeconds >= resolveSegmentedMinSeconds(args.env);
}
export function shouldStreamParallelCapture(args: {
@@ -4145,12 +4186,13 @@ async function executeRenderPipeline(input: {
// excluded routes are the ones whose concat-copy or capture loop the
// segment contract does not cover (webm VP9 concat is fragile, HDR and
// shader transitions run their own compositor).
useSegmentedCapture:
isSegmentedCaptureRequested(process.env) &&
workerCount === 1 &&
(outputFormat === "mp4" || outputFormat === "mov") &&
!hasHdrContent &&
!compiled.hasShaderTransitions,
useSegmentedCapture: shouldSegmentCapture({
env: process.env,
durationSeconds: job.duration,
outputFormat,
layeredOrEffectRoute: hasHdrContent || compiled.hasShaderTransitions,
streamingOk: shouldUseStreamingEncode(cfg, outputFormat, 1, job.duration),
}),
useLayeredComposite,
usePageSideCompositing: usePageSideCompositingForTransitions,
hasHdrContent,
@@ -4398,6 +4440,7 @@ async function executeRenderPipeline(input: {
dedupPerfs,
segmentFrames,
segmentDir,
workerCount: segmentedPlan.workerCount,
browserRecycleEverySegments: resolveSegmentBrowserRecycle(process.env),
completedSegments,
onSegmentComplete: (entry) => {
+15 -1
View File
@@ -34,7 +34,7 @@ Never run two of these concurrently on a dev Mac — the fleet limit is why the
| 2a | `HF_SEGMENTED_CAPTURE=true HF_SEGMENT_FRAMES=1500 render a-single --fps 30 -w 1` | log `Segmented capture complete: 6 segment(s)`; output 300.000 s and exactly 9000 frames; no per-frame PSNR dip at a segment boundary (see below) |
| 2b | kill the 2a render at ~40 %, rerun with `--resume` | log `resuming: N segments complete` + N × `segment skipped (resume)`; output byte-identical to an uninterrupted run; the segment dir is gone afterwards unless `--keep-segments` |
| 2c | 2a with `HF_SEGMENT_BROWSER_RECYCLE=1` | `segment browser recycled (cadence)` once per segment **after the first** (5 for 6 segments), each carrying the session's `rendererRssPeakMb`; output byte-identical to the single-session render |
| 2d | 2a with `-w 4` | four `segment worker` lines; output 300.000 s |
| 2d | 2a with `-w 3` | `Segmented capture complete: 6 segment(s)`; output 300.000 s, 9000 frames, byte-identical to the `-w 1` segmented render |
The Phase 0 mutation check is the same render with `PRODUCER_STREAMING_ENCODE_DURATION_CAP_ENABLED=true`: the gate line must flip to `"enabled":false,"reason":"duration_cap"` and the render must fail at the disk preflight on a host without ~75 GB free.
@@ -108,6 +108,20 @@ Determinism across restarts is also what makes the 2c retry safe: a segment
re-captured on a fresh session is indistinguishable from one that never
failed.
### What the 2d gate is really testing
That the worker count is invisible in the output. Measured 2026-09-17 with
`-w 3`: byte-identical to the single-worker segmented render, 3 m 14 s against
5 m 13 s (1.6x). Segments finish out of order, so a mismatch here would mean
the concat list had picked up completion order.
Do not run more than 3 Chrome fleets at once on a dev Mac (vault: kernel-panic
history) — that is why this gate is `-w 3` and not `-w 4`.
With the default recycle cadence of 3 and 6 segments over 3 workers, each
worker only captures 2 segments, so no `segment browser recycled` line appears
in this gate. Combine with `HF_SEGMENT_BROWSER_RECYCLE=1` to exercise both.
### Known cost: segmented capture is slower
The segmented stage runs the plain `captureFrameToBuffer` loop, while the