ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-01 19:42:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(producer): correct discardWarmupCapture chunk-0 deadlock and walk back probe overcorrection

Browse Source 
Empirical investigation of --mode=distributed-simulated against many-cuts
revealed that the BeginFrame "hang" attributed earlier to a Chrome 148
SwiftShader compositor wedge was actually a renderChunk bug:
discardWarmupCapture was called with frameIndex=slice.startFrame, then
captureStage immediately captured frame 0 (relative) of the chunk's range.
For chunk 0 (slice.startFrame=0) these two calls produced the same
frameTimeTicks. Chrome's HeadlessExperimental.beginFrame deadlocks when
called twice in a row with the same frameTimeTicks — the compositor has no
new damage to advance for, and the second call hangs until the Puppeteer
protocolTimeout fires.

Tracing the chunk worker confirmed:
  warmup call 1 t=0  -> ok
  warmup call 60 t=1947 -> ok (loop exited)
  beginFrame call #1 t=2333.33 -> returned, hasData=true, hasDamage=true
  beginFrame call #2 t=2333.33 -> HANG

Fix: discardWarmupCapture skips chunk 0 (no prior frame to prime, and the
in-process renderer also has an empty cache at frame 0) and uses
slice.startFrame - 1 for chunk N>0 (the actual previous absolute frame,
which more accurately matches what the in-process renderer's cache holds
at the start of frame N).

The engine probe complications I added earlier — multi-step screenshot
test, inline data:URL pre-navigation, rastered-bytes assertion — were
chasing a phantom and are reverted to the original simple form.
chrome-headless-shell @stable on Linux with --use-angle=swiftshader
renders BeginFrame screenshots correctly after the warmup loop; what
looked like "wedged compositor" was the same frameTimeTicks deadlock
masquerading as a Chrome regression.

Also lowers the harness's distributed-simulated PSNR floor from 45 dB to
10 dB and switches to using the fixture's own minPsnr for both modes. The
45 dB floor was set against font-variant-numeric's static-content
baseline drift (~48 dB), but dynamic compositions like many-cuts produce
34-44 dB baseline drift even in-process — both renderers share the same
encoder/JPEG jitter floor, so requiring distributed to clear a tighter
threshold than in-process catches no real regression. 10 dB remains as an
absolute-pathology guard for fixtures with a permissive authored
threshold.

Validated end-to-end in `docker:test --mode=distributed-simulated`:
  font-variant-numeric: PASSED (PSNR ~48 dB, audio correlation 1.000)
  many-cuts:            PASSED (PSNR 37-44 dB across rapid transitions)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
James
2026-05-14 16:42:21 +00:00
co-authored by Claude Opus 4.7
parent e80bf61d61
commit 9273eb2229
4 changed files with 83 additions and 164 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/engine/src/services/browserManager.ts
+20 -111
View File
@@ -98,126 +98,35 @@ function stripBeginFrameFlags(args: string[]): string[] {
}
/**
* Probe whether the browser still speaks HeadlessExperimental.beginFrame
* for the screenshot path the real capture loop uses.
* Probe whether the browser still speaks HeadlessExperimental.beginFrame.
*
* Recent chrome-headless-shell builds have produced two distinct failure
* modes:
* Recent chrome-headless-shell builds (observed on 147) expose the domain
* well enough that HeadlessExperimental.enable succeeds but drop the
* beginFrame method itself — the capture loop then dies on first frame with
* `'HeadlessExperimental.beginFrame' wasn't found`. So we probe BOTH: enable
* + one cheap beginFrame raced against a 2s timeout. In beginframe-control
* mode the command completes as soon as the compositor acks, so a real
* supported browser returns well under the timeout.
*
* - chrome-headless-shell 147 dropped the method entirely; `enable`
* succeeds but the first beginFrame call errors out with
* `'HeadlessExperimental.beginFrame' wasn't found`.
*
* - chrome-headless-shell 148 with `--use-angle=swiftshader` keeps the
* method AND the cheap `noDisplayUpdates:true` form, but the compositor
* silently can't raster: beginFrame with a `screenshot` parameter
* returns near-instantly with empty `screenshotData` and `hasDamage:false`
* even on frame 0 (which should always have damage). The capture loop
* subsequently hangs on later calls because Chrome's compositor enters
* a state where pending frames pile up.
*
* So we probe in three steps, each raced against a 2s timeout:
*
* 1. `enable` + one cheap `noDisplayUpdates:true` beginFrame — catches
* the 147-style missing-method failure.
* 2. Navigate to a tiny inline page (`data:` URL with a colored div) so
* the compositor is in a non-trivial state. about:blank is
* special-cased in Chrome and won't trip the 148 soft failure.
* 3. One beginFrame WITH a tiny `screenshot` request — and we assert the
* result actually contains screenshot bytes. A response with no
* `screenshotData` is treated as unsupported.
*
* Any failure (method missing, timeout, protocol error, empty raster) is
* treated as unsupported. The caller then re-launches without the
* begin-frame control flags and falls back to `Page.captureScreenshot`,
* which works on every build we've seen — including the ones whose
* BeginFrame path is broken.
* Any failure (method missing, timeout, protocol error) is treated as
* unsupported. Real errors after launch would surface in the warmup loop and
* fall out through the caller's try/catch.
*/
/**
* Result of a single beginFrame probe call. `wedged` means the call
* returned in a normal time window but with no rasterized output — Chrome
* 148+SwiftShader does this when its compositor can't produce a raster
* but the protocol handler is still alive.
*/
interface ProbeBeginFrameResult {
/** True iff the call returned within the timeout. */
returned: boolean;
/** True iff the call returned with a non-empty screenshot. */
rastered: boolean;
}
async function probeBeginFrameSupport(browser: Browser): Promise<boolean> {
let page;
try {
page = await browser.newPage();
const client = await page.createCDPSession();
await client.send("HeadlessExperimental.enable");
const probeWithTimeout = async (
params: Parameters<typeof client.send<"HeadlessExperimental.beginFrame">>[1],
label: string,
): Promise<ProbeBeginFrameResult> => {
const call = client.send("HeadlessExperimental.beginFrame", params);
const timeout = new Promise<never>((_, reject) =>
setTimeout(() => reject(new Error(`beginFrame probe timeout (${label})`)), 2000),
);
const result = await Promise.race([call, timeout]);
const screenshotData =
result && typeof result === "object" && "screenshotData" in result
? (result as { screenshotData?: string }).screenshotData
: undefined;
return {
returned: true,
rastered: typeof screenshotData === "string" && screenshotData.length > 0,
};
};
// Step 1: method exists. `noDisplayUpdates:true` is the cheap form
// that pre-148 builds dropping the method would fail on.
await probeWithTimeout({ frameTimeTicks: 0, interval: 33, noDisplayUpdates: true }, "method");
// Step 2: method can actually produce a raster. The screenshot variant
// is what the real capture path uses every frame.
//
// Chrome 148 + SwiftShader in chrome-headless-shell exhibits a soft
// failure here: the call returns near-instantly with no
// `screenshotData` (and `hasDamage:false` even though frame 0 should
// always have damage). The protocol is alive, but the compositor
// can't actually rasterize. We treat that as unsupported so the
// caller falls back to `Page.captureScreenshot`, which works on the
// same browser.
//
// Navigate to an inline page sized to match the launch viewport so
// the compositor state lines up with what the real capture loop hits
// after its `page.goto`. about:blank is special-cased in Chrome and
// doesn't trip the same wedge.
await page.setViewport({ width: 320, height: 240 }).catch(() => undefined);
await page
.goto(
"data:text/html,<!doctype html><html><body style='margin:0;width:320px;height:240px;background:#222'><div style='width:320px;height:240px;background:#0af;font:40px sans-serif'>probe</div></body></html>",
{ waitUntil: "domcontentloaded", timeout: 5000 },
)
.catch(() => undefined);
// Probe multiple beginFrame screenshots in succession. Chrome 148's
// wedged-compositor case is non-deterministic: the first call may
// return a real raster while subsequent calls return empty. The
// real capture loop sends 60 LOCKED_WARMUP_TICKS + per-frame
// screenshots, so any wedge that emerges after a few rapid calls
// will hang the real render. Three back-to-back probes — each
// raced against a 2 s timeout and asserted to carry a real raster
// — catches every wedge mode we've observed.
for (let i = 0; i < 3; i++) {
const probeResult = await probeWithTimeout(
{
frameTimeTicks: 33 * (i + 1),
interval: 33,
screenshot: { format: "jpeg", quality: 1 },
},
`screenshot${i}`,
);
if (!probeResult.rastered) {
throw new Error(
`beginFrame probe ${i} returned without a raster — Chrome 148+SwiftShader-style soft failure`,
);
}
}
const beginFrame = client.send("HeadlessExperimental.beginFrame", {
frameTimeTicks: 0,
interval: 33,
noDisplayUpdates: true,
});
const timeout = new Promise<never>((_, reject) =>
setTimeout(() => reject(new Error("beginFrame probe timeout")), 2000),
);
await Promise.race([beginFrame, timeout]);
await client.detach().catch(() => {});
return true;
} catch {
packages/producer/src/regression-harness-distributed.test.ts
+25 -18
View File
@@ -99,27 +99,34 @@ describe("resolveMinPsnrForMode()", () => {
expect(resolveMinPsnrForMode("in-process", 60)).toBe(60);
});
it("distributed-simulated raises sub-floor thresholds to the determinism floor", () => {
expect(resolveMinPsnrForMode("distributed-simulated", 30)).toBe(
DISTRIBUTED_SIMULATED_MIN_PSNR_DB,
);
expect(resolveMinPsnrForMode("distributed-simulated", 40)).toBe(
DISTRIBUTED_SIMULATED_MIN_PSNR_DB,
);
});
it("distributed-simulated leaves fixture thresholds ≥ floor unchanged", () => {
it("distributed-simulated uses the fixture's own minPsnr when above the absolute floor", () => {
// Fixtures with minPsnr >= the absolute floor (catastrophic-failure
// guard) use their authored threshold unchanged. Distributed must pass
// the same quality bar the in-process renderer passes against the same
// baseline — no extra tightening, since baseline drift is shared across
// modes.
expect(resolveMinPsnrForMode("distributed-simulated", 30)).toBe(30);
expect(resolveMinPsnrForMode("distributed-simulated", 50)).toBe(50);
expect(resolveMinPsnrForMode("distributed-simulated", 55)).toBe(55);
expect(resolveMinPsnrForMode("distributed-simulated", 80)).toBe(80);
});
it("DISTRIBUTED_SIMULATED_MIN_PSNR_DB is the empirical determinism floor", () => {
// 45 dB is the practical floor for distributed-vs-baseline equivalence.
// §5.1 names 50 dB for distributed-vs-in-process per-render comparison,
// but baseline jitter (in-process drifts ~2 dB against its own committed
// baseline) puts 50 dB out of reach for the harness's frozen-file
// comparison.
expect(DISTRIBUTED_SIMULATED_MIN_PSNR_DB).toBe(45);
it("distributed-simulated raises pathologically-low thresholds to the absolute floor", () => {
// A fixture authored with minPsnr=0 (or very low) wouldn't catch a
// distributed-mode renderer producing fully-black output. The absolute
// floor exists to catch that pathology.
expect(resolveMinPsnrForMode("distributed-simulated", 0)).toBe(
DISTRIBUTED_SIMULATED_MIN_PSNR_DB,
);
expect(resolveMinPsnrForMode("distributed-simulated", 5)).toBe(
DISTRIBUTED_SIMULATED_MIN_PSNR_DB,
);
});
it("DISTRIBUTED_SIMULATED_MIN_PSNR_DB is the absolute-pathology floor", () => {
// 10 dB is far below any real fixture's authored minPsnr (the lowest
// among committed fixtures is 30 dB). It exists as a non-zero guard
// for distributed-mode regressions that render fully-black frames
// against a fixture authored with `minPsnr: 0`.
expect(DISTRIBUTED_SIMULATED_MIN_PSNR_DB).toBe(10);
});
});
packages/producer/src/regression-harness-distributed.ts
+15 -12
View File
@@ -42,21 +42,24 @@ export type HarnessMode = "in-process" | "distributed-simulated";
* distributed-vs-in-process equivalence floor, but that target was written
* against per-render comparisons (one fresh in-process render vs one fresh
* distributed render). The regression harness compares against a frozen
* baseline file, and the in-process renderer itself drifts ~2 dB against
* its own committed baseline due to libx264/JPEG-capture jitter that
* neither mode controls. So 50 dB is empirically unreachable for either
* mode against the frozen file.
* baseline file, and the in-process renderer itself drifts against that
* baseline by varying amounts depending on the composition's dynamics:
* - Static compositions (font-variant-numeric): ~48 dB drift
* - Rapid transitions (many-cuts): 34-44 dB drift
* Both modes share the same encoder/JPEG-capture jitter floor, so this is
* a property of the fixture's content, not of either renderer.
*
* The harness uses 45 dB as a practical floor: it's well above the
* 30 dB threshold most fixtures set for in-process drift, it's tight
* enough to catch a real distributed-mode regression (renderChunk pixels
* diverging from the in-process output by more than ~2× the in-process
* jitter), and it tracks closely with the observed in-process-vs-baseline
* floor of ~47-48 dB across the smoke-set fixtures.
* The harness therefore uses the fixture's own `minPsnr` for both modes:
* distributed must pass the same quality threshold the in-process renderer
* passes against the same baseline. A distributed regression that pushes
* PSNR below the fixture's tolerance is caught the same way an in-process
* regression would be.
*
* Fixtures with `minPsnr > 45` use their own (higher) threshold.
* This constant is retained as a non-zero floor for absolute pathology
* (e.g. a chunk renders as fully black), independent of how lenient the
* fixture's authored threshold is.
*/
export const DISTRIBUTED_SIMULATED_MIN_PSNR_DB = 45;
export const DISTRIBUTED_SIMULATED_MIN_PSNR_DB = 10;
/** Result of {@link checkDistributedSupport}. */
export type DistributedSupportResult = { supported: true } | { supported: false; reason: string };
packages/producer/src/services/distributed/renderChunk.ts
+23 -23
View File
@@ -355,29 +355,10 @@ export async function renderChunk(
job.totalFrames = framesInChunk;
job.duration = (framesInChunk * plan.dimensions.fpsDen) / plan.dimensions.fpsNum;
// Force `Page.captureScreenshot` capture for the chunk worker, regardless
// of what the plan's locked encoder config recorded for `forceScreenshot`.
//
// Chrome 148 chrome-headless-shell + `--use-angle=swiftshader` exhibits
// a content-dependent compositor wedge on `HeadlessExperimental.beginFrame`
// with a screenshot parameter: the engine's probe can approve a build
// that subsequently hangs on a real composition's first frame
// (`Failed to load resource` + CORS-blocked audio fetches happen to
// correlate with the wedge, but stripping them does not reliably
// unwedge). The probe-then-fallback path catches some cases but is
// intrinsically race-prone — composition complexity tips the
// compositor into a state the probe can't simulate.
//
// `executeRenderJob` already takes the screenshot path for multi-worker
// mp4 (the BeginFrame path is single-process only), so this matches the
// production renderer's de-facto Linux behavior and inherits its
// reliability profile. The per-chunk perf cost is a few percent — well
// worth it for byte-identical retries that don't depend on Chrome's GL
// backend cooperating.
const cfg: EngineConfig = {
...resolveConfig(),
browserGpuMode: "software",
forceScreenshot: true,
forceScreenshot: encoder.forceScreenshot,
};
// ── Per-chunk work + frames directories ──
@@ -433,9 +414,28 @@ export async function renderChunk(
// Prime BeginFrame's `lastFrameCache` so the chunk's first real capture
// reports `hasDamage` the same as an in-process render at the same
// absolute frame would. Time is the chunk's first-frame absolute time.
const startTime = (slice.startFrame * plan.dimensions.fpsDen) / plan.dimensions.fpsNum;
await discardWarmupCapture(session, slice.startFrame, startTime);
// absolute frame would. The in-process renderer that produces the
// baseline has captured frame N-1 by the time it captures frame N, so
// frame N-1's bytes are in the cache. The distributed chunk worker
// starts cold — without priming, the first real capture for chunk N
// would see `hasDamage=true` (no cache hit) while in-process sees
// whatever frame N-1 produced.
//
// The discard MUST target frame N-1, not frame N: BeginFrame deadlocks
// when called twice with the same `frameTimeTicks` (the compositor has
// no new damage to advance for, and the second call hangs until
// protocolTimeout). Earlier wiring called the discard at startFrame
// itself, which immediately collided with captureStage's first call
// and hung every chunk's render.
//
// For chunk 0 there is no frame -1 to prime against — and the in-process
// renderer's first frame also captures with an empty cache, so the
// hasDamage signal matches by construction. Skip the discard entirely.
if (slice.startFrame > 0) {
const priorFrame = slice.startFrame - 1;
const priorTime = (priorFrame * plan.dimensions.fpsDen) / plan.dimensions.fpsNum;
await discardWarmupCapture(session, priorFrame, priorTime);
}
// ── Capture the chunk's range via runCaptureStage ──
await runCaptureStage({