ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-10 22:20:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(distributed): gate per-worker SwiftShader probe to worker 0 only (#956)

Browse Source 
After #916 moved `assertSwiftShader` from `renderChunk()`'s eager probe
session into `executeWorkerTask`, every parallel worker began running its
own `chrome://gpu` / canvas-WebGL probe. At `chunkWorkerCount=6` (texture
launch at chunks=3) that's 6 concurrent CDP page-loads per chunk × 3
chunks = 18 simultaneous probes. Bench data on dev (12 producer pods × 22
vCPU) showed c=3 worst-case wall-clock at 67.3s, 24.7s above c=6 worst
(42.6s) — pod_total inflates 100s → 147s uniformly across all three
chunks per slow iter, the signature of cluster-level CDP contention
rather than within-pod contention.

Workers within a chunk share the same Chrome binary, flags, and OS/driver
state on a single pod, so worker 0's success is representative for the
rest. Gate the probe via `shouldVerifyWorkerGpu(workerId, config)` so
only worker 0 navigates to the probe page; workers 1..N-1 skip it. The
fail-fast contract still holds at the chunk level (worker 0 still aborts
the chunk if SwiftShader didn't load) — just without the concurrent CDP
traffic.

Expected wall-clock impact: c=3 worst drops from ~67s to in line with
c=6 worst (~42-44s). c=6 (3 workers/pod) and c=8 (2 workers/pod) should
see smaller wins; c=12 (1 worker/pod, sequential branch) is unaffected.

Closes #955.
This commit is contained in:
James Russo
2026-05-18 22:30:13 -04:00
committed by GitHub
parent 7354d61371
commit 17f47f30dd
3 changed files with 63 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/producer/src/services/distributed/renderChunk.ts
+6 -4
View File
@@ -469,10 +469,12 @@ export async function renderChunk(
// Resolve worker count up-front so we can decide whether to bother
// pre-warming a probe session at all. The parallel branch
// (chunkWorkerCount > 1) closes the probe immediately and creates fresh
// per-worker sessions; `executeWorkerTask` now runs its own
// `assertSwiftShader` against each worker session (gated on
// `cfg.browserGpuMode === "software"`), so the safety contract holds
// without the eager pre-probe.
// per-worker sessions; `executeWorkerTask` runs `assertSwiftShader`
// on worker 0 only (gated on `cfg.browserGpuMode === "software"`), so
// the safety contract holds without the eager pre-probe and without
// every worker concurrently navigating to the GL probe page. See
// `heygen-com/hyperframes#955` for the worst-case wall regression that
// motivated gating the probe to worker 0.
//
// Capture-cost calibration based on shader transitions / renderModeHints
// is not threaded through to chunks yet; the in-process renderer's