mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
fix(producer): recover from worker crashes instead of hanging the render (#1132)
* fix(producer): recover from worker crashes instead of hanging the render Both the shader-transition and png-decode-blit worker pools freed a crashed worker's slot (busy=false, current=null) but left it in the slot list and never marked it dead. A later run() then selected the dead slot via slots.find(s => !s.busy) and dispatched to its terminated worker, where postMessage is a silent no-op (no throw, no reply) — so the task promise never settled. In the HDR hybrid capture loop, which pipelines blends across N DOM workers and awaits every dispatch, that wedges the whole render with no fail-fast. The crash handlers also never drained the queue, so a queued task could wait forever for a slot that had died. Mark a slot dead on error/exit, exclude dead slots from dispatch and from run()'s slot selection, and fail fast: when no live workers remain, reject queued tasks and reject new run() calls rather than hanging. This keeps the pools' existing no-respawn, fail-fast intent; it just actually fails fast instead of wedging. Adds crash-recovery tests to both pools via a fixture worker that throws on its first message, asserting the in-flight task, queued tasks, and subsequent run() calls all settle rather than hang. * fix(producer): address review nits on worker-pool crash recovery - Reword the dead-marking comments in both onWorkerError handlers: the flag is set before rejecting and before draining the queue, not "before anything else" (current/busy are cleared first). - Rename the shader pool's all-slots-die test to match the png pool's equivalent; the size-2 fixture crashes every worker, so there are no surviving workers serving. --------- Co-authored-by: Carlos Alcaraz <193642530+calcarazgre646@users.noreply.github.com>
This commit is contained in:
co-authored by
Carlos Alcaraz
parent
13b1bffe01
commit
0e895cbff7
@@ -22,6 +22,9 @@
|
||||
// Worker entry points loaded dynamically by their *Pool.ts companions.
|
||||
"packages/producer/src/services/pngDecodeBlitWorker.ts",
|
||||
"packages/producer/src/services/shaderTransitionWorker.ts",
|
||||
// Test fixture worker, spawned by path via the pools' workerEntryPath
|
||||
// option from the crash-recovery tests; has no import-graph referrer.
|
||||
"packages/producer/src/services/__fixtures__/crashOnMessageWorker.mjs",
|
||||
"scripts/*.{ts,mjs,js}",
|
||||
"scripts/*/run.mjs",
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user