* feat(parsers): mark tweens whose duration is not a static number as unresolved
* feat(parsers): detect timing mismatches between attributes and the gsap timeline
* refactor(parsers): simplify unresolved-duration handling
* fix(parsers): treat every unreadable duration shape as unknown
* fix(parsers): keep helpers that unrolled-away calls still run through
* fix(parsers): unroll keeps every helper a remaining call still reaches
* fix(parsers): unroll matches helper names by identifier and skips unresolved selectors
* test(parsers): pin the identifier boundary used to keep unroll helpers
The aggregator runs with always(), so a run cancelled by concurrency posted a red
required check. A cancelled lane now passes only when a newer run of this workflow
exists for the same branch; a manual cancel of the latest run, or any API error,
still fails.
Review follow-ups on the R3 hardening PR:
- A resume that finds every segment complete no longer opens a browser: the
stage closes the probe it was handed, concatenates the plan's segment list
and returns success. A stage test asserts the session factory is never
called on that path.
- retryAfterTargetLoss notes that its recycle also resets the worker's
cadence counter (intended: the fresh browser is as new as a cadence restart).
- The long-form README gains a Known cost section on the narrower retry
surface of segmented capture versus plain streaming, and an operational
note that renders/.hf-segments/<planHash>/ persists after failure by design
and that the plan hash now covers GPU encoder, output resolution and motion
blur.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Four renders on the 60 s fixture, one injected failure each: ffmpeg killed
mid-stream, one worker's Chrome killed, the same under the manual
HF_DE_PARALLEL_STREAM opt-in, and one worker frozen with SIGSTOP under a 5 s
stall window. All four retried on one worker and delivered all 1800 frames;
the worker death surfaced its own error in 0.9 s instead of a 60 s stall
relabel, and the opt-in cohort retried at one worker instead of N. Records
the one caveat: a frozen renderer cannot observe the pool abort, so that
retry waits for the pool to settle.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- HANDOFF section 4a: the single-worker fallback applies on default routing
(non-DE router and the manual opt-in); a DE-router plan keeps its
precomputed fallback. Adds the review's verdicts: three confirmed defects
fixed, one latent hardened, one split accepted, one low left open (a worker
wedged before its first capture is bounded only by player-ready and CDP
timeouts and is not retry-eligible).
- cli.mdx: the parallel-stream paragraph no longer promises a single-worker
retry for every case.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The streaming stage interleaves on forceParallelStream OR
HF_DE_PARALLEL_STREAM=true, but the plan only carried the two routers'
flags. A manual opt-in below the DE router's frame floor therefore retried
at N workers with forceParallelStream false, which the env var re-interleaved
on the second attempt, contradicting the single-worker guarantee the retry
comment and log line made. isParallelStreamForced folds the env var into the
plan flag at creation, so the plan, the retry target and the
deParallelStreamForced telemetry agree with the distribution the stage
picks. Found by the adversarial review of this branch.
Also from that review: the parallel-stall log and checkpoint no longer claim
a "single-worker" retry, which router-pinned plans do not make (they take
their precomputed routing fallback; the post-replan log carries the real
count); the fallbackReason doc comments list encoder_death and
parallel_stall; the peer-death test names the engine test that pins the
hook contract it replays.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The streaming stage's peer-death test replays the contract "hook first, with
the original CaptureFailure, then the peers abort and the pool rejects with
its own summary" inside its mock, and nothing on the engine side pinned it.
The gate now lives in createPoolFailureHandler, tested without a browser:
the hook sees the original failure before the peer signal flips, a hook
that throws can no longer skip the abort (the pool still reports the
classified failure, not the hook's error), failures after the first are
ignored, and a cancellation leaves the peers running. Mutation-checked:
removing the try/finally, swapping the order, or dropping the first-failure
guard each fails one test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- tests/long-form/README: the two-clip in-flow fixture, the per-frame luma
check over clip B, the 2026-09-18 matrix (branch -w 1 / -w 2 disk / -w 2
streaming / absolute CSS / published 0.8.44 all 0 of 100 black; 0.8.35 and
0.8.36 99 of 100 as the positive control; fixed in 0.8.37 by #3893), and
why the multi-worker trace's captureMode was not the engine's mode.
- HANDOFF: R3 row now reads "hardening built"; new section 4a lists the
guards, the 694 gate result and what the flip still needs; counts refreshed.
- cli.mdx: the parallel-stream paragraph says the opt-in path now retries a
stalled pool or a mid-stream encoder exit on a fresh single-worker session.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The capture-stage trace checkpoints read captureMode from the probe session
and, when none is open, assumed "beginframe". The multi-worker path closes
its probe before capture starts, so every macOS and Windows multi-worker
render logged captureMode:"beginframe" while each worker's own
[initSession:screenshot] prefix said otherwise. Found while gating the
PRINFRA-694 fixture on the R3 route. The fallback now goes through the
platform rule (resolveObservedCaptureMode) via a pure helper with tests;
the telemetry field was already derived correctly and is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
shouldRetryViaPinnedFallback is the routing-independent retry allowlist;
until now a parallel-capture stall and an encoder dying mid-stream both
fell through it and hard-failed the render on default routing. Two
predicates join the allowlist:
- isParallelCaptureStallError matches the stage's typed stall by name or by
its "[Render] Parallel <label> capture stalled" message.
- isRetryableEncoderDeath matches "Streaming encoder exited before frame N
was written" for N >= 2, so a session that streamed at least one frame is
retried on a fresh encoder, while a death before any frame (an ffmpeg
argument or codec problem that reproduces) and a host-interrupted encoder
(EncoderInterruptedError, which has its own path) are not.
The fallback reason, log line and checkpoint gain matching parallel_stall
and encoder_death branches, and the stage receives the capture label
(screenshot / drawElement / BeginFrame) so the stall names the engine.
Mutation-checked: removing the allowlist line, the frame threshold, or the
EncoderInterruptedError exclusion each fails one of the new tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
replanAfterFailure's default-routing fallback kept plan.workerCount for
every non-memory failure. For the parallel-stream router's plan that meant
retrying at N workers with forceParallelStream off, which is contiguous-chunk
streaming: worker k+1's first frame waits for all of worker k's, so
throughput serialises to roughly one worker while N Chrome processes run
and the no-progress watchdog can re-trip on the same render. The retry now
targets a single worker for interleaved plans, the fully hardened path
(typed stall, init-excluded clock, routing-independent retry).
Mutation-checked: dropping the forceParallelStream term fails the new test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three of the 5.5 % screenshot-cohort failure modes lived in the parallel
branch of the streaming stage:
- The no-progress watchdog started its clock when the pool was created, so
a fleet whose browsers took longer than HF_DE_STALL_MS to launch was
reported as stalled before any worker had a chance to capture. The timer
now arms on the first frame_capture phase (or first captured frame).
- The stall was a plain Error whose message said "drawElement" regardless
of the capture engine in use. It is now ParallelCaptureStallError with a
variant (drain | pool) and the caller-supplied capture label, so the
orchestrator can route it by name and the log names the engine that
stalled.
- A worker dying mid-render left the reorder buffer parked until the
watchdog fired, which relabelled the real cause as a stall. The stage now
passes an onWorkerFailure hook to executeParallelCapture and aborts the
writer with the worker's own failure the moment the pool declares it
fatal.
Each guard has a test that fails when it is removed (init-hang, hard-coded
label, no-op hook), checked by mutation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On the disk path a transient worker death is deliberately non-fatal: the
orchestrator's adaptive retry re-captures that worker's missing frames. The
streaming path has no per-worker retry and the dead worker's frames are
gone, yet it inherited the same gating — so its peers parked in the ordered
writer waiting for a frame that would never come, and sixty seconds later
the no-progress watchdog relabelled the death as a stall. That is the
`stuck at k/N` field signature.
isPoolFatalWorkerFailure splits the two: every non-cancelled failure ends a
streaming pool, and a new onWorkerFailure hook delivers the ORIGINAL failure
to the caller before peers are aborted, so the ordered writer can be released
with it rather than with a stall. Mutation-checked in both directions,
including that `cancelled` never propagates (the pool is already aborting).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
EPIPE (darwin/linux), EOF (win32's UV_EOF for a closed named pipe) and
ECONNRESET all mean the process reading our stdin has gone. Bucketed as
authoring, telemetry blamed the composition for a host event. Matched on
the errno code and on the message, since the code is lost when a worker
pool flattens the error to a string.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The field "write EPIPE" / "write EOF" render errors are this path:
writeFrame waits on once(stdin, "drain") for back-pressure to clear, ffmpeg
dies first, and Node rejects the drain wait with the stream's own error.
That rejection escaped writeFrame as the render error, so ffmpeg's exit code
and stderr — captured, but read only from the `false`-return path — were
discarded. With N workers feeding one encoder the writer is parked most of
the time, which is why the parallel cohort reads EPIPE where a single-worker
render reads "Streaming encoder exited before frame N: FFmpeg exited with
code …".
Wait for the exit to settle, then report it as the close race does. Both
halves are mutation-checked: rethrowing again fails the test, and returning
"exit" without awaiting settlement fails the synchronous getExitStatus read.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review notes on the Phase 2d PR: shouldSegmentCapture took workerCount and
never read it (any count segments), so the parameter documented an intent
the body did not have. Removed from the signature, call site and tests, and
the doc comment now says worker count is not a gate. The stage input's
sessionFactory doc says it is shared across workers when no per-worker
factory is given and must return a distinct session per create().
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three workers produced a byte-identical file to the single-worker segmented
render at 1.6x the speed, which is the property the gate exists to check:
segments finish out of order, so a mismatch would mean the concat list had
picked up completion order instead of the plan.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shouldSegmentCapture replaces the 2a single-worker gate: the kill switch and
the forced opt-in keep working, and a duration threshold decides the rest.
The threshold ships disabled (the default is Infinity) rather than at the
spec's 600 s. Flipping the route for every long render in the same release
that introduces the route is exactly what the soak gate exists to prevent;
SEGMENTED_MIN_SECONDS_AFTER_SOAK records the intended value, and a fleet can
opt in today with HF_SEGMENTED_MIN_SECONDS=600.
createCapturePlan no longer requires useStreamingEncode for a segmented
plan. That flag answers "one encoder for the whole render", which goes false
for multi-worker — requiring it would have silently dropped every
multi-worker segmented render onto the disk path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each worker owns a session (recycled and retried per 2c) and pulls whole
segments in index order. Segments finish out of order, so the concat list is
built from the plan rather than from completion order — ordering never
reaches the encoders.
Three guards, each mutation-checked: every worker settles before any session
is closed (closing one out from under a live worker orphans its ffmpeg and
races the CDP connection); a multi-worker render never takes the
single-encoder fallback, because other workers are already encoding work it
would discard; and the queue is shared, so no frame is captured twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default deps object captured the engine's closeCaptureSession at module
load, so a test that swaps the engine module after this file is evaluated
never saw the close. The Phase 2a test that checks the probe is closed after
a first-encoder spawn failure failed from this layer upward for that reason.
Calling through the import binding restores the check; production behaviour
is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Recycling the browser on every segment produced a byte-identical file to
the single-session render, which is what makes the target-loss retry safe:
a segment re-captured on a fresh session is indistinguishable from one that
never failed. Five launches cost ~5 s, and each recycle reported a renderer
RSS peak of ~1.4 GB being returned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HF_SEGMENT_BROWSER_RECYCLE defaults to 3 segments, which keeps a session
under the 10k-frame band where field crash rates pass 4%. segmentRetries
lands on live capture observability, so a render that retried still reports
it if it later dies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A segment boundary is the only safe point to hand capture a fresh browser,
which is what makes the long-render failure class survivable: the session
restarts every N segments, and a Chrome target loss mid-segment re-captures
that segment once on a new session instead of failing the render.
Retry is deliberately narrow. Encoder and authoring errors reproduce on a
fresh browser, so retrying them doubles the time to the same failure — all
three guards (retry exists, retry is classified, cadence boundary) are
mutation-checked. The partial segment file is deleted before the retry so a
later resume cannot see it, and onSegmentComplete still fires only after a
successful close, so the manifest never records a partial segment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only the Chrome target-loss class is worth retrying a segment for: encoder
and authoring failures reproduce on a fresh browser, so a retry just doubles
the time to the same failure. Both mutation-checked, including that a
non-Error is never classified as retryable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review findings on the Phase 2b PR. Three inputs that change segment bytes
were missing from computeSegmentPlanHash, so --resume could splice
mismatched segments: the hardware encoder (h264_nvenc and libx264 both
report codec "h264"), the device-scaled capture size (the hash used the
composition's CSS size), and the motion-blur options (sample count changes
every averaged frame). All three are hashed now, each with a test that
fails if it is dropped. The hash and the encoder read the image format
from one binding.
The plan -> options -> request -> config chain that once dropped --resume
and --keep-segments had no test. renderOptionsFromPlan is now a pure,
exported hop with a test, renderConfigFromRequest has one for the same
flags, and the batch path, which was still dropping both flags, carries
them too.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
execute.ts is where the plan becomes RenderOptions, and it dropped both new
flags, so --keep-segments deleted the segment directory anyway and --resume
never resumed. Nothing failed: the render just silently re-captured
everything. Caught by the Phase 2b gate, which is the only check that spans
plan -> options -> request -> config; the gate doc now says so.
Gate result recorded: killed at 2 of 6 segments, resume skipped both,
finished in 3m32 against 5m13, and the output was byte-identical to the
uninterrupted render.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Segments and their manifest live under renders/.hf-segments/<planHash>. A
resumed run re-validates every recorded segment before skipping it; a run
without --resume deletes the directory first, because a previous run's
segments must never be spliced into this output. The directory survives a
failure on purpose — that is the resume point — and is removed after a
successful render unless --keep-segments.
The frame-count check derives from probed duration x fps: the engine has no
exact-count reader and -count_frames decodes the whole file, while the only
thing this must catch is a segment missing frames, which shortens duration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
segmentOutputPath now takes the segment directory, so segments can live in
a stable resume directory instead of the disposable work dir. Skipped
segments are still concatenated in order, and the fallback-on-spawn-failure
rule moves from "segment 0" to "the first segment this run still has to
capture" — on a resume that is not index 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resume reuses encoded bytes, so a wrong reuse corrupts the output silently
instead of failing. Three guards, each mutation-checked: the plan hash
covers every input that determines segment bytes and is separator-joined so
adjacent fields cannot alias; a reused segment must still exist at its
recorded size; and its probed frame count must equal the slice length.
Type guards use `in` + Reflect.get rather than the plan's
`as Record<string, unknown>`, which the repo's no-cast rule forbids.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review finding on the Phase 2a PR: when the first segment's encoder failed
to spawn, the stage's finally had already closed the session it opened,
including a reused probe, but the orchestrator kept its probeSession
reference and handed the closed session to the streaming branch, which
would fail a stage later with a confusing "page closed". The fallback
branch now nulls the reference; the stage test pins that the probe is
closed on that path. Also covers the negative HF_SEGMENT_FRAMES clamp the
review noted was untested.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The spec's "PSNR >= 45 dB vs the single-encoder output" reads as a failure
on a healthy render: the measured average is 44.4 dB because segmented
capture forces a closed GOP with scene-cut detection off, and two CRF
encodes of identical frames differ regardless. The boundary frames are the
HIGHEST-PSNR frames in the series (48-62 dB forced IDRs), which is what the
gate should actually assert.
Also records the cost this buys nothing back on: the segmented stage runs
the plain capture loop rather than the depth-2 worker-encode pipeline, so
the same fixture takes 5m16 against streaming's 2m43.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Opt-in, single-worker, mp4/mov only, and not for HDR or shader-transition
routes — those run their own compositor and webm's VP9 concat-copy is
fragile across ffmpeg versions.
A first-segment encoder spawn failure replans onto the plain streaming
path and falls through to the branch below, so the opt-in cannot make a
render fail where the default would have succeeded. A failure after that
propagates: per-segment retry is Phase 2c.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One closed-GOP encoder per segment writing segments/segment_NNNNN.mp4,
then a concat-copy into the video-only file the assemble stage muxes. The
GOP is each segment's own frame count, so every concat boundary is an IDR;
the tail segment is shorter and says so.
A spawn failure on the first segment returns success:false so the caller
can still replan onto the single-encoder streaming path. A later one
throws: earlier segments are already encoded, and silently discarding them
would drop frames from the output.
Reuses the sequential stall watchdog from the streaming stage rather than
running unguarded; both helpers are exported for that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Inert until the orchestrator passes useSegmentedCapture. replanAfterFailure
handles the kind for every failure, including capture_failure, which would
otherwise reach the "cannot apply" throw guarding the non-streaming kinds:
a drawElement failure keeps segmenting on the screenshot engine, anything
else drops to a plain streaming render.
CapturePlanTarget deliberately does not gain the kind — routing fallbacks
are streaming/disk only, and adding it there would make
`useStreamingEncode: fallback.kind === "sdr_streaming"` silently false for
a segmented target.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Segmented capture needs the same concat-demuxer stream copy the chunked
encoder already does. Behaviour preserved, including the
external_interruption classification the caller maps to a retryable
failure: the helper reports it rather than swallowing it. The concat list
keeps living beside its inputs, because concurrent encodes share an output
directory but not a chunk directory.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-ups on the Phase 0 and Phase 1a PRs:
- The disk-headroom message blamed HDR and shader-transition output for a
throw those renders can never reach (they run hdr_layered, a separate
loop). It now lists png-sequence/gif, multi-worker screenshot capture,
and the three ways streaming is switched off, including the duration cap
and its threshold.
- RenderCaptureObservability.captureParallelStream still described the
pre-split semantics ("the kill switch was on"). It now says BeginFrame
routes by default, screenshot only with the env, that eligible_off sizes
the held-back screenshot cohort, and that the label can disagree with the
platform-only captureMode on Linux with system Chrome or DPR > 1.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Splits the non-DE parallel-stream router default by the capture mode the
render will actually use. BeginFrame (Linux headless-shell) defaults on:
its opt-in cohort fails at 0.05% over 11.5k renders. Screenshot capture
(macOS/Windows) stays behind HF_CAPTURE_PARALLEL_STREAM=true, because its
opt-in cohort fails at 5.5% against a ~1.1% baseline in two classes this
path owns — the parallel stall watchdog hard-fails with no retry, and a
dying encoder surfaces as a bare write EPIPE. Phase 1b flips it once those
are fixed.
resolveParallelCaptureMode mirrors the engine's preMode rather than reusing
resolveObservedCaptureMode, which is platform-only and would have
default-enabled Linux-with-system-Chrome and Linux-at-DPR>1, whose real
capture path is the screenshot one. The routed telemetry value now comes
from the same helper, so the cohort cannot disagree with the predicate that
routed it, and eligible-but-held-back renders keep emitting eligible_off.
The preflight message named --workers 1 as the fix for every disk-path
render; with the duration cap enabled a --workers 1 render lands there too.
It now names each override that can cause it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both events carry the live observability values, so a render_error that
never built a perf summary still reports the last known Chrome memory.
render_complete prefers the aggregate from RenderPerfSummary.chromeMemory,
which covers every worker rather than whichever session sampled last.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SystemInfo.getProcessInfo is served only by the browser target; a
page-target session rejects with "is only supported on the browser
target". The sampler swallows probe failures by design, so this shipped as
a permanently empty reading: a 10 s debug render reported chromeMemory
null with capturePath already correct. Against the browser target the same
render reports browser 105 MB / renderer 359 MB over 3 samples.
readChromePids is extracted so the target choice is pinned by a test whose
fake rejects a page session the way Chrome does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Live route through updateCaptureObservability survives a crash, which is
the case this telemetry exists for: the failure path never builds a
RenderPerfSummary. The aggregate route through CapturePerfSummary (max of
per-session peaks, sum of samples) feeds render_complete.
capturePath is recorded inside syncCapturePlan rather than beside the
streaming-encode gate log, so a render that replans onto disk reports the
path it actually captured on; capturePathForPlanKind is exhaustive over
CapturePlan["kind"] so a new plan kind cannot silently inherit the wrong
label.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Start a 2 s unref'd sampler when a session finishes initializing, stop it
in closeCaptureSession, and report peaks on CapturePerfSummary. Pids come
from puppeteer (browser) and CDP SystemInfo.getProcessInfo (renderer, GPU),
so the long-render "Target closed" class can be correlated with Chrome
memory instead of guessed at. Disable with HF_CHROME_MEMORY_SAMPLER=false;
cadence HF_CHROME_MEMORY_SAMPLE_MS.
The start hook is finalizeSessionInit rather than the tail of
initializeSession: that helper is already the single owner of "session is
ready" across both of initializeSession's exits, so the early screenshot
return cannot silently skip sampling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
