Miguel R4 blocker on #2359: my R3 fix at renderOrchestrator only updated
the observability copy, leaving the authoritative captureForceScreenshot
local at compileResult.forceScreenshot (false for auto→software). The
frameCapture side clamped its own local and correctly routed screenshot,
but downstream orchestrator code overwrote observability back to
beginframe from the still-false local at two sites:
- Parallel-stream label at renderOrchestrator.ts:2293 mis-labelled the
stream as 'beginframe' when actual capture was 'screenshot'.
- capture_strategy telemetry at renderOrchestrator.ts:2440-2450
overwrote the earlier observability correction, so the final
captureMode observation flipped back to 'beginframe' while the
engine actually captured screenshot.
Fix: extract the clamp into a caller-facing helper
applyConcreteGpuScreenshotClamp(current, resolvedGpuMode, cfg) that
returns the (possibly-promoted) new boolean. Callers assign it back to
their authoritative local, so routing + telemetry + strategy code read
one value.
Changes:
- packages/engine/src/config.ts: new exported
applyConcreteGpuScreenshotClamp; delegates to
shouldClampToScreenshotForConcreteGpu but computes the caller's
final value, not just the clamp decision. Reads the programmatic
opt-out from cfg.forceScreenshotExplicitlyOptedOut. Idempotent on
already-true input.
- packages/engine/src/index.ts: export the new helper.
- packages/engine/src/services/frameCapture.ts: replace the inline
OR expression with applyConcreteGpuScreenshotClamp.
- packages/producer/src/services/renderOrchestrator.ts: assign result
into the AUTHORITATIVE captureForceScreenshot local (was updating
only observability). Downstream parallel-stream label at :2293 and
capture_strategy telemetry at :2440-2450 now read the corrected
value.
Tests: 6 new caller-level cases for applyConcreteGpuScreenshotClamp
covering the exact matrix Miguel called out:
- resolved software + default false → promotes to true (screenshot)
- resolved software + programmatic opt-out → stays false (BeginFrame)
- resolved hardware + default false → stays false
- resolved software + already-true → stays true (idempotent)
- resolved software + env PRODUCER_FORCE_SCREENSHOT=false → stays false
- resolved software + undefined cfg → promotes to true (frameCapture path)
Local: 67/67 engine config tests pass (was 61). oxfmt clean.
Miguel R3 blocker on #2359: the runtime helper only checked the env opt-out
(PRODUCER_FORCE_SCREENSHOT=false), silently defeating the documented
programmatic escape hatch (overrides.forceScreenshot === false) on the
browserGpuMode:'auto' → software probe path. At the concrete-resolution
site the boolean forceScreenshot === false is ambiguous between default
and explicit opt-out — resolveConfig sees the provenance but the runtime
helper does not.
Fix: persist provenance on the resolved config.
- New INTERNAL EngineConfig field forceScreenshotExplicitlyOptedOut, set
by resolveConfig when EITHER env or programmatic explicit-false is
present. Purpose-documented in the type as 'not intended to be set by
callers'.
- shouldClampToScreenshotForConcreteGpu gains an opts.programmaticOptOut
parameter; returns false early when set. Env stays as the third arg
(backward compatibility with existing tests).
- frameCapture.ts and renderOrchestrator.ts pass
config.forceScreenshotExplicitlyOptedOut through at both call sites, so
the auto→software probe path preserves the same escape hatches as
literal browserGpuMode:'software'.
New tests: 5 additional cases across the helper (programmatic opt-out
alone; programmatic beats missing env) and resolveConfig provenance
(programmatic sets flag; env sets flag; neither leaves it undefined).
Local: 61/61 engine config tests pass (was 56).
The SDR-to-HDR extraction tests synthesize their HDR fixture with
-color_trc/-color_primaries flags and rely on the encoder propagating
them into the bitstream. The pinned Windows CI ffmpeg build drops the
transfer on that path, so after #2377 narrowed HDR detection to the
transfer function, the fixture probes as SDR on Windows and both tests
fail (hdrPreflightCount 0). Write the VUI directly with the
h264_metadata bitstream filter so the tag survives on every build.
Addresses Miguel's R1 blockers:
1. `browserGpuMode: "auto"` that runtime-probes to software slipped past the
`resolveConfig` clamp — that clamp only sees the pre-resolve string. Add
`shouldClampToScreenshotForConcreteGpu(resolvedGpuMode, currentForceScreenshot, env)`
in `packages/engine/src/config.ts` and apply it at BOTH concrete-resolution
sites:
- `packages/engine/src/services/frameCapture.ts`: downgrades `preMode`
from "beginframe" to "screenshot" when resolved GPU is software (respects
`PRODUCER_FORCE_SCREENSHOT=false` env opt-out), fixing the routing.
- `packages/producer/src/services/renderOrchestrator.ts`: updates
`captureObservability.forceScreenshot` (and thus `captureMode`) at the
same call site, fixing the observability truth on the auto → software
case.
2. New unit tests in `config.test.ts`:
- Documents the auto-branch gap (resolveConfig leaves auto as
forceScreenshot=false — the runtime companion closes it).
- 5 branch tests on `shouldClampToScreenshotForConcreteGpu` covering
software / hardware / already-forced / env-opt-out / non-"false" env
values.
Full suite: 56/56 pass.
Scope narrowing on Blocker 2: the distributed rendering path at
`packages/producer/src/services/distributed/plan.ts:753-754` and
`renderChunk.ts:462-466` explicitly hardcodes `browserGpuMode:"software",
forceScreenshot:false` post-resolveConfig and stays outside this PR's
invariant boundary. `compileStage` may still flip it to true for alpha
formats, but generic MP4 distributed renders on SwiftShader hosts remain
BeginFrame. That's a separate architectural cleanup (needs its own
behavior-change trace); the PR body now scopes the invariant to the
in-process CLI/orchestrator path.
When `browserGpuMode === "software"`, set `forceScreenshot = true` in
`resolveConfig`. Explicit opt-outs (`PRODUCER_FORCE_SCREENSHOT=false`
or `overrides.forceScreenshot === false`) are honored.
This is defense-in-depth on top of the existing platform gates:
1. Linux + software (SwiftShader host) skips BeginFrame, avoiding the
compositor stall on shader-heavy frames under CPU raster (same
motivation as the closed PR #822).
2. `renderOrchestrator`'s reported `captureMode` field is derived from
`cfg.forceScreenshot ? "screenshot" : "beginframe"` — without this
clamp it misreports `"beginframe"` for the actual screenshot capture
on darwin + software.
3. Any new BeginFrame or drawElement entry point that forgets to gate
on GPU mode still routes to screenshot here.
Does NOT fix SwiftShader-on-darwin text-rasterization artifacts (an
ANGLE-SwiftShader issue on macOS text — the fix there is to use
`--browser-gpu`, which routes to `--use-angle=metal`).
## What
Two fixes from adversarial testing of the DE parallel router (10 hostile comps, routed vs screenshot-baseline PSNR). The router itself held — both bugs are in general drawElement fast capture, and one slipped past self-verify.
### 1. Compile gate: ancestor background-image (`producer`)
`drawElementService`'s per-frame ancestor fill replicates what lies behind the captured subtree by walking up the DOM for the nearest non-transparent **`backgroundColor`**. A background-**image** (`linear-gradient`, `url()`) on `body`/`html`/a wrapper reads as transparent in that scan, so a deeper ancestor's solid color paints instead wherever the subtree leaves pixels uncovered.
Measured repro: body `linear-gradient` + html solid color + an element shrinking late in the comp → DE paints the html purple instead of the body gradient. 30.9 dB min frame vs baseline, visually unmistakable. Identical damage single-worker and parallel — general DE bug, in every wild DE render matching this (very common) authoring pattern.
Fix: `detectAncestorBackgroundImage()` in the compiler (DOM-aware — inline styles on the root's ancestor chain + `<style>` rules resolved via `querySelectorAll`, so class-selected wrappers are covered; backgrounds *inside* the root are deliberately not matched). New compile gate `ancestor_background_image`, same shape as the 3D/mix-blend gates, bypass `HF_FAST_CAPTURE_ANCESTOR_BG=true`.
### 2. Self-verify tail sample (`engine`)
The verify grid sampled at `(i+1)/(k+1)` → [20/40/60/80]% of the timeline. The damage above starts at ~79% and peaks after the last sample — verification **passed** on output that bottomed at 30.9 dB (threshold 32 dB would have caught it, it just never looked there).
Fix: `computeDeVerifySampleFractions()` — first k−1 samples evenly spaced, last pinned at 95%. Default k=4 grid becomes [25/50/75/95]%. Kills the whole late-onset damage class, not just this repro.
## Validation
- Repro comp (body gradient + shrink reveal): now gates → baseline route, 54.7 dB avg vs ground truth (was 41.6 avg / 30.9 min with the purple surround)
- Control comp (nested stacked fades, routed): still routes, verify grid `[90, 180, 270, 342] of 360`, passes, 60.6 dB avg — unchanged
- Full adversarial matrix context: 6/10 comps routed clean (49–68 dB min), blend/3D gated correctly, animated-canvas damage caught by verify at 16.5 dB with clean revert, video comps route legitimately (frames pre-extracted)
- Tests: 7 new detection cases (`htmlCompiler.test.ts`), 5 new grid cases (`frameCapture-verifySampleFractions.test.ts`); `compileStage.test.ts` + `frameCapture.test.ts` suites green
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Three defects found by max-effort code review of this branch:
1. The Bun OOM exact-match regex was defeated by this codebase's own
parallel-worker error wrapping. executeParallelCapture/formatWorkerFailure
(parallelCoordinator.ts) always wrap a worker's error as
"Worker N: <message>", optionally suffixed and joined with other workers'
segments, all prefixed "[Parallel] Capture failed: ". That wrapping
defeated the exact-message check for exactly the cohort (deParallelRouter
routed, N separate Chrome processes) the OOM-drops-to-1 fix targets — a
real OOM there would retry at the SAME worker count instead of dropping
to 1. Added a second pattern that recovers the signal by requiring
"out of memory" appear as the WHOLE content of a "Worker N: ..." segment
(bounded by end-of-string/"; "), preserving the same exact-match property
(no bare substring match) while surviving the wrapping. Verified against
the real wrapping logic, not a hand-typed guess at its shape.
2. shouldRetryViaPinnedFallback didn't exclude cancellation, so aborting a
render mid-capture on the pinned router/inversion cohort would detour
through spawning a fresh encoder/capture session before the outer catch's
RenderCancelledError branch ended the render — delaying "stop" with a
pointless resource spin-up/tear-down. Added an isCancellation param
(checked first, before isVerifyError) using the same
`err instanceof RenderCancelledError || abortSignal?.aborted` check the
outer catch already uses.
3. deFallbackReason (this PR's new "oom"/"capture_error" values) was set
locally but never mirrored into RenderCaptureObservability alongside
deSelfVerifyFallback, so a render that fails AFTER a fallback attempt
(perfSummary never built) was indistinguishable in render_error telemetry
from one that never attempted any fallback — undercutting the "how often
does the OOM retry fire on a render that still ultimately fails"
question this branch exists to answer. Threaded through
RenderCaptureObservability → RenderObservabilityTelemetryPayload →
renderObservabilityTelemetryPayload, mirroring the existing
deSelfVerifyFallback plumbing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found while testing the previous commit's OOM-drops-to-1-worker fallback
end-to-end: the producer's deployed runtime is Bun (JavaScriptCore), not
Node (V8) — see packages/gcp-cloud-run/Dockerfile's `bun dist/server.js`
entrypoint. All 7 MEMORY_EXHAUSTION_ERROR_PATTERNS are V8-specific allocation
failure signatures; JSC's equivalent for the same single-oversized-allocation
RangeErrors is the bare string "Out of memory" (verified against real Bun
behavior), which none of them match. Without this, isMemoryExhaustionError
returns false for genuine production OOM, so the memory-specific worker-count
reduction just added would never actually engage where it's deployed — every
OOM would fall through to the generic capture_error retry path instead.
Matches the FULL (trimmed) message only, not merely a substring — same
rationale as the existing V8 patterns' comment: "out of memory" also appears
in benign WebGL/GPU console noise that must not trip this classifier.
Verified end-to-end from a script inside the producer workspace (importing
the real @hyperframes/engine source, not a stale globally-cached npm dist a
script outside the workspace would otherwise resolve to): a genuine Bun
RangeError from new Uint8Array(Number.MAX_SAFE_INTEGER) now correctly
classifies as memory exhaustion and drives both resolveInversionRetryPlan
and resolveParallelRouterRetryPlan down to workerCount=1 on retry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Address PR #2093 review feedback (Miga, Rames D Jusso):
- The walker treated a repeating nested timeline (total > single) as an
opaque interval and never descended into it, so a tl.call() living
inside one would slip past hasTimelineCall detection entirely — the
"any tl.call() disqualifies" claim wasn't quite literal. Now recurses
for detection purposes even when the span is already opaque; the
parent-level interval still dominates for frame-animated-marking, so
this only widens what counts as "has a call()," never narrows the
existing interval coverage.
- Restored the totalDuration() vs duration() rationale comment that got
dropped when the tl.call() detection comment was added above it.
Real bug report: a mono count span driven by a GSAP tl.call() (a counter
going "0 sur 0" -> "1 sur 1" at a later beat) rendered the LATER value
baked in from frame 0 of an EARLIER, unrelated static-hold span, despite
the dedup log reporting "verified".
Root cause: computeStaticFrameSet's tween walker only tracks property
tweens, so a call()-driven textContent mutation carries no tracked
interval and the span around it looks fully static. verifyStaticFramesSafe
does catch genuine drift WITHIN a run it's checking, but a call() is a
one-shot side effect wired as both onComplete and onReverseComplete (GSAP
has no separate "undo" — crossing it in either direction fires the SAME
forward mutation). Verifying a LATER run forward-seeks past the call(),
permanently mutating the live page; an EARLIER run already passed its own
check before that happened, so nothing re-verifies it afterward. Real
capture then starts on the same corrupted page and bakes the wrong value
into the earlier span's reused buffer.
No reliable way to tell a DOM-mutating call() from a harmless one
(analytics ping, class toggle) without executing it, so this disqualifies
the whole comp on ANY call() — conservative, costs some dedup perf on
comps that use call() harmlessly, but correctness over speed.
Address two max-effort code-review findings on PR #2056 not covered by
the earlier review-gap commit:
- captureFrameToBufferPipelined's static-dedup reuse branch never
advanced session.lastEncodeResultFrame, unlike its sibling real-capture
branches. The gap-check window is computed from that watermark, so
every consecutive reuse in a static run rescanned an ever-widening
window instead of just the newest frame — O(n^2) total work over a
long static stretch instead of O(n).
- The "single-threaded, no race" justification on the shared
parallelGuard closure was wrong: the guard has real internal await
points (recapture, PSNR) between reading and writing its
sizes/absFloor/acceptedSmall state, so concurrent workers' calls do
interleave there (confirmed). Replaced with the actual reason it's
safe: absFloor only ratchets down, sizes is append-only and
order-independent for the median, and acceptedSmall's fast path
re-validates by exact byte-equality regardless of which worker wrote
the reference buffer.
Address PR #2056 review feedback:
- Fix totalFrames progress inflation for interleaved tasks — divide
each task's span by its frameStride to match the actual per-worker
frame count (captureFrameRange steps by stride), instead of summing
raw endFrame-startFrame which double(N)-counts interleaved tasks.
- Attach a no-op .catch to each frame's pipelined encodeResult at kick
time so an abandoned promise (loop exits early on abort/error before
draining it) can't surface as an unhandled rejection during teardown.
- Document why the pipelined branch's stride=1 path is validation-only
in production (HF_DE_PARALLEL_STREAM always uses interleaved
distribution) so a future refactor doesn't unknowingly widen it.
- Comment the intentional single shared parallelGuard/parallelStats
across workers (safe single-threaded, better rolling-median signal).
Step 2 of the DE engagement plan: multi-worker drawElement capture through
the streaming encoder, with the full runtime self-verification net riding
along — the confinement rule that kept the parallel clamp in place is now
satisfied on this path. Opt-in via HF_DE_PARALLEL_STREAM=true; default
routing (including the #2026 single-worker inversion) is unchanged.
Mechanism:
- distributeFramesInterleaved + WorkerTask.frameStride: worker i captures
frames i, i+N, i+2N... — seek-based capture makes stride free and the
ordered writer's reorder window shrinks from totalFrames/N to N (contiguous
chunks serialize workers behind the writer).
- Depth-2 pipelined worker-encode produce in the parallel worker loop (the
same shape as the sequential loop; frame k's in-page encode overlaps
k+stride's produce). HF_DE_PAR_DEBUG=1 traces the first frames per worker.
- Drain guard extracted to createDrainFrameGuard (session-parameterized):
every parallel frame gets the SAME blank-guard + PSNR self-verify as the
sequential drain, against its owning worker's pre-injection ground truth
(all sessions arm identical sample indices from
CaptureOptions.compositionDurationSeconds).
- FrameReorderBuffer.abort(err): a failed worker (e.g. verification error)
rejects all parked and future waiters — without this, peers park forever
in waitForFrame and the pool (which awaits ALL workers before surfacing
errors) deadlocks. Found by the verify-trip test; unit-tested.
- The typed DrawElementVerificationError is preserved past the pool's
error-string flattening so the orchestrator's verify-retry recognizes it.
- Static-dedup stride hazard fixed: lastEncodeResult reuse now requires EVERY
frame in (lastEncodeResultFrame, i] to be predicted-static (sequential
capture reduces to the old has(i) check).
- Workers get separate browser PROCESSES under the flag: pages co-tenant in
one browser starve non-active pages of BeginFrames on the paint-wait path
(measured 86s vs 30s on a 3,245-frame rAF comp).
Validation:
- Happy path W3: verify samples pass across workers (4x inf on the 2,381f
comp), output vs single-worker DE = 59.3dB (encode noise floor) — the
interleave + dedup-stride produce identical pixels.
- Verify-trip (marginal comp + HF_DE_VERIFY_MIN_DB=45): fails at frame 649
(32.2dB < 45), peers abort instead of deadlocking, whole render retries
via parallel screenshot, RENDER_OK in 42.6s.
- Canary suite 7/7 with the flag off (default paths untouched); producer
orchestrator tests 99/99; engine suite 909 passed (14 pre-existing main
failures, stash-A/B verified); reorder-buffer abort unit tests.
Perf note: capture-only parallel speedup measured 1.38x (W2) / 1.52x (W3)
over single-worker DE in the spike; end-to-end numbers on this machine are
currently noisy (separate-browser init overhead + bench load) — clean
benchmarks before any default routing change. The flag stays explicit
opt-in; promoting it into the router replaces the #2026 W=1 pin for the
same cohort.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Address max-effort code-review finding on PR #2045 (confirmed, not
addressed by the earlier review-gap commit): the script_failure bail
path skipped the composition-id enumeration entirely, so a render with
multiple sub-compositions sharing a failed script only logged the raw
failed URL(s), never which composition(s) were still waiting on it —
a real observability regression versus the pre-#2045 behavior, which
always logged the missing-id list on any non-ready outcome.
Now enumerate unregistered composition ids unconditionally and log them
alongside whichever reason (script_failure or natural timeout) fired.
Address PR #2045 review feedback:
- Share a SubTimelineWaitOutcome type (engine) end-to-end instead of
widening to string across CapturePerfSummary / RenderPerfSummary /
telemetry, so the three layers can't drift.
- Dedupe scriptLoadFailures on push — a 4xx response and its trailing
requestfailed both recorded the same URL, doubling the failed-URL
list in the fail-fast warning.
- Thread the sub-timeline-wait outcome into render_error (not just
render_complete): a render that fail-fasts and then fails downstream
(pollVideosReady, extract, encode) previously dropped this signal on
the floor. dedupPerfs is now function-scoped so the catch path can
read it, same treatment as the existing captureAttempts array.
pollSubCompositionTimelines waits for every [data-composition-id] host to
register window.__timelines[id]. When the script carrying that registration
fails to load (404 / request failure), the registration can never arrive —
but the poll still burned the full playerReadyTimeout (45s), then warned and
shipped a silently animation-less render. Wild scale: the capture-setup
histogram over 30 days of local renders decays smoothly (402/503/364/282/191
per 5s bucket) then spikes to 705 at the 45s bucket — ~1,000 renders/month
across 402 distinct users, ~15 user-hours of pure waiting.
- Sessions now record failed SCRIPT resources (requestfailed + HTTP>=400
response, listeners that already existed for diagnostics) in
session.scriptLoadFailures.
- pollSubCompositionTimelines takes a failure getter and cuts the wait to a
2s grace once any script failed, with a loud warning naming the URL(s).
Late-registering fetch-async comps are unaffected: no script failure means
the full timeout still applies, and a registration landing inside the
grace window still wins (tested).
- Outcome telemetry: session.subTimelineWaitOutcome ("ready" | "timeout" |
"script_failure") -> CapturePerfSummary -> RenderPerfSummary.subTimelineWait
(worst across sessions) -> render_complete sub_timeline_wait, so the wild
rate becomes directly trackable instead of setup-histogram forensics.
Validation: the discovery comp (0768f038, its animations.js unreachable)
drops from ~72s to 23.1s total — poll cut at 2.1s with the script named;
healthy comp reports "ready". Canary suite 7/7 (PSNRs identical). 4 new
poll unit tests; engine suite 907 passed (14 failures are PRE-EXISTING on
main at v0.7.42 — 18 fail on a clean checkout, stash A/B verified).
tsc/oxlint/oxfmt clean.
Corpus note: 258/1,762 corpus comps (14%) reference local scripts missing
from the corpus fetch — their historical eval INIT timings measured this
timeout, not the engine. Capture-stage ratios remain valid (both paths paid
it equally).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Address PR #2082 review feedback:
- Route studio thumbnail + render call sites through preferManagedChrome
so studio renders no longer silently fall back to whatever system
Chrome happens to be installed.
- `hyperframes browser ensure` now resolves through the same
preferManagedChrome path render uses, so it reports what render will
actually pick instead of any system Chrome it happens to find.
- Point the unsupported-Chrome fallback log at `browser ensure --force`
instead of `doctor`, which doesn't check Chrome/drawElement capability.
- Fix stale findFromCache comment: the HF pin is now a Dev-channel build
that can be newer than a user's puppeteer-cache Stable install.
canvas.drawElementImage is an unlaunched Dev/Canary-only Blink feature
(~151+). The CLI's pinned CHROME_VERSION fallback was still 131.0.6778.85 —
a puppeteer 24→25.2.1 bump that pinned it to Chrome Dev 151.0.7912.0 was
written on 2026-06-29 but never merged (orphaned local commit, no PR). Any
render on that pin, or on the shared puppeteer-cache binary, or on system
Chrome (Stable, no drawElementImage at all) got a canvas.getContext("2d")
missing the method and crashed mid-capture with "ctx.drawElementImage is
not a function" instead of falling back (HF#2060).
Three changes:
- Bump puppeteer/puppeteer-core to ^25.2.1 across every package that
depends on it, and CHROME_VERSION to 152.0.7928.2 (today's Dev channel;
confirmed via direct probe to implement drawElementImage, unlike 131).
- `ensureBrowser({ preferManagedChrome: true })`, always used by `render`:
resolve straight to our pinned/cached build, skipping both the shared
puppeteer-cache preference and system Chrome. Rendering shouldn't depend
on whatever arbitrary Chrome a machine happens to have — that's exactly
how this regressed (any Mac with Chrome.app installed bypassed the CLI's
pin entirely).
- A runtime capability probe in the engine, right before any other
drawElement work: if `drawElementImage` isn't a function on the injected
canvas, route to the existing screenshot-fallback gate instead of
crashing. This is the real backstop — it protects every resolution path
(env override, stale cache entry, a future Chrome regression), not just
the ones `preferManagedChrome` reaches.
Verified end-to-end: rendering against chrome-headless-shell 131 (confirmed
to lack drawElementImage) now falls back cleanly and produces a valid MP4
instead of crashing; rendering against a capable build still engages
drawElement normally. 922 engine tests + 1373 CLI tests pass.
Fixes#2060.
Miguel Ángel