Review feedback on #2411 (Rames): the crash-survival RenderCaptureObservability
mirror passed deFallbackFailedDb raw/unrounded while the render_complete
perfSummary path rounded to 1 decimal — the same underlying PSNR could ship two
different values to PostHog depending on which event fired. Extracted the
existing inline round/clamp expression (previously duplicated for verifyMinDb
and fallbackFailedDb) into a shared roundDb helper, applied once at the single
point deFallbackFailedDb is derived from the thrown error so both downstream
consumers agree.
Also threads verifyThresholdDb (captured on the error but never propagated,
per the nit) through DrawElementPerfInput/RenderCaptureObservability/render.ts/
telemetry as de_fallback_threshold_db on both events — the HF_DE_VERIFY_MIN_DB
value the failing dB breached, letting ops read "28.4dB failed a 32dB
threshold" directly instead of cross-referencing config.
de_fallback_reason only told you the fallback happened (blank/psnr/oom/
capture_error), not the failing PSNR or frame index — that data existed as
text inside the thrown error's message and was discarded on the way to
telemetry. DrawElementVerificationError now carries structured
frameIndex/failedDb/verifyThresholdDb; the orchestrator reads them via the
new getDrawElementVerificationDetails helper instead of regexing message
text, and both telemetry surfaces (the render_complete perfSummary path and
the crash-survival RenderCaptureObservability mirror) emit
de_fallback_failed_db / de_fallback_frame_index.
Needed to distinguish "32dB vs the 32dB threshold, tune it" from "12dB real
corruption, investigate" during the parallel-router soak — currently that
distinction is invisible.
Three defects found by max-effort code review of this branch:
1. The Bun OOM exact-match regex was defeated by this codebase's own
parallel-worker error wrapping. executeParallelCapture/formatWorkerFailure
(parallelCoordinator.ts) always wrap a worker's error as
"Worker N: <message>", optionally suffixed and joined with other workers'
segments, all prefixed "[Parallel] Capture failed: ". That wrapping
defeated the exact-message check for exactly the cohort (deParallelRouter
routed, N separate Chrome processes) the OOM-drops-to-1 fix targets — a
real OOM there would retry at the SAME worker count instead of dropping
to 1. Added a second pattern that recovers the signal by requiring
"out of memory" appear as the WHOLE content of a "Worker N: ..." segment
(bounded by end-of-string/"; "), preserving the same exact-match property
(no bare substring match) while surviving the wrapping. Verified against
the real wrapping logic, not a hand-typed guess at its shape.
2. shouldRetryViaPinnedFallback didn't exclude cancellation, so aborting a
render mid-capture on the pinned router/inversion cohort would detour
through spawning a fresh encoder/capture session before the outer catch's
RenderCancelledError branch ended the render — delaying "stop" with a
pointless resource spin-up/tear-down. Added an isCancellation param
(checked first, before isVerifyError) using the same
`err instanceof RenderCancelledError || abortSignal?.aborted` check the
outer catch already uses.
3. deFallbackReason (this PR's new "oom"/"capture_error" values) was set
locally but never mirrored into RenderCaptureObservability alongside
deSelfVerifyFallback, so a render that fails AFTER a fallback attempt
(perfSummary never built) was indistinguishable in render_error telemetry
from one that never attempted any fallback — undercutting the "how often
does the OOM retry fire on a render that still ultimately fails"
question this branch exists to answer. Threaded through
RenderCaptureObservability → RenderObservabilityTelemetryPayload →
renderObservabilityTelemetryPayload, mirroring the existing
deSelfVerifyFallback plumbing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
render_error previously carried zero DE-cohort context — a hard failure while
routed (worker crash, OOM, capture timeout from the fixed 3-worker pin
overriding calibration) was indistinguishable from any other failure. The
data existed (RenderCaptureObservability is mutated live and survives into
job.errorDetails on the failure path) but was never projected into the
render_error payload, which only ever drew de_* fields from perfSummary
(success-only).
- RenderCaptureObservability now also records dePreInversionWorkers /
dePreRouterWorkers — the worker count calibration would have picked absent
the experiment — so a resource-pressure failure can be correlated with the
router overriding a lower calibrated count.
- New capture-sourced de_* fields on RenderObservabilityTelemetryPayload,
shared by trackRenderComplete and trackRenderError. Explicit
perfSummary-sourced fields still win on render_complete (spread moved
first in the event object) — this is purely a failure-path fallback.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Address PR #2045 review feedback:
- Share a SubTimelineWaitOutcome type (engine) end-to-end instead of
widening to string across CapturePerfSummary / RenderPerfSummary /
telemetry, so the three layers can't drift.
- Dedupe scriptLoadFailures on push — a 4xx response and its trailing
requestfailed both recorded the same URL, doubling the failed-URL
list in the fail-fast warning.
- Thread the sub-timeline-wait outcome into render_error (not just
render_complete): a render that fail-fasts and then fails downstream
(pollVideosReady, extract, encode) previously dropped this signal on
the floor. dedupPerfs is now function-scoped so the catch path can
read it, same treatment as the existing captureAttempts array.
Follow-up to the render-reliability batch (#1841/#1842/#1843). Threads two capture-reliability counters through the existing observability → CLI-telemetry pipeline (no new PostHog wiring) so #1842's hardening is measurable on dashboard 1783183:
- transient-retry burn (CaptureAttemptSummary.reason gains "transient-retry"; counted into RenderCaptureObservability.transientRetries on BOTH the recovered and the still-failed paths via a shared helper).
- OOM classification (memoryExhaustionDetected set when describeMemoryExhaustion classifies the failure).
Surfaced as capture_transient_retries + capture_memory_exhaustion_detected render-event props. Tests cover the attempt tagging and the payload mapping.
Further follow-up (different subsystems): encoder-frame-0-exit signal, and P1-3 pre-flight-rejection / P1-4 cli_env_check counters.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Vance Ingalls