Commit Graph
5 Commits
Author SHA1 Message Date
vanceingalls 591235f05c feat(producer): emit eligible_off cohort-sizing signal when parallel-stream router is off 2026-07-11 22:38:04 +00:00
vanceingalls 88c21ccd8b feat(cli): capture_parallel_stream telemetry on success and failure paths 2026-07-11 22:37:25 +00:00
Vance IngallsandClaude Sonnet 5 a355fb2f6b fix(producer,engine,cli): oom wrapping, cancellation, fallback-reason gaps
Three defects found by max-effort code review of this branch:

1. The Bun OOM exact-match regex was defeated by this codebase's own
   parallel-worker error wrapping. executeParallelCapture/formatWorkerFailure
   (parallelCoordinator.ts) always wrap a worker's error as
   "Worker N: <message>", optionally suffixed and joined with other workers'
   segments, all prefixed "[Parallel] Capture failed: ". That wrapping
   defeated the exact-message check for exactly the cohort (deParallelRouter
   routed, N separate Chrome processes) the OOM-drops-to-1 fix targets — a
   real OOM there would retry at the SAME worker count instead of dropping
   to 1. Added a second pattern that recovers the signal by requiring
   "out of memory" appear as the WHOLE content of a "Worker N: ..." segment
   (bounded by end-of-string/"; "), preserving the same exact-match property
   (no bare substring match) while surviving the wrapping. Verified against
   the real wrapping logic, not a hand-typed guess at its shape.

2. shouldRetryViaPinnedFallback didn't exclude cancellation, so aborting a
   render mid-capture on the pinned router/inversion cohort would detour
   through spawning a fresh encoder/capture session before the outer catch's
   RenderCancelledError branch ended the render — delaying "stop" with a
   pointless resource spin-up/tear-down. Added an isCancellation param
   (checked first, before isVerifyError) using the same
   `err instanceof RenderCancelledError || abortSignal?.aborted` check the
   outer catch already uses.

3. deFallbackReason (this PR's new "oom"/"capture_error" values) was set
   locally but never mirrored into RenderCaptureObservability alongside
   deSelfVerifyFallback, so a render that fails AFTER a fallback attempt
   (perfSummary never built) was indistinguishable in render_error telemetry
   from one that never attempted any fallback — undercutting the "how often
   does the OOM retry fire on a render that still ultimately fails"
   question this branch exists to answer. Threaded through
   RenderCaptureObservability → RenderObservabilityTelemetryPayload →
   renderObservabilityTelemetryPayload, mirroring the existing
   deSelfVerifyFallback plumbing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 20:31:24 -07:00
Vance IngallsandClaude Sonnet 5 ec921e143b feat(producer,cli): full telemetry visibility for DE parallel-router/inversion failures
render_error previously carried zero DE-cohort context — a hard failure while
routed (worker crash, OOM, capture timeout from the fixed 3-worker pin
overriding calibration) was indistinguishable from any other failure. The
data existed (RenderCaptureObservability is mutated live and survives into
job.errorDetails on the failure path) but was never projected into the
render_error payload, which only ever drew de_* fields from perfSummary
(success-only).

- RenderCaptureObservability now also records dePreInversionWorkers /
  dePreRouterWorkers — the worker count calibration would have picked absent
  the experiment — so a resource-pressure failure can be correlated with the
  router overriding a lower calibrated count.
- New capture-sourced de_* fields on RenderObservabilityTelemetryPayload,
  shared by trackRenderComplete and trackRenderError. Explicit
  perfSummary-sourced fields still win on render_complete (spread moved
  first in the event object) — this is purely a failure-path fallback.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 15:31:58 -07:00
James RussoandClaude Opus 4.8 733f88cb1f feat(producer,cli): render-reliability telemetry counters for capture hardening (#1850)
Follow-up to the render-reliability batch (#1841/#1842/#1843). Threads two capture-reliability counters through the existing observability → CLI-telemetry pipeline (no new PostHog wiring) so #1842's hardening is measurable on dashboard 1783183:

- transient-retry burn (CaptureAttemptSummary.reason gains "transient-retry"; counted into RenderCaptureObservability.transientRetries on BOTH the recovered and the still-failed paths via a shared helper).
- OOM classification (memoryExhaustionDetected set when describeMemoryExhaustion classifies the failure).

Surfaced as capture_transient_retries + capture_memory_exhaustion_detected render-event props. Tests cover the attempt tagging and the payload mapping.

Further follow-up (different subsystems): encoder-frame-0-exit signal, and P1-3 pre-flight-rejection / P1-4 cli_env_check counters.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 20:57:39 -07:00