fix(producer): record routing state on the failure path

de_parallel_router is present on 95.4% of render_complete events and 0.83%
of render_error. Capture context itself survives failures fine (capture_mode
is on 98.6% of them), so this is not renders failing before capture — the
routing state specifically is being dropped.

Cause is ordering. deParallelRouter is assigned twice: once before the
capture-observability update, and again inside syncCapturePlan where routing
is actually resolved — including the 'reverted' case, which the earlier
assignment cannot know. The update in between recorded whatever was true
first, so a render that failed while routed reported no routing state at all.
The existing comment at the earlier call site says it is recorded there
precisely so hard failures carry it; that intent was correct and the value
just arrived too late.

This matters for the #2840 ramp specifically. The per-install circuit breaker
only arms on a revert, which requires the render to finish and self-detect —
it cannot catch a crash or hang. Those are exactly the failure modes a
percentage ramp exists to bound, and they were the ones telemetry could not
see.

Also makes the ffprobe contract sweep resilient per entry. A dangling symlink
under packages/studio/data/projects threw ENOENT on stat and aborted the whole
traversal, so every package sorting after 'studio' — both studio-server
callers included — silently stopped being checked. main is currently red on
this. The manifest assertion is what caught it, which is what it was added
for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
2026-08-06 22:41:26 -07:00
co-authored by Claude Opus 5
parent 02a24055c7
commit a3d13e2673
2 changed files with 21 additions and 2 deletions
@@ -3264,6 +3264,15 @@ async function executeRenderPipeline(input: {
usePageSideCompositing: capturePlan.usePageSideCompositing,
hasHdrContent: capturePlan.hasHdrContent,
forceScreenshot: capturePlan.forceScreenshot,
// Re-recorded here because `syncCapturePlan` above is where routing is
// actually decided — including "reverted", which the earlier update
// could not know. Without this, capture observability keeps whatever
// was true before the plan resolved, so a render that failed while
// routed reports no routing state at all: `de_parallel_router` was
// present on 95% of render_complete events and 0.8% of render_error.
// The failure path is the one the rollout is watching.
deWorkerInversion,
deParallelRouter,
});
observability.checkpoint("capture_strategy", "resolved", {
plan: capturePlan.kind,
@@ -173,8 +173,18 @@ function discoverCallers(): { found: string[]; unclassified: string[]; shell: st
for (const entry of readdirSync(dir)) {
if (SKIP_DIRS.has(entry) || entry.startsWith(".")) continue;
const abs = join(dir, entry);
if (statSync(abs).isDirectory()) walk(abs);
else if (isSourceFile(entry)) classify(abs);
// Per-entry, because a single unreadable one used to abort the whole
// traversal: a dangling symlink under packages/studio/data/projects
// threw ENOENT on stat, so every package sorting after `studio` —
// including both studio-server callers — silently stopped being
// checked. Skipping the entry keeps the sweep complete; the manifest
// assertion is what caught the truncation.
try {
if (statSync(abs).isDirectory()) walk(abs);
else if (isSourceFile(entry)) classify(abs);
} catch {
/* unreadable entry (dangling symlink, permissions) — not a caller */
}
}
};
for (const root of SWEEP_ROOTS) {