mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-12 15:20:13 +00:00
fix(producer): record routing state on the failure path
de_parallel_router is present on 95.4% of render_complete events and 0.83% of render_error. Capture context itself survives failures fine (capture_mode is on 98.6% of them), so this is not renders failing before capture — the routing state specifically is being dropped. Cause is ordering. deParallelRouter is assigned twice: once before the capture-observability update, and again inside syncCapturePlan where routing is actually resolved — including the 'reverted' case, which the earlier assignment cannot know. The update in between recorded whatever was true first, so a render that failed while routed reported no routing state at all. The existing comment at the earlier call site says it is recorded there precisely so hard failures carry it; that intent was correct and the value just arrived too late. This matters for the #2840 ramp specifically. The per-install circuit breaker only arms on a revert, which requires the render to finish and self-detect — it cannot catch a crash or hang. Those are exactly the failure modes a percentage ramp exists to bound, and they were the ones telemetry could not see. Also makes the ffprobe contract sweep resilient per entry. A dangling symlink under packages/studio/data/projects threw ENOENT on stat and aborted the whole traversal, so every package sorting after 'studio' — both studio-server callers included — silently stopped being checked. main is currently red on this. The manifest assertion is what caught it, which is what it was added for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
02a24055c7
commit
a3d13e2673
@@ -3264,6 +3264,15 @@ async function executeRenderPipeline(input: {
|
||||
usePageSideCompositing: capturePlan.usePageSideCompositing,
|
||||
hasHdrContent: capturePlan.hasHdrContent,
|
||||
forceScreenshot: capturePlan.forceScreenshot,
|
||||
// Re-recorded here because `syncCapturePlan` above is where routing is
|
||||
// actually decided — including "reverted", which the earlier update
|
||||
// could not know. Without this, capture observability keeps whatever
|
||||
// was true before the plan resolved, so a render that failed while
|
||||
// routed reports no routing state at all: `de_parallel_router` was
|
||||
// present on 95% of render_complete events and 0.8% of render_error.
|
||||
// The failure path is the one the rollout is watching.
|
||||
deWorkerInversion,
|
||||
deParallelRouter,
|
||||
});
|
||||
observability.checkpoint("capture_strategy", "resolved", {
|
||||
plan: capturePlan.kind,
|
||||
|
||||
@@ -173,8 +173,18 @@ function discoverCallers(): { found: string[]; unclassified: string[]; shell: st
|
||||
for (const entry of readdirSync(dir)) {
|
||||
if (SKIP_DIRS.has(entry) || entry.startsWith(".")) continue;
|
||||
const abs = join(dir, entry);
|
||||
if (statSync(abs).isDirectory()) walk(abs);
|
||||
else if (isSourceFile(entry)) classify(abs);
|
||||
// Per-entry, because a single unreadable one used to abort the whole
|
||||
// traversal: a dangling symlink under packages/studio/data/projects
|
||||
// threw ENOENT on stat, so every package sorting after `studio` —
|
||||
// including both studio-server callers — silently stopped being
|
||||
// checked. Skipping the entry keeps the sweep complete; the manifest
|
||||
// assertion is what caught the truncation.
|
||||
try {
|
||||
if (statSync(abs).isDirectory()) walk(abs);
|
||||
else if (isSourceFile(entry)) classify(abs);
|
||||
} catch {
|
||||
/* unreadable entry (dangling symlink, permissions) — not a caller */
|
||||
}
|
||||
}
|
||||
};
|
||||
for (const root of SWEEP_ROOTS) {
|
||||
|
||||
Reference in New Issue
Block a user