mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
feat(producer,cli): full telemetry visibility for DE parallel-router/inversion failures
render_error previously carried zero DE-cohort context — a hard failure while routed (worker crash, OOM, capture timeout from the fixed 3-worker pin overriding calibration) was indistinguishable from any other failure. The data existed (RenderCaptureObservability is mutated live and survives into job.errorDetails on the failure path) but was never projected into the render_error payload, which only ever drew de_* fields from perfSummary (success-only). - RenderCaptureObservability now also records dePreInversionWorkers / dePreRouterWorkers — the worker count calibration would have picked absent the experiment — so a resource-pressure failure can be correlated with the router overriding a lower calibrated count. - New capture-sourced de_* fields on RenderObservabilityTelemetryPayload, shared by trackRenderComplete and trackRenderError. Explicit perfSummary-sourced fields still win on render_complete (spread moved first in the event object) — this is purely a failure-path fallback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
3b93f516b4
commit
ec921e143b
@@ -44,8 +44,12 @@ export interface RenderCaptureObservability {
|
||||
deSelfVerifyFallback?: boolean;
|
||||
/** Auto-parallel inversion outcome: "inverted" (fired, held) | "reverted" (fired, self-verify retry rolled back). */
|
||||
deWorkerInversion?: "inverted" | "reverted";
|
||||
/** Worker count the resolver would have used absent the inversion; undefined if it never fired. */
|
||||
dePreInversionWorkers?: number;
|
||||
/** DE parallel-router outcome: "routed" (fired, held) | "reverted" (fired, self-verify retry rolled back). */
|
||||
deParallelRouter?: "routed" | "reverted";
|
||||
/** Worker count the resolver would have used absent the router; undefined if it never fired. */
|
||||
dePreRouterWorkers?: number;
|
||||
protocolTimeoutMs?: number;
|
||||
pageNavigationTimeoutMs?: number;
|
||||
playerReadyTimeoutMs?: number;
|
||||
|
||||
@@ -1970,7 +1970,18 @@ export async function executeRenderJob(
|
||||
);
|
||||
workerCount = 1;
|
||||
}
|
||||
updateCaptureObservability({ workerCount, deWorkerInversion, deParallelRouter });
|
||||
updateCaptureObservability({
|
||||
workerCount,
|
||||
deWorkerInversion,
|
||||
deParallelRouter,
|
||||
// Recorded here (not just in the success-path perfSummary) so a hard
|
||||
// failure while routed/inverted still tells us what worker count the
|
||||
// resolver would have used absent the experiment — the DE-router pin
|
||||
// to 3 workers regardless of calibration is the leading suspect for
|
||||
// any resource-pressure failure unique to this cohort.
|
||||
dePreInversionWorkers: deWorkerInversion ? preRoutingWorkerCount : undefined,
|
||||
dePreRouterWorkers: deParallelRouter ? preRoutingWorkerCount : undefined,
|
||||
});
|
||||
observability.checkpoint("worker_resolution", "resolved", {
|
||||
workerCount,
|
||||
deWorkerInversion: deWorkerInversion ?? "none",
|
||||
|
||||
Reference in New Issue
Block a user