Commit Graph
21 Commits
Author SHA1 Message Date
Vance IngallsandClaude Opus 5 e9de2fa14f refactor(producer): make the short-comp band baseline-first and its attribution decisive
Recut after pre-registering the read exposed two flaws in the first cut:

1. Attribution was wrong. de_short_band keyed on frame count + element
   ceiling alone, so a webm render, a compile-gated comp, or a forced
   screenshot at 400f reported "applied" while its routing was untouched —
   poisoning the measurement cohort with unaffected renders and diluting
   any effect toward zero. Now the predicate is evaluated twice (900 floor
   vs band floor) and the band is DECISIVE only when the calls disagree:
   every other eligibility condition passed and only the floor differed.
   The cohort contains exactly the renders whose routing the band decides.

2. A same-release flip is unfalsifiable. composition_element_count ships
   WITH the routing change, so the before-period cannot be filtered to the
   same cohort as the after-period — the comparison would show a speedup
   even if the change did nothing (the after-cohort excludes big comps by
   construction; the before-cohort includes them). Routing is therefore
   gated behind HF_DE_SHORT_BAND_ROUTE, default OFF: this release computes
   and emits the full band decision on every render ("applied" is the
   counterfactual "would have inverted"), a follow-up flips the default.
   Identical cohort selector on both sides of the boundary, and the
   skipped/oversize renders in the same frame band form a concurrent
   control — a difference-in-differences that absorbs secular drift
   (content mix, version-correlated populations, hardware), which a plain
   before/after cannot.

Also: countElementTags now counts HTML void elements. Counting only
closers read an image gallery as a tiny comp and opened the band on
exactly the content most likely to lose it (images skew expensive to
paint). Opening tags stay uncounted — inline scripts' `a < b` would
false-positive. Counter semantics are frozen while the baseline is read:
the distribution the baseline release records must be measured by the
same counter that later gates.

Revert-rate baseline for the pre-registered read, measured over 14d
fleet-wide: the 900+ inversion runs 31,756 inverted / 1,705 reverted =
5.1%. At the benched 1.16-1.24x win and ~1.8x revert cost, expected net
for the band is ~12%. Kill criteria for the flip release: DiD <= 0,
in-band revert rate > 5.1% baseline, or band fallback rate > DE baseline.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 10:02:05 -07:00
Vance IngallsandClaude Opus 5 0749cd9ff8 feat(producer): open the DE single-worker inversion to short comps under an element ceiling
31% of fleet renders (24h, v0.7.78+) are DE-eligible comps clamped to
parallel screenshot purely because they sit under the 900-frame inversion
floor — the median fleet render is ~250-600 frames, below every DE entry
threshold. This opens a 250-899 frame band, gated on composition size.

Measured, not assumed. A controlled sweep (fixed synthetic content,
{250,400,600,900}f, single-DE vs parallel-screenshot-W4, 3 reps, capture
mode verified per row, AC power, load-gated) showed single-DE winning
1.16-1.24x at every size — but only for content in constant motion. A
follow-up 2x2 found motion and DOM size pull in OPPOSITE directions, so
neither alone predicts the winner (ratio = ss4/de1, >1 means DE wins):

     24 movers /     0 nodes -> 1.05
    320 movers /     0 nodes -> 1.24
    320 movers /  7000 nodes -> 1.09
     24 movers /  7000 nodes -> 0.96
     24 movers / 20000 nodes -> 0.71
     24 movers / 40000 nodes -> 0.55

DE's wall-clock scales ~0.50ms/element against parallel screenshot's
~0.22ms — drawElement repaints the whole tree per frame while fan-out
amortizes it — so the downside is NOT bounded and a bare floor drop would
have handed a 1.8x regression to large comps. Since motion only ever helps
DE, an element ceiling calibrated at the lowest-motion case is safe at
every motion level; crossover there is ~3.9k, and the default sits at 2500.

The predicate is untouched; the call site picks the floor. Above the
ceiling, or at 900+ frames, behaviour is bit-identical to today — the
change can only add inversions in the new band, never remove one.

Instrumentation, since this ships at full exposure rather than cohorted:
`composition_element_count` on EVERY render (the fleet distribution of the
gate variable is unknown — without it we cannot tell whether 2500 opens the
band for most short comps or almost none, nor re-derive the threshold from
real content), and `de_short_band` = applied | skipped_elements, unset when
the frame count made the band irrelevant, so a fleet perf shift is
attributable to this change rather than to content mix.

Safety is unchanged and already proven on this path: per-frame PSNR
self-verify with screenshot fallback, exactly as the 900+ band has shipped
default-on. Knobs: HF_DE_SHORT_MIN_FRAMES, HF_DE_SHORT_MAX_ELEMENTS (0
disables the band).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 02:36:40 -07:00
Vance IngallsandClaude Opus 5 4520cd240b fix(engine): bucket gpu_renderer + cover the failure cohort (review)
Three review findings on the win32 drawElement PR:

1. gpu_renderer shipped the raw UNMASKED_RENDERER_WEBGL string — unbounded,
   driver-authored, GPU-model-specific, and |-joined across parallel
   sessions, i.e. high cardinality by construction, against this file's own
   convention of sanitizing engine-sourced strings (deGateReason is a
   bucket; error messages go through redactTelemetryString). Now bucketed at
   the source by classifyGpuRenderer to <backend>/<vendor>
   (metal/apple, d3d11/nvidia, swiftshader/other, ...), which is the whole
   analytic signal the win32 rollout needs and nothing else. The raw string
   never leaves the engine.

2. gpu_renderer reached render_complete only, so a crashed render — the
   cohort the field exists to attribute — carried no backend. It now rides
   RenderCaptureObservability (deGpuRenderer, sourced from the live probe
   session like the de_* counters), so both render_complete and
   render_error carry it and a hard failure still reports its GPU backend.
   On render_complete the perfSummary value still wins by spread order.

3. Restore the fallow-ignore-next-line suppression above
   __resetDeParallelRouterTrialStateForTests: CLI test files are not fallow
   entry points, so removing it fails the CI dead-code audit (local
   pre-commit passed only because of its changed-file scope).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 00:55:10 -07:00
James 9b63646c8a fix(producer): type video extraction failures 2026-07-26 19:47:33 +00:00
Via e6cdf4abb1 feat(engine): opt-in per-frame timing on fast-capture fallback path
Field-signal baseline: >=2 fallbacks/hr on darwin/arm64 from filter:blur
and filter:drop-shadow triggers. Fallback path perf is currently untimed,
so we can't know if the overhead is 10% or 10x. This PR adds opt-in
per-frame timing (HF_PROFILE_FALLBACK_CAPTURE=true) that emits p50/p95/p99
+ trigger reason via the observeRenderStage telemetry channel extended in
#2510. Diagnostic surface only -- no perf fix, no behavior change on
healthy paths.

Stack: PR #9 (final) of 9 (base via/escape-hatch-fallback-reproducer).

Signed-off-by: Via
2026-07-16 00:28:40 +00:00
Via 971bcf39ae feat(producer): calibration-aware heartbeat + worker-death terminal-error contract
Field signals ts=1784019503 (heartbeat reports 0 frames during 64s
browser calibration — reads as broken but is healthy) and ts=1784042064
(1292s Windows render hard-exited during video frame extraction with
no final error string — silent worker crash).

Add calibrating/capturing state to heartbeat labels; surface synthetic
terminal error on unexpected worker exit when no explicit error was
emitted.

Stack: PR #6 of 9 (base via/overlay-count-lint).
Signed-off-by: Via <vance@heygen.com>
2026-07-15 23:14:49 +00:00
Via 69b393865a feat(producer): parity telemetry gate for per-clip frame-count invariant
Field signals ts=1784139267 (win32 15-injection blank later clips,
injection-count-scaled) + ts=1784144554 (darwin 147-authored-clip
visibility gap, authored-clip-count-scaled). check/snapshot passes,
final MP4 has blank/missing clips. Extends #2474's videoCount:0 probe
with per-clip captured-vs-expected frame ratio + threshold fail-loud
gate at render finalization.

Stack: PR #4 of 9 (base via/darwin-goto-nav-timeout-hint).
Signed-off-by: Via <via@heygen.com>
2026-07-15 22:46:45 +00:00
Vance Ingalls af923d947c fix(producer,cli): round the fallback dB consistently, thread the verify threshold through
Review feedback on #2411 (Rames): the crash-survival RenderCaptureObservability
mirror passed deFallbackFailedDb raw/unrounded while the render_complete
perfSummary path rounded to 1 decimal — the same underlying PSNR could ship two
different values to PostHog depending on which event fired. Extracted the
existing inline round/clamp expression (previously duplicated for verifyMinDb
and fallbackFailedDb) into a shared roundDb helper, applied once at the single
point deFallbackFailedDb is derived from the thrown error so both downstream
consumers agree.

Also threads verifyThresholdDb (captured on the error but never propagated,
per the nit) through DrawElementPerfInput/RenderCaptureObservability/render.ts/
telemetry as de_fallback_threshold_db on both events — the HF_DE_VERIFY_MIN_DB
value the failing dB breached, letting ops read "28.4dB failed a 32dB
threshold" directly instead of cross-referencing config.
2026-07-14 23:29:49 -07:00
Vance Ingalls a5cbb78ff9 feat(engine,producer,cli): capture failing dB/frame index on drawElement verify fallback
de_fallback_reason only told you the fallback happened (blank/psnr/oom/
capture_error), not the failing PSNR or frame index — that data existed as
text inside the thrown error's message and was discarded on the way to
telemetry. DrawElementVerificationError now carries structured
frameIndex/failedDb/verifyThresholdDb; the orchestrator reads them via the
new getDrawElementVerificationDetails helper instead of regexing message
text, and both telemetry surfaces (the render_complete perfSummary path and
the crash-survival RenderCaptureObservability mirror) emit
de_fallback_failed_db / de_fallback_frame_index.

Needed to distinguish "32dB vs the 32dB threshold, tune it" from "12dB real
corruption, investigate" during the parallel-router soak — currently that
distinction is invisible.
2026-07-14 23:29:49 -07:00
vanceingalls 591235f05c feat(producer): emit eligible_off cohort-sizing signal when parallel-stream router is off 2026-07-11 22:38:04 +00:00
vanceingalls 6d5a7711f3 feat(producer): route eligible parallel screenshot/beginframe renders to streaming 2026-07-11 22:34:46 +00:00
Miguel Ángel 87618eef4c fix(telemetry): expose stalled render stages (#2220)
* fix(telemetry): expose stalled render stages

* fix(telemetry): preserve capture data on terminal stage events

* fix(telemetry): fix calibration TDZ crash, tag encode/assemble, extend heartbeat cadence

capture_calibration referenced captureStageObservationData before its
declaration (later in the same scope), which would throw a ReferenceError
for any render hitting the calibration path. Hoist the closure and split
workerCount's declaration from its resolution so calibration can safely
read it as undefined before capture strategy resolves worker count.

Also address the two non-blocking review items: wire encode/assemble
stages through captureStageObservationData for consistent tagging, and
extend the heartbeat schedule to repeat every 120s after the initial
30/60/120s ramp instead of going dark on stalls beyond two minutes.
2026-07-10 23:59:08 -04:00
Vance IngallsandClaude Opus 4.8 b02703b7c9 fix(cli,producer): opt-in trial polarity + narrowed-fallback-flag docs (review)
Two non-blocking review notes from Rames, both addressed:

1. Trial polarity inverted to OPT-IN: disableDeParallelRouterTrial →
   enableDeParallelRouterTrial. renderLocal is exported, so any programmatic
   consumer (future studio-server path, test harness, distributed runner)
   previously inherited the trial and its process-wide env-var/module-latch
   state without knowing to disable it — and concurrent invocation races
   that state. Now only the CLI's own sequential call sites opt in (the
   single top-level render, and batch at concurrency 1); everyone else gets
   no trial by default. The doc comment names the sequential-invocation
   assumption explicitly.

2. deSelfVerifyFallback semantic narrowing documented at both declarations
   (RenderCaptureObservability + RenderPerfSummary.drawElement): since the
   pinned-fallback retry was widened, the flag means verify-triggered
   SPECIFICALLY — OOM/capture_error fallbacks report false with
   deFallbackReason carrying the reason. Dashboards keyed on
   de_self_verify_fallback=true as "any fallback fired" must migrate to
   de_fallback_reason IS NOT NULL (also called out in the PR body for the
   observability rebuild to pick up).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 17:24:13 -07:00
Vance IngallsandClaude Sonnet 5 a355fb2f6b fix(producer,engine,cli): oom wrapping, cancellation, fallback-reason gaps
Three defects found by max-effort code review of this branch:

1. The Bun OOM exact-match regex was defeated by this codebase's own
   parallel-worker error wrapping. executeParallelCapture/formatWorkerFailure
   (parallelCoordinator.ts) always wrap a worker's error as
   "Worker N: <message>", optionally suffixed and joined with other workers'
   segments, all prefixed "[Parallel] Capture failed: ". That wrapping
   defeated the exact-message check for exactly the cohort (deParallelRouter
   routed, N separate Chrome processes) the OOM-drops-to-1 fix targets — a
   real OOM there would retry at the SAME worker count instead of dropping
   to 1. Added a second pattern that recovers the signal by requiring
   "out of memory" appear as the WHOLE content of a "Worker N: ..." segment
   (bounded by end-of-string/"; "), preserving the same exact-match property
   (no bare substring match) while surviving the wrapping. Verified against
   the real wrapping logic, not a hand-typed guess at its shape.

2. shouldRetryViaPinnedFallback didn't exclude cancellation, so aborting a
   render mid-capture on the pinned router/inversion cohort would detour
   through spawning a fresh encoder/capture session before the outer catch's
   RenderCancelledError branch ended the render — delaying "stop" with a
   pointless resource spin-up/tear-down. Added an isCancellation param
   (checked first, before isVerifyError) using the same
   `err instanceof RenderCancelledError || abortSignal?.aborted` check the
   outer catch already uses.

3. deFallbackReason (this PR's new "oom"/"capture_error" values) was set
   locally but never mirrored into RenderCaptureObservability alongside
   deSelfVerifyFallback, so a render that fails AFTER a fallback attempt
   (perfSummary never built) was indistinguishable in render_error telemetry
   from one that never attempted any fallback — undercutting the "how often
   does the OOM retry fire on a render that still ultimately fails"
   question this branch exists to answer. Threaded through
   RenderCaptureObservability → RenderObservabilityTelemetryPayload →
   renderObservabilityTelemetryPayload, mirroring the existing
   deSelfVerifyFallback plumbing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 20:31:24 -07:00
Vance IngallsandClaude Sonnet 5 ec921e143b feat(producer,cli): full telemetry visibility for DE parallel-router/inversion failures
render_error previously carried zero DE-cohort context — a hard failure while
routed (worker crash, OOM, capture timeout from the fixed 3-worker pin
overriding calibration) was indistinguishable from any other failure. The
data existed (RenderCaptureObservability is mutated live and survives into
job.errorDetails on the failure path) but was never projected into the
render_error payload, which only ever drew de_* fields from perfSummary
(success-only).

- RenderCaptureObservability now also records dePreInversionWorkers /
  dePreRouterWorkers — the worker count calibration would have picked absent
  the experiment — so a resource-pressure failure can be correlated with the
  router overriding a lower calibrated count.
- New capture-sourced de_* fields on RenderObservabilityTelemetryPayload,
  shared by trackRenderComplete and trackRenderError. Explicit
  perfSummary-sourced fields still win on render_complete (spread moved
  first in the event object) — this is purely a failure-path fallback.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 15:31:58 -07:00
Vance Ingalls 0393ba5be2 feat(producer): default-off router for verified parallel drawElement
Promotes the opt-in HF_DE_PARALLEL_STREAM mechanism (#2056) into the
auto-routing decision, gated behind its own default-off flag
(HF_DE_PARALLEL_ROUTER). This is the next step from the 2026-07-08
parallel-DE benchmark verdict: par3/single 1.16-1.36x on real-work
comps >=2,000 frames, no comp anywhere losing to single-worker.

shouldPreferParallelDrawElement mirrors shouldPreferSingleWorkerDrawElement
(#2026) but takes priority over it when both are eligible — its higher
default threshold (HF_DE_PARALLEL_MIN_FRAMES=2000 vs the inversion's 900)
means it only ever picks up the long tail the inversion's own benchmark
didn't cover. Fixed at 3 workers (benchmark-validated; not calibration-
derived), same shape as the inversion pinning to a fixed 1.

resolveParallelRouterRetryPlan mirrors resolveInversionRetryPlan for the
self-verify-failure rollback path: falls back to the ordinary (non-DE)
parallel-disk path at the pre-router worker count. The caller must clear
HF_DE_PARALLEL_STREAM before recomputing useStreamingEncode or the retry
would keep resolving to the parallel-streaming shape.

New telemetry (de_parallel_router, de_pre_router_workers) tags which
render used the router, separate from de_worker_inversion — needed
before the planned telemetry soak can segment revert-rate and
de_verify_min_db to the parallel cohort specifically; today there's no
way to tell those apart from ordinary single-worker DE renders.

Verified end-to-end: HF_DE_PARALLEL_ROUTER=true on a 2,381-frame comp
resolves to 3 workers with 3 separate drawElement sessions and renders
successfully; without the flag, behavior is unchanged (falls through to
the existing single-worker inversion, workerCount=1) — no regression to
current production routing. 114 orchestrator tests pass (15 new).
2026-07-08 21:48:34 -07:00
Vance IngallsandClaude Fable 5 924727a0b4 feat(producer,cli): drawElement priority inversion — single-worker streaming over auto-parallel (#2026)
* feat(producer,cli): drawElement priority inversion — single-worker streaming over auto-parallel

clamp:parallel eats 50% of local renders (1,326/fortnight; DE engagement
stuck at 3.8%) by routing multi-worker renders to unverified screenshot
capture. Benchmarks (2026-07-08, 4 comps x W1/W2/W3/W5) show that above the
~900-frame amortization crossover, single-worker VERIFIED drawElement
streaming beats screenshot-parallel at EVERY worker count (2,380f: 66s vs
109-127s; 3,600f: 33s vs 39-56s; parallel scaling flattens past W2), while
below it DE's fixed init cost loses by <=2.2s.

- shouldPreferSingleWorkerDrawElement (exported predicate + 7 unit tests):
  inverts an AUTO-resolved multi-worker render to workerCount=1 when the
  comp matches the benchmarked configuration — default-on DE (darwin
  hardware clamp upstream), no compile gate, no forced-screenshot hint,
  mp4 output, single-worker streaming eligible, and totalFrames >=
  HF_DE_SINGLE_MIN_FRAMES (default 900; 0 disables). Explicit --workers N
  is always honored.
- Inverted renders keep the probe session and land on the worker-encode
  streaming drain — the ONLY path with runtime self-verification, so this
  moves ~40% of previously-clamped renders onto the verified fast path.
  Comps that later hit an init-time gate (~1.5% of local renders) render
  single-worker screenshot streaming; accepted trade.
- Telemetry: de_worker_inversion on render_complete (orchestrator ->
  perfSummary.workerInversion -> CLI), plus the worker_resolution
  observability checkpoint now records deWorkerInversion.

Validation: e2e matrix on 2,381f comp — auto->5 workers inverted to 1,
DE verified 4x inf PSNR, RENDER_OK; short comp (360f) auto stays 5-worker;
explicit WORKERS=3 honored; HF_DE_SINGLE_MIN_FRAMES=0 disables. Canary
suite 7/7 (PSNRs identical). renderOrchestrator tests 86/86.
tsc/oxlint/oxfmt clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(producer): review fixes — inversion routing guards, calibration skip, retry revert

Max code-review round on the inversion (13 confirmed findings):

- Streaming spawn-failure disk fallback now clamps default-on drawElement
  (deClampReason=disk_path, DE-mode probe closed) exactly like the
  pre-capture clamp — previously it carried useDrawElement=true onto the
  unverified disk path, the hole the verified-path confinement exists to
  close, newly reachable for every inverted render.
- Predicate gained the routing knowledge it was blind to: layered/HDR and
  shader-transition comps (drawElement never runs there), supersampling
  (deviceScaleFactor>1 init gate), a probe session whose init gates already
  disengaged DE, and the PRODUCER_EXPERIMENTAL_FAST_CAPTURE=true explicit
  parallel-DE opt-in (honored like --workers N).
- Eligibility is evaluated BEFORE capture calibration and skips it when the
  inversion pins workers to 1 regardless of the estimate — the throwaway
  calibration browser + sample captures cost ~41s on the 2,381-frame
  benchmark comp (auto render: 111.6s -> 70.1s total).
- Self-verify retry reverts the inversion: the re-render returns to the
  pre-inversion parallel screenshot path (disk) instead of single-worker
  screenshot streaming, the slowest shape for exactly the comps drawElement
  damages.
- HF_DE_SINGLE_MIN_FRAMES="" (set-but-empty) now falls back to the 900
  default instead of aliasing the 0 kill switch.
- Timeout advisory uses the RESOLVED worker count — an inverted render that
  times out no longer prints "Retry with --workers 1" (the configuration
  that just failed).
- Telemetry: deWorkerInversion recorded in capture observability (failed
  renders are attributable), emitted as literal false when not fired
  (queryable denominator), and the drawElement perf input shape is one
  exported DrawElementPerfInput type instead of three copies.
- Tests: requestedWorkers undefined (the value production actually passes)
  + the four new predicate guards; 91/91.

Validation: e2e auto render — calibration skipped (deInversionEligible),
inversion fires, DE verified 4x inf, total 70.1s (was 111.6s);
HF_DE_SINGLE_MIN_FRAMES=0 restores calibration + parallel; canary suite
7/7 (PSNRs identical); tsc/oxlint/oxfmt clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(producer,cli): review round 2 — loss-cohort telemetry, retry-plan helper, boundary tests

- de_worker_inversion is now a tri-state string ("inverted" | "reverted" |
  "none") instead of a boolean: the self-verify retry marks the render
  "reverted" rather than resetting to false, so the dashboard can segment
  the lost-inversion cohort first-class instead of inferring it from
  deSelfVerifyFallback + frame-count joins (james-russo #1).
- The retry rollback is extracted to resolveInversionRetryPlan (pure,
  exported) with unit coverage: pre-inversion worker-count restore,
  streaming re-resolution (multi-worker retry -> disk), "reverted" state,
  null when never inverted (james-russo #2).
- WOULD_RESOLVE_MULTI_WORKER named constant replaces the bare sentinel 2
  (james-russo #5); minFrames: -1 boundary case added (miga #3).

94/94 renderOrchestrator tests; tsc/oxlint/oxfmt clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(producer,cli): emit de_pre_inversion_workers for the parallel counterfactual

The ramp-down decision needs "did DE beat the parallel render it displaced",
not just "did DE beat single-worker screenshot". Emitting the worker count
the auto-resolution chose BEFORE the inversion pinned it to 1 makes the
parallel counterfactual computable per render (screenshot ms/frame from the
verify samples / W x the measured parallel-efficiency curve). Set only when
the inversion fired.

Smoke: 2,381f auto render -> de_worker_inversion="inverted",
de_pre_inversion_workers=5, mode=drawelement, verify armed 4. 99/99 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 15:48:07 -07:00
Vance IngallsandClaude Fable 5 ec06f4bf89 feat(engine,producer): drawElement fast-capture default-on with runtime self-verification safety net (#1998)
* feat(engine,producer): drawElement fast-capture default-on with runtime self-verification safety net

Flip useDrawElement + worker-encode defaults on (HF_DE_BATCH default 4),
clamped in resolveConfig to hosts where drawElement can engage (macOS +
hardware-GPU browser) so page-side shader compositing is untouched
everywhere else; explicit env opt-in keeps attempt-and-gate semantics.

Safety net makes default-on safe: the compile/init gates catch predictable
incompatibility; this catches the intermittent residue no static analysis
can see (stale paints, dropped background images, transient blank frames).

- engine: captureDeVerificationFrames — K=4 (HF_DE_VERIFY) ground-truth
  screenshots at init, after gates + armStaticDedup, BEFORE canvas
  injection (post-injection screenshots show the canvas bitmap, not the
  DOM). Runs the video-injection hook per sample; double-captures so
  rAF-driven text counters settle (a single immediate screenshot captures
  stale text and false-positives). Skips png, <10 frames, implausible
  __hf.duration (infinite-repeat GSAP sentinel).
- producer: guardFrame on both worker-encode drains — rolling-median blank
  guard with retry-once at drain (byte-identical retry ⇒ deterministic dark
  frame, accepted; retry save/restores the static-dedup anchor) + ffmpeg
  PSNR self-verify vs ground truth (HF_DE_VERIFY_MIN_DB, default 32dB;
  natural agreement ≥45dB, damage ≤25dB). Breach dumps the frame pair to
  tmpdir and throws DrawElementVerificationError.
- orchestrator: one-shot retry — on verification error the whole render
  re-runs with forceScreenshot (slower, never wrong); telemetry flag
  deSelfVerifyFallback.
- tooling: de-canary-suite.sh (7-comp release gate with expected verdicts),
  de-gatecheck.sh (init-only corpus routing classifier), we-render.mjs.

Validated: canary suite 7/7; 611-comp routing sample 54% drawelement /
37.5% gated / 8.3% comp-defect; 12/12 risk-band renders clean on bare
defaults (48/48 verify samples); engine suite 888 passed; caught two real
intermittent damage classes in the wild (background-image drop, root-props
offset) that previously shipped silently.

Kill switches: PRODUCER_EXPERIMENTAL_FAST_CAPTURE=false,
HF_DE_WORKER_ENCODE=false, HF_DE_BATCH=0, HF_DE_VERIFY=0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(engine,producer): harden the drawElement self-verification net (max code-review findings)

15 confirmed findings from the adversarial review of the default-on flip;
the load-bearing five:

- Ground-truth capture no longer scrubs GSAP state: seek(0) + forced frame
  FIRST (lazy .from()/overlap tweens record start values on first seek —
  mid-timeline scrubs corrupted them for the whole render, and since DE
  frames and truth shared the corruption, PSNR passed on damaged output),
  then ascending even-spread fractions, page left at frame 0.
- Default-on drawElement is confined to the verified path: resolveConfig
  requires worker-encode (the drain that runs the net), the orchestrator
  disengages the default when the render takes the disk path or parallel
  capture (no drain verification there), and closes a drawElement-initialized
  probe session rather than letting the unverified path reuse it. Explicit
  PRODUCER_EXPERIMENTAL_FAST_CAPTURE=true keeps old attempt-and-gate behavior.
- Blank-frame retry can no longer splice wrong-frame pixels: recapture goes
  through recaptureDrawElementFrameForVerify — no static-dedup shortcut
  (lastEncodeResult runs ahead of the drain) and no "No cached paint record"
  screenshot fallback (post-injection that captures the canvas = the LAST
  drawn frame); any recapture failure falls back the whole render.
- Verify indices derive from the producer-resolved duration
  (CaptureOptions.compositionDurationSeconds) instead of raw __hf.duration,
  so samples always land inside the drained range.
- The platform clamp accepts "auto" GPU mode — the stock CLI resolves auto,
  and the literal-"hardware" clamp made default-on a no-op for the primary
  audience (masked in validation by explicitly-set env).

Also: NaN-safe env parses (HF_DE_VERIFY / HF_DE_VERIFY_MIN_DB / HF_DE_BATCH);
video comps skip verification when the session has no frame injector (probe
sessions — black-video truth false-positived); psnr infrastructure failures
skip the sample instead of failing the render; boundary-saturated sample
indices are skipped; shader-transition comps prefer page-side compositing
over default drawElement and compile-gated comps get page-side compositing
restored; observability.clearFailure un-brands the recovered first streaming
attempt; canary suite exempts known-marginal "any" comps from the cross-path
PSNR gate; dead we-render options removed; clamp tests pin their env.

Validated: canary suite 7/7; auto-GPU bare render engages the full stack;
disk-path and worker-encode-off renders disengage default drawElement;
malformed HF_DE_VERIFY_MIN_DB still verifies at the default threshold;
engine suite 890 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(engine,producer): review fixes — PSC intent, verify-threshold clamp, fail-closed canaries

Addresses miguel-heygen's review on #1998:

- Page-side compositing restore preserves explicit caller intent (blocker):
  resolveConfig now records pageSideCompositingAutoDisabled only when IT
  turned page-side compositing off because drawElement was on; the
  compile-time drawElement gates restore page-side compositing only when
  that flag is set. An explicit enablePageSideCompositing:false from the
  programmatic API or HF_PAGE_SIDE_COMPOSITING=false stays off. Pinned by
  two config tests.
- HF_DE_VERIFY_MIN_DB clamped to [10, 60] with a warning on out-of-range
  values: below ~10dB the check passes severe damage; above ~60dB natural
  encoder differences force a screenshot fallback on every verified render.
- de-canary-suite.sh + de-gatecheck.sh run under set -euo pipefail with
  explicit `|| true` on expected-nonzero commands (render exits handled by
  the suite's own checks, grep no-match, kill/pkill/wait races) and a hard
  FAIL when the PSNR compare produces no value — release canaries fail
  closed. Full suite re-run green (7/7) under the new flags.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 16:29:30 -07:00
James RussoandClaude Opus 4.8 733f88cb1f feat(producer,cli): render-reliability telemetry counters for capture hardening (#1850)
Follow-up to the render-reliability batch (#1841/#1842/#1843). Threads two capture-reliability counters through the existing observability → CLI-telemetry pipeline (no new PostHog wiring) so #1842's hardening is measurable on dashboard 1783183:

- transient-retry burn (CaptureAttemptSummary.reason gains "transient-retry"; counted into RenderCaptureObservability.transientRetries on BOTH the recovered and the still-failed paths via a shared helper).
- OOM classification (memoryExhaustionDetected set when describeMemoryExhaustion classifies the failure).

Surfaced as capture_transient_retries + capture_memory_exhaustion_detected render-event props. Tests cover the attempt tagging and the payload mapping.

Further follow-up (different subsystems): encoder-frame-0-exit signal, and P1-3 pre-flight-rejection / P1-4 cli_env_check counters.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 20:57:39 -07:00
Miguel Ángel f622e5a7ba fix(engine): restore fast screenshot path for viewport captures (#1670) 2026-06-23 15:13:24 -04:00
Miguel Ángel 29d6f1eac9 fix(render): add end-to-end observability (#1248) 2026-06-06 23:55:58 -04:00