refactor(producer): make the short-comp band baseline-first and its attribution decisive

Recut after pre-registering the read exposed two flaws in the first cut:

1. Attribution was wrong. de_short_band keyed on frame count + element
   ceiling alone, so a webm render, a compile-gated comp, or a forced
   screenshot at 400f reported "applied" while its routing was untouched —
   poisoning the measurement cohort with unaffected renders and diluting
   any effect toward zero. Now the predicate is evaluated twice (900 floor
   vs band floor) and the band is DECISIVE only when the calls disagree:
   every other eligibility condition passed and only the floor differed.
   The cohort contains exactly the renders whose routing the band decides.

2. A same-release flip is unfalsifiable. composition_element_count ships
   WITH the routing change, so the before-period cannot be filtered to the
   same cohort as the after-period — the comparison would show a speedup
   even if the change did nothing (the after-cohort excludes big comps by
   construction; the before-cohort includes them). Routing is therefore
   gated behind HF_DE_SHORT_BAND_ROUTE, default OFF: this release computes
   and emits the full band decision on every render ("applied" is the
   counterfactual "would have inverted"), a follow-up flips the default.
   Identical cohort selector on both sides of the boundary, and the
   skipped/oversize renders in the same frame band form a concurrent
   control — a difference-in-differences that absorbs secular drift
   (content mix, version-correlated populations, hardware), which a plain
   before/after cannot.

Also: countElementTags now counts HTML void elements. Counting only
closers read an image gallery as a tiny comp and opened the band on
exactly the content most likely to lose it (images skew expensive to
paint). Opening tags stay uncounted — inline scripts' `a < b` would
false-positive. Counter semantics are frozen while the baseline is read:
the distribution the baseline release records must be measured by the
same counter that later gates.

Revert-rate baseline for the pre-registered read, measured over 14d
fleet-wide: the 900+ inversion runs 31,756 inverted / 1,705 reverted =
5.1%. At the benched 1.16-1.24x win and ~1.8x revert cost, expected net
for the band is ~12%. Kill criteria for the flip release: DiD <= 0,
in-band revert rate > 5.1% baseline, or band fallback rate > DE baseline.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
2026-07-29 10:02:05 -07:00
co-authored by Claude Opus 5
parent 0749cd9ff8
commit e9de2fa14f
4 changed files with 130 additions and 88 deletions
@@ -81,13 +81,17 @@ export interface RenderCaptureObservability {
*/
compositionElementCount?: number;
/**
* Why the short-comp band did or did not apply to this render:
* "applied" (frames landed in 250-899 and the element count cleared the
* ceiling), "skipped_elements" (band was open by frame count but the comp
* was too large), or undefined when the frame count made the band
* irrelevant either way. Distinguishes "the new routing chose this" from
* "the pre-existing 900 floor chose this" — otherwise a fleet perf shift is
* unattributable.
* Short-comp band decision, emitted only when the band is DECISIVE — every
* other inversion-eligibility condition passed and only the floor (250 vs
* 900) differed. "applied": the element count cleared the ceiling too, so
* with routing enabled (HF_DE_SHORT_BAND_ROUTE) this render inverts; in the
* baseline release the same value is the COUNTERFACTUAL "would have
* inverted". "skipped_elements": the element ceiling was the only blocker.
* Unset: the band could not have affected this render (ineligible for some
* other reason, or already inverting at 900+). The selector is computed
* identically before and after the routing flip, and the skipped/oversize
* renders form the concurrent control for the difference-in-differences
* read — that is the entire point of the field.
*/
deShortBand?: "applied" | "skipped_elements";
/** DE parallel-router outcome: "routed" (fired, held) | "reverted" (fired, self-verify retry rolled back). */