From c43b4e2d951c229935f65a4e224ef8a66c43dd12 Mon Sep 17 00:00:00 2001 From: Vance Ingalls Date: Tue, 28 Jul 2026 12:04:28 -0700 Subject: [PATCH] docs(canary): make calibration check 4 compare per-surface values MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit As written, check 4's condition (dual-surface AND value flipped anywhere) was a strict subset of check 3's (value flipped) — if check 3 read zero, check 4 was vacuously zero and added no independent signal. Compare the value each surface actually reported instead, and state plainly that any disagreement is also a check-3 flip: this check's job is attributing such a flip to binding divergence. Matching fix applied to the live PostHog tile (insight jQi7QdW1, dashboard 1918875). Co-Authored-By: Claude Opus 5 (1M context) --- docs/contributing/canary-rollouts.mdx | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/contributing/canary-rollouts.mdx b/docs/contributing/canary-rollouts.mdx index cc9574aaf5..f3cf7f5763 100644 --- a/docs/contributing/canary-rollouts.mdx +++ b/docs/contributing/canary-rollouts.mdx @@ -181,17 +181,23 @@ SELECT count() AS installs_that_flipped FROM ( **4. Cross-surface agreement — do CLI and Studio agree for the same install?** A CLI-launched Studio adopts the CLI's id, so the same install must report the -same cohort on both. Disagreement means the two bindings have diverged. +same cohort on both. Disagreement means the two bindings have diverged. Note +this compares the *value reported per surface* — any disagreement here is also +a check-3 flip, so this check's job is attribution: it isolates the flips that +are binding divergence rather than within-surface instability. ```sql -SELECT count() AS installs_disagreeing FROM ( - SELECT distinct_id +SELECT countIf(cli_val != studio_val) AS installs_disagreeing FROM ( + SELECT + distinct_id, + anyIf(properties['$feature/canary-calibration-10'], NOT startsWith(event, 'studio')) AS cli_val, + anyIf(properties['$feature/canary-calibration-10'], startsWith(event, 'studio')) AS studio_val FROM events WHERE timestamp >= now() - INTERVAL 14 DAY AND JSONHas(properties, '$feature/canary-calibration-10') GROUP BY distinct_id - HAVING uniqExact(startsWith(event, 'studio')) > 1 -- seen on both surfaces - AND uniqExact(properties['$feature/canary-calibration-10']) > 1 + HAVING countIf(startsWith(event, 'studio')) > 0 -- seen on both surfaces + AND countIf(NOT startsWith(event, 'studio')) > 0 ) ```