docs(canary): make calibration check 4 compare per-surface values

As written, check 4's condition (dual-surface AND value flipped anywhere)
was a strict subset of check 3's (value flipped) — if check 3 read zero,
check 4 was vacuously zero and added no independent signal. Compare the
value each surface actually reported instead, and state plainly that any
disagreement is also a check-3 flip: this check's job is attributing such
a flip to binding divergence. Matching fix applied to the live PostHog
tile (insight jQi7QdW1, dashboard 1918875).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
2026-07-30 15:14:59 -07:00
co-authored by Claude Opus 5
parent 6f6f01c8ea
commit c43b4e2d95
+11 -5
View File
@@ -181,17 +181,23 @@ SELECT count() AS installs_that_flipped FROM (
**4. Cross-surface agreement — do CLI and Studio agree for the same install?**
A CLI-launched Studio adopts the CLI's id, so the same install must report the
same cohort on both. Disagreement means the two bindings have diverged.
same cohort on both. Disagreement means the two bindings have diverged. Note
this compares the *value reported per surface* — any disagreement here is also
a check-3 flip, so this check's job is attribution: it isolates the flips that
are binding divergence rather than within-surface instability.
```sql
SELECT count() AS installs_disagreeing FROM (
SELECT distinct_id
SELECT countIf(cli_val != studio_val) AS installs_disagreeing FROM (
SELECT
distinct_id,
anyIf(properties['$feature/canary-calibration-10'], NOT startsWith(event, 'studio')) AS cli_val,
anyIf(properties['$feature/canary-calibration-10'], startsWith(event, 'studio')) AS studio_val
FROM events
WHERE timestamp >= now() - INTERVAL 14 DAY
AND JSONHas(properties, '$feature/canary-calibration-10')
GROUP BY distinct_id
HAVING uniqExact(startsWith(event, 'studio')) > 1 -- seen on both surfaces
AND uniqExact(properties['$feature/canary-calibration-10']) > 1
HAVING countIf(startsWith(event, 'studio')) > 0 -- seen on both surfaces
AND countIf(NOT startsWith(event, 'studio')) > 0
)
```