test(studio): the playback gate reports timing and fails only on counts

Timing was meant to be reporting-only, but a ratio over 1.15x still exited
1 and turned the job red; identical builds cross it in 31% of draws. It now
prints a warning (a workflow warning in CI) and exits 0. Only the counts,
a Chrome other than the ceilings', or a broken run fail the step.

On a Chrome mismatch the evidence JSON is printed before the exit, so new
ceilings can be taken from it. The mismatch rule has one owner,
perf-ratchet.mjs browserMismatch, which lower, the check notes and the
script all call. playbackVerdict is exported and tested: a 1.3x ratio
exits 0 with a warning, a mismatch exits 1, and a count rise fails the
ratchet.
This commit is contained in:
Miguel Ángel
2026-09-27 16:55:55 -04:00
parent 480ac61067
commit 5aed93b5c6
4 changed files with 145 additions and 72 deletions
+6 -5
View File
@@ -915,8 +915,8 @@ jobs:
packages/studio/tests/e2e/perf-ceilings.json studio-open /tmp/studio-open-counts.json \
/tmp/base-perf-ceilings.json
- name: Count and time playback with the Catalog open
# Counts go through perf-ceilings.json like studio-open. Time is only ever a ratio
# against the base branch's Studio on this runner, so a slow runner cannot fail it.
# Counts go through perf-ceilings.json like studio-open and can fail the step. Time is a
# ratio against the base branch's Studio on this runner, reported as a warning only.
# Both servers stay up; they sit idle while the other one is measured.
if: success() || failure()
run: |
@@ -949,13 +949,14 @@ jobs:
STUDIO_BASE_URL="$BASE_URL" \
node packages/studio/tests/e2e/studio-playback.mjs \
| tee /tmp/studio-playback.json
TIMING_STATUS=$?
PLAYBACK_STATUS=$?
node -e 'const w = require("/tmp/studio-playback.json").ab?.warning; if (w) console.log(`::warning::${w}`)' || true
node packages/studio/tests/e2e/perf-ratchet.mjs check \
packages/studio/tests/e2e/perf-ceilings.json studio-playback /tmp/studio-playback.json \
/tmp/base-perf-ceilings.json
COUNTS_STATUS=$?
if (( TIMING_STATUS != 0 || COUNTS_STATUS != 0 )); then
echo "FAIL: playback timing=${TIMING_STATUS}, counts=${COUNTS_STATUS}"
if (( PLAYBACK_STATUS != 0 || COUNTS_STATUS != 0 )); then
echo "FAIL: playback run=${PLAYBACK_STATUS}, counts=${COUNTS_STATUS}"
exit 1
fi
- name: Upload gate evidence