Commit Graph
2 Commits
Author SHA1 Message Date
Miguel Ángel ca54631464 test(studio): measure render drift against a producer frame in the edit accuracy bench (#4764)
* test(studio): measure render drift against a producer frame in the edit accuracy bench

Finds the target's box in a producer-captured frame by its flat colour, with sub-pixel edges from area moments, and compares it with the reloaded preview. The baseline gains the render column from the same full run on main.

* test(studio): mark the pixel box scan as intentionally complex for fallow

* test(studio): regenerate the edit accuracy baseline with render drift on main

Same full run on main 14e87005d as #4760's baseline, with the render column: render passes 618/660, 70/660 pass, 314/660 pass everything but smoothness.

* test(studio): fail only render drift when the producer cannot render a case

A producer capture failure used to turn the case into a harness error that wiped every accuracy metric. It now sets render to null with renderError, fails render alone, and is counted in the table.

* test(studio): shorten the render drift box comment to two lines
2026-09-30 21:01:40 +00:00
Miguel Ángel 0c6f5fd80e test(studio): an edit accuracy benchmark for manual editing (#4760)
* test(studio): start an edit accuracy benchmark for manual editing (work in progress)

* test(studio): an edit accuracy benchmark for manual editing

Drives real pointer and keyboard gestures in the built CLI Studio on generated one-element projects and scores tracking, drop jump, reload drift, undo fidelity and smoothness in composition px.

* test(studio): move render drift out of the first benchmark PR

Render drift against a producer frame ships in the stacked follow-up so this PR stays near the size budget.

* test(studio): keep the edit accuracy bench clean under fallow

Builds the grid as a product of its axes, drops unused exports and marks the harness functions that fallow scores as complex.

* test(studio): report edit accuracy with the Studio and bench commits it measured

Moves scoring and the report files into report.mjs, stamps both commits at start, counts undo byte failures apart from the worst box, and survives Studio swapping the preview iframe mid-read.

* test(studio): bank the edit accuracy baseline measured on main

660 cases on main ae914334f: 0 pass all six checks, 304 pass everything but smoothness. Five cases that hit a harness error in the full run were rerun and spliced in.

* test(studio): state the edit accuracy build prerequisite in the runner header

* test(studio): keep the edit accuracy baseline one line per case under the formatter

* test(studio): store edit accuracy baseline values rounded up

Rounding to the nearest 0.01 stored seven 0.502 px tracking results as 0.5, which reads as a pass at the 0.5 px limit.

* test(studio): score edit smoothness on dropped frames and main-thread work per frame

* test(studio): measure edit accuracy on a still preview and score press jump

Snapshots wait for the preview frames and the box to hold still for 1 s, so Studio's asynchronous reload after a save is measured, not raced. A preview that never settles fails the metrics it feeds instead of erroring.

Adds press jump (box at press against the box before it, 0.5 px). Smoothness counts only dropped frames beyond a blank-page control driven the same way in the same Chrome; unknown trace data fails smoothness alone.

The runner takes the suite lock per chunk of 8 cases (--lock), reads the port Studio announces, keeps undo history and tmp projects inside the case dir and removes them on a signal, and stamps the CLI tree's commit and bundle hash.

* test(studio): stamp the edit accuracy build with the Studio bundle as well as the CLI

The CLI bundle hashed the same for two different Studio builds; Studio's index.html names its hashed assets, so the stamp now covers it.

* test(studio): regenerate the edit accuracy baseline on main after the settle fix

Full grid on main 14e87005d with the settle fix, press jump and the smoothness control: 70/660 pass, 314/660 pass everything but smoothness, 0 harness errors, 0 unsettled. Scored without render, which #4764 adds.

* test(studio): wait for each edit accuracy undo write before sending redo

Undo and redo each wait up to 15 s for their own write, like the settle; redo is sent only once undo landed, and a write that never lands fails undo with undoTimeout instead of scoring an untested redo. The reload waits for the files to hold still. The headline is now the accuracy count, with smoothness on its own line against the control. The settle compares each read with the window's first read, and a CLI outside a git tree no longer stops the run.
2026-09-30 17:07:51 +00:00