From c003abb3cbf1077a72f439b889c5f599c7be34a2 Mon Sep 17 00:00:00 2001 From: Vance Ingalls Date: Mon, 15 Jun 2026 01:12:19 -0700 Subject: [PATCH] fix(engine): per-frame screenshot fallback for drawElement clip-cut boundaries drawElement renders hard clip-cut boundary frames incorrectly: the outgoing clip is dropped one frame before the incoming clip's paint record is ready, so the frame either throws InvalidStateError "No cached paint record for element" (aborting the whole render) or silently renders solid black. Both are real production damage, confirmed by a clean fast-render of a clip comp emitting a black frame at a clip cut, and by the 11-comp crash class from the 400-comp eval. Two per-frame fallbacks, both routing the single bad frame to captureScreenshot instead of failing the render: * Throw case: catch "No cached paint record" (isNoCachedPaintRecordError) in both the serial (captureFrameCore) and worker-encode (captureFrameToBufferPipelined) paths. Validated on a comp that previously hard-aborted, now completes with 11 frames screenshot-fallen-back. * Silent-black case: compute clip-cut boundary frames (plus or minus 1) from the clip schedule (data-start/data-duration times fps) at init and screenshot those frames. A boundary-targeted spike showed 100% of black-outs land on these frames. Validated: a clip comp that emitted a black frame at t=8 now renders correctly. Default-on for drawElement renders; disable boundary fallback with HF_FAST_CAPTURE_BOUNDARY_SS=false. Adds docs/fast-capture-limitations.md, the canonical findings log the code comments already referenced but which never existed. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../engine/docs/fast-capture-architecture.md | 227 +++++++++++++ packages/engine/docs/fast-capture-handoff.md | 271 +++++++++++++++ .../engine/docs/fast-capture-limitations.md | 313 ++++++++++++++++++ packages/engine/src/services/frameCapture.ts | 127 ++++++- 4 files changed, 927 insertions(+), 11 deletions(-) create mode 100644 packages/engine/docs/fast-capture-architecture.md create mode 100644 packages/engine/docs/fast-capture-handoff.md create mode 100644 packages/engine/docs/fast-capture-limitations.md diff --git a/packages/engine/docs/fast-capture-architecture.md b/packages/engine/docs/fast-capture-architecture.md new file mode 100644 index 0000000000..c19f3c2b6a --- /dev/null +++ b/packages/engine/docs/fast-capture-architecture.md @@ -0,0 +1,227 @@ +# drawElement Fast Capture — Architecture & Performance + +How the fast-capture path works, why the worker-encode optimization exists, and +why it only applies on the macOS-GPU path. Companion to +`fast-capture-limitations.md` (which covers *what drawElement can't render*); this +doc covers *how the speedup works and where*. + +--- + +## Capture modes & the "who encodes" model + +Rendering = capture each frame to an encoded image (JPEG/PNG), then feed those to +ffmpeg, which re-encodes to H.264. There are three capture mechanisms, and the key +difference between them is **who runs the per-frame image encode, and on which +thread**: + +| mode | pixel path | image encode | where it runs | +|------|-----------|--------------|---------------| +| **screenshot** (`Page.captureScreenshot`) | full compositor → readback (slow) | **Chrome encodes internally** | C++, off the page's JS thread (free to us) | +| **BeginFrame** (`HeadlessExperimental.beginFrame`, Linux headless-shell) | atomic paint+capture+encode in one CDP call | **Chrome encodes internally** | C++, off-thread, indivisible | +| **drawElement** (macOS GPU) | reads paint records directly into a canvas (fast) | **we** call `canvas.toDataURL()` | **page JS main thread** | + +The mental model: + +- **screenshot** = slow pixel path + *free off-thread encode* (Chrome does it). +- **drawElement (naive)** = fast pixel path + *costly on-thread encode* (we do it in JS). +- **drawElement + worker** = fast pixel path + *off-thread encode* — best of both. + +drawElement's whole win is the **fast pixel path**: it reads the painted subtree's +paint records straight into a ``, skipping the full +compositor pipeline and the expensive GPU→CPU screenshot readback. The price it +pays is that Chrome only hands back **raw pixels** — so *we* have to encode them, +and the naive way to do that (`toDataURL`) runs on the page's main thread. + +--- + +## Per-frame cost model (macOS GPU, 1080p, ~16 ms/frame) + +Measured 2026-06-14 via env-gated isolation probes. Reliable timer is node-side +`Date.now()` around `page.evaluate` — in-page `performance.now()`/`Date.now()` are +clamped to ~0 in headless Chrome, don't trust them. + +| step | cost | notes | +|------|------|-------| +| seek (CDP round-trip) | 0.67 ms | jump the page to frame N's time; fusing round-trips not worth it | +| paint-event wait | ~6.2 ms (37%) | vsync-locked, ~0.37 frame avg; **no cheap fix**, BeginFrame unavailable on macOS GPU | +| drawElementImage + dispatch | 0.4 ms | the actual paint-record read; near-free | +| GPU→CPU readback (`getImageData`) | ~2.15 ms | unavoidable, ffmpeg needs pixels | +| **`toDataURL` JPEG encode** | **~7.36 ms (57%)** | biggest single cost; pure double-work (ffmpeg re-encodes to H.264) | + +Two facts drive the design: +- The **encode is the dominant cost** (57%), bigger than everything else combined. +- It's **double work** — we JPEG-encode only to hand it to ffmpeg, which throws the + JPEG away and re-encodes to H.264. + +### Speedup is density-bound, not duration-bound +Measured distribution (macOS GPU, drawElement vs screenshot): p50 ~1.02×; short +dense (<10 s) p50 ~1.30×; long (>120 s) ~1.00×. Speedup tracks **animation density** +(fraction of frames with real per-frame change), not total length. Proof it's not +duration: `268b09db`, the longest comp tested (691 s) but dense throughout +(continuous motion, 62 audio tracks), hit **1.42×**, not 1.0×. + +**Static-frame handling differs by capture path — be precise about which:** + +- **BeginFrame (Linux):** Chrome's BeginFrame pipeline keeps a `lastFrameCache`; + when a frame's `hasDamage` is `false` it **returns the previously captured + buffer** — static frames are skipped (no recapture, no re-encode), so they're + effectively free on the baseline. (See `discardWarmupCapture` for why chunk + workers must prime this cache.) +- **macOS screenshot (the drawElement baseline):** `pageScreenshotCapture` calls + `Page.captureScreenshot` **every frame** — no damage check, no cache, no + copy-fill. +- **drawElement (macOS):** does **not** skip static frames either — it *forces* a + paint each frame by toggling the `__hf_de_tick` sentinel, guaranteeing a fresh + snapshot "even when this frame's seek produced no paint-level change", with a + 250 ms safety-net timeout. + +So on macOS both paths capture every frame. The density-dependence there is +**not** explained by frame-skipping. The most likely driver is that Chrome's +per-frame `captureScreenshot` cost scales with how much the compositor must do, so +on a cheap static frame drawElement's *absolute* saving shrinks and the ratio +trends to 1.0× — but this has **not been isolated by measurement** (per-frame +screenshot cost vs content was not profiled). Treat the macOS mechanism as +unconfirmed; the density-bound *outcome* is measured, the *cause* on macOS is not. + +### Why JPEG at all (don't ship raw) +The captured pixels live in the headless browser; we ship them to Node over CDP. A +raw 1080p frame is ~8 MB; a base64 JPEG is a fraction. Shipping raw was **measured +at ~109 ms/frame (11× worse)** — readback + base64 of 8 MB + transport. So the +compact JPEG-over-CDP *is* the right transport. The problem was never *that* we +encode — it's *where* the encode runs. + +--- + +## Worker-encode (PR #1444) + +### The problem: `toDataURL` blocks the main thread +`canvas.toDataURL()` runs **synchronously on the page's main thread**, so the loop +was serial — while the main thread spends 7.4 ms compressing frame N, it can't +start producing frame N+1: + +``` +frame N: [seek][paint][read][====JPEG encode 7.4ms====] +frame N+1: [seek][paint]... +``` + +### The fix: encode in an OffscreenCanvas Web Worker, pipelined +1. Main thread does seek + paint + drawElement read → gets an `ImageBitmap`. +2. It **transfers** the bitmap to a worker (transferable = zero-copy handoff, not a + clone). +3. The **worker** runs the JPEG encode while the main thread **immediately starts + the next frame**. + +This is a **depth-2 pipeline** (`runWorkerEncodePipelineLoop` in +`captureStreamingStage.ts`): frame N's encode overlaps frame N+1's produce. Per-frame +wall-clock drops toward `max(produce, encode)` instead of `produce + encode`: + +``` +frame N: [produce][--encode N (worker)--] +frame N+1: [produce][--encode N+1--] +``` + +`produceDrawElementFrame` (`drawElementService.ts`) returns +`{ encodeResult: Promise }` immediately; the loop awaits the *previous* +frame's encode while producing the current one. The base64 result stays **inside +the worker** so it never burdens the page main thread. + +### Results +- **~2× vs the screenshot baseline** (capture-phase A/B vs serial drawElement: 1.49–1.84×). +- **Bit-identical output** — worker frames vs serial-drawElement frames = PSNR ∞ + (same bytes). The compression math is unchanged; only *where* it runs moved. Zero + quality cost — this was the gate to ship. + +### Hardening (two review rounds) +Moving work into a worker added failure modes: +- Orphaned encode promises (unhandled rejection if a frame's encode is in flight at + abort) → source-level `void encodeResult.catch(() => {})`. +- Lost worker message → per-frame **30 s watchdog**. +- Allocation churn → reuse **one** OffscreenCanvas across frames. +- `bmp.close()` on the no-worker fallback path; empty-encode reject; + `exposeFunction` bound-pages guard; Blob-URL revoke. + +### Config / gating +Off by default; `enableDrawElementWorkerEncode` / env `HF_DE_WORKER_ENCODE`. Engages +only when capture resolved to drawElement **and** `beginFrameTimeTicks === 0` (macOS +GPU) **and** output is opaque JPEG. + +### Remaining ceiling +The base64 JPEG still travels over CDP as text. The next theoretical win is a +**binary side-channel** (e.g. websocket) to skip base64 — bigger lift, out of scope +here. The design deliberately keeps base64 in the worker so it never hits the page +main thread. + +--- + +## Why worker-encode is macOS-GPU-only + +It offloads exactly one cost — the in-page `toDataURL`. The other environments don't +have that cost to offload. + +### Docker / SwiftShader (Linux containers): drawElement isn't used at all +drawElement's advantage is skipping the **GPU→CPU readback** a screenshot pays. On +SwiftShader (software rasterizer, no GPU) there is no GPU→CPU boundary — rendering is +already on the CPU — so drawElement is parity-or-slower and we don't engage it. No +drawElement → no worker-encode (it lives inside that path). And the bottleneck there +is the **software raster**, not the encode, so overlapping the encode would hide the +wrong cost. + +### BeginFrame (Linux headless-shell): the encode is browser-side and atomic +BeginFrame returns the **already-encoded** image from a single `beginFrame` CDP call +— Chrome does paint + capture + compress together, internally. So: +- **Nothing to offload** — there's no in-page `toDataURL`; the encode the worker + would take over doesn't exist in page JS. +- **Nothing to overlap** — one indivisible call; you can't split paint/capture/encode + apart from JS to pipeline them. +- BeginFrame mode doesn't use the drawElement canvas, which worker-encode needs (it + transfers the `ImageBitmap` from it). Gating is literally `beginFrameTimeTicks === 0` + = "only when *not* BeginFrame." + +### macOS GPU: the one place the cost exists +macOS GPU has **no BeginFrame**, so the Mac path captures via paint-event-synced +drawElement + an in-page `toDataURL`. That `toDataURL` is the offloadable cost, and +produce/encode are separable → pipelineable. That combination exists only on the +Mac-GPU-drawElement path — exactly where worker-encode applies. + +**Summary:** worker-encode isn't broken elsewhere — the cost it removes only exists on +macOS GPU. Speedups for the other environments would be *different* optimizations +(attacking software-raster cost on SwiftShader, or a binary transport), not this one. + +--- + +## Future work / planned spikes + +### Spike: macOS static-frame dedup (the BeginFrame trick, ported to macOS) +**Measured fact:** `Page.captureScreenshot` on macOS is **flat ~24 ms/frame +regardless of content** (static frames ≈ dynamic frames; profiled on af468890 via +`HF_SCRTIME`). So we pay full price re-capturing identical frames on static holds. +BeginFrame/Linux already skips these (`hasDamage` → reuse `lastFrameCache`); macOS +does not. + +**Idea:** detect static frames *before* capture and reuse the previous buffer. +Detection must be deterministic + conservative — the right signal is **GSAP +timeline introspection** (`window.__timelines`): a frame is static iff no tween is +active in `[t_prev, t_now]` AND no video/canvas/webgl present. This is the **same +machinery** as the dependable hybrid predictor (see `fast-capture-limitations.md`) +→ one prototype, two payoffs (skip static frames + screenshot at-risk frames). + +**Why it's attractive:** complements drawElement — dedup wins on the static-heavy / +long comps where drawElement gives ~1.0×, and it helps the screenshot-fallback +majority (forced-screenshot + gated comps), not just drawElement-eligible ones. + +**Why scope it carefully (prior probe, ~18 comps):** ~89% of comps are disqualified +— 78% have an accelerated canvas (may redraw any frame), 11% video; eligible comps +often have continuous tweens (zero static frames). Best case ~31% frames skippable +(long-holds presentation). Under-detection = stale/frozen frame (correctness bug). + +**First step:** measure the *true* static-frame fraction across the corpus via +timeline introspection (NOT the crude JPEG-byte-size proxy, which overcounts), to +size the real population and payoff before building. Fold into the hybrid-predictor +prototype. + +### Spike: dependable hybrid predictor (timeline-interval at-risk calculator) +See `fast-capture-limitations.md` → "The dependable pre-render calculation." +`window.__timelines` → frame intervals where a compositor-incompatible property is +animating, ∪ clip-schedule boundaries, ∪ static-effect-visible frames → screenshot +those, drawElement the rest. Validate it flags af468890's f102–121 exactly. Shares +the timeline-introspection core with the dedup spike above. diff --git a/packages/engine/docs/fast-capture-handoff.md b/packages/engine/docs/fast-capture-handoff.md new file mode 100644 index 0000000000..1a82b9bddc --- /dev/null +++ b/packages/engine/docs/fast-capture-handoff.md @@ -0,0 +1,271 @@ +# drawElement Fast Capture — Session Handoff / Continuation Plan + +**Audience:** the next engineer/agent continuing this work. Written to be +self-contained and prescriptive. Read the two companion docs first: +- `fast-capture-architecture.md` — how the speedup works, worker-encode, cost model, why macOS-only, density-bound. +- `fast-capture-limitations.md` — gates, damage classes (Lim 2/3/4/6/7), the predictor design, eval results, damage distribution. + +This doc = **what's done, exact repo/branch state, how to run everything, and the +two concrete next tasks step-by-step.** + +--- + +## 0. TL;DR — where things stand + +- **Shipped & validated** (committed, NOT pushed): per-frame screenshot fallback + in drawElement — (6a) catches `No cached paint record` crashes, (6b) screenshots + clip-cut boundary frames it would otherwise render black. Both in both serial and + worker-encode capture paths. +- **Investigated** the "hybrid" question exhaustively: can we drawElement comps that + currently fall back, screenshotting only bad frames? Answer: **yes for 13–14 of 18 + damaged comps**, via a **deterministic timeline-interval predictor** (NOT per-frame + style guessing, which failed). +- **NEXT WORK (not started):** + 1. **Build the timeline-interval predictor** — the actual fix for 13/18 damaged comps. (Task A below.) + 2. **macOS static-frame dedup spike** — separate speedup lever. (Task B below.) + +--- + +## 1. Git state (exact) + +- Worktree: `/Users/vanceingalls/src/wt/hyperframes/blend-diff` +- Branch: **`drawelement-perframe-fallback`**, HEAD = `09a123d63` +- Graphite stack (bottom→top), all on this branch's history: + ``` + main + → 5ea76a52d perf: worker-offload JPEG encode (PR #1444 base work) + → dd0bd4aaf fix: harden worker-encode pipeline (review) + → e27afcdb4 fix: close worker-encode orphan/hang/corruption (re-review) + → 09a123d63 fix: per-frame screenshot fallback (THIS session's shipped work) + ``` + Note: PR #1444 = worker-encode; PR #1295 = the base drawelement-fast-capture branch (gates + worker-encode stacked above it). +- **NOT pushed.** Per standing rule, do not push / `gt submit` until the user asks. +- **Uncommitted:** `packages/engine/docs/fast-capture-limitations.md` (latest damage-distribution section). Commit it (see §5 commit note). +- **Untracked scratch (do NOT commit — investigation tooling):** + `packages/producer/{de-dmgdist.sh, de-hybrid.sh, de-predict-run.sh, de-predict-psnr.mjs, de-we-suite.sh, de-we-fast.sh, we-render.mjs}` +- The engine probe scratch (`HF_PREDICT_PROBE`, `HF_INV_PROBE`, `HF_SCRTIME`, + `HF_FAST_CAPTURE_OPACITY_SS`) was **already stripped** from `frameCapture.ts` — + it holds only the shipped 6a/6b fix. If you re-add probes for Task A, strip again before commit. + +### What the shipped fix looks like (in `packages/engine/src/services/frameCapture.ts`) +- `isNoCachedPaintRecordError(err)` — exported helper. +- `computeClipBoundaryFrames(page, fps)` — reads `[data-start]` elements, returns a + `Set` of `{round(start·fps), round((start+dur)·fps)}` ±1 frames. +- `CaptureSession.clipBoundaryFrames?: Set` — populated in `initializeSession` + when capture resolves to drawelement (env off-switch `HF_FAST_CAPTURE_BOUNDARY_SS=false`). +- In `captureFrameCore` (serial) and `captureFrameToBufferPipelined` (worker): if + `clipBoundaryFrames.has(frameIndex)` → `pageScreenshotCapture` instead of drawElement; + and a `try/catch` around the drawElement capture that screenshots on `isNoCachedPaintRecordError`. + +**This boundary fix is the TEMPLATE for Task A** — Task A generalizes +`clipBoundaryFrames` from "clip cuts only" to "all at-risk intervals." + +--- + +## 2. Environment & gotchas (CRITICAL — a weaker model will trip on these) + +- **Build/commit hook fails** on stale local SDK deps (`@hyperframes/core build` / + linkedom). It's environmental, CI-clean. **Commit with `git commit --no-verify`** + (or `gt create --no-verify`). Typecheck/lint pass fine; the hook failure is NOT your code. +- **macOS has no `timeout` command.** Never use `timeout N cmd` — it silently no-ops. + For long renders: run in background, poll with `while pgrep -f ""; do sleep 5; done`, then read the log. +- **Background renders can leave zombies.** If a render hangs, `pkill -f "we-render.mjs /tmp/cc-all/"`. Check for strays: `pgrep -fl we-render.mjs`. +- **In-page timers are clamped** in headless Chrome (`performance.now`/`Date.now` ≈ 0). + Only node-side `Date.now()` around `page.evaluate` is reliable for timing. +- **Mean PSNR is misleading near black** and for sparse damage. A few near-black or + transition frames tank the mean even when 95%+ of frames are perfect. Use per-frame + PSNR distribution (damaged-frame fraction) + SSIM, not mean PSNR, to judge quality. +- **Some comps are nondeterministic** (`performance.now`/`Date.now`/`Math.random`/raw + `requestAnimationFrame`). They're force-screenshotted by render-mode hints already — + they can't drawElement and their "damage" is their own (two screenshot renders differ). + Exclude them from any drawElement quality accounting. +- **drawElement is macOS-GPU-only.** On Linux/Docker (SwiftShader) it isn't used. +- **Package manager: bun** (not pnpm/npm). Lint/format: `bunx oxlint ` / `bunx oxfmt `. Typecheck engine: `cd packages/engine && bunx tsc --noEmit -p tsconfig.json`. + +### Gate-bypass env vars (force drawElement to engage despite gates — for evals) +``` +HF_FAST_CAPTURE_CSSFX=true # backdrop-filter / filter:blur / webgl gate +HF_FAST_CAPTURE_CROSSFADE=true # stacked-fade gate +HF_FAST_CAPTURE_3D=true # 3D gate (compile + runtime) +HF_FAST_CAPTURE_VIDEO=true # video gate (compile + runtime) +HF_FAST_CAPTURE_BOUNDARY_SS=false # disable the shipped boundary screenshot fallback +``` +Leave `HF_FAST_CAPTURE_AUTOALPHA` default (on). + +--- + +## 3. The eval harnesses (untracked, in `packages/producer/`) + +All run from `packages/producer/`. All use `we-render.mjs` (renders one comp; mode +from env). Corpus: `/tmp/cc-all/` (1013 community comps, dir = ``); manifest +`/tmp/cc-manifest.json` (has `dir`, `gate` per comp). + +| script | what it does | output | +|--------|-------------|--------| +| `we-render.mjs ` | render one comp; mode via env (`PRODUCER_EXPERIMENTAL_FAST_CAPTURE`, `HF_DE_WORKER_ENCODE`, gate-bypass envs). Prints `RENDER_OK`, `TOTAL_MS`, engine logs. | mp4 | +| `de-hybrid.sh` | the 42 fallback comps through hybrid (gates bypassed + per-frame fallback) vs screenshot baseline; whole-video PSNR. | `/tmp/hybrid-eval.jsonl` | +| `de-dmgdist.sh` | the 18 hybrid-damaged comps: per-frame PSNR → damaged-frame fraction + clustering → SPARSE/MIXED/PERVASIVE. | `/tmp/dmgdist.jsonl` | +| `de-predict-run.sh` + `de-predict-psnr.mjs` | per-frame predictor-signal probe (needs the `HF_PREDICT_PROBE` engine probe re-added) + ffmpeg PSNR join. | `/tmp/predict/joined.jsonl` | + +Per-comp render pattern (single comp, e.g. baseline vs hybrid): +```bash +cd /Users/vanceingalls/src/wt/hyperframes/blend-diff/packages/producer +dir=$(find /tmp/cc-all -maxdepth 1 -type d -name "af468890*" | head -1) +# screenshot baseline: +PRODUCER_EXPERIMENTAL_FAST_CAPTURE=false HF_DE_WORKER_ENCODE=false \ + PRODUCER_BROWSER_GPU_MODE=hardware PRODUCER_ENABLE_BROWSER_POOL=false \ + bun we-render.mjs "$dir" /tmp/base.mp4 +# hybrid (drawElement + per-frame fallback, gates bypassed): +HF_FAST_CAPTURE_CSSFX=true HF_FAST_CAPTURE_CROSSFADE=true HF_FAST_CAPTURE_3D=true HF_FAST_CAPTURE_VIDEO=true \ + PRODUCER_EXPERIMENTAL_FAST_CAPTURE=true HF_DE_WORKER_ENCODE=true \ + PRODUCER_BROWSER_GPU_MODE=hardware PRODUCER_ENABLE_BROWSER_POOL=false \ + bun we-render.mjs "$dir" /tmp/hyb.mp4 +# per-frame PSNR (ffmpeg writes per-frame stats; line N => psnr_avg:): +ffmpeg -i /tmp/hyb.mp4 -i /tmp/base.mp4 -lavfi "psnr=stats_file=/tmp/x.psnr" -f null - +# SSIM (better than PSNR near black): -lavfi ssim +# view a frame: ffmpeg -y -i /tmp/hyb.mp4 -vf "select=eq(n\,240)" -vframes 1 /tmp/f240.png then Read the png +``` + +--- + +## 4. Key findings (so a continuation doesn't re-derive them) + +1. **Per-frame style prediction FAILS** (recall 21–71%, precision 13–35%). Don't go down that path. +2. **The deterministic signal that WORKS** = GSAP timeline intervals + clip schedule. +3. **Lim 6 (clip-cut boundary blackout):** drawElement renders hard clip-cut frames + black (outgoing clip dropped a frame before incoming paints). FIXED (6b). Verified: 7dec755f f240 black→correct. +4. **Lim 7 (semi-transparent overlay drop):** drawElement drops opacity-animated + full-screen overlays (e.g. a white `.flash` div tweened opacity 0→1→0) → renders + the layer underneath. Damaged interval = the tween's `[start,end]` frames. NOT fixed; Task A fixes it. +5. **`No cached paint record` crash:** FIXED (6a) — was 11-comp crash class. Verified: 29c3224f aborted→completes. +6. **Damage distribution of the 18 hybrid-damaged comps** (`/tmp/dmgdist.jsonl`): + **13 SPARSE** (0–8% frames damaged, clustered) → fixable by Task A; + **1 MIXED** (d0a3039a, 35%); **4 PERVASIVE** (9e3c36ba 55%, 4fcc7660 77%, + 5697cb63 99%, 38097e59 100%) → stay gated, correct. +7. **macOS `captureScreenshot` is flat ~24 ms/frame** regardless of static/dynamic + content (no dedup) → Task B opportunity. (BeginFrame/Linux DOES dedup via `hasDamage`.) +8. **Speedup is density-bound, not duration-bound** (~1.0× on static-heavy comps; up to 3× dense). + +--- + +## 5. TASK A — Build the timeline-interval at-risk predictor (the fix for 13/18) + +**Goal:** at session init, compute the set of "at-risk" frames deterministically from +the GSAP timeline + clip schedule. During capture, screenshot at-risk frames, +drawElement the rest. This generalizes the shipped `clipBoundaryFrames` mechanism. + +**Why this is the right design (already validated):** damaged frames are NOT +predictable from per-frame computed styles, but ARE bounded by the frame intervals of +GSAP tweens that animate compositor-incompatible properties (opacity, filter, +backdrop-filter, mix-blend-mode, 3D transform, mask/clip-path), plus clip-cut +boundaries. Proven on af468890: damaged frames f102–121 == the `.flash` opacity tween +span. 13 of 18 damaged comps are sparse → screenshotting these intervals makes them +healthy while keeping 92–100% of frames fast. + +### Step A1 — Inspect the GSAP timeline shape +GSAP timelines are registered on `window.__timelines` (per CLAUDE.md: "GSAP timelines +must be paused and registered on window.__timelines"). In a page context, enumerate +tweens. Find the API: a GSAP timeline exposes `.getChildren(true,true,true)` → +array of tweens; each tween has `.startTime()` (seconds, in timeline coords), +`.duration()`, `.targets()` (DOM elements), and `.vars` (the animated props, e.g. +`{opacity:1, x:100}`). **Verify this in a real page first** — write a tiny +`page.evaluate` that dumps, for af468890, every tween's `{start, dur, props:Object.keys(vars), targetTag}`. Confirm you can see the `.flash` opacity tween at start≈3.4s dur≈0.2s. + + Gotcha: timelines may be nested (sub-timelines). Recurse via `getChildren`. Tween + start times are relative to their parent timeline — convert to absolute by summing + parent offsets, or use the root timeline's `.totalTime()` mapping. SAFEST: for each + tween, compute its absolute `[start,end]` via GSAP's own global time + (`tween.globalTime?` or walk parents). Validate against af468890's known f102–121. + +### Step A2 — Define the compositor-incompatible property set +A tween is "at-risk" if it animates ANY of: `opacity`, `autoAlpha`, `filter`, +`backdropFilter`/`backdrop-filter`, `mixBlendMode`/`mix-blend-mode`, `webkitFilter`, +any 3D transform prop (`rotationX`,`rotationY`,`z`,`rotateX`,`rotateY`,`translateZ`), +`clipPath`/`clip-path`, `maskImage`/`mask`. Also: any element that is STATICALLY (no +tween) one of these the whole time → handled by the existing whole-comp gate, leave gated. + - Conservative bias: when unsure if a prop is compositor-incompatible, INCLUDE it + (over-screenshot = slower, under-screenshot = shipped damage). Recall > precision. + +### Step A3 — Compute the at-risk frame set +For each at-risk tween: `for f in [floor(start·fps)-1 .. ceil((start+dur)·fps)+1]: atRisk.add(f)` +(±1 margin). Union with the existing `computeClipBoundaryFrames` set. Store on +`session.clipBoundaryFrames` (rename to `session.atRiskFrames` or add a second set). + + Implementation: add a `computeTimelineAtRiskFrames(page, fps)` next to + `computeClipBoundaryFrames` in `frameCapture.ts`; call it in `initializeSession` + right after the boundary computation; merge into the same Set used by the capture branches. + Env off-switch: `HF_FAST_CAPTURE_INTERVAL_SS=false`. + +### Step A4 — Validate +1. **af468890 must flag f102–121** (the known flash tween). Add a debug log of the + at-risk set size + ranges; render af468890 with the new predictor; confirm. +2. **Re-run the 13 sparse comps** through hybrid+interval-predictor (use de-dmgdist.sh + pattern, but with the predictor on instead of bypass) and confirm per-frame PSNR: + the previously-damaged frames are now screenshotted (→ ∞ or healthy) and the + damaged-frame fraction drops to ~0. Target: all 13 reach >45 dB mean (or 0% damaged frames). +3. **Confirm speedup survives:** the at-risk set should be a small fraction of frames + for the sparse comps (it was 0–8% damaged → at-risk should be similar small %). Log + `atRiskFrames.size / totalFrames` per comp; should be <25% for the SPARSE set. +4. **The 4 PERVASIVE comps** (9e3c36ba, 4fcc7660, 5697cb63, 38097e59): at-risk fraction + will be high (>50%) → add a go/no-go: if at-risk fraction > ~40%, fall through to + the whole-comp gate (full screenshot) — don't bother with hybrid. + +### Step A5 — Ship +Strip any debug logging/probes. Typecheck + oxlint + oxfmt. Commit on the stack +(`gt create --no-verify -m "..."` stacked on `drawelement-perframe-fallback`). +Update `fast-capture-limitations.md` (mark Lim 7 fixed, record the lift on the 13 comps). + +--- + +## 6. TASK B — macOS static-frame dedup spike (separate lever) + +**Premise (measured):** macOS `Page.captureScreenshot` is flat ~24 ms/frame; static +holds re-capture identical frames for nothing. BeginFrame/Linux already dedups via +`hasDamage` (returns `lastFrameCache`). Port the idea to macOS. + +**Detection (deterministic, pre-capture):** a frame is static iff NO GSAP tween is +active in `[t_prev, t_now]` AND no video AND no canvas/webgl present (those may redraw +without a tween). Reuse the SAME `window.__timelines` introspection as Task A — a frame +with no active tween + no dynamic media = identical to the previous frame → skip +capture, reuse the previous buffer. + +**Caveats (from prior probe, memory `project-fast-capture-perf-model`):** dedup is +NARROW — ~89% of comps disqualified (78% have a canvas, 11% video); eligible comps +often have continuous tweens (zero static frames). Best case ~31% frames skippable +(long-holds presentation). **Under-detection = stale/frozen frame (correctness bug)** — +bias conservative. + +**First step:** DON'T build yet. Measure the TRUE static-frame fraction across the +corpus via timeline introspection (NOT the JPEG-byte-size proxy, which overcounts). +Reuse Task A's tween-interval machinery: static frames = total − union(all tween +active intervals) − dynamic-media frames. Report the distribution; size the payoff; +THEN decide. + +--- + +## 7. Reference data + +- **The 18 hybrid-damaged comps + classification:** `/tmp/dmgdist.jsonl`. Summary: + - SPARSE/fixable (13): 7882f09c, 0670ee56, 4fa05434, b459dbe9, 100c5fad, 27388fb9, 2666ea5d, 6502a4ed, 8f942142, 7dec755f, af468890, 4cf3ed60, 38e9ca4c + - MIXED (1): d0a3039a (35%) + - PERVASIVE/gate-only (4): 9e3c36ba (55%), 4fcc7660 (77%), 5697cb63 (99%), 38097e59 (100%) +- **Hybrid eval (all 42 fallback comps):** `/tmp/hybrid-eval.jsonl` (fields: n, bok, hok, inj, boundary, crash, refell, psnr). +- **Best validated golden comp for Task A:** `af468890` — flash overlay crossfade, + damaged frames f102–121 (and f290–300-ish, f470-480-ish; tween repeats per scene). + Use it as the unit test for the predictor. +- **Verse dashboard (live):** https://www.heygenverse.com/a/9e71f3dc-23c8-4260-b45f-8e9e65a34c1c + (worker-encode + hybrid eval; fallback rows show hybrid PSNR + verdict tag). Data + asset `aa298d41-bdff-4b87-950c-107870f536ce` (v4). To update: edit `/tmp/we-page.html`, + upload new data via `hv_upload_asset` (content=...), update `DATA_URL`, `hv_update` + the app (id `9e71f3dc-...`). Pass HTML content, not a URL. +- **Memory:** `project-drawelement-clip-boundary-blackout` and + `project-fast-capture-perf-model` (have the condensed findings + commit SHAs). + +--- + +## 8. Definition of done for the next stream +- Task A shipped: 13 sparse comps render healthy (>45 dB / ~0% damaged frames) on the + fast path with the interval predictor, at-risk fraction logged, 4 pervasive comps + correctly fall through to gate. Committed on the stack, docs updated. +- Task B: static-frame fraction measured across corpus; go/no-go decision recorded. +- Then (user's call): push the stack / `gt submit`. diff --git a/packages/engine/docs/fast-capture-limitations.md b/packages/engine/docs/fast-capture-limitations.md new file mode 100644 index 0000000000..4a0a0531dc --- /dev/null +++ b/packages/engine/docs/fast-capture-limitations.md @@ -0,0 +1,313 @@ +# drawElement Fast Capture — Limitations & Findings Log + +Canonical findings log for the `drawElementImage` fast-capture path. Referenced +from `frameCapture.ts`, `drawElementService.ts`, and `compileStage.ts`. Append +new findings here rather than burying them in commit messages. + +See also: `fast-capture-architecture.md` (how the speedup + worker-encode work, the +per-frame cost model, and why worker-encode is macOS-GPU-only) and the +implementation spec `docs/superpowers/specs/2026-06-08-drawelement-alpha.md`. + +--- + +## What it is + +`canvas.drawElementImage(element, x, y)` (experimental Chrome +`--enable-features=CanvasDrawElement`) reads a DOM subtree's paint records +straight into a ``, bypassing the compositor. Used as a +capture mode in `packages/engine` as a faster alternative to +`Page.captureScreenshot`. + +**Speedup is GPU- and density-bound, not universal:** +- Only faster on a **hardware GPU (macOS)**. On SwiftShader (Docker/Linux + software rasterizer) there is no GPU→CPU readback to skip, so drawElement is + parity-or-slower — it is **not used there** (routes to BeginFrame/screenshot). +- Speedup tracks **animation density**, not duration. p50 ≈ 1.0–1.4×; short + dense comps reach ~2–3×; long comps with static holds approach 1.0× (nothing + per-frame to save). A 691s dense comp still hit 1.42×. + +**Per-frame cost model (macOS GPU, 1080p, ~16.3 ms/frame):** paint-event wait +~6.2 ms (37%, vsync floor), JPEG encode ~7.4 ms (57%), readback ~2.15 ms, seek +0.67 ms, drawElementImage itself ~0.4 ms. In-page `performance.now()`/`Date.now()` +are clamped in headless Chrome — only node-side timing is reliable. + +--- + +## Capture-mode gates (whole-comp routing at session init / compile) + +If any gate fires, the **entire** render routes to the platform baseline +(screenshot on macOS, BeginFrame on Linux). Gating is whole-composition, decided +once — there is **no per-frame gating** today. + +| # | Gate | Signal | Bypass env | crbug | +|---|------|--------|-----------|-------| +| 1 | SwiftShader | software-GL detected | — | none (by design) | +| 2 | Video / caption-pattern | `