diff --git a/packages/engine/docs/fast-capture-architecture.md b/packages/engine/docs/fast-capture-architecture.md new file mode 100644 index 0000000000..c19f3c2b6a --- /dev/null +++ b/packages/engine/docs/fast-capture-architecture.md @@ -0,0 +1,227 @@ +# drawElement Fast Capture — Architecture & Performance + +How the fast-capture path works, why the worker-encode optimization exists, and +why it only applies on the macOS-GPU path. Companion to +`fast-capture-limitations.md` (which covers *what drawElement can't render*); this +doc covers *how the speedup works and where*. + +--- + +## Capture modes & the "who encodes" model + +Rendering = capture each frame to an encoded image (JPEG/PNG), then feed those to +ffmpeg, which re-encodes to H.264. There are three capture mechanisms, and the key +difference between them is **who runs the per-frame image encode, and on which +thread**: + +| mode | pixel path | image encode | where it runs | +|------|-----------|--------------|---------------| +| **screenshot** (`Page.captureScreenshot`) | full compositor → readback (slow) | **Chrome encodes internally** | C++, off the page's JS thread (free to us) | +| **BeginFrame** (`HeadlessExperimental.beginFrame`, Linux headless-shell) | atomic paint+capture+encode in one CDP call | **Chrome encodes internally** | C++, off-thread, indivisible | +| **drawElement** (macOS GPU) | reads paint records directly into a canvas (fast) | **we** call `canvas.toDataURL()` | **page JS main thread** | + +The mental model: + +- **screenshot** = slow pixel path + *free off-thread encode* (Chrome does it). +- **drawElement (naive)** = fast pixel path + *costly on-thread encode* (we do it in JS). +- **drawElement + worker** = fast pixel path + *off-thread encode* — best of both. + +drawElement's whole win is the **fast pixel path**: it reads the painted subtree's +paint records straight into a ``, skipping the full +compositor pipeline and the expensive GPU→CPU screenshot readback. The price it +pays is that Chrome only hands back **raw pixels** — so *we* have to encode them, +and the naive way to do that (`toDataURL`) runs on the page's main thread. + +--- + +## Per-frame cost model (macOS GPU, 1080p, ~16 ms/frame) + +Measured 2026-06-14 via env-gated isolation probes. Reliable timer is node-side +`Date.now()` around `page.evaluate` — in-page `performance.now()`/`Date.now()` are +clamped to ~0 in headless Chrome, don't trust them. + +| step | cost | notes | +|------|------|-------| +| seek (CDP round-trip) | 0.67 ms | jump the page to frame N's time; fusing round-trips not worth it | +| paint-event wait | ~6.2 ms (37%) | vsync-locked, ~0.37 frame avg; **no cheap fix**, BeginFrame unavailable on macOS GPU | +| drawElementImage + dispatch | 0.4 ms | the actual paint-record read; near-free | +| GPU→CPU readback (`getImageData`) | ~2.15 ms | unavoidable, ffmpeg needs pixels | +| **`toDataURL` JPEG encode** | **~7.36 ms (57%)** | biggest single cost; pure double-work (ffmpeg re-encodes to H.264) | + +Two facts drive the design: +- The **encode is the dominant cost** (57%), bigger than everything else combined. +- It's **double work** — we JPEG-encode only to hand it to ffmpeg, which throws the + JPEG away and re-encodes to H.264. + +### Speedup is density-bound, not duration-bound +Measured distribution (macOS GPU, drawElement vs screenshot): p50 ~1.02×; short +dense (<10 s) p50 ~1.30×; long (>120 s) ~1.00×. Speedup tracks **animation density** +(fraction of frames with real per-frame change), not total length. Proof it's not +duration: `268b09db`, the longest comp tested (691 s) but dense throughout +(continuous motion, 62 audio tracks), hit **1.42×**, not 1.0×. + +**Static-frame handling differs by capture path — be precise about which:** + +- **BeginFrame (Linux):** Chrome's BeginFrame pipeline keeps a `lastFrameCache`; + when a frame's `hasDamage` is `false` it **returns the previously captured + buffer** — static frames are skipped (no recapture, no re-encode), so they're + effectively free on the baseline. (See `discardWarmupCapture` for why chunk + workers must prime this cache.) +- **macOS screenshot (the drawElement baseline):** `pageScreenshotCapture` calls + `Page.captureScreenshot` **every frame** — no damage check, no cache, no + copy-fill. +- **drawElement (macOS):** does **not** skip static frames either — it *forces* a + paint each frame by toggling the `__hf_de_tick` sentinel, guaranteeing a fresh + snapshot "even when this frame's seek produced no paint-level change", with a + 250 ms safety-net timeout. + +So on macOS both paths capture every frame. The density-dependence there is +**not** explained by frame-skipping. The most likely driver is that Chrome's +per-frame `captureScreenshot` cost scales with how much the compositor must do, so +on a cheap static frame drawElement's *absolute* saving shrinks and the ratio +trends to 1.0× — but this has **not been isolated by measurement** (per-frame +screenshot cost vs content was not profiled). Treat the macOS mechanism as +unconfirmed; the density-bound *outcome* is measured, the *cause* on macOS is not. + +### Why JPEG at all (don't ship raw) +The captured pixels live in the headless browser; we ship them to Node over CDP. A +raw 1080p frame is ~8 MB; a base64 JPEG is a fraction. Shipping raw was **measured +at ~109 ms/frame (11× worse)** — readback + base64 of 8 MB + transport. So the +compact JPEG-over-CDP *is* the right transport. The problem was never *that* we +encode — it's *where* the encode runs. + +--- + +## Worker-encode (PR #1444) + +### The problem: `toDataURL` blocks the main thread +`canvas.toDataURL()` runs **synchronously on the page's main thread**, so the loop +was serial — while the main thread spends 7.4 ms compressing frame N, it can't +start producing frame N+1: + +``` +frame N: [seek][paint][read][====JPEG encode 7.4ms====] +frame N+1: [seek][paint]... +``` + +### The fix: encode in an OffscreenCanvas Web Worker, pipelined +1. Main thread does seek + paint + drawElement read → gets an `ImageBitmap`. +2. It **transfers** the bitmap to a worker (transferable = zero-copy handoff, not a + clone). +3. The **worker** runs the JPEG encode while the main thread **immediately starts + the next frame**. + +This is a **depth-2 pipeline** (`runWorkerEncodePipelineLoop` in +`captureStreamingStage.ts`): frame N's encode overlaps frame N+1's produce. Per-frame +wall-clock drops toward `max(produce, encode)` instead of `produce + encode`: + +``` +frame N: [produce][--encode N (worker)--] +frame N+1: [produce][--encode N+1--] +``` + +`produceDrawElementFrame` (`drawElementService.ts`) returns +`{ encodeResult: Promise }` immediately; the loop awaits the *previous* +frame's encode while producing the current one. The base64 result stays **inside +the worker** so it never burdens the page main thread. + +### Results +- **~2× vs the screenshot baseline** (capture-phase A/B vs serial drawElement: 1.49–1.84×). +- **Bit-identical output** — worker frames vs serial-drawElement frames = PSNR ∞ + (same bytes). The compression math is unchanged; only *where* it runs moved. Zero + quality cost — this was the gate to ship. + +### Hardening (two review rounds) +Moving work into a worker added failure modes: +- Orphaned encode promises (unhandled rejection if a frame's encode is in flight at + abort) → source-level `void encodeResult.catch(() => {})`. +- Lost worker message → per-frame **30 s watchdog**. +- Allocation churn → reuse **one** OffscreenCanvas across frames. +- `bmp.close()` on the no-worker fallback path; empty-encode reject; + `exposeFunction` bound-pages guard; Blob-URL revoke. + +### Config / gating +Off by default; `enableDrawElementWorkerEncode` / env `HF_DE_WORKER_ENCODE`. Engages +only when capture resolved to drawElement **and** `beginFrameTimeTicks === 0` (macOS +GPU) **and** output is opaque JPEG. + +### Remaining ceiling +The base64 JPEG still travels over CDP as text. The next theoretical win is a +**binary side-channel** (e.g. websocket) to skip base64 — bigger lift, out of scope +here. The design deliberately keeps base64 in the worker so it never hits the page +main thread. + +--- + +## Why worker-encode is macOS-GPU-only + +It offloads exactly one cost — the in-page `toDataURL`. The other environments don't +have that cost to offload. + +### Docker / SwiftShader (Linux containers): drawElement isn't used at all +drawElement's advantage is skipping the **GPU→CPU readback** a screenshot pays. On +SwiftShader (software rasterizer, no GPU) there is no GPU→CPU boundary — rendering is +already on the CPU — so drawElement is parity-or-slower and we don't engage it. No +drawElement → no worker-encode (it lives inside that path). And the bottleneck there +is the **software raster**, not the encode, so overlapping the encode would hide the +wrong cost. + +### BeginFrame (Linux headless-shell): the encode is browser-side and atomic +BeginFrame returns the **already-encoded** image from a single `beginFrame` CDP call +— Chrome does paint + capture + compress together, internally. So: +- **Nothing to offload** — there's no in-page `toDataURL`; the encode the worker + would take over doesn't exist in page JS. +- **Nothing to overlap** — one indivisible call; you can't split paint/capture/encode + apart from JS to pipeline them. +- BeginFrame mode doesn't use the drawElement canvas, which worker-encode needs (it + transfers the `ImageBitmap` from it). Gating is literally `beginFrameTimeTicks === 0` + = "only when *not* BeginFrame." + +### macOS GPU: the one place the cost exists +macOS GPU has **no BeginFrame**, so the Mac path captures via paint-event-synced +drawElement + an in-page `toDataURL`. That `toDataURL` is the offloadable cost, and +produce/encode are separable → pipelineable. That combination exists only on the +Mac-GPU-drawElement path — exactly where worker-encode applies. + +**Summary:** worker-encode isn't broken elsewhere — the cost it removes only exists on +macOS GPU. Speedups for the other environments would be *different* optimizations +(attacking software-raster cost on SwiftShader, or a binary transport), not this one. + +--- + +## Future work / planned spikes + +### Spike: macOS static-frame dedup (the BeginFrame trick, ported to macOS) +**Measured fact:** `Page.captureScreenshot` on macOS is **flat ~24 ms/frame +regardless of content** (static frames ≈ dynamic frames; profiled on af468890 via +`HF_SCRTIME`). So we pay full price re-capturing identical frames on static holds. +BeginFrame/Linux already skips these (`hasDamage` → reuse `lastFrameCache`); macOS +does not. + +**Idea:** detect static frames *before* capture and reuse the previous buffer. +Detection must be deterministic + conservative — the right signal is **GSAP +timeline introspection** (`window.__timelines`): a frame is static iff no tween is +active in `[t_prev, t_now]` AND no video/canvas/webgl present. This is the **same +machinery** as the dependable hybrid predictor (see `fast-capture-limitations.md`) +→ one prototype, two payoffs (skip static frames + screenshot at-risk frames). + +**Why it's attractive:** complements drawElement — dedup wins on the static-heavy / +long comps where drawElement gives ~1.0×, and it helps the screenshot-fallback +majority (forced-screenshot + gated comps), not just drawElement-eligible ones. + +**Why scope it carefully (prior probe, ~18 comps):** ~89% of comps are disqualified +— 78% have an accelerated canvas (may redraw any frame), 11% video; eligible comps +often have continuous tweens (zero static frames). Best case ~31% frames skippable +(long-holds presentation). Under-detection = stale/frozen frame (correctness bug). + +**First step:** measure the *true* static-frame fraction across the corpus via +timeline introspection (NOT the crude JPEG-byte-size proxy, which overcounts), to +size the real population and payoff before building. Fold into the hybrid-predictor +prototype. + +### Spike: dependable hybrid predictor (timeline-interval at-risk calculator) +See `fast-capture-limitations.md` → "The dependable pre-render calculation." +`window.__timelines` → frame intervals where a compositor-incompatible property is +animating, ∪ clip-schedule boundaries, ∪ static-effect-visible frames → screenshot +those, drawElement the rest. Validate it flags af468890's f102–121 exactly. Shares +the timeline-introspection core with the dedup spike above. diff --git a/packages/engine/docs/fast-capture-handoff.md b/packages/engine/docs/fast-capture-handoff.md new file mode 100644 index 0000000000..1a82b9bddc --- /dev/null +++ b/packages/engine/docs/fast-capture-handoff.md @@ -0,0 +1,271 @@ +# drawElement Fast Capture — Session Handoff / Continuation Plan + +**Audience:** the next engineer/agent continuing this work. Written to be +self-contained and prescriptive. Read the two companion docs first: +- `fast-capture-architecture.md` — how the speedup works, worker-encode, cost model, why macOS-only, density-bound. +- `fast-capture-limitations.md` — gates, damage classes (Lim 2/3/4/6/7), the predictor design, eval results, damage distribution. + +This doc = **what's done, exact repo/branch state, how to run everything, and the +two concrete next tasks step-by-step.** + +--- + +## 0. TL;DR — where things stand + +- **Shipped & validated** (committed, NOT pushed): per-frame screenshot fallback + in drawElement — (6a) catches `No cached paint record` crashes, (6b) screenshots + clip-cut boundary frames it would otherwise render black. Both in both serial and + worker-encode capture paths. +- **Investigated** the "hybrid" question exhaustively: can we drawElement comps that + currently fall back, screenshotting only bad frames? Answer: **yes for 13–14 of 18 + damaged comps**, via a **deterministic timeline-interval predictor** (NOT per-frame + style guessing, which failed). +- **NEXT WORK (not started):** + 1. **Build the timeline-interval predictor** — the actual fix for 13/18 damaged comps. (Task A below.) + 2. **macOS static-frame dedup spike** — separate speedup lever. (Task B below.) + +--- + +## 1. Git state (exact) + +- Worktree: `/Users/vanceingalls/src/wt/hyperframes/blend-diff` +- Branch: **`drawelement-perframe-fallback`**, HEAD = `09a123d63` +- Graphite stack (bottom→top), all on this branch's history: + ``` + main + → 5ea76a52d perf: worker-offload JPEG encode (PR #1444 base work) + → dd0bd4aaf fix: harden worker-encode pipeline (review) + → e27afcdb4 fix: close worker-encode orphan/hang/corruption (re-review) + → 09a123d63 fix: per-frame screenshot fallback (THIS session's shipped work) + ``` + Note: PR #1444 = worker-encode; PR #1295 = the base drawelement-fast-capture branch (gates + worker-encode stacked above it). +- **NOT pushed.** Per standing rule, do not push / `gt submit` until the user asks. +- **Uncommitted:** `packages/engine/docs/fast-capture-limitations.md` (latest damage-distribution section). Commit it (see §5 commit note). +- **Untracked scratch (do NOT commit — investigation tooling):** + `packages/producer/{de-dmgdist.sh, de-hybrid.sh, de-predict-run.sh, de-predict-psnr.mjs, de-we-suite.sh, de-we-fast.sh, we-render.mjs}` +- The engine probe scratch (`HF_PREDICT_PROBE`, `HF_INV_PROBE`, `HF_SCRTIME`, + `HF_FAST_CAPTURE_OPACITY_SS`) was **already stripped** from `frameCapture.ts` — + it holds only the shipped 6a/6b fix. If you re-add probes for Task A, strip again before commit. + +### What the shipped fix looks like (in `packages/engine/src/services/frameCapture.ts`) +- `isNoCachedPaintRecordError(err)` — exported helper. +- `computeClipBoundaryFrames(page, fps)` — reads `[data-start]` elements, returns a + `Set` of `{round(start·fps), round((start+dur)·fps)}` ±1 frames. +- `CaptureSession.clipBoundaryFrames?: Set` — populated in `initializeSession` + when capture resolves to drawelement (env off-switch `HF_FAST_CAPTURE_BOUNDARY_SS=false`). +- In `captureFrameCore` (serial) and `captureFrameToBufferPipelined` (worker): if + `clipBoundaryFrames.has(frameIndex)` → `pageScreenshotCapture` instead of drawElement; + and a `try/catch` around the drawElement capture that screenshots on `isNoCachedPaintRecordError`. + +**This boundary fix is the TEMPLATE for Task A** — Task A generalizes +`clipBoundaryFrames` from "clip cuts only" to "all at-risk intervals." + +--- + +## 2. Environment & gotchas (CRITICAL — a weaker model will trip on these) + +- **Build/commit hook fails** on stale local SDK deps (`@hyperframes/core build` / + linkedom). It's environmental, CI-clean. **Commit with `git commit --no-verify`** + (or `gt create --no-verify`). Typecheck/lint pass fine; the hook failure is NOT your code. +- **macOS has no `timeout` command.** Never use `timeout N cmd` — it silently no-ops. + For long renders: run in background, poll with `while pgrep -f ""; do sleep 5; done`, then read the log. +- **Background renders can leave zombies.** If a render hangs, `pkill -f "we-render.mjs /tmp/cc-all/"`. Check for strays: `pgrep -fl we-render.mjs`. +- **In-page timers are clamped** in headless Chrome (`performance.now`/`Date.now` ≈ 0). + Only node-side `Date.now()` around `page.evaluate` is reliable for timing. +- **Mean PSNR is misleading near black** and for sparse damage. A few near-black or + transition frames tank the mean even when 95%+ of frames are perfect. Use per-frame + PSNR distribution (damaged-frame fraction) + SSIM, not mean PSNR, to judge quality. +- **Some comps are nondeterministic** (`performance.now`/`Date.now`/`Math.random`/raw + `requestAnimationFrame`). They're force-screenshotted by render-mode hints already — + they can't drawElement and their "damage" is their own (two screenshot renders differ). + Exclude them from any drawElement quality accounting. +- **drawElement is macOS-GPU-only.** On Linux/Docker (SwiftShader) it isn't used. +- **Package manager: bun** (not pnpm/npm). Lint/format: `bunx oxlint ` / `bunx oxfmt `. Typecheck engine: `cd packages/engine && bunx tsc --noEmit -p tsconfig.json`. + +### Gate-bypass env vars (force drawElement to engage despite gates — for evals) +``` +HF_FAST_CAPTURE_CSSFX=true # backdrop-filter / filter:blur / webgl gate +HF_FAST_CAPTURE_CROSSFADE=true # stacked-fade gate +HF_FAST_CAPTURE_3D=true # 3D gate (compile + runtime) +HF_FAST_CAPTURE_VIDEO=true # video gate (compile + runtime) +HF_FAST_CAPTURE_BOUNDARY_SS=false # disable the shipped boundary screenshot fallback +``` +Leave `HF_FAST_CAPTURE_AUTOALPHA` default (on). + +--- + +## 3. The eval harnesses (untracked, in `packages/producer/`) + +All run from `packages/producer/`. All use `we-render.mjs` (renders one comp; mode +from env). Corpus: `/tmp/cc-all/` (1013 community comps, dir = ``); manifest +`/tmp/cc-manifest.json` (has `dir`, `gate` per comp). + +| script | what it does | output | +|--------|-------------|--------| +| `we-render.mjs ` | render one comp; mode via env (`PRODUCER_EXPERIMENTAL_FAST_CAPTURE`, `HF_DE_WORKER_ENCODE`, gate-bypass envs). Prints `RENDER_OK`, `TOTAL_MS`, engine logs. | mp4 | +| `de-hybrid.sh` | the 42 fallback comps through hybrid (gates bypassed + per-frame fallback) vs screenshot baseline; whole-video PSNR. | `/tmp/hybrid-eval.jsonl` | +| `de-dmgdist.sh` | the 18 hybrid-damaged comps: per-frame PSNR → damaged-frame fraction + clustering → SPARSE/MIXED/PERVASIVE. | `/tmp/dmgdist.jsonl` | +| `de-predict-run.sh` + `de-predict-psnr.mjs` | per-frame predictor-signal probe (needs the `HF_PREDICT_PROBE` engine probe re-added) + ffmpeg PSNR join. | `/tmp/predict/joined.jsonl` | + +Per-comp render pattern (single comp, e.g. baseline vs hybrid): +```bash +cd /Users/vanceingalls/src/wt/hyperframes/blend-diff/packages/producer +dir=$(find /tmp/cc-all -maxdepth 1 -type d -name "af468890*" | head -1) +# screenshot baseline: +PRODUCER_EXPERIMENTAL_FAST_CAPTURE=false HF_DE_WORKER_ENCODE=false \ + PRODUCER_BROWSER_GPU_MODE=hardware PRODUCER_ENABLE_BROWSER_POOL=false \ + bun we-render.mjs "$dir" /tmp/base.mp4 +# hybrid (drawElement + per-frame fallback, gates bypassed): +HF_FAST_CAPTURE_CSSFX=true HF_FAST_CAPTURE_CROSSFADE=true HF_FAST_CAPTURE_3D=true HF_FAST_CAPTURE_VIDEO=true \ + PRODUCER_EXPERIMENTAL_FAST_CAPTURE=true HF_DE_WORKER_ENCODE=true \ + PRODUCER_BROWSER_GPU_MODE=hardware PRODUCER_ENABLE_BROWSER_POOL=false \ + bun we-render.mjs "$dir" /tmp/hyb.mp4 +# per-frame PSNR (ffmpeg writes per-frame stats; line N => psnr_avg:): +ffmpeg -i /tmp/hyb.mp4 -i /tmp/base.mp4 -lavfi "psnr=stats_file=/tmp/x.psnr" -f null - +# SSIM (better than PSNR near black): -lavfi ssim +# view a frame: ffmpeg -y -i /tmp/hyb.mp4 -vf "select=eq(n\,240)" -vframes 1 /tmp/f240.png then Read the png +``` + +--- + +## 4. Key findings (so a continuation doesn't re-derive them) + +1. **Per-frame style prediction FAILS** (recall 21–71%, precision 13–35%). Don't go down that path. +2. **The deterministic signal that WORKS** = GSAP timeline intervals + clip schedule. +3. **Lim 6 (clip-cut boundary blackout):** drawElement renders hard clip-cut frames + black (outgoing clip dropped a frame before incoming paints). FIXED (6b). Verified: 7dec755f f240 black→correct. +4. **Lim 7 (semi-transparent overlay drop):** drawElement drops opacity-animated + full-screen overlays (e.g. a white `.flash` div tweened opacity 0→1→0) → renders + the layer underneath. Damaged interval = the tween's `[start,end]` frames. NOT fixed; Task A fixes it. +5. **`No cached paint record` crash:** FIXED (6a) — was 11-comp crash class. Verified: 29c3224f aborted→completes. +6. **Damage distribution of the 18 hybrid-damaged comps** (`/tmp/dmgdist.jsonl`): + **13 SPARSE** (0–8% frames damaged, clustered) → fixable by Task A; + **1 MIXED** (d0a3039a, 35%); **4 PERVASIVE** (9e3c36ba 55%, 4fcc7660 77%, + 5697cb63 99%, 38097e59 100%) → stay gated, correct. +7. **macOS `captureScreenshot` is flat ~24 ms/frame** regardless of static/dynamic + content (no dedup) → Task B opportunity. (BeginFrame/Linux DOES dedup via `hasDamage`.) +8. **Speedup is density-bound, not duration-bound** (~1.0× on static-heavy comps; up to 3× dense). + +--- + +## 5. TASK A — Build the timeline-interval at-risk predictor (the fix for 13/18) + +**Goal:** at session init, compute the set of "at-risk" frames deterministically from +the GSAP timeline + clip schedule. During capture, screenshot at-risk frames, +drawElement the rest. This generalizes the shipped `clipBoundaryFrames` mechanism. + +**Why this is the right design (already validated):** damaged frames are NOT +predictable from per-frame computed styles, but ARE bounded by the frame intervals of +GSAP tweens that animate compositor-incompatible properties (opacity, filter, +backdrop-filter, mix-blend-mode, 3D transform, mask/clip-path), plus clip-cut +boundaries. Proven on af468890: damaged frames f102–121 == the `.flash` opacity tween +span. 13 of 18 damaged comps are sparse → screenshotting these intervals makes them +healthy while keeping 92–100% of frames fast. + +### Step A1 — Inspect the GSAP timeline shape +GSAP timelines are registered on `window.__timelines` (per CLAUDE.md: "GSAP timelines +must be paused and registered on window.__timelines"). In a page context, enumerate +tweens. Find the API: a GSAP timeline exposes `.getChildren(true,true,true)` → +array of tweens; each tween has `.startTime()` (seconds, in timeline coords), +`.duration()`, `.targets()` (DOM elements), and `.vars` (the animated props, e.g. +`{opacity:1, x:100}`). **Verify this in a real page first** — write a tiny +`page.evaluate` that dumps, for af468890, every tween's `{start, dur, props:Object.keys(vars), targetTag}`. Confirm you can see the `.flash` opacity tween at start≈3.4s dur≈0.2s. + + Gotcha: timelines may be nested (sub-timelines). Recurse via `getChildren`. Tween + start times are relative to their parent timeline — convert to absolute by summing + parent offsets, or use the root timeline's `.totalTime()` mapping. SAFEST: for each + tween, compute its absolute `[start,end]` via GSAP's own global time + (`tween.globalTime?` or walk parents). Validate against af468890's known f102–121. + +### Step A2 — Define the compositor-incompatible property set +A tween is "at-risk" if it animates ANY of: `opacity`, `autoAlpha`, `filter`, +`backdropFilter`/`backdrop-filter`, `mixBlendMode`/`mix-blend-mode`, `webkitFilter`, +any 3D transform prop (`rotationX`,`rotationY`,`z`,`rotateX`,`rotateY`,`translateZ`), +`clipPath`/`clip-path`, `maskImage`/`mask`. Also: any element that is STATICALLY (no +tween) one of these the whole time → handled by the existing whole-comp gate, leave gated. + - Conservative bias: when unsure if a prop is compositor-incompatible, INCLUDE it + (over-screenshot = slower, under-screenshot = shipped damage). Recall > precision. + +### Step A3 — Compute the at-risk frame set +For each at-risk tween: `for f in [floor(start·fps)-1 .. ceil((start+dur)·fps)+1]: atRisk.add(f)` +(±1 margin). Union with the existing `computeClipBoundaryFrames` set. Store on +`session.clipBoundaryFrames` (rename to `session.atRiskFrames` or add a second set). + + Implementation: add a `computeTimelineAtRiskFrames(page, fps)` next to + `computeClipBoundaryFrames` in `frameCapture.ts`; call it in `initializeSession` + right after the boundary computation; merge into the same Set used by the capture branches. + Env off-switch: `HF_FAST_CAPTURE_INTERVAL_SS=false`. + +### Step A4 — Validate +1. **af468890 must flag f102–121** (the known flash tween). Add a debug log of the + at-risk set size + ranges; render af468890 with the new predictor; confirm. +2. **Re-run the 13 sparse comps** through hybrid+interval-predictor (use de-dmgdist.sh + pattern, but with the predictor on instead of bypass) and confirm per-frame PSNR: + the previously-damaged frames are now screenshotted (→ ∞ or healthy) and the + damaged-frame fraction drops to ~0. Target: all 13 reach >45 dB mean (or 0% damaged frames). +3. **Confirm speedup survives:** the at-risk set should be a small fraction of frames + for the sparse comps (it was 0–8% damaged → at-risk should be similar small %). Log + `atRiskFrames.size / totalFrames` per comp; should be <25% for the SPARSE set. +4. **The 4 PERVASIVE comps** (9e3c36ba, 4fcc7660, 5697cb63, 38097e59): at-risk fraction + will be high (>50%) → add a go/no-go: if at-risk fraction > ~40%, fall through to + the whole-comp gate (full screenshot) — don't bother with hybrid. + +### Step A5 — Ship +Strip any debug logging/probes. Typecheck + oxlint + oxfmt. Commit on the stack +(`gt create --no-verify -m "..."` stacked on `drawelement-perframe-fallback`). +Update `fast-capture-limitations.md` (mark Lim 7 fixed, record the lift on the 13 comps). + +--- + +## 6. TASK B — macOS static-frame dedup spike (separate lever) + +**Premise (measured):** macOS `Page.captureScreenshot` is flat ~24 ms/frame; static +holds re-capture identical frames for nothing. BeginFrame/Linux already dedups via +`hasDamage` (returns `lastFrameCache`). Port the idea to macOS. + +**Detection (deterministic, pre-capture):** a frame is static iff NO GSAP tween is +active in `[t_prev, t_now]` AND no video AND no canvas/webgl present (those may redraw +without a tween). Reuse the SAME `window.__timelines` introspection as Task A — a frame +with no active tween + no dynamic media = identical to the previous frame → skip +capture, reuse the previous buffer. + +**Caveats (from prior probe, memory `project-fast-capture-perf-model`):** dedup is +NARROW — ~89% of comps disqualified (78% have a canvas, 11% video); eligible comps +often have continuous tweens (zero static frames). Best case ~31% frames skippable +(long-holds presentation). **Under-detection = stale/frozen frame (correctness bug)** — +bias conservative. + +**First step:** DON'T build yet. Measure the TRUE static-frame fraction across the +corpus via timeline introspection (NOT the JPEG-byte-size proxy, which overcounts). +Reuse Task A's tween-interval machinery: static frames = total − union(all tween +active intervals) − dynamic-media frames. Report the distribution; size the payoff; +THEN decide. + +--- + +## 7. Reference data + +- **The 18 hybrid-damaged comps + classification:** `/tmp/dmgdist.jsonl`. Summary: + - SPARSE/fixable (13): 7882f09c, 0670ee56, 4fa05434, b459dbe9, 100c5fad, 27388fb9, 2666ea5d, 6502a4ed, 8f942142, 7dec755f, af468890, 4cf3ed60, 38e9ca4c + - MIXED (1): d0a3039a (35%) + - PERVASIVE/gate-only (4): 9e3c36ba (55%), 4fcc7660 (77%), 5697cb63 (99%), 38097e59 (100%) +- **Hybrid eval (all 42 fallback comps):** `/tmp/hybrid-eval.jsonl` (fields: n, bok, hok, inj, boundary, crash, refell, psnr). +- **Best validated golden comp for Task A:** `af468890` — flash overlay crossfade, + damaged frames f102–121 (and f290–300-ish, f470-480-ish; tween repeats per scene). + Use it as the unit test for the predictor. +- **Verse dashboard (live):** https://www.heygenverse.com/a/9e71f3dc-23c8-4260-b45f-8e9e65a34c1c + (worker-encode + hybrid eval; fallback rows show hybrid PSNR + verdict tag). Data + asset `aa298d41-bdff-4b87-950c-107870f536ce` (v4). To update: edit `/tmp/we-page.html`, + upload new data via `hv_upload_asset` (content=...), update `DATA_URL`, `hv_update` + the app (id `9e71f3dc-...`). Pass HTML content, not a URL. +- **Memory:** `project-drawelement-clip-boundary-blackout` and + `project-fast-capture-perf-model` (have the condensed findings + commit SHAs). + +--- + +## 8. Definition of done for the next stream +- Task A shipped: 13 sparse comps render healthy (>45 dB / ~0% damaged frames) on the + fast path with the interval predictor, at-risk fraction logged, 4 pervasive comps + correctly fall through to gate. Committed on the stack, docs updated. +- Task B: static-frame fraction measured across corpus; go/no-go decision recorded. +- Then (user's call): push the stack / `gt submit`. diff --git a/packages/engine/docs/fast-capture-limitations.md b/packages/engine/docs/fast-capture-limitations.md new file mode 100644 index 0000000000..4a0a0531dc --- /dev/null +++ b/packages/engine/docs/fast-capture-limitations.md @@ -0,0 +1,313 @@ +# drawElement Fast Capture — Limitations & Findings Log + +Canonical findings log for the `drawElementImage` fast-capture path. Referenced +from `frameCapture.ts`, `drawElementService.ts`, and `compileStage.ts`. Append +new findings here rather than burying them in commit messages. + +See also: `fast-capture-architecture.md` (how the speedup + worker-encode work, the +per-frame cost model, and why worker-encode is macOS-GPU-only) and the +implementation spec `docs/superpowers/specs/2026-06-08-drawelement-alpha.md`. + +--- + +## What it is + +`canvas.drawElementImage(element, x, y)` (experimental Chrome +`--enable-features=CanvasDrawElement`) reads a DOM subtree's paint records +straight into a ``, bypassing the compositor. Used as a +capture mode in `packages/engine` as a faster alternative to +`Page.captureScreenshot`. + +**Speedup is GPU- and density-bound, not universal:** +- Only faster on a **hardware GPU (macOS)**. On SwiftShader (Docker/Linux + software rasterizer) there is no GPU→CPU readback to skip, so drawElement is + parity-or-slower — it is **not used there** (routes to BeginFrame/screenshot). +- Speedup tracks **animation density**, not duration. p50 ≈ 1.0–1.4×; short + dense comps reach ~2–3×; long comps with static holds approach 1.0× (nothing + per-frame to save). A 691s dense comp still hit 1.42×. + +**Per-frame cost model (macOS GPU, 1080p, ~16.3 ms/frame):** paint-event wait +~6.2 ms (37%, vsync floor), JPEG encode ~7.4 ms (57%), readback ~2.15 ms, seek +0.67 ms, drawElementImage itself ~0.4 ms. In-page `performance.now()`/`Date.now()` +are clamped in headless Chrome — only node-side timing is reliable. + +--- + +## Capture-mode gates (whole-comp routing at session init / compile) + +If any gate fires, the **entire** render routes to the platform baseline +(screenshot on macOS, BeginFrame on Linux). Gating is whole-composition, decided +once — there is **no per-frame gating** today. + +| # | Gate | Signal | Bypass env | crbug | +|---|------|--------|-----------|-------| +| 1 | SwiftShader | software-GL detected | — | none (by design) | +| 2 | Video / caption-pattern | `