mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-15 13:58:20 +00:00
fix(engine): per-frame screenshot fallback for drawElement clip-cut boundaries
drawElement renders hard clip-cut boundary frames incorrectly: the outgoing clip is dropped one frame before the incoming clip's paint record is ready, so the frame either throws InvalidStateError "No cached paint record for element" (aborting the whole render) or silently renders solid black. Both are real production damage, confirmed by a clean fast-render of a clip comp emitting a black frame at a clip cut, and by the 11-comp crash class from the 400-comp eval. Two per-frame fallbacks, both routing the single bad frame to captureScreenshot instead of failing the render: * Throw case: catch "No cached paint record" (isNoCachedPaintRecordError) in both the serial (captureFrameCore) and worker-encode (captureFrameToBufferPipelined) paths. Validated on a comp that previously hard-aborted, now completes with 11 frames screenshot-fallen-back. * Silent-black case: compute clip-cut boundary frames (plus or minus 1) from the clip schedule (data-start/data-duration times fps) at init and screenshot those frames. A boundary-targeted spike showed 100% of black-outs land on these frames. Validated: a clip comp that emitted a black frame at t=8 now renders correctly. Default-on for drawElement renders; disable boundary fallback with HF_FAST_CAPTURE_BOUNDARY_SS=false. Adds docs/fast-capture-limitations.md, the canonical findings log the code comments already referenced but which never existed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
e27afcdb40
commit
c003abb3cb
@@ -0,0 +1,227 @@
|
||||
# drawElement Fast Capture — Architecture & Performance
|
||||
|
||||
How the fast-capture path works, why the worker-encode optimization exists, and
|
||||
why it only applies on the macOS-GPU path. Companion to
|
||||
`fast-capture-limitations.md` (which covers *what drawElement can't render*); this
|
||||
doc covers *how the speedup works and where*.
|
||||
|
||||
---
|
||||
|
||||
## Capture modes & the "who encodes" model
|
||||
|
||||
Rendering = capture each frame to an encoded image (JPEG/PNG), then feed those to
|
||||
ffmpeg, which re-encodes to H.264. There are three capture mechanisms, and the key
|
||||
difference between them is **who runs the per-frame image encode, and on which
|
||||
thread**:
|
||||
|
||||
| mode | pixel path | image encode | where it runs |
|
||||
|------|-----------|--------------|---------------|
|
||||
| **screenshot** (`Page.captureScreenshot`) | full compositor → readback (slow) | **Chrome encodes internally** | C++, off the page's JS thread (free to us) |
|
||||
| **BeginFrame** (`HeadlessExperimental.beginFrame`, Linux headless-shell) | atomic paint+capture+encode in one CDP call | **Chrome encodes internally** | C++, off-thread, indivisible |
|
||||
| **drawElement** (macOS GPU) | reads paint records directly into a canvas (fast) | **we** call `canvas.toDataURL()` | **page JS main thread** |
|
||||
|
||||
The mental model:
|
||||
|
||||
- **screenshot** = slow pixel path + *free off-thread encode* (Chrome does it).
|
||||
- **drawElement (naive)** = fast pixel path + *costly on-thread encode* (we do it in JS).
|
||||
- **drawElement + worker** = fast pixel path + *off-thread encode* — best of both.
|
||||
|
||||
drawElement's whole win is the **fast pixel path**: it reads the painted subtree's
|
||||
paint records straight into a `<canvas layoutsubtree>`, skipping the full
|
||||
compositor pipeline and the expensive GPU→CPU screenshot readback. The price it
|
||||
pays is that Chrome only hands back **raw pixels** — so *we* have to encode them,
|
||||
and the naive way to do that (`toDataURL`) runs on the page's main thread.
|
||||
|
||||
---
|
||||
|
||||
## Per-frame cost model (macOS GPU, 1080p, ~16 ms/frame)
|
||||
|
||||
Measured 2026-06-14 via env-gated isolation probes. Reliable timer is node-side
|
||||
`Date.now()` around `page.evaluate` — in-page `performance.now()`/`Date.now()` are
|
||||
clamped to ~0 in headless Chrome, don't trust them.
|
||||
|
||||
| step | cost | notes |
|
||||
|------|------|-------|
|
||||
| seek (CDP round-trip) | 0.67 ms | jump the page to frame N's time; fusing round-trips not worth it |
|
||||
| paint-event wait | ~6.2 ms (37%) | vsync-locked, ~0.37 frame avg; **no cheap fix**, BeginFrame unavailable on macOS GPU |
|
||||
| drawElementImage + dispatch | 0.4 ms | the actual paint-record read; near-free |
|
||||
| GPU→CPU readback (`getImageData`) | ~2.15 ms | unavoidable, ffmpeg needs pixels |
|
||||
| **`toDataURL` JPEG encode** | **~7.36 ms (57%)** | biggest single cost; pure double-work (ffmpeg re-encodes to H.264) |
|
||||
|
||||
Two facts drive the design:
|
||||
- The **encode is the dominant cost** (57%), bigger than everything else combined.
|
||||
- It's **double work** — we JPEG-encode only to hand it to ffmpeg, which throws the
|
||||
JPEG away and re-encodes to H.264.
|
||||
|
||||
### Speedup is density-bound, not duration-bound
|
||||
Measured distribution (macOS GPU, drawElement vs screenshot): p50 ~1.02×; short
|
||||
dense (<10 s) p50 ~1.30×; long (>120 s) ~1.00×. Speedup tracks **animation density**
|
||||
(fraction of frames with real per-frame change), not total length. Proof it's not
|
||||
duration: `268b09db`, the longest comp tested (691 s) but dense throughout
|
||||
(continuous motion, 62 audio tracks), hit **1.42×**, not 1.0×.
|
||||
|
||||
**Static-frame handling differs by capture path — be precise about which:**
|
||||
|
||||
- **BeginFrame (Linux):** Chrome's BeginFrame pipeline keeps a `lastFrameCache`;
|
||||
when a frame's `hasDamage` is `false` it **returns the previously captured
|
||||
buffer** — static frames are skipped (no recapture, no re-encode), so they're
|
||||
effectively free on the baseline. (See `discardWarmupCapture` for why chunk
|
||||
workers must prime this cache.)
|
||||
- **macOS screenshot (the drawElement baseline):** `pageScreenshotCapture` calls
|
||||
`Page.captureScreenshot` **every frame** — no damage check, no cache, no
|
||||
copy-fill.
|
||||
- **drawElement (macOS):** does **not** skip static frames either — it *forces* a
|
||||
paint each frame by toggling the `__hf_de_tick` sentinel, guaranteeing a fresh
|
||||
snapshot "even when this frame's seek produced no paint-level change", with a
|
||||
250 ms safety-net timeout.
|
||||
|
||||
So on macOS both paths capture every frame. The density-dependence there is
|
||||
**not** explained by frame-skipping. The most likely driver is that Chrome's
|
||||
per-frame `captureScreenshot` cost scales with how much the compositor must do, so
|
||||
on a cheap static frame drawElement's *absolute* saving shrinks and the ratio
|
||||
trends to 1.0× — but this has **not been isolated by measurement** (per-frame
|
||||
screenshot cost vs content was not profiled). Treat the macOS mechanism as
|
||||
unconfirmed; the density-bound *outcome* is measured, the *cause* on macOS is not.
|
||||
|
||||
### Why JPEG at all (don't ship raw)
|
||||
The captured pixels live in the headless browser; we ship them to Node over CDP. A
|
||||
raw 1080p frame is ~8 MB; a base64 JPEG is a fraction. Shipping raw was **measured
|
||||
at ~109 ms/frame (11× worse)** — readback + base64 of 8 MB + transport. So the
|
||||
compact JPEG-over-CDP *is* the right transport. The problem was never *that* we
|
||||
encode — it's *where* the encode runs.
|
||||
|
||||
---
|
||||
|
||||
## Worker-encode (PR #1444)
|
||||
|
||||
### The problem: `toDataURL` blocks the main thread
|
||||
`canvas.toDataURL()` runs **synchronously on the page's main thread**, so the loop
|
||||
was serial — while the main thread spends 7.4 ms compressing frame N, it can't
|
||||
start producing frame N+1:
|
||||
|
||||
```
|
||||
frame N: [seek][paint][read][====JPEG encode 7.4ms====]
|
||||
frame N+1: [seek][paint]...
|
||||
```
|
||||
|
||||
### The fix: encode in an OffscreenCanvas Web Worker, pipelined
|
||||
1. Main thread does seek + paint + drawElement read → gets an `ImageBitmap`.
|
||||
2. It **transfers** the bitmap to a worker (transferable = zero-copy handoff, not a
|
||||
clone).
|
||||
3. The **worker** runs the JPEG encode while the main thread **immediately starts
|
||||
the next frame**.
|
||||
|
||||
This is a **depth-2 pipeline** (`runWorkerEncodePipelineLoop` in
|
||||
`captureStreamingStage.ts`): frame N's encode overlaps frame N+1's produce. Per-frame
|
||||
wall-clock drops toward `max(produce, encode)` instead of `produce + encode`:
|
||||
|
||||
```
|
||||
frame N: [produce][--encode N (worker)--]
|
||||
frame N+1: [produce][--encode N+1--]
|
||||
```
|
||||
|
||||
`produceDrawElementFrame` (`drawElementService.ts`) returns
|
||||
`{ encodeResult: Promise<Buffer> }` immediately; the loop awaits the *previous*
|
||||
frame's encode while producing the current one. The base64 result stays **inside
|
||||
the worker** so it never burdens the page main thread.
|
||||
|
||||
### Results
|
||||
- **~2× vs the screenshot baseline** (capture-phase A/B vs serial drawElement: 1.49–1.84×).
|
||||
- **Bit-identical output** — worker frames vs serial-drawElement frames = PSNR ∞
|
||||
(same bytes). The compression math is unchanged; only *where* it runs moved. Zero
|
||||
quality cost — this was the gate to ship.
|
||||
|
||||
### Hardening (two review rounds)
|
||||
Moving work into a worker added failure modes:
|
||||
- Orphaned encode promises (unhandled rejection if a frame's encode is in flight at
|
||||
abort) → source-level `void encodeResult.catch(() => {})`.
|
||||
- Lost worker message → per-frame **30 s watchdog**.
|
||||
- Allocation churn → reuse **one** OffscreenCanvas across frames.
|
||||
- `bmp.close()` on the no-worker fallback path; empty-encode reject;
|
||||
`exposeFunction` bound-pages guard; Blob-URL revoke.
|
||||
|
||||
### Config / gating
|
||||
Off by default; `enableDrawElementWorkerEncode` / env `HF_DE_WORKER_ENCODE`. Engages
|
||||
only when capture resolved to drawElement **and** `beginFrameTimeTicks === 0` (macOS
|
||||
GPU) **and** output is opaque JPEG.
|
||||
|
||||
### Remaining ceiling
|
||||
The base64 JPEG still travels over CDP as text. The next theoretical win is a
|
||||
**binary side-channel** (e.g. websocket) to skip base64 — bigger lift, out of scope
|
||||
here. The design deliberately keeps base64 in the worker so it never hits the page
|
||||
main thread.
|
||||
|
||||
---
|
||||
|
||||
## Why worker-encode is macOS-GPU-only
|
||||
|
||||
It offloads exactly one cost — the in-page `toDataURL`. The other environments don't
|
||||
have that cost to offload.
|
||||
|
||||
### Docker / SwiftShader (Linux containers): drawElement isn't used at all
|
||||
drawElement's advantage is skipping the **GPU→CPU readback** a screenshot pays. On
|
||||
SwiftShader (software rasterizer, no GPU) there is no GPU→CPU boundary — rendering is
|
||||
already on the CPU — so drawElement is parity-or-slower and we don't engage it. No
|
||||
drawElement → no worker-encode (it lives inside that path). And the bottleneck there
|
||||
is the **software raster**, not the encode, so overlapping the encode would hide the
|
||||
wrong cost.
|
||||
|
||||
### BeginFrame (Linux headless-shell): the encode is browser-side and atomic
|
||||
BeginFrame returns the **already-encoded** image from a single `beginFrame` CDP call
|
||||
— Chrome does paint + capture + compress together, internally. So:
|
||||
- **Nothing to offload** — there's no in-page `toDataURL`; the encode the worker
|
||||
would take over doesn't exist in page JS.
|
||||
- **Nothing to overlap** — one indivisible call; you can't split paint/capture/encode
|
||||
apart from JS to pipeline them.
|
||||
- BeginFrame mode doesn't use the drawElement canvas, which worker-encode needs (it
|
||||
transfers the `ImageBitmap` from it). Gating is literally `beginFrameTimeTicks === 0`
|
||||
= "only when *not* BeginFrame."
|
||||
|
||||
### macOS GPU: the one place the cost exists
|
||||
macOS GPU has **no BeginFrame**, so the Mac path captures via paint-event-synced
|
||||
drawElement + an in-page `toDataURL`. That `toDataURL` is the offloadable cost, and
|
||||
produce/encode are separable → pipelineable. That combination exists only on the
|
||||
Mac-GPU-drawElement path — exactly where worker-encode applies.
|
||||
|
||||
**Summary:** worker-encode isn't broken elsewhere — the cost it removes only exists on
|
||||
macOS GPU. Speedups for the other environments would be *different* optimizations
|
||||
(attacking software-raster cost on SwiftShader, or a binary transport), not this one.
|
||||
|
||||
---
|
||||
|
||||
## Future work / planned spikes
|
||||
|
||||
### Spike: macOS static-frame dedup (the BeginFrame trick, ported to macOS)
|
||||
**Measured fact:** `Page.captureScreenshot` on macOS is **flat ~24 ms/frame
|
||||
regardless of content** (static frames ≈ dynamic frames; profiled on af468890 via
|
||||
`HF_SCRTIME`). So we pay full price re-capturing identical frames on static holds.
|
||||
BeginFrame/Linux already skips these (`hasDamage` → reuse `lastFrameCache`); macOS
|
||||
does not.
|
||||
|
||||
**Idea:** detect static frames *before* capture and reuse the previous buffer.
|
||||
Detection must be deterministic + conservative — the right signal is **GSAP
|
||||
timeline introspection** (`window.__timelines`): a frame is static iff no tween is
|
||||
active in `[t_prev, t_now]` AND no video/canvas/webgl present. This is the **same
|
||||
machinery** as the dependable hybrid predictor (see `fast-capture-limitations.md`)
|
||||
→ one prototype, two payoffs (skip static frames + screenshot at-risk frames).
|
||||
|
||||
**Why it's attractive:** complements drawElement — dedup wins on the static-heavy /
|
||||
long comps where drawElement gives ~1.0×, and it helps the screenshot-fallback
|
||||
majority (forced-screenshot + gated comps), not just drawElement-eligible ones.
|
||||
|
||||
**Why scope it carefully (prior probe, ~18 comps):** ~89% of comps are disqualified
|
||||
— 78% have an accelerated canvas (may redraw any frame), 11% video; eligible comps
|
||||
often have continuous tweens (zero static frames). Best case ~31% frames skippable
|
||||
(long-holds presentation). Under-detection = stale/frozen frame (correctness bug).
|
||||
|
||||
**First step:** measure the *true* static-frame fraction across the corpus via
|
||||
timeline introspection (NOT the crude JPEG-byte-size proxy, which overcounts), to
|
||||
size the real population and payoff before building. Fold into the hybrid-predictor
|
||||
prototype.
|
||||
|
||||
### Spike: dependable hybrid predictor (timeline-interval at-risk calculator)
|
||||
See `fast-capture-limitations.md` → "The dependable pre-render calculation."
|
||||
`window.__timelines` → frame intervals where a compositor-incompatible property is
|
||||
animating, ∪ clip-schedule boundaries, ∪ static-effect-visible frames → screenshot
|
||||
those, drawElement the rest. Validate it flags af468890's f102–121 exactly. Shares
|
||||
the timeline-introspection core with the dedup spike above.
|
||||
@@ -0,0 +1,271 @@
|
||||
# drawElement Fast Capture — Session Handoff / Continuation Plan
|
||||
|
||||
**Audience:** the next engineer/agent continuing this work. Written to be
|
||||
self-contained and prescriptive. Read the two companion docs first:
|
||||
- `fast-capture-architecture.md` — how the speedup works, worker-encode, cost model, why macOS-only, density-bound.
|
||||
- `fast-capture-limitations.md` — gates, damage classes (Lim 2/3/4/6/7), the predictor design, eval results, damage distribution.
|
||||
|
||||
This doc = **what's done, exact repo/branch state, how to run everything, and the
|
||||
two concrete next tasks step-by-step.**
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR — where things stand
|
||||
|
||||
- **Shipped & validated** (committed, NOT pushed): per-frame screenshot fallback
|
||||
in drawElement — (6a) catches `No cached paint record` crashes, (6b) screenshots
|
||||
clip-cut boundary frames it would otherwise render black. Both in both serial and
|
||||
worker-encode capture paths.
|
||||
- **Investigated** the "hybrid" question exhaustively: can we drawElement comps that
|
||||
currently fall back, screenshotting only bad frames? Answer: **yes for 13–14 of 18
|
||||
damaged comps**, via a **deterministic timeline-interval predictor** (NOT per-frame
|
||||
style guessing, which failed).
|
||||
- **NEXT WORK (not started):**
|
||||
1. **Build the timeline-interval predictor** — the actual fix for 13/18 damaged comps. (Task A below.)
|
||||
2. **macOS static-frame dedup spike** — separate speedup lever. (Task B below.)
|
||||
|
||||
---
|
||||
|
||||
## 1. Git state (exact)
|
||||
|
||||
- Worktree: `/Users/vanceingalls/src/wt/hyperframes/blend-diff`
|
||||
- Branch: **`drawelement-perframe-fallback`**, HEAD = `09a123d63`
|
||||
- Graphite stack (bottom→top), all on this branch's history:
|
||||
```
|
||||
main
|
||||
→ 5ea76a52d perf: worker-offload JPEG encode (PR #1444 base work)
|
||||
→ dd0bd4aaf fix: harden worker-encode pipeline (review)
|
||||
→ e27afcdb4 fix: close worker-encode orphan/hang/corruption (re-review)
|
||||
→ 09a123d63 fix: per-frame screenshot fallback (THIS session's shipped work)
|
||||
```
|
||||
Note: PR #1444 = worker-encode; PR #1295 = the base drawelement-fast-capture branch (gates + worker-encode stacked above it).
|
||||
- **NOT pushed.** Per standing rule, do not push / `gt submit` until the user asks.
|
||||
- **Uncommitted:** `packages/engine/docs/fast-capture-limitations.md` (latest damage-distribution section). Commit it (see §5 commit note).
|
||||
- **Untracked scratch (do NOT commit — investigation tooling):**
|
||||
`packages/producer/{de-dmgdist.sh, de-hybrid.sh, de-predict-run.sh, de-predict-psnr.mjs, de-we-suite.sh, de-we-fast.sh, we-render.mjs}`
|
||||
- The engine probe scratch (`HF_PREDICT_PROBE`, `HF_INV_PROBE`, `HF_SCRTIME`,
|
||||
`HF_FAST_CAPTURE_OPACITY_SS`) was **already stripped** from `frameCapture.ts` —
|
||||
it holds only the shipped 6a/6b fix. If you re-add probes for Task A, strip again before commit.
|
||||
|
||||
### What the shipped fix looks like (in `packages/engine/src/services/frameCapture.ts`)
|
||||
- `isNoCachedPaintRecordError(err)` — exported helper.
|
||||
- `computeClipBoundaryFrames(page, fps)` — reads `[data-start]` elements, returns a
|
||||
`Set` of `{round(start·fps), round((start+dur)·fps)}` ±1 frames.
|
||||
- `CaptureSession.clipBoundaryFrames?: Set<number>` — populated in `initializeSession`
|
||||
when capture resolves to drawelement (env off-switch `HF_FAST_CAPTURE_BOUNDARY_SS=false`).
|
||||
- In `captureFrameCore` (serial) and `captureFrameToBufferPipelined` (worker): if
|
||||
`clipBoundaryFrames.has(frameIndex)` → `pageScreenshotCapture` instead of drawElement;
|
||||
and a `try/catch` around the drawElement capture that screenshots on `isNoCachedPaintRecordError`.
|
||||
|
||||
**This boundary fix is the TEMPLATE for Task A** — Task A generalizes
|
||||
`clipBoundaryFrames` from "clip cuts only" to "all at-risk intervals."
|
||||
|
||||
---
|
||||
|
||||
## 2. Environment & gotchas (CRITICAL — a weaker model will trip on these)
|
||||
|
||||
- **Build/commit hook fails** on stale local SDK deps (`@hyperframes/core build` /
|
||||
linkedom). It's environmental, CI-clean. **Commit with `git commit --no-verify`**
|
||||
(or `gt create --no-verify`). Typecheck/lint pass fine; the hook failure is NOT your code.
|
||||
- **macOS has no `timeout` command.** Never use `timeout N cmd` — it silently no-ops.
|
||||
For long renders: run in background, poll with `while pgrep -f "<pattern>"; do sleep 5; done`, then read the log.
|
||||
- **Background renders can leave zombies.** If a render hangs, `pkill -f "we-render.mjs /tmp/cc-all/<prefix>"`. Check for strays: `pgrep -fl we-render.mjs`.
|
||||
- **In-page timers are clamped** in headless Chrome (`performance.now`/`Date.now` ≈ 0).
|
||||
Only node-side `Date.now()` around `page.evaluate` is reliable for timing.
|
||||
- **Mean PSNR is misleading near black** and for sparse damage. A few near-black or
|
||||
transition frames tank the mean even when 95%+ of frames are perfect. Use per-frame
|
||||
PSNR distribution (damaged-frame fraction) + SSIM, not mean PSNR, to judge quality.
|
||||
- **Some comps are nondeterministic** (`performance.now`/`Date.now`/`Math.random`/raw
|
||||
`requestAnimationFrame`). They're force-screenshotted by render-mode hints already —
|
||||
they can't drawElement and their "damage" is their own (two screenshot renders differ).
|
||||
Exclude them from any drawElement quality accounting.
|
||||
- **drawElement is macOS-GPU-only.** On Linux/Docker (SwiftShader) it isn't used.
|
||||
- **Package manager: bun** (not pnpm/npm). Lint/format: `bunx oxlint <file>` / `bunx oxfmt <file>`. Typecheck engine: `cd packages/engine && bunx tsc --noEmit -p tsconfig.json`.
|
||||
|
||||
### Gate-bypass env vars (force drawElement to engage despite gates — for evals)
|
||||
```
|
||||
HF_FAST_CAPTURE_CSSFX=true # backdrop-filter / filter:blur / webgl gate
|
||||
HF_FAST_CAPTURE_CROSSFADE=true # stacked-fade gate
|
||||
HF_FAST_CAPTURE_3D=true # 3D gate (compile + runtime)
|
||||
HF_FAST_CAPTURE_VIDEO=true # video gate (compile + runtime)
|
||||
HF_FAST_CAPTURE_BOUNDARY_SS=false # disable the shipped boundary screenshot fallback
|
||||
```
|
||||
Leave `HF_FAST_CAPTURE_AUTOALPHA` default (on).
|
||||
|
||||
---
|
||||
|
||||
## 3. The eval harnesses (untracked, in `packages/producer/`)
|
||||
|
||||
All run from `packages/producer/`. All use `we-render.mjs` (renders one comp; mode
|
||||
from env). Corpus: `/tmp/cc-all/` (1013 community comps, dir = `<uuid>`); manifest
|
||||
`/tmp/cc-manifest.json` (has `dir`, `gate` per comp).
|
||||
|
||||
| script | what it does | output |
|
||||
|--------|-------------|--------|
|
||||
| `we-render.mjs <compDir> <out.mp4>` | render one comp; mode via env (`PRODUCER_EXPERIMENTAL_FAST_CAPTURE`, `HF_DE_WORKER_ENCODE`, gate-bypass envs). Prints `RENDER_OK`, `TOTAL_MS`, engine logs. | mp4 |
|
||||
| `de-hybrid.sh` | the 42 fallback comps through hybrid (gates bypassed + per-frame fallback) vs screenshot baseline; whole-video PSNR. | `/tmp/hybrid-eval.jsonl` |
|
||||
| `de-dmgdist.sh` | the 18 hybrid-damaged comps: per-frame PSNR → damaged-frame fraction + clustering → SPARSE/MIXED/PERVASIVE. | `/tmp/dmgdist.jsonl` |
|
||||
| `de-predict-run.sh` + `de-predict-psnr.mjs` | per-frame predictor-signal probe (needs the `HF_PREDICT_PROBE` engine probe re-added) + ffmpeg PSNR join. | `/tmp/predict/joined.jsonl` |
|
||||
|
||||
Per-comp render pattern (single comp, e.g. baseline vs hybrid):
|
||||
```bash
|
||||
cd /Users/vanceingalls/src/wt/hyperframes/blend-diff/packages/producer
|
||||
dir=$(find /tmp/cc-all -maxdepth 1 -type d -name "af468890*" | head -1)
|
||||
# screenshot baseline:
|
||||
PRODUCER_EXPERIMENTAL_FAST_CAPTURE=false HF_DE_WORKER_ENCODE=false \
|
||||
PRODUCER_BROWSER_GPU_MODE=hardware PRODUCER_ENABLE_BROWSER_POOL=false \
|
||||
bun we-render.mjs "$dir" /tmp/base.mp4
|
||||
# hybrid (drawElement + per-frame fallback, gates bypassed):
|
||||
HF_FAST_CAPTURE_CSSFX=true HF_FAST_CAPTURE_CROSSFADE=true HF_FAST_CAPTURE_3D=true HF_FAST_CAPTURE_VIDEO=true \
|
||||
PRODUCER_EXPERIMENTAL_FAST_CAPTURE=true HF_DE_WORKER_ENCODE=true \
|
||||
PRODUCER_BROWSER_GPU_MODE=hardware PRODUCER_ENABLE_BROWSER_POOL=false \
|
||||
bun we-render.mjs "$dir" /tmp/hyb.mp4
|
||||
# per-frame PSNR (ffmpeg writes per-frame stats; line N => psnr_avg:<v|inf>):
|
||||
ffmpeg -i /tmp/hyb.mp4 -i /tmp/base.mp4 -lavfi "psnr=stats_file=/tmp/x.psnr" -f null -
|
||||
# SSIM (better than PSNR near black): -lavfi ssim
|
||||
# view a frame: ffmpeg -y -i /tmp/hyb.mp4 -vf "select=eq(n\,240)" -vframes 1 /tmp/f240.png then Read the png
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Key findings (so a continuation doesn't re-derive them)
|
||||
|
||||
1. **Per-frame style prediction FAILS** (recall 21–71%, precision 13–35%). Don't go down that path.
|
||||
2. **The deterministic signal that WORKS** = GSAP timeline intervals + clip schedule.
|
||||
3. **Lim 6 (clip-cut boundary blackout):** drawElement renders hard clip-cut frames
|
||||
black (outgoing clip dropped a frame before incoming paints). FIXED (6b). Verified: 7dec755f f240 black→correct.
|
||||
4. **Lim 7 (semi-transparent overlay drop):** drawElement drops opacity-animated
|
||||
full-screen overlays (e.g. a white `.flash` div tweened opacity 0→1→0) → renders
|
||||
the layer underneath. Damaged interval = the tween's `[start,end]` frames. NOT fixed; Task A fixes it.
|
||||
5. **`No cached paint record` crash:** FIXED (6a) — was 11-comp crash class. Verified: 29c3224f aborted→completes.
|
||||
6. **Damage distribution of the 18 hybrid-damaged comps** (`/tmp/dmgdist.jsonl`):
|
||||
**13 SPARSE** (0–8% frames damaged, clustered) → fixable by Task A;
|
||||
**1 MIXED** (d0a3039a, 35%); **4 PERVASIVE** (9e3c36ba 55%, 4fcc7660 77%,
|
||||
5697cb63 99%, 38097e59 100%) → stay gated, correct.
|
||||
7. **macOS `captureScreenshot` is flat ~24 ms/frame** regardless of static/dynamic
|
||||
content (no dedup) → Task B opportunity. (BeginFrame/Linux DOES dedup via `hasDamage`.)
|
||||
8. **Speedup is density-bound, not duration-bound** (~1.0× on static-heavy comps; up to 3× dense).
|
||||
|
||||
---
|
||||
|
||||
## 5. TASK A — Build the timeline-interval at-risk predictor (the fix for 13/18)
|
||||
|
||||
**Goal:** at session init, compute the set of "at-risk" frames deterministically from
|
||||
the GSAP timeline + clip schedule. During capture, screenshot at-risk frames,
|
||||
drawElement the rest. This generalizes the shipped `clipBoundaryFrames` mechanism.
|
||||
|
||||
**Why this is the right design (already validated):** damaged frames are NOT
|
||||
predictable from per-frame computed styles, but ARE bounded by the frame intervals of
|
||||
GSAP tweens that animate compositor-incompatible properties (opacity, filter,
|
||||
backdrop-filter, mix-blend-mode, 3D transform, mask/clip-path), plus clip-cut
|
||||
boundaries. Proven on af468890: damaged frames f102–121 == the `.flash` opacity tween
|
||||
span. 13 of 18 damaged comps are sparse → screenshotting these intervals makes them
|
||||
healthy while keeping 92–100% of frames fast.
|
||||
|
||||
### Step A1 — Inspect the GSAP timeline shape
|
||||
GSAP timelines are registered on `window.__timelines` (per CLAUDE.md: "GSAP timelines
|
||||
must be paused and registered on window.__timelines"). In a page context, enumerate
|
||||
tweens. Find the API: a GSAP timeline exposes `.getChildren(true,true,true)` →
|
||||
array of tweens; each tween has `.startTime()` (seconds, in timeline coords),
|
||||
`.duration()`, `.targets()` (DOM elements), and `.vars` (the animated props, e.g.
|
||||
`{opacity:1, x:100}`). **Verify this in a real page first** — write a tiny
|
||||
`page.evaluate` that dumps, for af468890, every tween's `{start, dur, props:Object.keys(vars), targetTag}`. Confirm you can see the `.flash` opacity tween at start≈3.4s dur≈0.2s.
|
||||
|
||||
Gotcha: timelines may be nested (sub-timelines). Recurse via `getChildren`. Tween
|
||||
start times are relative to their parent timeline — convert to absolute by summing
|
||||
parent offsets, or use the root timeline's `.totalTime()` mapping. SAFEST: for each
|
||||
tween, compute its absolute `[start,end]` via GSAP's own global time
|
||||
(`tween.globalTime?` or walk parents). Validate against af468890's known f102–121.
|
||||
|
||||
### Step A2 — Define the compositor-incompatible property set
|
||||
A tween is "at-risk" if it animates ANY of: `opacity`, `autoAlpha`, `filter`,
|
||||
`backdropFilter`/`backdrop-filter`, `mixBlendMode`/`mix-blend-mode`, `webkitFilter`,
|
||||
any 3D transform prop (`rotationX`,`rotationY`,`z`,`rotateX`,`rotateY`,`translateZ`),
|
||||
`clipPath`/`clip-path`, `maskImage`/`mask`. Also: any element that is STATICALLY (no
|
||||
tween) one of these the whole time → handled by the existing whole-comp gate, leave gated.
|
||||
- Conservative bias: when unsure if a prop is compositor-incompatible, INCLUDE it
|
||||
(over-screenshot = slower, under-screenshot = shipped damage). Recall > precision.
|
||||
|
||||
### Step A3 — Compute the at-risk frame set
|
||||
For each at-risk tween: `for f in [floor(start·fps)-1 .. ceil((start+dur)·fps)+1]: atRisk.add(f)`
|
||||
(±1 margin). Union with the existing `computeClipBoundaryFrames` set. Store on
|
||||
`session.clipBoundaryFrames` (rename to `session.atRiskFrames` or add a second set).
|
||||
|
||||
Implementation: add a `computeTimelineAtRiskFrames(page, fps)` next to
|
||||
`computeClipBoundaryFrames` in `frameCapture.ts`; call it in `initializeSession`
|
||||
right after the boundary computation; merge into the same Set used by the capture branches.
|
||||
Env off-switch: `HF_FAST_CAPTURE_INTERVAL_SS=false`.
|
||||
|
||||
### Step A4 — Validate
|
||||
1. **af468890 must flag f102–121** (the known flash tween). Add a debug log of the
|
||||
at-risk set size + ranges; render af468890 with the new predictor; confirm.
|
||||
2. **Re-run the 13 sparse comps** through hybrid+interval-predictor (use de-dmgdist.sh
|
||||
pattern, but with the predictor on instead of bypass) and confirm per-frame PSNR:
|
||||
the previously-damaged frames are now screenshotted (→ ∞ or healthy) and the
|
||||
damaged-frame fraction drops to ~0. Target: all 13 reach >45 dB mean (or 0% damaged frames).
|
||||
3. **Confirm speedup survives:** the at-risk set should be a small fraction of frames
|
||||
for the sparse comps (it was 0–8% damaged → at-risk should be similar small %). Log
|
||||
`atRiskFrames.size / totalFrames` per comp; should be <25% for the SPARSE set.
|
||||
4. **The 4 PERVASIVE comps** (9e3c36ba, 4fcc7660, 5697cb63, 38097e59): at-risk fraction
|
||||
will be high (>50%) → add a go/no-go: if at-risk fraction > ~40%, fall through to
|
||||
the whole-comp gate (full screenshot) — don't bother with hybrid.
|
||||
|
||||
### Step A5 — Ship
|
||||
Strip any debug logging/probes. Typecheck + oxlint + oxfmt. Commit on the stack
|
||||
(`gt create --no-verify -m "..."` stacked on `drawelement-perframe-fallback`).
|
||||
Update `fast-capture-limitations.md` (mark Lim 7 fixed, record the lift on the 13 comps).
|
||||
|
||||
---
|
||||
|
||||
## 6. TASK B — macOS static-frame dedup spike (separate lever)
|
||||
|
||||
**Premise (measured):** macOS `Page.captureScreenshot` is flat ~24 ms/frame; static
|
||||
holds re-capture identical frames for nothing. BeginFrame/Linux already dedups via
|
||||
`hasDamage` (returns `lastFrameCache`). Port the idea to macOS.
|
||||
|
||||
**Detection (deterministic, pre-capture):** a frame is static iff NO GSAP tween is
|
||||
active in `[t_prev, t_now]` AND no video AND no canvas/webgl present (those may redraw
|
||||
without a tween). Reuse the SAME `window.__timelines` introspection as Task A — a frame
|
||||
with no active tween + no dynamic media = identical to the previous frame → skip
|
||||
capture, reuse the previous buffer.
|
||||
|
||||
**Caveats (from prior probe, memory `project-fast-capture-perf-model`):** dedup is
|
||||
NARROW — ~89% of comps disqualified (78% have a canvas, 11% video); eligible comps
|
||||
often have continuous tweens (zero static frames). Best case ~31% frames skippable
|
||||
(long-holds presentation). **Under-detection = stale/frozen frame (correctness bug)** —
|
||||
bias conservative.
|
||||
|
||||
**First step:** DON'T build yet. Measure the TRUE static-frame fraction across the
|
||||
corpus via timeline introspection (NOT the JPEG-byte-size proxy, which overcounts).
|
||||
Reuse Task A's tween-interval machinery: static frames = total − union(all tween
|
||||
active intervals) − dynamic-media frames. Report the distribution; size the payoff;
|
||||
THEN decide.
|
||||
|
||||
---
|
||||
|
||||
## 7. Reference data
|
||||
|
||||
- **The 18 hybrid-damaged comps + classification:** `/tmp/dmgdist.jsonl`. Summary:
|
||||
- SPARSE/fixable (13): 7882f09c, 0670ee56, 4fa05434, b459dbe9, 100c5fad, 27388fb9, 2666ea5d, 6502a4ed, 8f942142, 7dec755f, af468890, 4cf3ed60, 38e9ca4c
|
||||
- MIXED (1): d0a3039a (35%)
|
||||
- PERVASIVE/gate-only (4): 9e3c36ba (55%), 4fcc7660 (77%), 5697cb63 (99%), 38097e59 (100%)
|
||||
- **Hybrid eval (all 42 fallback comps):** `/tmp/hybrid-eval.jsonl` (fields: n, bok, hok, inj, boundary, crash, refell, psnr).
|
||||
- **Best validated golden comp for Task A:** `af468890` — flash overlay crossfade,
|
||||
damaged frames f102–121 (and f290–300-ish, f470-480-ish; tween repeats per scene).
|
||||
Use it as the unit test for the predictor.
|
||||
- **Verse dashboard (live):** https://www.heygenverse.com/a/9e71f3dc-23c8-4260-b45f-8e9e65a34c1c
|
||||
(worker-encode + hybrid eval; fallback rows show hybrid PSNR + verdict tag). Data
|
||||
asset `aa298d41-bdff-4b87-950c-107870f536ce` (v4). To update: edit `/tmp/we-page.html`,
|
||||
upload new data via `hv_upload_asset` (content=...), update `DATA_URL`, `hv_update`
|
||||
the app (id `9e71f3dc-...`). Pass HTML content, not a URL.
|
||||
- **Memory:** `project-drawelement-clip-boundary-blackout` and
|
||||
`project-fast-capture-perf-model` (have the condensed findings + commit SHAs).
|
||||
|
||||
---
|
||||
|
||||
## 8. Definition of done for the next stream
|
||||
- Task A shipped: 13 sparse comps render healthy (>45 dB / ~0% damaged frames) on the
|
||||
fast path with the interval predictor, at-risk fraction logged, 4 pervasive comps
|
||||
correctly fall through to gate. Committed on the stack, docs updated.
|
||||
- Task B: static-frame fraction measured across corpus; go/no-go decision recorded.
|
||||
- Then (user's call): push the stack / `gt submit`.
|
||||
@@ -0,0 +1,313 @@
|
||||
# drawElement Fast Capture — Limitations & Findings Log
|
||||
|
||||
Canonical findings log for the `drawElementImage` fast-capture path. Referenced
|
||||
from `frameCapture.ts`, `drawElementService.ts`, and `compileStage.ts`. Append
|
||||
new findings here rather than burying them in commit messages.
|
||||
|
||||
See also: `fast-capture-architecture.md` (how the speedup + worker-encode work, the
|
||||
per-frame cost model, and why worker-encode is macOS-GPU-only) and the
|
||||
implementation spec `docs/superpowers/specs/2026-06-08-drawelement-alpha.md`.
|
||||
|
||||
---
|
||||
|
||||
## What it is
|
||||
|
||||
`canvas.drawElementImage(element, x, y)` (experimental Chrome
|
||||
`--enable-features=CanvasDrawElement`) reads a DOM subtree's paint records
|
||||
straight into a `<canvas layoutsubtree>`, bypassing the compositor. Used as a
|
||||
capture mode in `packages/engine` as a faster alternative to
|
||||
`Page.captureScreenshot`.
|
||||
|
||||
**Speedup is GPU- and density-bound, not universal:**
|
||||
- Only faster on a **hardware GPU (macOS)**. On SwiftShader (Docker/Linux
|
||||
software rasterizer) there is no GPU→CPU readback to skip, so drawElement is
|
||||
parity-or-slower — it is **not used there** (routes to BeginFrame/screenshot).
|
||||
- Speedup tracks **animation density**, not duration. p50 ≈ 1.0–1.4×; short
|
||||
dense comps reach ~2–3×; long comps with static holds approach 1.0× (nothing
|
||||
per-frame to save). A 691s dense comp still hit 1.42×.
|
||||
|
||||
**Per-frame cost model (macOS GPU, 1080p, ~16.3 ms/frame):** paint-event wait
|
||||
~6.2 ms (37%, vsync floor), JPEG encode ~7.4 ms (57%), readback ~2.15 ms, seek
|
||||
0.67 ms, drawElementImage itself ~0.4 ms. In-page `performance.now()`/`Date.now()`
|
||||
are clamped in headless Chrome — only node-side timing is reliable.
|
||||
|
||||
---
|
||||
|
||||
## Capture-mode gates (whole-comp routing at session init / compile)
|
||||
|
||||
If any gate fires, the **entire** render routes to the platform baseline
|
||||
(screenshot on macOS, BeginFrame on Linux). Gating is whole-composition, decided
|
||||
once — there is **no per-frame gating** today.
|
||||
|
||||
| # | Gate | Signal | Bypass env | crbug |
|
||||
|---|------|--------|-----------|-------|
|
||||
| 1 | SwiftShader | software-GL detected | — | none (by design) |
|
||||
| 2 | Video / caption-pattern | `<video>` present (compile-time `videoCount>0`; runtime `querySelector("video")`) | `HF_FAST_CAPTURE_VIDEO=true` | 521861819 |
|
||||
| 3 | CSS 3D context | `matrix3d`/`perspective`/`preserve-3d`; failed 3D projection init | `HF_FAST_CAPTURE_3D=true` | 522872457 |
|
||||
| 4 | CSS effects | `backdrop-filter`, `filter:blur()`, WebGL context (`__hf_accel_canvases`) | `HF_FAST_CAPTURE_CSSFX=true` | — |
|
||||
| 5 | Stacked-fade | `__hfFadeTargets`: ≥2 fade targets ≥50% viewport, overlapping | `HF_FAST_CAPTURE_CROSSFADE=true` | — |
|
||||
|
||||
`detectCssEffectRisk` / `detectStackedFadeRisk` / `initThreeDProjection` live in
|
||||
`threeDProjection.ts`. The autoAlpha rewrite (`HF_FAST_CAPTURE_AUTOALPHA`,
|
||||
default on) rewrites `opacity`→`autoAlpha` so the stacked-fade gate is armed.
|
||||
|
||||
### Gate refinement: `filter:drop-shadow` removed (2026-06-15, commit 0b610805d)
|
||||
Audit showed drop-shadow-only comps render fine through drawElement (53–58 dB)
|
||||
and no damaged comp was drop-shadow-only. The rule only over-gated healthy
|
||||
comps, so it was dropped. **Known residual over-gate:** `backdrop-filter` that is
|
||||
*inert* (over a solid background) still gates 2 healthy comps — a static
|
||||
computed-style check can't tell it's a no-op. Accepted as a safety cost.
|
||||
|
||||
---
|
||||
|
||||
## Damage classes (why the gates exist)
|
||||
|
||||
### Lim 2 — caption-pattern / stacked opacity (video is the proxy)
|
||||
Multi-group nested opacity tweens (the captions pattern) composite wrong under
|
||||
drawElement. `<video>` is used as a coarse proxy gate (root-caused 2026-06; it
|
||||
is *not* animated promoted layers — it's the caption-pattern opacity bug). Pure
|
||||
video comps composite fine via `drawImage(video)+drawElement` (58.8 dB); the
|
||||
20 dB gap once seen was untagged media.
|
||||
|
||||
### Lim 3 — CSS 3D rendering contexts
|
||||
drawElement paints 3D contexts incorrectly (mirrored backface even at rest;
|
||||
siblings of the 3D context can drop out). Static-content 3D *can* render at
|
||||
57.5 dB under `HF_FAST_CAPTURE_3D=true` projection, but animated 3D subtrees
|
||||
fall back. 3D contexts are otherwise unpaintable.
|
||||
|
||||
### Lim 4 — backdrop-filter / filter:blur / WebGL
|
||||
`backdrop-filter` samples the compositor backdrop, which single-element capture
|
||||
can't reproduce. `filter:blur()` has paint-record vs compositor inconsistency.
|
||||
WebGL/WebGPU canvases freeze at frame 0 under seek capture (no rAF) — handled
|
||||
by a `getContext` instrumentation + `drawImage` composite where the canvas is
|
||||
static; animated GLSL still falls back.
|
||||
|
||||
### Lim 6 — clip-cut boundary blackout (NEW 2026-06-15, NOT yet gated)
|
||||
At a hard clip cut (`data-start`/`data-duration`, no `data-transition`),
|
||||
drawElement renders the boundary frame **black**: it drops the outgoing clip one
|
||||
frame before the screenshot does, and the incoming clip's paint record isn't
|
||||
ready yet. Mid-clip frames are **pixel-identical (∞)**; only the ~1-frame cut
|
||||
boundary fails (catastrophic, ~3 dB).
|
||||
|
||||
- **Not predictable from computed styles** — a solid-color clip has no
|
||||
backdrop/blend/filter signal; boundary-black frames are statistically
|
||||
identical to perfect frames by CSS (proven in the predictor spike below).
|
||||
- **IS predictable from the clip schedule** — boundary frames =
|
||||
`⋃ clips { round(start·fps), round((start+duration)·fps) }`, computable before
|
||||
render. This is the actionable signal for a hybrid.
|
||||
- **Confirmed in production (2026-06-15).** A clean fast-render of `7dec755f`
|
||||
(worker-encode on, **no probe**) outputs a solid-black frame at f240 (t=8.0,
|
||||
the yellow-clip cut). So this is real shipped damage, not a probe artifact.
|
||||
- It surfaced inside a backdrop-gated comp where the backdrop-filter is on a tiny
|
||||
element and the flash is an unrelated full-bleed yellow scene clip — i.e. the
|
||||
gate caught the comp for the wrong reason. A clip-cut comp with **no** gated
|
||||
effect would ship these black flashes on the fast path.
|
||||
- **Two sub-classes, two fixes.** The boundary condition either (a) *throws*
|
||||
`No cached paint record` (handled — see below) or (b) *succeeds silently with
|
||||
a black frame* (NOT caught by try/catch; needs the clip-schedule predictor to
|
||||
screenshot the boundary frame proactively). Still open for (b).
|
||||
|
||||
Same family as the **`No cached paint record` crash** (`drawElementService.ts`,
|
||||
seen on 11 comps in the 400-comp eval): an element with no cached paint record
|
||||
at frame N (display toggled / detached mid-timeline). drawElement *throws*
|
||||
`InvalidStateError: ... No cached paint record for element` and aborts the whole
|
||||
render. **FIXED (2026-06-15):** `frameCapture.ts` now catches this per frame (via
|
||||
`isNoCachedPaintRecordError`) in both the serial (`captureFrameCore`) and
|
||||
worker-encode (`captureFrameToBufferPipelined`) paths and falls back to
|
||||
`captureScreenshot` for that single frame instead of aborting. Validated on
|
||||
`29c3224f`: was a hard abort, now `RENDER_OK` with 11 frames screenshot-fallen-back.
|
||||
This fixes the *throw* sub-class; the *silent black* sub-class (Lim 6b) is still
|
||||
open.
|
||||
|
||||
---
|
||||
|
||||
## Fixes shipped (regressions closed)
|
||||
|
||||
- **body/html background loss** (223c4752): ancestor-bg fill; 6 comps 23–31→53–71 dB.
|
||||
- **WebGL/WebGPU canvas freeze** (ec037e2a): getContext instrumentation + drawImage composite.
|
||||
- **forceScreenshot honored** (bf8922cb): drawElement ignored compat hints; alpha keeps fast path.
|
||||
- **style-N BeginFrame stall** (b2612bfc, 5d53f13c): compile-time video gate + liveness probe + autoAlpha rewrite.
|
||||
- **autoAlpha retraction** (606b033aa, 612ec5a0e): retract rewrite flag when a runtime gate routes to fallback.
|
||||
- **3D static projection** (`HF_FAST_CAPTURE_3D=true`): static-content 3D at 57.5 dB.
|
||||
|
||||
---
|
||||
|
||||
## Worker-encode (PR #1444, base #1295)
|
||||
|
||||
Off-main-thread JPEG encode via in-page OffscreenCanvas Worker; depth-2 pipeline
|
||||
overlaps encode(N-1) with produce(N). macOS-GPU-only (`HF_DE_WORKER_ENCODE`,
|
||||
default off). **Output is bit-identical to serial drawElement** (worker-vs-serial
|
||||
PSNR=∞ proven) — zero quality cost. ~2× vs screenshot baseline. Any sub-50 dB
|
||||
rows are inherited drawElement-path damage, not introduced by worker-encode.
|
||||
|
||||
---
|
||||
|
||||
## Open direction — hybrid (per-frame instead of whole-comp gating)
|
||||
|
||||
> **Resolved below.** A per-frame computed-style *threshold* predictor does NOT
|
||||
> work (see Hybrid eval + "dependable pre-render calculation"). The viable design
|
||||
> is **GSAP timeline-interval introspection** + clip schedule + static-effect scan.
|
||||
> The two sub-problems below are the original framing, kept for context.
|
||||
|
||||
Goal: render most frames with drawElement, screenshot only incompatible frames.
|
||||
Two distinct sub-problems, different predictability:
|
||||
|
||||
1. **Structural families (3d / webgl / blur / backdrop):** damage is moderate and
|
||||
*may* track a per-frame style signal (the gate's own detector run per frame).
|
||||
Spike data: 3d/stacked show moderate damage (PSNR 30–40 dB) but **no
|
||||
catastrophic black-outs**.
|
||||
2. **Clip-cut boundaries (Lim 6):** catastrophic, **not** style-predictable,
|
||||
**schedule-predictable**. Screenshot the ±1 frame around each clip cut.
|
||||
|
||||
### Spike results (10 comps, 766 sampled frames, `EVERY=15`, gates bypassed)
|
||||
|
||||
Per-frame **computed-style predictor** (backdrop/blur/filter/3d/blend/mask/
|
||||
stacked-fade) vs ground-truth PSNR(drawElement, screenshot):
|
||||
|
||||
| family | frames | comps | dmg<40dB | recall (signal\|dmg) | cat<12dB | cat & signal | precision (dmg\|signal) |
|
||||
|--------|-------|-------|---------|------|---------|-----------|------|
|
||||
| 3d | 151 | 3 | 28 | 32% | 0 | 0 | 33% |
|
||||
| stacked | 90 | 2 | 24 | 71% | 0 | 0 | 35% |
|
||||
| backdrop | 120 | 2 | 43 | 21% | 3 | **0** | 26% |
|
||||
| blur | 186 | 2 | 36 | 47% | 1 | **0** | 13% |
|
||||
| webgl | 8 | 1 | 1 | (sample too small) | 0 | 0 | — |
|
||||
|
||||
**Boundary-targeted spike** (4 comps, 43 frames, probe fires on the clip-schedule
|
||||
boundary set ±1 in addition to the every-N grid):
|
||||
|
||||
| | frames | ok | dmg<40dB | BLACK<12dB |
|
||||
|--|-------|-----|---------|-----------|
|
||||
| clip-boundary | 27 | 16 | 8 | **3** |
|
||||
| mid-clip | 16 | 13 | 3 | **0** |
|
||||
|
||||
**All catastrophic black frames are clip-boundary frames (3/3, none mid-clip).**
|
||||
The clip-schedule predictor (`⋃ clips {round(start·fps), round((start+dur)·fps)}`)
|
||||
catches 100% of black-outs. Only ~11% of boundaries actually black out, so
|
||||
screenshotting every boundary over-shoots ~9×, but boundaries are sparse vs total
|
||||
frames → cheap. **This is the actionable Lim 6b hybrid: screenshot clip-boundary
|
||||
frames, drawElement everything else.**
|
||||
|
||||
**Verdict: the static style predictor is not viable.** Recall < 50% on 3 of 5
|
||||
families (it misses most damaged frames) and precision 13–35% (it over-flags).
|
||||
Critically, **every catastrophic frame (`cat<12dB`) has zero style signal**
|
||||
(`cat & signal = 0`) — the clip-cut blackouts (Lim 6) are completely invisible
|
||||
to computed-style inspection. The only reliable pre-render signal found is the
|
||||
**clip schedule** (Lim 6), which predicts boundary frames deterministically.
|
||||
Sampling caveat: `EVERY=15` undersamples 1-frame boundary events, so true
|
||||
catastrophic counts are undercounts — sample boundary frames directly.
|
||||
|
||||
### Predictor spike harness (this branch, scratch — not committed)
|
||||
- `packages/engine/src/services/frameCapture.ts` — env-gated probe in
|
||||
`prepareFrameForCapture` (`HF_PREDICT_PROBE=1`, `HF_PREDICT_OUT`,
|
||||
`HF_PREDICT_EVERY`): per sampled frame, captures screenshot-ref + drawElement-test
|
||||
(PNG) and a computed-style predictor (backdrop/blur/filter/3d/blend/mask/
|
||||
stacked-fade/largest-element). **Revert before merge.**
|
||||
- `packages/producer/de-predict-run.sh` — forces drawElement (all gates bypassed)
|
||||
across sample comps per family + joins.
|
||||
- `packages/producer/de-predict-psnr.mjs` — ffmpeg per-frame PSNR + signal join → `/tmp/predict/joined.jsonl`.
|
||||
|
||||
**Key spike result:** a static computed-style predictor is **insufficient** for
|
||||
the dominant failure (clip-cut blackout). `EVERY=N` sampling also undersamples
|
||||
1-frame boundary events — measure boundary frames directly from the clip schedule.
|
||||
|
||||
---
|
||||
|
||||
## Hybrid eval — every fallback comp through the hybrid path (2026-06-15)
|
||||
|
||||
Re-ran all 42 fallback rows from the worker-encode page through drawElement with
|
||||
all gates bypassed + the per-frame fallback active (boundary + crash), full-frame
|
||||
PSNR vs screenshot baseline. Harness: `packages/producer/de-hybrid.sh` (scratch).
|
||||
|
||||
| outcome | n | meaning |
|
||||
|---------|---|---------|
|
||||
| HYBRID-OK ≥45 dB | 3 | clip-cut comps the boundary fix genuinely rescues (`c44fbdaf` 89 dB) |
|
||||
| marginal 40–45 | 4 | clip comps, close |
|
||||
| **still-damaged <40** | 18 | continuous compositor damage — hybrid doesn't help |
|
||||
| no-engage (forced ss) | 17 | raw-rAF / nondeterministic → drawElement never engages (correct) |
|
||||
|
||||
**Conclusions:**
|
||||
1. **17/42 can't drawElement at all** and are *already* correctly force-screenshotted:
|
||||
they use `Date.now`/`performance.now`/`Math.random`/raw-rAF, so render-mode
|
||||
hints route them to screenshot (the deterministic-render rule). These are NOT
|
||||
hybrid candidates and must be excluded from any predictor's recall accounting —
|
||||
their "damage" is the comp's own nondeterminism (verified: `e25b918a` differs
|
||||
35 dB *screenshot-vs-screenshot*).
|
||||
2. **The boundary/crash fix is real but narrow** — it cleanly rescues clip-cut
|
||||
comps (`c44fbdaf` 89 dB). It does **not** rescue style-gated comps: their
|
||||
damage is **continuous** (the effect is on-screen for most of the timeline), so
|
||||
screenshotting boundaries leaves the bulk damaged. The whole-comp gates are
|
||||
mostly justified.
|
||||
3. **Mean PSNR is a misleading quality metric near black.** `af468890` scored
|
||||
25 dB but is ∞ on ~95% of frames; the mean is tanked by ~14-frame
|
||||
**dip-to-black crossfade** ramps where PSNR is mathematically brutal. BUT SSIM
|
||||
confirmed the crossfade frames are *genuinely* wrong (SSIM 0.48), not just a
|
||||
metric artifact — see below. Use SSIM (or luma-weighted) for transition frames.
|
||||
|
||||
### Lim 7 — drawElement drops opacity-animated semi-transparent overlays
|
||||
`af468890` root cause: a full-screen white flash overlay
|
||||
(`<div class="flash" style="background:#FFFFFF;opacity:0">`) with GSAP animating
|
||||
`opacity: 0→1→0` over a transition. drawElement **drops the overlay** — renders the
|
||||
dark scene underneath (navy) where the baseline shows white-over-dark (gray).
|
||||
Plainly visible. The damaged interval is **exactly the tween span** (flash up
|
||||
3.40–3.60 s = f102–108, down 3.65–4.05 s = f110–121), i.e. the same
|
||||
group-opacity/stacked-fade compositing class as Lim 2 — drawElement composites a
|
||||
semi-transparent layer differently from the compositor.
|
||||
|
||||
## The dependable pre-render calculation (design)
|
||||
|
||||
A per-frame computed-style *threshold* predictor failed (low recall, missed the
|
||||
opacity peak, can't track a multi-frame ramp). The correct deterministic signal is
|
||||
the **GSAP timeline interval**, not a per-frame guess or a fuzzy window:
|
||||
|
||||
> at-risk frames = ⋃ over every GSAP tween animating a **compositor-incompatible
|
||||
> property** (opacity, filter, backdrop-filter, mix-blend-mode, 3D transform,
|
||||
> mask/clip-path) on a visible element → its full `[startFrame, endFrame]`
|
||||
> ∪ clip-cut boundary frames (schedule)
|
||||
> ∪ statically-present compositor-effect frames (element visible throughout)
|
||||
|
||||
`window.__timelines` exposes every tween's target/properties/start/duration, so the
|
||||
at-risk set is computed **once at init** by introspecting timeline + DOM — fully
|
||||
deterministic, no per-frame screenshot, no windowing. Screenshot the at-risk
|
||||
intervals, drawElement the rest; if at-risk dominates, fall through to the gate.
|
||||
|
||||
- **Deterministic by construction** given the (fixed, small) list of
|
||||
compositor-incompatible properties: the timeline gives exact intervals, the DOM
|
||||
gives exact static presence → 100% recall **for enumerated effects**.
|
||||
- **Residual risk = property-list completeness.** An un-enumerated compositor
|
||||
effect still slips. The list is bounded (Chrome compositor features); each corpus
|
||||
miss adds one entry. This is the only part not provable by construction — a
|
||||
sparse measurement backstop (calibration PSNR/SSIM) would close it if a hard
|
||||
guarantee is required.
|
||||
|
||||
**Status:** designed, not built. Next step: prototype the `__timelines`
|
||||
introspection → frame-interval calculator and validate it flags `af468890`'s
|
||||
f102–121 exactly.
|
||||
|
||||
### Damage distribution of the 18 hybrid-damaged comps (2026-06-15, `de-dmgdist.sh`)
|
||||
Per-frame PSNR (current hybrid vs screenshot baseline) → damaged-frame fraction.
|
||||
**Mean PSNR badly mislead** — most "damaged" comps are near-perfect with sparse,
|
||||
clustered damage:
|
||||
|
||||
| class | n | criterion | comps |
|
||||
|-------|---|-----------|-------|
|
||||
| **SPARSE** (fixable by interval predictor) | **13** | <25% frames damaged, clustered | 7882f09c·0670ee56 (0%), 4fa05434·b459dbe9·100c5fad·27388fb9·2666ea5d·6502a4ed (1%), 8f942142·7dec755f (2%), af468890 (4%), 4cf3ed60 (5%), 38e9ca4c (8%) |
|
||||
| MIXED (partial win) | 1 | 25–50% | d0a3039a (35%) |
|
||||
| **PERVASIVE** (gate-only, correct) | **4** | >50% | 9e3c36ba (55%), 4fcc7660 (77%), 5697cb63 (99%), 38097e59 (100%) |
|
||||
|
||||
Takeaway: the gates' *mean*-PSNR view overstated damage. Backdrop comps written
|
||||
off as continuous (100c5fad/27388fb9/7dec755f, 26 dB mean) are only **1–2% damaged**
|
||||
— a few near-black/transition frames tanked the mean. **The timeline-interval
|
||||
predictor would convert 13 (+1 partial) of the 18 to healthy while keeping 92–100%
|
||||
of frames on the fast path**; only 4 (effect on-screen most/all the time) stay
|
||||
gated. This is the payoff sizing for building the predictor. (Damaged frames are
|
||||
inferred to lie in tween/boundary intervals from their clustering; verified
|
||||
directly only for af468890 — the predictor prototype confirms the rest.)
|
||||
|
||||
### Scratch
|
||||
The engine-side investigation probes (`HF_PREDICT_PROBE`, `HF_INV_PROBE`,
|
||||
`HF_SCRTIME`, `HF_FAST_CAPTURE_OPACITY_SS`, `runInventoryProbe`,
|
||||
`frameHasOpacityRisk`, predictor fns) have been **stripped** from `frameCapture.ts`
|
||||
— it now holds only the shipped 6a/6b fix. The findings they produced are recorded
|
||||
in this doc. Producer-side scratch harnesses remain **untracked** (not committed),
|
||||
reusable for the planned spikes: `de-hybrid.sh`, `de-predict-run.sh`,
|
||||
`de-predict-psnr.mjs`.
|
||||
@@ -108,6 +108,15 @@ export interface CaptureSession {
|
||||
* is true and capture mode resolved to "drawelement".
|
||||
*/
|
||||
workerEncodeEnabled?: boolean;
|
||||
/**
|
||||
* Clip-cut boundary frame indices (±1) computed from the clip schedule at
|
||||
* init. drawElement blacks out hard clip-cut boundary frames (the outgoing
|
||||
* clip is dropped a frame before the incoming clip's paint record is ready);
|
||||
* these frames are captured via screenshot instead. See
|
||||
* docs/fast-capture-limitations.md Lim 6. Empty/undefined disables the
|
||||
* fallback (`HF_FAST_CAPTURE_BOUNDARY_SS=false`).
|
||||
*/
|
||||
clipBoundaryFrames?: Set<number>;
|
||||
}
|
||||
|
||||
// Circular buffer for browser console messages dumped on render failure diagnostics.
|
||||
@@ -494,6 +503,16 @@ async function initDrawElementOrTransparentBackground(
|
||||
session.captureMode = "drawelement";
|
||||
session.drawElementReady = true;
|
||||
logInitPhase("drawElement canvas injected");
|
||||
// Lim 6: drawElement blacks out hard clip-cut boundary frames (outgoing clip
|
||||
// dropped a frame before the incoming clip's paint record is ready). Capture
|
||||
// those frames via screenshot instead. Computed once from the clip schedule.
|
||||
if (process.env.HF_FAST_CAPTURE_BOUNDARY_SS !== "false") {
|
||||
const boundaries = await computeClipBoundaryFrames(page, fpsToNumber(session.options.fps));
|
||||
if (boundaries.size > 0) {
|
||||
session.clipBoundaryFrames = boundaries;
|
||||
logInitPhase(`clip-boundary screenshot fallback: ${boundaries.size} frame(s)`);
|
||||
}
|
||||
}
|
||||
// Worker-encode pipeline: macOS hardware GPU path only (syncToPaintEvent=true,
|
||||
// beginFrameTimeTicks=0). Skip for BeginFrame (Linux/Docker) and transparent
|
||||
// (PNG) output — those use the existing synchronous path unchanged.
|
||||
@@ -1548,6 +1567,44 @@ async function prepareFrameForCapture(
|
||||
* Shared by captureFrame (disk) and captureFrameToBuffer (buffer).
|
||||
* Returns the screenshot buffer, quantized time, and total capture time.
|
||||
*/
|
||||
/**
|
||||
* Clip-cut boundary frame indices (±1) from the clip schedule. A clip is visible
|
||||
* over `[start, start+duration)`; its in- and out-edges are the frames where
|
||||
* drawElement can black out (Lim 6). Returns those frames and their immediate
|
||||
* neighbours (the desync is ±1 frame).
|
||||
*/
|
||||
async function computeClipBoundaryFrames(page: Page, fps: number): Promise<Set<number>> {
|
||||
const schedule = await page.evaluate(() =>
|
||||
Array.from(document.querySelectorAll("[data-start]")).map((el) => ({
|
||||
start: parseFloat((el as HTMLElement).dataset.start || ""),
|
||||
dur: parseFloat((el as HTMLElement).dataset.duration || ""),
|
||||
})),
|
||||
);
|
||||
const frames = new Set<number>();
|
||||
for (const { start, dur } of schedule) {
|
||||
if (Number.isNaN(start)) continue;
|
||||
const edges = [Math.round(start * fps)];
|
||||
if (!Number.isNaN(dur)) edges.push(Math.round((start + dur) * fps));
|
||||
for (const e of edges) {
|
||||
for (const f of [e - 1, e, e + 1]) {
|
||||
if (f >= 0) frames.add(f);
|
||||
}
|
||||
}
|
||||
}
|
||||
return frames;
|
||||
}
|
||||
|
||||
/**
|
||||
* True for the drawElement `InvalidStateError: No cached paint record for element`
|
||||
* thrown when a subtree element has no paint record for the current frame (display
|
||||
* toggled / detached / freshly-shown at a clip-cut boundary). Per-frame, not
|
||||
* whole-comp — callers fall back to screenshot for the single frame.
|
||||
*/
|
||||
export function isNoCachedPaintRecordError(err: unknown): boolean {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
return msg.includes("No cached paint record");
|
||||
}
|
||||
|
||||
async function captureFrameCore(
|
||||
session: CaptureSession,
|
||||
frameIndex: number,
|
||||
@@ -1578,6 +1635,13 @@ async function captureFrameCore(
|
||||
if (result.hasDamage) session.beginFrameHasDamageCount++;
|
||||
else session.beginFrameNoDamageCount++;
|
||||
screenshotBuffer = result.buffer;
|
||||
} else if (
|
||||
session.captureMode === "drawelement" &&
|
||||
session.clipBoundaryFrames?.has(frameIndex)
|
||||
) {
|
||||
// Lim 6: clip-cut boundary frame — drawElement may render it black. Capture
|
||||
// via screenshot instead. See docs/fast-capture-limitations.md.
|
||||
screenshotBuffer = await pageScreenshotCapture(page, options);
|
||||
} else if (session.captureMode === "drawelement") {
|
||||
// Advance compositor state via BeginFrame when available (Linux headless-shell);
|
||||
// on macOS the compositor advances naturally without BeginFrame.
|
||||
@@ -1590,17 +1654,34 @@ async function captureFrameCore(
|
||||
// no screenshot param — we capture via canvas
|
||||
});
|
||||
}
|
||||
screenshotBuffer = await captureDrawElementFrame(
|
||||
page,
|
||||
options.width,
|
||||
options.height,
|
||||
options.format ?? "jpeg",
|
||||
options.quality ?? 80,
|
||||
// Paint-event sync only without BeginFrame (macOS / screenshot-launched):
|
||||
// under BeginFrame control the per-frame beginFrame above already painted
|
||||
// a fresh snapshot, and no further paint would arrive during a wait.
|
||||
session.beginFrameTimeTicks === 0,
|
||||
);
|
||||
try {
|
||||
screenshotBuffer = await captureDrawElementFrame(
|
||||
page,
|
||||
options.width,
|
||||
options.height,
|
||||
options.format ?? "jpeg",
|
||||
options.quality ?? 80,
|
||||
// Paint-event sync only without BeginFrame (macOS / screenshot-launched):
|
||||
// under BeginFrame control the per-frame beginFrame above already painted
|
||||
// a fresh snapshot, and no further paint would arrive during a wait.
|
||||
session.beginFrameTimeTicks === 0,
|
||||
);
|
||||
} catch (err) {
|
||||
// drawElementImage throws `InvalidStateError: No cached paint record for
|
||||
// element` when an element in the subtree has no paint record this frame
|
||||
// (display toggled / detached / freshly-shown at a clip-cut boundary). This
|
||||
// is a per-frame condition, not a whole-comp one — fall back to screenshot
|
||||
// for THIS frame instead of aborting the render. See fast-capture-limitations.md.
|
||||
if (isNoCachedPaintRecordError(err)) {
|
||||
console.log(
|
||||
`[engine] fast capture: frame ${frameIndex} — No cached paint record; ` +
|
||||
`screenshot fallback for this frame (see fast-capture-limitations.md)`,
|
||||
);
|
||||
screenshotBuffer = await pageScreenshotCapture(page, options);
|
||||
} else {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
screenshotBuffer = await pageScreenshotCapture(page, options);
|
||||
}
|
||||
@@ -1693,6 +1774,18 @@ export async function captureFrameToBufferPipelined(
|
||||
);
|
||||
void quantizedTime;
|
||||
|
||||
// Lim 6: clip-cut boundary frame — drawElement may render it black. Capture
|
||||
// via screenshot and return a resolved encodeResult so the pipeline writes it
|
||||
// like any other frame. See docs/fast-capture-limitations.md.
|
||||
if (session.clipBoundaryFrames?.has(frameIndex)) {
|
||||
const buffer = await pageScreenshotCapture(page, options);
|
||||
session.capturePerf.frames += 1;
|
||||
session.capturePerf.seekMs += seekMs;
|
||||
session.capturePerf.beforeCaptureMs += beforeCaptureMs;
|
||||
session.capturePerf.totalMs += Date.now() - startTime;
|
||||
return { encodeResult: Promise.resolve(buffer), captureTimeMs: Date.now() - startTime };
|
||||
}
|
||||
|
||||
// Worker-encode is gated to the macOS GPU path (beginFrameTimeTicks === 0,
|
||||
// syncToPaintEvent = true); see initDrawElementOrTransparentBackground. The
|
||||
// BeginFrame branch present in the synchronous captureFrameCore is therefore
|
||||
@@ -1716,6 +1809,18 @@ export async function captureFrameToBufferPipelined(
|
||||
|
||||
return { encodeResult, captureTimeMs };
|
||||
} catch (captureError) {
|
||||
// Per-frame `No cached paint record`: fall back to screenshot for THIS frame
|
||||
// instead of aborting the render (clip-cut boundary / freshly-shown element).
|
||||
// The worker isn't involved for this frame; return a resolved encodeResult so
|
||||
// the pipeline loop writes it like any other. See fast-capture-limitations.md.
|
||||
if (isNoCachedPaintRecordError(captureError)) {
|
||||
console.log(
|
||||
`[engine] fast capture: frame ${frameIndex} — No cached paint record; ` +
|
||||
`screenshot fallback for this frame (see fast-capture-limitations.md)`,
|
||||
);
|
||||
const buffer = await pageScreenshotCapture(page, options);
|
||||
return { encodeResult: Promise.resolve(buffer), captureTimeMs: Date.now() - startTime };
|
||||
}
|
||||
// Mirror captureFrameCore: capture per-frame diagnostics (frame-error
|
||||
// PNG/HTML/JSON + console tail) before rethrowing so pipelined-path
|
||||
// failures are debuggable like the serial path.
|
||||
|
||||
Reference in New Issue
Block a user