From 9769ba2c7bcd75e938ca5267b6568b457cb98fae Mon Sep 17 00:00:00 2001 From: Hblee Date: Thu, 6 Aug 2026 16:27:38 -0700 Subject: [PATCH] feat(sdk): expose a paint query for transparent-composition hit-testing (#3070) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(sdk): expose a paint query for transparent-composition hit-testing A host layering a transparent composition over other content has to know whether a point carries ink before it decides to swallow a click. The adapter only answered "what element is here", so AI Studio wrote its own answer and could not reach the per-pixel alpha the adapter already samples for . Add PreviewAdapter.paintsAt, plus the pieces it is built from on a new ./adapters/iframe subpath so a host with different hit-test policy can compose its own walk. The walk is geometric rather than elementsFromPoint-based: that stack omits pointer-events:none nodes, and a decorative overlay carrying it still paints, so a z-stack query would report no ink over visible artwork — the direction that makes a composition vanish from under the cursor. fullBleedFraction is an option rather than a constant because "a layer covering the whole frame is background, not artwork" is host policy, not a fact about the composition. * fix(sdk): scope the full-bleed frame to the root under the point compositionFrameArea took the smallest [data-composition-id] in the whole document, so an unrelated sub-composition sized the reference frame for points nowhere near it: a 300x300 badge in a corner made every mid-size painter in a 1920x1080 outer frame read as full-bleed, and the composition went click-through under artwork the user can plainly see. That is the direction the fail-safe exists to avoid, and the docs already described the intended behaviour — the innermost root CONTAINING the point. Also read the alpha channel instead of matching known transparent spellings. Only the `transparent` keyword computes to rgba(0, 0, 0, 0); a faded-out white stays rgba(255, 255, 255, 0), which the set counted as painted. That erred toward absorbing clicks rather than losing them, so it was a false positive rather than a hazard, but it is wrong. Both are pinned by tests that fail when the fix is reverted. * fix(sdk): stop the paint query answering "no ink" over visible artwork Four cases where the walk landed on the wrong side of its own fail-safe. The full-bleed veto tested the winner's border-box area even when the alpha sampler had just read an opaque pixel there, so a full-frame transparent PNG or SVG overlay — the case this feature exists for — reported background over visibly opaque artwork, with no fullBleedFraction that worked. Ink now carries how it was established, and a measured pixel is never vetoed. An image whose pixels could NOT be read stays inferred, so a tainted CDN overlay still yields to the veto rather than absorbing every click. Composition roots were excluded from candidacy outright, so a root carrying a background answered false even at fraction 0, where the docs promise every painting box counts. Roots are candidates now; the veto discounts them without a special case, since a root's box is the frame. An with a clear pixel early-returned past its own background, padding plate and border, which any other element would have counted. A same-origin iframe mid-navigation exposes a readable but empty document, so the !doc guard never fired and a loading composition answered a confident "no ink" — the exact failure the null convention exists to prevent. Also: the sort comparator's epsilon tie was intransitive, leaving the smallest-first guarantee (and the lazy single-sample property that rides on it) engine-dependent; the guide's pass-through recipe called a function that does not exist and hand-waved the coordinate mapping that makes it correct; and the reference now states the under-counts alongside the over-counts, the walk's blindness to runtime-mounted content, and compositionPaintsAt's preconditions. Each fix is pinned by a test that fails when the fix is reverted. * refactor(sdk)!: invert the paint query to isProvablyEmptyAt paintsAt handed callers three falsy bottom values with opposite safe readings: false meant "no ink, pass the click through", null meant "not knowable, treat as painted", and undefined from an adapter without the method also meant painted. The idiomatic `if (!preview.paintsAt?.(x, y)) passThrough()` therefore did the dangerous thing for two of the three, and the convention needed defending in the interface docstring, the reference and the guide — plus a dedicated comment and a pinning test on the headless adapter to stop null regressing to false. Inverting the polarity collapses the tri-state to a plain boolean and makes the safe reading structural: true only when the composition was readable and nothing painted there, so ink, an unreadable or still-loading document, and a missing implementation all land on "keep the composition clickable". The prose stays as rationale, but nothing depends on a reader remembering it. PaintsAtOptions becomes PaintQueryOptions, since it now describes the walk that both the adapter method and the exported compositionPaintsAt share rather than one method's arguments. compositionPaintsAt keeps its ink-positive name: it answers the other question, and its docstring points callers who need the fail-safe contract at the adapter. Nothing is released yet, so no consumer is on the old name. --- .fallowrc.jsonc | 6 + docs/sdk/guides/canvas-integration.mdx | 59 ++ docs/sdk/reference/adapters.mdx | 58 +- packages/sdk/package-subpaths.json | 6 + packages/sdk/package.json | 9 + packages/sdk/src/adapters/headless.test.ts | 12 + packages/sdk/src/adapters/headless.ts | 16 +- packages/sdk/src/adapters/iframe.test.ts | 601 +++++++++++++++++++++ packages/sdk/src/adapters/iframe.ts | 348 +++++++++++- packages/sdk/src/adapters/types.ts | 45 ++ packages/sdk/src/index.ts | 7 +- 11 files changed, 1159 insertions(+), 8 deletions(-) create mode 100644 packages/sdk/src/adapters/headless.test.ts diff --git a/.fallowrc.jsonc b/.fallowrc.jsonc index fd1cc03ff..0fcb2140f 100644 --- a/.fallowrc.jsonc +++ b/.fallowrc.jsonc @@ -544,6 +544,12 @@ // this PR only teaches the server scan to prefer its reported PID, but that // line shift makes fallow re-flag the inherited probe clones. "packages/cli/src/server/portUtils.ts", + // iframe.test.ts: the remaining clone groups are pre-existing per-case arrange + // blocks in the selection and draft-loop suites (build an adapter, wire a spy, + // act). Appending the paint-query suite shifts their line numbers and re-flags + // them; each block states its own setup on purpose, which a shared fixture + // would hide. + "packages/sdk/src/adapters/iframe.test.ts", // gsapParserAcorn.motionEval.test.ts: parallel arrange/act/assert cases for // the staggered-collection honesty pass (.from reveal vs .to landing on the // rest pose). Each asserts a distinct keyframe shape; collapsing the shared diff --git a/docs/sdk/guides/canvas-integration.mdx b/docs/sdk/guides/canvas-integration.mdx index 80b17a983..87fbb3792 100644 --- a/docs/sdk/guides/canvas-integration.mdx +++ b/docs/sdk/guides/canvas-integration.mdx @@ -99,6 +99,65 @@ iframeDoc.addEventListener("click", (e) => { `resolveNearestHfElement` returns `null` when the walk exits the tree without finding a `[data-hf-id]` node, when the matching node carries `[data-hf-root]` (the root is transparent to selection), or when `isVisible` returns `false` for that node. +## Transparent compositions over other content + +A composition authored as an overlay — a small graphic on an otherwise-empty 1080×1920 frame, layered over a video or an avatar — is still a rectangular DOM box covering every pixel of the frame. Without help it swallows every click, and whatever sits beneath it becomes unreachable. + +`preview.isProvablyEmptyAt(x, y)` is the question you need answered: is this point provably free of ink, so a click may safely reach what sits beneath? Toggle `pointer-events` on your wrapper from the answer, and let the browser deliver the event to the right target: + +```typescript +const wrapper = document.querySelector("#composition-wrapper")!; + +/** + * Host-page pointer coordinates → the iframe document's own client space, which is what + * the paint query samples against. The iframe renders at the composition's native size and is + * CSS-scaled to fit, so the on-screen scale has to be divided out — skip this and you + * sample the wrong pixel, and pass-through toggles over the wrong regions. + */ +function toCompositionPoint(clientX: number, clientY: number) { + const rect = iframe.getBoundingClientRect(); + if (!rect.width || !rect.height) return null; + const scaleX = rect.width / compositionNativeWidth; + const scaleY = rect.height / compositionNativeHeight; + if (!scaleX || !scaleY) return null; + return { x: (clientX - rect.left) / scaleX, y: (clientY - rect.top) / scaleY }; +} + +function updatePassThrough(clientX: number, clientY: number, altKey: boolean) { + const point = toCompositionPoint(clientX, clientY); + // Alt is the escape hatch for grabbing the composition itself in an empty region. + // Every uncertain case — no point, still loading, adapter without the method — is + // falsy here, so the composition stays clickable rather than vanishing. + const passThrough = + !!point && !altKey && !!preview.isProvablyEmptyAt?.(point.x, point.y); + wrapper.style.pointerEvents = passThrough ? "none" : ""; +} +``` + + + Listen on the **host document**, not on the wrapper or the iframe. The first time this + sets `pointer-events: none` the wrapper stops receiving events, so a listener attached + there can never turn it back on — the pass-through state sticks. + + +Three things are easy to get wrong here: + + + + The browser picks an event's target before any handler runs, so flipping `pointer-events` inside `mousedown` cannot retarget the click already in flight. Sample the pointer position on `mousemove` and keep the decision current. + + + Animated artwork moves under a stationary cursor. Anything that can change the answer — pointer movement, the Alt key, and the playhead — has to re-run the query from the last known position. Coalesce those triggers into one `requestAnimationFrame` query rather than answering each separately, and short-circuit before the query when the pointer is outside the composition's box: it is a walk over the document, so it does not belong on an ungated per-event path. + + + `isProvablyEmptyAt` is true only when it has established there is no ink. A document that hasn't loaded, an adapter without the method, and a point you couldn't map all come back falsy — which keeps the composition clickable. Don't invert it into a "does it paint" variable; that reintroduces the bug the polarity removes. + + + +Pass `{ fullBleedFraction: 0.9 }` if your editor treats a layer covering nearly the whole frame as background rather than artwork — a common choice, since a full-bleed wrapper is usually scaffolding rather than something the user is pointing at. + +Do not reimplement this with `elementsFromPoint`. That stack omits `pointer-events: none` nodes, and a decorative overlay carrying `pointer-events: none` still paints — a z-stack query would report no ink over visible artwork and pass the click through anyway. The paint walk covers element boxes geometrically for that reason. + ## Draft loop: 60fps drag without model mutations The draft loop keeps the model clean during a drag. The SDK is **not** in the 60fps path — you call `preview.applyDraft` on every `pointermove` and `preview.commitPreview` once on `pointerup`. The model sees exactly one `moveElement` op per drag, rather than hundreds. diff --git a/docs/sdk/reference/adapters.mdx b/docs/sdk/reference/adapters.mdx index a8b6a6760..93b962243 100644 --- a/docs/sdk/reference/adapters.mdx +++ b/docs/sdk/reference/adapters.mdx @@ -87,6 +87,7 @@ Injectable preview surface adapter. Decouples the SDK from the host's rendering ```typescript interface PreviewAdapter { elementAtPoint(x: number, y: number, opts?: { atTime?: number }): ElementAtPointResult | null; + isProvablyEmptyAt?(x: number, y: number, opts?: PaintQueryOptions): boolean; applyDraft(id: string, props: DraftProps): void; commitPreview(): void; cancelPreview(): void; @@ -100,6 +101,24 @@ interface PreviewAdapter { Synchronous hit-test at composition coordinates `(x, y)`. Returns the nearest `[data-hf-id]` element under the point, or `null` for a transparent hit (the composition root, an opacity-0 element, or nothing at all). Requires a same-origin iframe — cross-origin access throws a DOMException. The `atTime` option reflects GSAP state at the current playhead; seeking to a speculative time is not supported. + + Optional. Is `(x, y)` provably free of ink — is it safe to let a click pass through to whatever sits beneath? This is the question a host has to answer before a transparent composition layered over other content swallows a click: is the user pointing **at** artwork, or through an empty gap? Geometry alone cannot tell — a composition is mostly full-bleed wrapper `
`s that cover every pixel of the frame without painting anything. + + **True only when the composition was readable and nothing painted there.** Ink present, a document still loading or unreadable, and an adapter that doesn't implement the method (`preview.isProvablyEmptyAt?.(x, y)` → `undefined` → falsy) all come back falsy. That polarity is deliberate: it puts the burden of proof on passing the click through, so every way of failing keeps the composition clickable rather than making it vanish from under the cursor. The obvious call site is safe by construction: + + ```typescript + if (preview.isProvablyEmptyAt?.(x, y)) passThrough(); + ``` + + Ink is a computed-style test — background colour, background image, visible border, the element's own text, or intrinsic media — with one exception: `` (and the `` inside a ``) routes through per-pixel alpha, so a transparent PNG paints only where its pixels do. A pixel-verified hit is never discounted by `fullBleedFraction`: box area is not ink area, so a full-frame transparent overlay stays clickable where it is actually opaque. + + **Known over-counts** (report ink that isn't there, so a click selects the composition): a `background-image` that is itself mostly transparent reads as painting across its whole box; `