mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-19 19:36:04 +00:00
50b4ddafac02abdc2846217afd5de052d3364a15
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9825614d91 |
fix(studio): Studio's own text reads without em or en dashes (#5414)
* fix(studio): text in Studio reads without em or en dashes * fix(studio): the dash check also catches HTML entities, and peak reads Unavailable * test(studio): the dash check test covers every entity form * fix(studio): messages Studio shows from lint, sdk, core and the renderer drop dashes too * fix(producer): the gate check stops early on the engine's new fallback wording |
||
|
|
84e36fc16a |
feat(runtime): data-vfx-chain backdrop capture and two-source kernels (luma-matte, noise) (#4330)
* feat(runtime): data-vfx-chain backdrop capture and ref second sources
Interface v1.1's two remaining capture shapes, in the runtime.
`backdrop`: a host resolves its source from a sibling
`<canvas layoutsubtree class="hf-vfx-src" data-vfx-for="<host id>">`
before its own child wrapper, so the same kernel reads either its own
pixels or every layer below it — which element the wrapper holds is the
exporter's placement, not a def's declaration. `VfxEntry.capture` now
reports what was resolved rather than what the defs asked for.
The trap that shape creates: a `backdrop` wrapper lives OUTSIDE the host,
and children of a `layoutsubtree` canvas are never painted by the page
compositor. Skipping a hidden host the way a `self` host is skipped would
therefore delete every layer below the adjustment instead of passing it
through. A hidden backdrop host now still captures and KEEPS the bitmap,
painting no kernel.
`ref` params: a node's second source, captured into `u_src2` from the
wrapper the exporter emitted around the referenced element. Per node, not
per entry, and shared when two nodes name the same element. Every source
is drawn at the host's device size so `u_src`/`u_src2`/`.hf-vfx-out`
share one coordinate space, and each source's canvas gets its own
`paint` wait — paint records are per canvas. A source the clip runtime
has hidden captures EMPTY rather than waiting out a paint that cannot
come and then throwing "No cached paint record" every frame; an empty
matte is also what After Effects shows outside a matte layer's window.
Also: a `.hf-vfx-in` that measures 0×0 is now reported once instead of
capturing nothing in silence, and one that is not the canvas's immediate
child is refused at registration instead of throwing every frame. Both
are measured Chrome rules (vault `layoutsubtree-capture-rules`).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(runtime): luma-matte and noise two-source vfx kernels
`luma-matte` is the first def with a `ref` param: `out = src · coverage(u_src2)`
with Alpha / Alpha Inverted / Luma / Luma Inverted coverage. It is not one
After Effects effect but the runtime form of all of them — unsupported
track-matte sources, stencils, silhouettes, Set Matte — which the exporter
distinguishes by choosing what the host wraps and what `matte` names. `src`
is premultiplied, so one scalar multiply covers all four channels, and Luma
reads the matte over black, which is what a premultiplied texel already holds.
`noise` ports `ADBE Noise` from `AEFX_Noise.metallib`: Bob Jenkins' 1997
integer mix over `(x, y, seed32)`, the two glibc-style LCG rounds, and the
`0x7F80`/`0xFF` byte the IR XORs out of them — no table, no state, so it is
seek-safe by construction. Reading the metallib's struct metadata rather
than the deep dive's prose settles two of its open questions:
`ADBE Noise-0002` is `inUseColorNoise` (a checkbox), not a Uniform/Gaussian
popup, and the mystery seed floats are `inRandomSeed_High`/`_Low`, the two
halves of one 32-bit seed. What the CPU puts in those halves per frame is
still unprobed (plan Task 4.3), so the kernel seeds the low half with the
frame index; the tests assert distribution and seek determinism, not parity.
The shader declares `precision highp int` — the whole port rests on 32-bit
wraparound and a fragment shader's integers default to mediump, which ESSL
guarantees only 16 bits of.
Both defs declare `capture: "self"`, which says a texture is needed and
nothing about whose: `noise` on an adjustment layer is the same kernel, and
the runtime reads the `backdrop` placement off the DOM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(vfx): browser checks for backdrop capture, luma-matte and noise
Task 2.6 Step 5 and Task 2.7's owed browser checks, folded into the
existing puppeteer harness: an identity backdrop kernel composites two
stacked blocks captured through a data-vfx-for sibling wrapper back
over themselves (PSNR >= 40dB vs a plain-DOM control panel); a backdrop
host outside its own clip window still keeps those blocks on the
wrapper canvas's own bitmap via capturePassThrough; luma-matte's four
modes are checked against alpha- and luma-varying matte strips at four
sample points each; noise is checked for amount=0 identity, same-time
determinism, and a changed pattern when its only seed (seek time)
changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* feat(runtime): external displacement maps and visible ref sources
Two gaps the exporter side found in the `ref` plumbing.
1. `displacement-map` had no `map` ref param, so the exporter's
`params.map = "<element id>"` was dropped by `normalizeVfxParams` (it
iterates the def) and the node silently self-displaced. `map` is now a
ref param, bound to `u_src2` and selected by a new `u_hasSrc2` uniform
the runtime sets from whether the ref resolved. Refs gain
`optional: true` — empty `map` is the self-referential form every
corpus instance uses, while an empty `matte` stays a loud failure — and
a ref naming its own host resolves to self rather than capturing the
same pixels twice (on a backdrop host it would otherwise have read the
layers below as the map).
2. Interface v1.1 assumed a ref source is invisible in After Effects.
retro-wave `Logo Anim` layer 5 is layer 4's displacement map AND an
enabled, opaque layer, and the runtime cleared its wrapper's bitmap
after the upload, deleting it from the frame. A wrapper marked
`data-vfx-ref-visible` now keeps its bitmap and is captured at the
REF's own box instead of the host's, so it paints at its own size and
resolution. The three capture shapes are one `{ upload, keepBitmap }`
mode: texture only, bitmap only (a backdrop pass-through), or both.
Sampling is unaffected by the box choice, and the browser test proves it
rather than assuming it: `drawElementImage` scales the element into the
whole canvas, so a 320px-wide matte's four strips still land as four
40px bands in a 160px host. The v1 rule — a ref is scaled into the host's
box, not placed in composition space — is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(runtime): capture a visible ref while its host is hidden; review nits
Addresses PR #4330 review feedback (jrusso1020, CHANGES_REQUESTED).
Blocking: both the engine-mode and preview-mode capture paths treated
"host not paintable" as binary — a backdrop entry got a pass-through
capture, everything else was skipped outright, including a ref source
that lives on a different, independently-visible element (e.g.
retro-wave's Logo Anim layer, which is simultaneously layer 4's
displacement map and its own fully opaque visible layer). Adds
visibleRefSources()/captureVisibleRefsOnly() and a unified
FrameCaptureMode discriminator so a visible ref still gets captured
even while its own host is hidden, on both paths.
Non-blocking, addressed:
- luma-matte's mandatory ref that names its own host now resolves to
an explicit RefResolution {kind:"self"} and binds u_src2 to unit 0,
instead of relying on an unbound sampler's implicit default.
displacement-map's optional self-ref is unaffected (still folds into
{kind:"none"}, using its existing u_hasSrc2 shader branch).
- noise shader now enforces rgb<=a (valid premultiplied alpha) after
adding noise, independent of u_clipping — documented why u_clipping
is currently a permanent no-op on this runtime's 8-bit render
targets.
- added a hash-pinning test for hfJenkinsMix; documented, with a
brute-force proof, which one of the reviewer's suggested mutations
is unobservable within this fixture's reachable seed range.
- fixed two misplaced doc comments.
fallow flagged resolveVfxCapture/resolveRefSource as newly over the
complexity threshold after this diff; extracted resolveEmptyRefParam()
and captureHiddenEntry() to bring both back under.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(runtime): capture a visible ref while a hidden BACKDROP host passes through
Re-review of #4330 at
|
||
|
|
b32b9a7dab |
feat(runtime): data-vfx-chain — a seek-deterministic per-pixel effect element (#4306)
* feat(core): data-vfx-chain defs and chain parsing (mirrors data-fx-chain)
Mirrors audioFx.ts's shape for the per-pixel VFX chain: a versioned
JSON chain attribute (data-vfx-chain), a def registry with typed
params (number/enum/bool/ref), and parse/serialize/normalize helpers.
HF_VFX holds one def, fractal-noise, with a placeholder frag; the
kernel and its consumers (packages/core/src/runtime/vfx.ts) land in
Task 1.2/1.3 of the vfx-chain-runtime plan, so vfx.ts is added to
.fallowrc.jsonc's entry list for now.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* feat(runtime): paint data-vfx-chain canvases on every seek
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(vfx): fractal-noise kernel (Basic/fBm approximation) with CPU reference
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(vfx): self capture through layoutsubtree with the page-composite readiness protocol
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(vfx): wave-warp kernel (Sine)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(vfx): displacement-map kernel (self-referential map)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(vfx): chain order is nesting order; preceding CSS effects are captured
...and following ones apply outside. The plan's one-line message for Task 2.4
is 105 characters; commitlint caps the header at 100, so the tail moved here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(vfx): seek determinism and CPU-reference PSNR for fractal-noise
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(vfx): size the output canvas from the host's layout box, not its transformed AABB
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* feat(telemetry): vfx node count, capture class and types on render_complete
Threads vfxNodeCount, vfxCapture and vfxTypes from a static scan of the
compiled composition's data-vfx-chain hosts (mirrors the existing
compositionElementCount/colorGradingCount scan) through
RenderPerfSummary.drawElement into the render_complete event, next to
de_capture_mode. Exports chainCapture/parseVfxChain/HfVfxCapture from
@hyperframes/core's index for the producer to consume. The PostHog
dashboard tile is not created here — left for the coordinator.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* docs(vfx): data-vfx-chain guide — element shape, capture table, Studio flag, known constraints
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(vfx): register chain hosts that arrive with a sub-composition mount
`initVfx(document.body, fps)` ran once, synchronously, three lines after
`void loadExternalCompositions(...)` — so every `data-vfx-chain` host that
enters the DOM only when a `data-composition-src` mount resolves was never
registered. An unregistered host paints nothing and reports nothing: no
`[HyperFrames] composition script error`, the `.hf-vfx-out` canvas keeps its
300x150 default, and `paintVfx` never even arms the page-composite flag.
Measured on a real multi-file export served raw: `#main-l5-text`'s output
canvas stayed 300x150 and `window.__timelines` never gained the sub-comp.
Re-scan in the loader's `.finally`, before `maybePublishRenderReady()`, so
registration precedes readiness. `initVfx` now releases the outgoing
registry's programs, capture textures and ping-pong targets first — the
output canvas hands back the same WebGL2 context on a re-scan, so the
previous pass's GL objects would otherwise leak.
Two browser cases in `vfxDeterminism.test.ts`, both against the real runtime
bundle: a host whose `.hf-vfx-in` IS the mount (already worked — the mount
replaces the wrapper's children, not the wrapper, so the registered source
node survives), and a host that arrives INSIDE a mounted sub-composition
(failed with `armed: false` and no error; passes now).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(vfx): runtime tsconfig typecheck passes
Under noUncheckedIndexedAccess, resolveCaptureSource's null return
widened VfxEntry.src past its VfxCaptureSource | undefined type; the
function is internal-only, so its return type moves to
VfxCaptureSource | undefined instead. Indexing PingPong.framebuffers
by a non-literal index produced | undefined against
WebGLFramebuffer | null, coalesced to null. No behavior change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(vfx): structural DOM predicates instead of realm-bound instanceof
vfx.ts checked `instanceof HTMLCanvasElement`/`HTMLElement` directly, which
returns false for inner-realm nodes when the Studio previews across an
iframe realm. domRealm.ts had no canvas predicate yet, so this adds
isCanvasElement mirroring the existing isImageElement pattern, then swaps
all three vfx.ts checks to the structural predicates the lint guard requires.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(vfx): paint self-capture hosts under hyperframes snapshot, not only the engine path
`paintVfx(t, { engineMode: true })` armed the page-composite protocol and
returned, betting that every capturing chain runs under `frameCapture.ts`.
`hyperframes snapshot` (and check/compare/validate/layout, and Studio's
thumbnail capture) seek through the same `renderSeek`, never read
`__hf_page_composite_pending` and never call `__hf_page_composite_resolve` —
so a `self` chain painted nothing there, in silence, because its content
lives in a `layoutsubtree` canvas the browser never paints directly.
Engine mode now arms the preview-side capture as well and paints on whichever
completes first. A seek token stops a superseded async capture from repainting
a newer frame, and a registry-identity check stops it painting an entry
`initVfx` already released. `resolveVfxCapture` stays non-idempotent on
purpose: the engine resolves only after its own before-capture hooks have
mutated the DOM, so its capture is still the authoritative one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(vfx): an exporter-emitted .hf-vfx-out gets the box the spec fixes for it
findOrCreateOut adopted a canvas the exporter had already emitted without
applying the layout cssText, so it kept the default position:static;
display:inline and wrapped to the line AFTER the layer-wide .hf-vfx-src
canvas — painting a second, misplaced copy of the layer one full layer
height below the host. Measured on retro-wave #main-l6-text: .hf-vfx-src
rect top 315.1 h 93.0, .hf-vfx-out rect top 409.2, both style=null,
position static (ae-mcp findings Task 3.5b). Hosts with no emitted
out-canvas were never affected, which is why only this one host moved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(telemetry): vfx_node_count counts hosts, not chain nodes — rename to vfx_host_count
The field increments once per `data-vfx-chain` attribute occurrence, never
per node inside a chain's JSON, so the name promised something the scan does
not measure. Renames the internal `vfxNodeCount` and the wire property
`vfx_node_count` to `vfxHostCount`/`vfx_host_count` across the orchestrator
scan, the perf summary, the CLI render path and the telemetry event. No
dashboard consumes the field yet, so the wire rename costs nothing now and
would cost a migration later.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(vfx): join the preview capture to the seek-completion barrier, not race it
`paintVfx` fired `capturePreviewThenPaint` and forgot it, so nothing the CLI
awaits waited for a `self` chain's paint. Every snapshot-family command
(`snapshot`, `check`, `compare`, `validate`, `layout`) seeks through
`seekCompositionTimeline`, which drains `window.__hfWaitForSeekCompletion`
and then allows one settle race before screenshotting — and only the
`"hf-seek"` event's `waitUntil()` could put anything in that barrier. The
screenshot therefore landed on however many hosts happened to have finished,
which varied with host count and machine speed.
Extracts the "track, untrack on settle, retain the first rejection" block
`dispatch()` built inline into an exported `registerSeekCompletion`, and has
`paintVfx` register the preview capture through it. Also awaits the per-host
paints with `Promise.all` instead of sequentially: N hosts cost about two
animation frames now, not 2N.
Measured on the pre-fix bundle with two `self` hosts, both output canvases
read `[0,0,0,0]` at the moment the barrier returned; they painted only
because the settle race that follows happened to be long enough. Both tests
added here fail on that bundle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(vfx): report WebGL context loss loudly instead of silently no-op painting
One WebGL2 context per `data-vfx-chain` host, against a browser cap that is
commonly 16 and shared with `colorGrading.ts`'s per-element contexts. A lost
context makes every later call on it do nothing, per spec — so the chain kept
"painting" a blank canvas with no error, which is exactly what the module's
loud-failure contract exists to prevent.
Adds the `webglcontextlost` listener pair's first half, following
`colorGrading.ts`'s precedent: `preventDefault()`, mark the entry, report
once through `reportVfxError`. The entry is then skipped by `paintVfx`,
`capturePreviewThenPaint` and `resolveVfxCapture` — silently, since the one
loud report already happened and repeating it per frame is spam.
Restore is left unhandled on purpose, with the reasoning in the doc comment:
`initVfx`'s existing re-scan already rebuilds every GL object, and a second
narrower rebuild path for `webglcontextrestored` would duplicate it. The
shared-context redesign the review floated is likewise out of scope here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* perf(vfx): cache uniform locations at pass-build time instead of querying every paint
`setPassUniforms` asked the driver for `u_size`, `u_t`, `u_fps` and every
param's location on every paint of every pass, and `bindPass` did the same
for `u_src` on every bind. `buildPasses` already constructs each `VfxPass`
once at init, right after the program links, and a linked program's
locations never change — so they are resolved there now and read from
`pass.locations` in the hot path.
Same values, same `null`-means-optimised-out semantics (`gl.uniform*(null,
…)` is a defined no-op, which is what the per-frame query produced too), so
the existing vfx.test.ts and the browser determinism suite cover it
unchanged — both still green, the latter against a rebuilt runtime bundle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(vfx): bound the paint wait per host so a stalled compositor can't hang the seek barrier
`paintVfx` registers `capturePreviewThenPaint` into the shared seek-completion
barrier (
|