Address Copilot round-3 review: the previous engine-mode timeline used
`tl.set(toId, opacity:1, T)` + `tl.set(fromId, opacity:0, T+dur)` for
every transition. That keeps BOTH scenes at opacity:1 throughout the
transition window. The Node-side layered compositor handles this fine —
it captures each scene separately, masks opacity per layer, and runs the
blend itself — but the page-side compositing path (one opaque RGB
screenshot per frame, opt-in via EngineConfig.enablePageSideCompositing)
relies on the page to produce a correct frame. With `shader === undefined`
the page-side compositor skips the entry, so the screenshot would show
both scenes stacked at 100% opacity (visible ghosting) instead of a blend.
Fix: schedule an actual opacity-crossfade tween in `initEngineMode`
when `t.shader === undefined`. Shader transitions keep the existing
opacity-flip pattern because the Node-side compositor needs both scenes
fully visible to capture them. The crossfade is harmless in the layered
Node path because `applyDomLayerMask` overrides per-scene opacity during
each capture anyway.
Also corrects docstrings in engineModePageComposite.ts and at the
installPageSideCompositor call site that previously claimed the GSAP
timeline "handles the blend" — it now actually does.
Co-authored-by: Cursor <cursoragent@cursor.com>
Three follow-up fixes from the Copilot review on commit 8cad2173:
1. Use strict `t.shader === undefined` instead of `!t.shader` (Copilot c4)
in both the WebGL program compile loop and the page-side compositor.
An empty-string `shader: ""` from a vanilla-JS caller (the IIFE bundle
is hand-loaded via <script> tags in user HTML) should reach the shader
registry and surface a loud "unknown shader" error, not silently
degrade to a crossfade.
2. Graceful degradation when shader compile fails (Copilot c5). The
previous `continue` dropped the transition from `cachedTransitions`,
which also dropped its scene-visibility timeline entries and broke
scene progression. Now: log a warning and downgrade to the CSS
crossfade fallback (prog=null, fallback=true) so the opacity timeline
still runs and the composition keeps playing.
3. Preserve index-to-scene-pair correlation when calling the page-side
compositor (Copilot c6). The earlier filter `transitions.filter(t =>
!!t.shader)` shifted indices, so a shader transition at original index
2 (sitting between CSS crossfades) would be paired with scenes[1] and
scenes[2] inside `installPageSideCompositor` instead of the correct
scenes[2] and scenes[3]. The compositor now accepts the full array,
makes `PageCompositeTransitionConfig.shader` optional, and skips
CSS-only entries internally while keeping `transitions[i]` aligned
with `scenes[i]`/`scenes[i+1]`.
Co-authored-by: Cursor <cursoragent@cursor.com>
Three follow-on fixes after the optional-shader change rebased onto current
main (PR #832 introduced page-side compositing and the producer's hf#732
layered pipeline since this PR was opened).
shader-transitions/hyper-shader.ts
- Treat `cache.prog === null` as the canonical immutable marker for
CSS-only transitions via a new `isCssOnlyTransition()` helper.
- `disposeCachedTransition()` now restores the always-ready CSS fallback
state for prog=null caches instead of zeroing `fallback`/`ready` — the
previous behaviour, combined with `markScenesDirty()` re-running the
prewarm/capture pipeline, could put a CSS-only cache through the WebGL
path and reach `renderShader(state.prog!)` with a null prog (Copilot
review on lines 1168 + 1319).
- `markScenesDirty()` skips CSS-only caches; they have no shader to
recompile and no texture pyramid to recapture.
- `ensureTransitionCachesReady()` filters CSS-only caches out of the
prewarm work list.
- `tickShader()` now routes on `cache.fallback || cache.prog === null`
and threads a narrowed non-null `prog` local into `renderShader()`,
removing the unsound `state.prog!` non-null assertion.
- `initEngineMode()` filters CSS-only transitions before passing them to
`installPageSideCompositor()`, which expects `shader: ShaderName`
(required). Page-side compositing is shader-only; CSS crossfades stay
on the GSAP opacity timeline.
producer/render/stages/captureHdrHybridLoop.ts
producer/render/stages/captureHdrSequentialLoop.ts
- Guard `activeTransition.shader` against undefined: when omitted, route
the Node-side blend through `crossfade` (the engine's canonical
opacity blend, equivalent to `applyFallbackTransition()` on the page).
- The hybrid path also bypasses the worker pool when `shaderName` is
absent and runs `crossfade` inline.
This addresses the Copilot review comments and unblocks the 5 failing CI
jobs (Build, Typecheck, CLI smoke, Windows tests, Windows render) which
all rooted in 4 TS errors at these exact sites.
Co-authored-by: Cursor <cursoragent@cursor.com>
Allow omitting the shader field in TransitionConfig to get a smooth CSS
opacity crossfade instead of a WebGL effect. HyperShader manages all scene
visibility regardless of transition type, so shader and CSS crossfade
transitions can now be mixed freely in the same composition.
When shader is omitted:
- No WebGL program is compiled or cached for that transition
- The existing applyFallbackTransition() path handles the crossfade
- No texture prewarming needed — transition is marked ready immediately
Tested: verified with a 3-scene composition (sdf-iris + CSS crossfade)
rendered to MP4. Both transition types render correctly.
engine/src/types.ts: HfTransitionMeta.shader is now optional to match
* ci: run fallow audit in lefthook pre-commit
Mirrors the same `fallow audit --base ... --fail-on-issues` check that
runs in CI, but locally against HEAD so issues surface at commit time
instead of after the push round-trip.
Scoped to `packages/**` source files via the glob — non-code edits
(README, docs, top-level configs) skip the hook entirely.
Measured locally: ~5s in parallel with the existing lint/format/typecheck
checks. Doesn't extend wall-clock time because typecheck (~11s) is the
long pole, and lefthook runs commands in parallel.
The default `--gate new-only` means inherited findings don't block the
commit — same gate behavior as CI, so local pre-commit and PR audit
agree.
* refactor: delete orphan declarations flagged by fallow
After fallow's auto-fix de-exports unused symbols, oxlint surfaces them
as no-unused-vars. This PR deletes those orphan declarations outright.
Biggest cleanup: studio/src/icons/SystemIcons.tsx shrinks from 132 to 57
lines — 33 unused icon wrappers and their phosphor-icon imports deleted.
Other deletions across 14 more files covering paired getter/setters,
helper functions, dead env constants, internal components with no
callers, and cascading unused imports.
Cascade-causing files held back for follow-up PRs: renderOrchestrator
barrel of captureCost re-exports, telemetry/portUtils/remote barrels,
Button.tsx + ui/index.ts (would orphan whole file), studioMotion
type re-exports.
Test plan: typecheck clean across 8 packages, oxlint + oxfmt clean,
fallow audit exit 0 (remaining findings inherited), cli + studio
vitest suites pass.
Adds a Blocks tab to the Studio left sidebar with the full 78-item registry
catalog (58 blocks + 20 components). Users can browse by category, search by
title/description, preview CDN-hosted poster thumbnails with video-on-hover,
and install items on-demand with one click or drag-to-timeline.
Core changes:
- BlockCategory type + resolveBlockCategory() for 7 categories (Captions, VFX,
Transitions, Effects, Social, Data, Scenes)
- Registry API routes: GET /api/registry/blocks (catalog) + POST install
- StudioApiAdapter extended with listRegistryCatalog + installRegistryBlock
- Vite adapter reads from disk; CLI adapter fetches from GitHub (24h cache)
- BlockParam interface + params on 6 blocks for future parameter controls
Studio UI:
- 4th sidebar tab "Blocks" with responsive grid, category pills, search bar
- BlockCard: CDN poster thumbnail, video autoplay on hover, duration + WebGL badges
- On-demand install: blocks append as sub-compositions on timeline; components
overlay at start=0 spanning full duration with transparent background patching
- TIMELINE_BLOCK_MIME drag-and-drop to timeline
- BlockParamsPanel (Phase 3 scaffold) auto-opens for parameterized blocks
Registry manifests:
- All 58 blocks backfilled with preview: { video, poster } CDN URLs
- All 20 components normalized to object format + poster URLs added
- 6 blocks annotated with params (Liquid Glass/Background, Portal, Chart,
Logo Outro, Magnetic)
- flowchart-vertical preview generated and uploaded to CDN
Run `fallow fix --auto-fixable` to remove `export` keywords from symbols
fallow's reachability analysis identifies as unused. Keeps only the cases
where the symbol is still referenced internally in its own file (so
removing `export` doesn't surface a new oxlint `no-unused-vars` error).
Result: fallow dead-code findings drop from 276 → 208 (68 fewer unused
exports), with no behavior change — each symbol is still defined and used
exactly the same way within its file.
Reverted ~20 files where fallow's auto-fix would have created cascading
"declared but never used" lint errors — those are cases where the symbol
isn't used at all, and properly cleaning them up means deleting the
declaration, not just dropping `export`. Better to land that as a
separate, narrower PR rather than mixing it into a mechanical de-export.
Also reverted four false positives where fallow missed real consumers:
- `captureCost.ts` (renderOrchestrator has two separate import blocks
from the same module; fallow only saw the first)
- `propertyPanelHelpers.ts`, `domEditingLayers.ts` (real internal uses
fallow's reachability missed)
- `render.ts` (functions imported via `await import()` dynamic import,
which fallow's static analysis doesn't follow)
Test plan: bun run --filter '*' typecheck (clean), oxlint + oxfmt clean,
cli/core/studio/engine vitest suites pass (335 + 917 + 576 + 605 tests).
* fix(cli): prefer puppeteer cache + numeric version sort (staff review)
Two correctness fixes from PR #821 self-review:
1. Cache priority order. Previous order was hyperframes-managed cache →
puppeteer cache. HF cache is pinned to CHROME_VERSION (131-era) which
lags 17+ releases behind upstream; if a user separately installed a
newer chrome-headless-shell via @puppeteer/browsers install, the CLI
would silently hand engine the older HF-cache binary while engine's
own resolveHeadlessShellPath would have picked the newer one. Flip
the priority so puppeteer cache wins, matching engine semantics.
2. Numeric (not lexicographic) version sort. `readdirSync.sort().reverse()`
over names like `linux-148.0.7778.97` and `linux-99.0.6533.123` would
return `linux-99...` first because character '9' outranks '1'. Parse
each name into integer segments and compare them numerically.
Tests: add both-caches-populated and linux-148-beats-linux-99 cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* perf(engine): page-side compositing for shader transitions (opt-in spike)
Add an opt-in `--page-side-compositing` flag (CLI) backed by a new engine
config field `enablePageSideCompositing` and env var `HF_PAGE_SIDE_COMPOSITING`.
When set, SDR shader-transition compositions skip the Node-side layered blend
(the hf#677 chain) and instead run the shader inside Chrome via a page-side
WebGL canvas; the engine then captures ONE opaque RGB frame per output frame
via the existing streaming capture path.
This is the strongest non-beginFrame perf lever for Mac users, who cannot
take the beginFrame `~5×` path (Chromium structural limit, crbug.com/40656275).
Stacks on top of the hf#677 1.95× baseline.
Default OFF — existing fixture pins (byte-exact MP4 output) are preserved.
Opt-in path is intentionally PSNR-pinned, not byte-equal (WebGL is f32; Node
is f64). HDR content forces the existing layered path regardless.
Implementation:
- engine: new `EngineConfig.enablePageSideCompositing` (default false).
- producer/fileServer: new `HF_PAGE_SIDE_COMPOSITING_STUB` early-page script
injected into the served HTML head when the flag is on.
- producer/renderOrchestrator: when the flag + no HDR + no png-sequence,
route SDR transitions through the streaming path instead of the layered
HDR stage.
- shader-transitions: new `engineModePageComposite.ts` installs a fullscreen
WebGL compositor overlay and wraps `window.__hf.seek` so each seek inside
a transition window captures both scenes via the Chromium
`drawElementImage` API to GL textures, runs the fragment shader, and
displays the composited result on the overlay canvas. The engine takes
one screenshot per frame and sees the composited overlay.
- cli: new `--page-side-compositing` flag sets `HF_PAGE_SIDE_COMPOSITING=true`
before producer load.
- scripts/page-side-compositing-smoke: bundled-CLI smoke that renders a
representative fixture with and without the flag, validates the canary
strings are in the shipped bundles, and writes a wall-time pair.
Determinism trade documented in the engine config doc-comment. The smoke
script enforces the bundled-CLI validation discipline from prior perf work
(see internal feedback note `validate_bundled_cli_not_dev_path`).
Runtime requirement: Chromium's `CanvasDrawElement` feature (already
enabled by the engine's `--enable-features=CanvasDrawElement` launch flag).
When the runtime feature is unavailable, the page-side installer logs a
warning and falls back to opacity-flip mode — the engine still takes the
streaming path; the transition window degrades to a hard scene swap. Vance
will validate on Mac Chrome where the feature is supported.
Co-Authored-By: Vai <vai@heygen.com>
* fix(shader-transitions): use html2canvas for page-side compositor capture
The original drawElementImage approach fails in engine render mode because
the virtual-time shim prevents Chromium from generating paint records for
cloned elements. drawElementImage requires a cached paint record from the
browser's compositor — clones created at capture time never receive one
because (a) shimmed rAFs deadlock inside the seek wrapper, (b) original
rAFs don't produce real paints under virtual-time control, and
(c) layoutsubtree canvases don't apply CSS stylesheet rules to children.
Switch scene capture to html2canvas (foreignObjectRendering: false), the
same JS-based renderer already used by the preview-mode fallback path in
capture.ts. html2canvas reads computed styles and renders via its own
canvas drawing pipeline with no dependency on the browser paint cycle.
Also fixes:
- Engine seek must return the result so Puppeteer awaits async seek
promises (frameCapture.ts).
- GSAP opacity cache: compositor must restore scene opacity before seek,
not after — GSAP caches inline values and skips re-writes.
- Support check gates on WebGL availability, not drawElementImage.
Perf: 15-scene shader-perf fixture (28s, 14 transitions, 30fps)
Baseline (Node-side layered): 137s
Page-side (html2canvas+WebGL): 33s → 4.1× speedup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(shader-transitions): simplify review fixes for page-side compositor
- Use uploadTexture (zeroes canvas backing store after upload) to prevent
~2.2GB transient memory pressure across 280 html2canvas calls per render
- Add ignoreElements + stabilizeTransformedBoxShadows to html2canvas call,
matching the preview-path capture.ts behavior
- Parallelize from/to scene captures with Promise.all
- Wrap post-capture render in try/finally so opacity is always restored
- Fix WebGL context leak in isPageSideCompositingSupported probe
- Remove dead ResolvedTransition.index field
- Export stabilizeTransformedBoxShadows from capture.ts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(producer): unify page-side compositing gating and Docker forwarding
Addresses three issues from staff review:
1. ignoreElements filter stripped all in-scene canvases (Chart.js, D3,
p5.js) — narrowed to data-no-capture only since the compositor canvas
is a body sibling never in the scene subtree.
2. Docker mode silently dropped --page-side-compositing — thread
pageSideCompositing through DockerRenderOptions/buildDockerRunArgs
with regression tests.
3. Fragmented gating across 4 independent sites could disagree:
- Stub injection gated only on cfg flag (leaked into HDR/alpha)
- Probe-created fileServer never got the stub
- needsAlpha (WebM/MOV) not excluded from the gate
- WebGL-unavailable fallback claimed layered path would run but
orchestrator had already disabled it
Fix: compute stub injection at the same site as the layered-bypass
decision (after hasHdrContent is known), using addPreHeadScript on
the already-running fileServer. Single predicate now gates both
decisions, including !needsAlpha.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* perf(engine): two-phase drawElementImage capture for page-side compositing
Replace html2canvas with native drawElementImage for scene capture in
the page-side compositor. drawElementImage reads from the browser's own
paint cache, giving pixel-identical output to the preview path.
The blocker was that cloned elements inside layoutsubtree canvases have
no cached paint record under virtual time — the compositor only paints
when explicitly triggered. Fix: split the seek+composite into two phases
with an engine-forced paint between them.
Phase 1 (seek wrapper, page-side):
- GSAP seek positions the timeline
- Clone FROM/TO scenes into visible layoutsubtree staging canvases
- Set window.__hf_page_composite_pending flag
Engine paint force (frameCapture.ts):
- Detect pending flag after seek returns
- Fire micro Page.captureScreenshot (1x1 clip) via CDP to force the
browser compositor to paint all visible elements including staging
canvas children
Phase 2 (page.evaluate, page-side):
- drawElementImage reads the now-valid paint records
- Upload textures to WebGL, run shader, show GL overlay
Key insight: staging canvases must be visible (not opacity:0) for the
browser to paint their children. They sit at z-index:-9998, behind
the main DOM and covered by the GL overlay during transitions.
Perf: 15-scene fixture (28s, 14 transitions, 30fps):
Baseline (Node-side layered): 137s
html2canvas + WebGL: 33s (3.7×)
drawElementImage + WebGL: 21s (6.6×)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* perf(engine): optimize two-phase compositor hot path
- uploadTextureSource instead of uploadTexture: eliminates ~2.3GB of
canvas buffer alloc/dealloc churn (persistent staging canvases don't
need the one-shot zeroing behavior)
- Fold hasPending check into seek page.evaluate: eliminates one CDP
round-trip per frame (~700 unnecessary IPC calls on non-transition
frames)
- Fix renderShader error handling: on failure, leave source scenes
visible as fallback instead of hiding both scenes + GL overlay
(which produced black frames)
- Move mutable state declarations above resolveComposite to prevent
TDZ risk on refactor
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(engine): staff review — staging cleanup, pending flag, beginFrame guard
- Clear staging canvas children when leaving transition window (prevents
visible clone bleed-through on transparent compositions)
- Clear __hf_page_composite_pending on all resolveComposite exit paths
- Guard micro-screenshot paint force against beginFrame mode (CDP
Page.captureScreenshot conflicts with beginFrame compositor control)
- Update CLI flag description: document video/canvas limitation, remove
stale PSNR claim
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(engine): default-on page-side compositing for SDR shader transitions
Page-side compositing is now enabled by default for SDR shader-transition
renders without video content. The 6.6× speedup applies automatically —
no flag needed.
Auto-disables when:
- HDR content detected
- Alpha output (WebM/MOV/PNG-sequence)
- Composition contains <video> elements (cloneNode loses playback state)
- beginFrame capture mode (Linux headless)
Use --no-page-side-compositing to force the Node-side layered path.
Changes:
- Engine config: enablePageSideCompositing defaults to true
- CLI: flag default flipped to true; --no-page-side-compositing disables
- Orchestrator: added composition.videos.length === 0 gate
- Docker: forwards --no-page-side-compositing when explicitly disabled
- Config tests updated for new default
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(engine): support video elements on page-side compositing fast path
Three-phase capture protocol lets shader transitions render video scenes
without falling back to the slow Node-side layered pipeline:
1. Seek → compositor records transition metadata, sets pending flag
2. onBeforeCapture → video frame injector updates <img> replacements
3. prepare → cloneNode picks up current video frames, img.decode() awaits
4. micro-screenshot → forces browser to paint cloned elements
5. resolve → drawElementImage reads paint records, shader composites
Key changes:
- Remove `composition.videos.length === 0` gate from orchestrator
- Split compositor resolve into prepare (clone) + resolve (shader)
- Move onBeforeCapture before compositor prepare in frameCapture.ts
- Await img.decode() on cloned data-URI images to prevent stale frames
- Stop manipulating scene opacity in compositor (GL canvas overlay suffices)
- Add gsap.set declaration for shader-transitions ambient types
- Add video_missing_timing_attrs lint rule for <video> without id/data-start/data-end
Performance: compositions with video now render at 7.5s (6 workers) instead
of 2m38s on the layered path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(core): auto-inject data-start on video/audio so frame extraction works without explicit attrs
The timing compiler now injects data-start="0" on <video> and <audio>
elements that lack it. This makes discoverMediaFromBrowser() find the
element (it queries video[data-start]), so the frame extraction pipeline
activates automatically. Videos "just work" without requiring authors to
add data-start, data-end, or id attributes.
Also removes the video_missing_timing_attrs lint rule — the compiler
handles the missing attributes automatically, so the lint rule would
only false-positive.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(core): add data-hf-auto-start sentinel on auto-injected video timing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(producer): add discoverVideoVisibilityFromTimeline for runtime video discovery
Seeks the GSAP timeline in Puppeteer to discover when each video's parent
scene is visible (opacity > 0). Uses coarse sampling at 100ms steps followed
by binary search refinement to frame-level precision (1/60s). Only processes
videos with the data-hf-auto-start sentinel so author-specified timing is
never overridden.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(producer): integrate runtime video visibility discovery into probe stage
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(producer): trigger browser probe for auto-start videos, remove debug logging
The probe stage was skipping browser launch when composition duration was
already known, which meant discoverVideoVisibilityFromTimeline never ran.
Now needsBrowser also checks for data-hf-auto-start sentinel in compiled HTML.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(scripts): use mkdtempSync for smoke test work directory
Replaces hardcoded /tmp/hf-page-side-smoke with a unique temp directory
via mkdtempSync to resolve CodeQL "insecure temporary file" alert.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style: format smoke test script
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Vai <vai@heygen.com>
Miguel Ángel