Add a colocated README describing the five HDR regression fixtures, the
three layers of HDR coverage we have today (engine vitests, the
hdr-smoke metadata script, and the partial visual harness integration),
and the known gaps - notably that hdr-smoke is not in CI and there are
no committed pixel goldens, which is why the SDR opacity yoyo bug on
<video>-backed clips slipped through initially.
Add a pointer from CONTRIBUTING.md so HDR-touching contributors find it
from the standard "Running Tests" section.
Made-with: Cursor
Commit eac396a3 renamed hdrVideoStartTimes → hdrLayerStartTimes across the
file, but it predated the diagnostic-block typo fix on
feat/hdr-layered-compositing (4f6e9673). After the restack, line 1470 ended
up using the (now non-existent) hdrVideoStartTimes name and broke the
producer typecheck/build on this branch. Restore the rename so the
diagnostic block matches the local declaration.
Made-with: Cursor
Expands the regression composition to better cover the SDR-opacity
fix and to read more clearly when reviewing the rendered output:
* Replace the previous 36-byte placeholder media (Git LFS pointers
that were never resolved on disk) with real BT.2020 PQ clips
drawn from a YouTube HDR demo (M8hv1Oah2uQ) and BT.709 SDR clips
drawn from a YouTube SDR sample (SnUBb-FAlCY). Total ~40 MB of
binaries — kept inline rather than tracked through LFS so the
fixture matches the rest of the hdr-regression suite.
* Add `data-start` / `data-duration` to every HUD slate, caption,
and overlay element so they actually reach the layered compositor
during their owning scene window. Without these the elements were
computed off-stage by the renderer and the slates / pip captions
in scenes 1 and 2 never appeared in the final video.
* Paint scene 6 on a white field so an opacity yoyo on a framed clip
is visible — on the default near-black root the dip-to-translucent
reads as no change at all.
* Drop opacity from the scene-6 transform yoyo on `#s6-f3`. Earlier
scenes already cover the opacity-yoyo path; scene 6 is now a
dedicated transform showcase (entry opacity fade-in stays so the
clip enters cleanly).
Made-with: Cursor
Adds a focused 6-second composition that runs the same opacity timeline
on an HDR clip and an SDR clip side-by-side: a `tl.set` to opacity 0,
an entry fade-in (0→1), then an opacity yoyo (1↔0.15). Both wrappers
use the same selectors and share the same tween, so any divergence
between the HDR (compositor) path and the SDR (DOM-injected <img>)
path is immediately visible as a left/right asymmetry.
This fixture would have caught the SDR-opacity regression fixed in the
previous commit — without it, the issue only surfaced as a vague
"opacity isn't doing anything" report on the larger hdr-feature-stack
composition. The fixture is also wired into hdr-smoke so CI exercises
the same code path on every change.
Assets are symlinked to the existing hdr-feature-stack videos to
avoid duplicating ~25 MB of binary content.
Made-with: Cursor
A six-scene composition that exercises every native HDR compositing
feature in one render so regressions in any one path get caught by the
existing `bun run hdr-smoke` flow:
- scene 1: pure HDR video + sRGB DOM badge overlay
- scene 2: HDR background + SDR picture-in-picture + sRGB DOM headline
- scene 3: 16-bit HDR PNG image + sRGB DOM caption
- scene 4: two HDR videos masked by border-radius (circle + rounded)
- scene 5: three z-ordered overlapping cards (HDR, SDR, HDR) with sRGB tags
- scene 6: three transformed clips (translate/rotate, scale/skew,
opacity/translateY) — exercises per-element compositor layers
The five inter-scene transitions cover four shader programs
(domain-warp, cross-warp-morph, flash-through-white, gravitational-lens)
so the engine-render-mode init path in hyper-shader.ts gets coverage too.
Asset symlinks reuse source media that hdr-pq and sdr-baseline already
ship, so this does not add any new binary blobs to the repo. The
vendored shader-transitions.js is a local build of
@hyperframes/shader-transitions@0.4.6 — the published CDN bundle at
this version is missing the window.__hf.transitions write the engine
needs for native HDR compositing. Once a fixed build ships the vendor
file goes away and the HTML can pull from CDN again.
Wires the fixture into hdr-smoke.ts with the same probe expectations as
the other HDR fixtures (yuv420p10le, smpte2084, bt2020, requireHdrSideData),
and adds **/vendor/ to the oxlint ignore patterns so vendored library
bundles don't trip lint.
Made-with: Cursor
`page.evaluate` callbacks with nested `function` declarations crashed
with `ReferenceError: __name is not defined` whenever the host was
bundled by tsx/bun. esbuild's `keepNames` mode wraps every function
declaration — including ones inside the body of an evaluate callback —
with a `__name(fn, "name")` call to preserve `Function.prototype.name`.
The helper is injected into the host bundle but never serialized into
the function string Puppeteer ships to the browser, so the browser
context sees a free reference and throws.
Install a no-op identity `globalThis.__name` shim via
`page.evaluateOnNewDocument` during session init. We pass a string
literal (not a function) because esbuild does not transform string
contents — defining the shim inline would itself get wrapped with
`__name(...)` and produce a use-before-define cycle. Running it via
`evaluateOnNewDocument` guarantees the shim is in place before any
page script (including subsequent `page.evaluate` callbacks) executes.
Document the constraint at the call site in `videoFrameInjector.ts`
so future edits don't try to redefine the shim inside an evaluate
callback.
Made-with: Cursor
x265 emits HDR10 mastering display + content light level metadata as
in-band HEVC SEI messages, but FFmpeg's `mov` muxer doesn't extract
those into the container-level `mdcv` (Mastering Display Color Volume)
and `clli` (Content Light Level Info) boxes that ingest pipelines
read. Without them, YouTube, Apple AirPlay, and most HDR TVs see only
stream-level color tagging (`colr`) and treat the file as SDR BT.2020,
silently tone-mapping the output.
Add `mp4HdrBoxes.ts`, which surgically inserts `mdcv` + `clli` boxes
inside the HEVC sample entry (`hvc1`/`hev1`), bumps every parent box's
size, and rewrites every `stco`/`co64` chunk offset that points past
the insertion site so the file stays decodable. Reference: ISO/IEC
14496-15 (NAL-structured video) and ISO/IEC 23001-8 (CICP).
Wire the injection in two places:
- `streamingEncoder.ts`: post-encode for direct HDR renders, so
standalone calls into the engine emit YouTube-ready files.
- `renderOrchestrator.ts`: post-mux for the producer pipeline,
because FFmpeg's mp4 muxer rebuilds the container during
mux/faststart and drops the boxes we injected into the
intermediate video-only file.
Failures degrade to a warning — the file is still playable; only HDR
recognition on strict ingests is affected. Covered by 50+ unit tests
in `mp4HdrBoxes.test.ts` (parser fuzzing, box layout, stco/co64
rewriting, malformed-input rejection).
Made-with: Cursor
DOM overlays composited onto HDR video frames were oversaturated because
`blitRgba8OverRgb48le` mapped sRGB 8-bit values directly through an
HDR OETF (HLG or PQ) without first converting from BT.709 to BT.2020
color primaries. Treating sRGB values as if they already lived in the
much wider BT.2020 gamut pushed saturated colors well past the
designer's intent — e.g. sRGB pure blue (0,0,255) landed on BT.2020
blue, which is far more vivid than what was specified.
Replace `buildSrgbToHdrLut` with the full pipeline:
sRGB 8-bit
→ linear BT.709 (sRGB EOTF, 256-entry LUT)
→ linear BT.2020 (3×3 BT.2087-0 primary matrix)
→ HDR signal 16-bit (HLG/PQ OETF, 4096-entry LUT)
The matrix rows sum to 1.0, so neutral content (R=G=B) is invariant —
text and grayscale UI render identically. Chromatic content (icons,
accent colors, progress bars) is now color-accurate against BT.2020
HDR video. PQ scales relative to 203 nits SDR white per BT.2408 so
SDR overlays sit at conventional brightness inside the HDR frame.
Verified end-to-end with hdr-smoke (sdr-baseline, hdr-pq,
mixed-sdr-hdr): sampled red overlay pixels (#C1121F) in the rendered
PQ output match the calculated post-conversion 16-bit values
(~30837/18894/15630), versus the old buggy pipeline's
~33820/10529/13663 (visibly oversaturated).
Removes the `getSrgbToHdrLut` public export — the LUT is now an
internal pipeline stage, not a single-step conversion.
Made-with: Cursor
Adds a standalone smoke harness for the HDR encode pipeline plus a
guard against the silent-decode-failure regression in blitHdrVideoLayer.
- packages/producer/scripts/hdr-smoke.ts: end-to-end harness that
renders the hdr-pq and mixed-sdr-hdr regression fixtures, then
ffprobes both stream-level and frame-level side-data so we can
catch missing MasteringDisplay / MaxCLL SEI in CI without
hand-running ffprobe. Frame probe uses
-show_frames -read_intervals %+#1 since x265 emits HDR metadata
as in-band SEI prefix NAL units, not container-level boxes.
- renderOrchestrator.test.ts: new "logs decode errors via the
supplied logger" test pinning blitHdrVideoLayer's failure path
to log.warn (matches production behaviour — silent failures
were the original PR-314 review concern).
- renderOrchestrator.ts: doc-only — clarify that blitHdrVideoLayer
is exported so the time→frame math, last-frame freeze, border-
radius detection, and affine-vs-region branch can be unit tested
without spinning up the full producer.
- .gitignore: ignore packages/producer/tests/hdr-regression/_renders
and the generated hdr-full-demo workdir so smoke runs don't
pollute git status.
Writes to ffmpeg's stdin pipe that race with ffmpeg's exit emit EINVAL
or EPIPE as unhandled 'error' events on the Socket, crashing the
process. Add a no-op error handler on stdin — the exit handler already
captures the failure via the result object.
Surfaced during HDR regression renders where the last frame's write
coincides with ffmpeg finishing input processing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds the test cases the PR #314 review flagged as missing:
- empty image list / no <img> elements
- duration="0", negative, NaN, and Infinity rejection
- missing data-start defaults to 0
- duplicate ids are preserved (documents current contract)
Made-with: Cursor
Implements the decode-once cache claimed in the PR review reply:
HDR image layers now reuse decoded rgb48le buffers across the frames
they're visible. Cache is render-scoped (cleared per job) and keyed
by framePath::sourceTransfer::targetTransfer because convertTransfer
mutates the buffer in-place.
Only image layers receive the cache. Video layers would bloat memory
(every frame has a unique path: ~37 MB × 300 frames at 1080p ≈ 11 GB).
Images decode once and are blitted on every visible frame.
Also promotes the blitHdrVideoLayer catch block from log.debug to
log.warn — a blit failure means a missing/dropped HDR layer, which
is user-visible and shouldn't be silent at the default log level.
Made-with: Cursor
- Tolerate per-image HDR probe failures; warn on missing paths.
- Warn when HDR <img> source is animated (stills-only for now).
- Namespace HDR image frame dirs as hdr_img_<id> to avoid collisions.
- Drop image id from HDR layer set when extraction fails.
- Promote HDR frame extraction failure from info to warn.
- Rename hdrVideoStartTimes → hdrLayerStartTimes.
- Make CompiledComposition.images optional.
- Fix NaN data-duration slipping through parseImageElements.
- Remove dead setAttribute mutation in parseImageElements.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ImageElement type and parseImageElements() parser for <img> elements.
When --hdr is set, images are probed alongside videos for HDR color
space. HDR images get single-frame 16-bit PNG extraction and route
through the existing blitHdrVideoLayer path. SDR images unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Commit 188ebcca removed the opacity copy from `injectVideoFramesBatch` on
the assumption that the <img> sibling would inherit GSAP's opacity from
a shared wrapper. That breaks any composition where GSAP animates opacity
directly on the <video> element itself: the <img> has no animated
ancestor and renders at full opacity throughout any fade, even when the
user's intent is partial or zero opacity.
The CI `style-7-prod` and `style-8-prod` regressions caught this:
the <video id="aroll"> fade-in from 3.0-3.5s rendered as a hard cut
because the <img> inherited opacity 1 regardless of GSAP's tween.
Restore the old explicit copy from `computedStyle.opacity` to the
<img>'s inline opacity, with the `|| 1` fallback intentionally
preserved. The fallback is load-bearing: GSAP's seek does not re-apply
tweens that have already completed, so post-fade frames read opacity 0
from the stale `opacity: 0 !important` we apply to hide the native
<video>. The `|| 1` recovers the tween's end-state opacity 1 for
those frames, matching the final on-screen intent and the existing
baseline renders.
Handles both DOM shapes:
- GSAP on wrapper: video's own computed opacity is 1, img set to 1,
wrapper's opacity applies via stacking as before.
- GSAP on <video>: video's computed opacity is the tween value, copied
to img directly since they are siblings.
Fixes:
- style-7-prod: 0 failed frames (was 2 @ t=3.17, 3.33)
- style-8-prod: 0 failed frames (was 2 @ t=3.05, 3.24)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The diagnostic logging block in executeRenderJob's HDR layer composite
path referenced an undeclared `hdrLayerStartTimes` map. The correct
variable, declared and populated earlier in the same function, is
`hdrVideoStartTimes`. The typo was introduced alongside the DOM-layer
masking work and broke the producer build/typecheck on CI.
Made-with: Cursor
SDR clips inside an HDR composition were rendering at full opacity even
when the user had animated their wrapper opacity (e.g. fade-in or
yoyo). Two bugs in the per-layer screenshot path conspired to drop the
GSAP-applied opacity on the floor:
1. removeDomLayerMask was unconditionally calling
`el.style.removeProperty("opacity")` on every wrapper after each
layer capture. applyDomLayerMask only ever sets `visibility`, so the
only inline opacity present is the value GSAP wrote. Stripping it
between layer captures means that on the next capture (at the same
timestamp), GSAP's `totalTime(t, false)` no-ops because the timeline
is already at that time — the opacity is never restored, and the
wrapper renders fully opaque.
2. injectVideoFramesBatch was reading the source <video>'s computed
opacity via `parseFloat(computedStyle.opacity) || 1` and copying it
onto the injected <img>. Because syncVideoFrameVisibility forces the
<video> to `opacity: 0 !important` to hide it during capture, the
computed value is always 0, which `|| 1` then silently flips to
full opacity. The <img> is a sibling of the <video> inside the same
wrapper, so it should inherit opacity from the wrapper directly
instead of having a value hard-set on it.
Fix both: drop the opacity removal in removeDomLayerMask, skip opacity
when copying visual properties from <video> to <img>, and explicitly
clear any stale inline opacity on the <img> so it inherits from the
wrapper that GSAP is animating.
Made-with: Cursor
The HDR layered compositor blits z-ordered layers over a shared canvas. DOM
layers used a full-page screenshot from `captureAlphaPng`, which captures
*every* painted pixel on the page — root background, sibling-scene content,
overlay UI elements that aren't part of the current layer. Those opaque
pixels were then blitted over the canvas, overwriting any HDR content
composited beneath in earlier layers.
The previous workaround toggled `display:none` on hide ids via
`hideVideoElements`/`showVideoElements`. That correctly hid native videos
but did nothing about the root composition's background or about overlay
elements that the layer grouping considered part of a different layer.
This commit replaces the workaround with a precise CSS mask installed
before each DOM screenshot:
1. `applyDomLayerMask` injects a stylesheet that hides every `body *` and
re-shows the layer's elements (and their descendants and their injected
`__render_frame_*` siblings) with `visibility: visible !important`. CSS
visibility is *not* multiplicative through descendants — a child with
`visibility: visible` overrides an ancestor's `visibility: hidden`, so
deeply nested layer content still paints even though every intermediate
ancestor is hidden by the mass-hide rule.
2. Non-layer data-start ids are inline-hidden with
`visibility: hidden !important`. Inline `!important` beats stylesheet
`!important`, so this overrides the show rule for elements that fall
under a show selector but should NOT paint — most importantly HDR
videos and other-layer SDR videos that live as descendants of `#root`.
3. `removeDomLayerMask` tears the stylesheet down and clears the inline
`visibility`/`opacity` properties so subsequent video frame injection
gets a clean slate.
Crucially the mask only sets `visibility`, never `opacity`. CSS opacity
*is* multiplicative — `opacity: 0` on `#root` would zero out every
descendant including layer videos, even with `visibility: visible`. We
also extend `initTransparentBackground` to force the composition root
(`[data-composition-id]`) transparent in addition to `html`/`body`,
because compositions almost always set `#root { background: ... }` and
that background paints across the whole viewport otherwise.
Both compositing paths use the new helpers:
- The per-layer DOM branch (`compositeToBuffer`) for normal frames.
- The transition path (single DOM screenshot per scene) so transition
frames also get a clean per-scene capture.
Adds extensive `KEEP_TEMP=1`-gated diagnostics to `compositeToBuffer`:
per-layer pixel-add accounting, dumps of every captured DOM PNG, and a
periodic raw `rgb48le` snapshot of the composite buffer. These were
essential to diagnosing the root-overwrite bug and stay zero-cost in
normal renders. Also stops the workDir / per-video frame-dir cleanup
when `KEEP_TEMP=1` so the dumps survive past frame N.
Made-with: Cursor
Adds:
- 8 new sampleRgb48le bilinear-interpolation tests covering boundary
pixels, sub-pixel weights, edge clamping, and odd-byte-offset Buffers.
- uint16-alignment-audit.test.ts documenting the alignment requirement
for Uint16Array views over Buffer slices vs. readUInt16LE/writeUInt16LE.
Background: ~105 hot-loop sites in shader transitions still use
readUInt16LE/writeUInt16LE. Switching to Uint16Array views would cut
overhead but requires guaranteed even byteOffsets — these tests document
the contract before any future refactor lands.
Commit c6b4619c ("feat(hdr): shader transitions, --hdr flag, and SDR
rendering fixes") accidentally removed two pieces of the deterministic
rendering pipeline:
1. The `VIRTUAL_TIME_SHIM` injected via `createFileServer.preHeadScripts`,
which freezes `Date.now()` and `requestAnimationFrame` so RAF-driven
animations advance only when `window.__hf.seek(t)` is called.
2. The `applyRenderModeHints` function and its post-`compileForRender`
call site, which auto-forces screenshot capture mode for compositions
the compiler flagged as needing it (RAF, iframes, etc.).
Without (1), RAF animations advanced by wall-clock between the main-loop
seek and the per-DOM-layer seek inside `compositeToBuffer`, producing the
sawtooth PSNR pattern on `raf-ball-render-compat` (high PSNR at integer
seconds, ~24 dB everywhere else). Without (2), `iframe-render-compat`
lost its automatic fallback to screenshot mode and the child-document
motion stopped being captured.
Both helpers are still produced by `htmlCompiler` and exercised by
`renderOrchestrator.test.ts` — the orchestrator just stopped calling
them. Restored:
- Re-import `VIRTUAL_TIME_SHIM` from `./fileServer.js`
- Pass `preHeadScripts: [VIRTUAL_TIME_SHIM]` to both `createFileServer`
call sites (probe + main render)
- Re-add `applyRenderModeHints` (matching the test expectations) and
call it immediately after `compileForRender`
- Persist `renderModeHints` in `summary.json` and the
"Compiled composition metadata" log line
Fixes the `iframe-render-compat` and `raf-ball-render-compat` regression
failures on `feat/hdr-layered-compositing`.
Made-with: Cursor
- Tighten the all-transitions smoke test thresholds: at progress=0 we now
require the center pixel R-channel > 35000 (was > 25000) and at
progress=1 < 15000 (was < 25000). The old midpoint of 25000 sat exactly
halfway between the test from-pixel (40000) and to-pixel (10000), so a
half-blended transition would silently pass.
- Add a runtime assertion in HyperShader.init() that every scene id
resolves to a DOM element with the .scene class. Without this, missing
ids silently no-op when textures + querySelectorAll(.scene) run later.
Addresses deferred review feedback from PR #268.
## Summary
HDR video elements with GSAP animations (position, scale, rotation, opacity) and CSS border-radius rendered without any transforms applied — the video just sat at (0,0) full-size. This PR adds affine transform support and rounded-corner masking for natively-composited HDR video.
## What it does
**Affine blit with bilinear interpolation:**
- `blitRgb48leAffine()` — Takes a 4x4 DOMMatrix and maps each destination pixel back to source coordinates via the inverse transform. Bilinear interpolation between the 4 nearest source pixels produces smooth edges under rotation and non-integer scaling. Optional opacity and border-radius parameters.
- `parseTransformMatrix()` — Parses CSS `matrix(a,b,c,d,e,f)` strings into `[a,b,c,d,e,f]` tuples.
**Accumulated viewport matrix:**
- `getViewportMatrix()` — Walks the `offsetParent` chain from element to viewport, accumulating position offsets and CSS transforms at each level. Correctly handles `transform-origin` using the CSS sandwich: `translate(origin) × M × translate(-origin)`. This is critical because GSAP animates transforms on wrapper divs, not directly on the video element.
**Effective opacity:**
- `getEffectiveOpacity()` — Multiplies opacity values walking up the ancestor chain. Uses `Number.isNaN()` (not `|| 1`) so opacity:0 isn't incorrectly treated as 1.
**Border-radius masks:**
- `roundedRectAlpha()` — Per-pixel anti-aliased rounded-rectangle mask with support for independent corner radii.
- `getEffectiveBorderRadius()` — Walks ancestors for `overflow:hidden` + border-radius. Resolves percentage values (e.g., `50%` for circles) via `offsetWidth`/`offsetHeight`.
**Layout dimensions for extraction:**
- Uses `offsetWidth`/`offsetHeight` (unaffected by CSS transforms) instead of `getBoundingClientRect()` (which returns the transformed bounding box and wobbles under rotation).
## Files changed
| File | What changed |
|------|-------------|
| `packages/engine/src/utils/alphaBlit.ts` | `blitRgb48leAffine()`, `parseTransformMatrix()`, `roundedRectAlpha()`, `cornerAlpha()` |
| `packages/engine/src/services/videoFrameInjector.ts` | `getViewportMatrix()`, `getEffectiveOpacity()`, `getEffectiveBorderRadius()`, `layoutWidth`/`layoutHeight` on `ElementStackingInfo` |
| `packages/producer/src/services/renderOrchestrator.ts` | Affine blit path, extraction at layout dimensions, border-radius parameter passing |
## How to test
Render a composition with an HDR video that has GSAP scale + rotation animation and a `border-radius: 50%` wrapper (circle mask). The video should rotate smoothly with round edges — no wobble, no sharp corners.
## Stack position
**5 of 6** — Stacked on #289 (z-ordered layers). Adds transform and masking support to the HDR blit that the layer compositor uses.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
* refactor(engine): restructure frame reorder buffer with Map-keyed storage
Rewrites createFrameReorderBuffer to use a Map<number, Array<() => void>>
keyed by frame index instead of a flat Array<{frame, resolve}> scanned on
every advance. O(1) lookups in enqueue/flush, fast-paths for the matching-
cursor and overshoot cases, and a small fix: waitForAllDone now coexists
with the writer still waiting on the final frame instead of colliding on
the same waiter slot.
Also adds 5 unit tests (there were none before) covering the fast-path,
out-of-order gating, multi-waiter-per-frame semantics, waitForAllDone
normal path, and the overshoot case.
Comment tweaks on buildChromeArgs — the flag profile is the standard
headless-for-capture set (Puppeteer / Playwright / Chrome headless-shell
all converge on similar flags); rephrased for clarity.
* refactor(cli): simplify port availability probe with async/await
Rewrites isPortAvailableOnHost from a single new-Promise callback into an
async/await form with an intermediate `bindError: ErrnoException | null`
variable. Makes the bind-then-release flow explicit as two sequential
awaits, and broadens the non-EADDRINUSE errno commentary (EADDRNOTAVAIL
for disabled IPv6, EACCES for privileged ports, EAFNOSUPPORT for missing
address families — all treated as "this host doesn't apply", not "port
occupied").
No behavior change to existing callers; all four portUtils tests still
pass.
* docs: add CREDITS.md and surface website-to-hyperframes skill
- New CREDITS.md acknowledging prior art in the browser-based video
rendering space (Remotion) and the ecosystem HyperFrames builds on
(Puppeteer, FFmpeg, GSAP, Hono). Standard OSS practice.
- Adds the `website-to-hyperframes` skill to the skills tables in
README.md, docs/guides/prompting.mdx, and the project template at
packages/cli/src/templates/_shared/CLAUDE.md. The skill ships in
skills/ but was missing from every table.
- Adds `/hyperframes-registry` to the prose mention in the repo
CLAUDE.md.
* feat(hdr): add z-ordered multi-layer compositing with PQ support
Per-frame z-order analysis groups elements into DOM and HDR layers,
composited bottom-to-top. Adjacent DOM elements merge into single
screenshots. PQ (HDR10/smpte2084) support via sRGB-to-PQ LUT with
203-nit SDR reference white. queryElementStacking walks DOM for
effective z-index, groupIntoLayers splits on HDR/DOM boundaries.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(hdr): address review feedback across stack
- Document groupIntoLayers tie-break (V8 stable sort → DOM order).
- Expand layerCompositor docstring: merge rationale, visibility inclusion.
- Add tests: empty input, negative z-index, stable tie-break at equal z.
- Document getEffectiveZIndex CSS stacking-context limitations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
Compositions with HDR video AND DOM overlays (text, graphics, SDR video) couldn't render both correctly — either HDR data was lost (Chrome captures sRGB only) or DOM overlays were missing (FFmpeg pass-through skips Chrome). This PR adds in-memory alpha compositing that combines both.
## What it does
**Per-frame two-pass capture:**
1. **DOM pass** — Chrome screenshots the page with a transparent background (CDP alpha). HDR videos are hidden, leaving transparent holes where they go.
2. **HDR pass** — Pre-extracted native HLG/PQ frames (16-bit PNG from FFmpeg) are read from disk.
3. **Composite** — DOM pixels (sRGB RGBA8) are alpha-composited over HDR pixels (rgb48le) in Node.js memory, with sRGB→HLG/PQ conversion via a 256-entry lookup table.
**Key components:**
- `decodePng()` / `decodePngToRgb48le()` — Pure Node.js PNG decoders (no native dependencies). Support all 5 PNG filter types.
- `blitRgba8OverRgb48le()` — Alpha composite with per-pixel sRGB→HDR LUT conversion. Fast paths for alpha=0 (skip) and alpha=255 (overwrite).
- `initTransparentBackground()` + `captureAlphaPng()` — Split CDP transparent background setup (once) from per-frame screenshot capture (eliminates 2 CDP round-trips per frame).
- Single-pass FFmpeg extraction — All HDR frames extracted in one sequential FFmpeg run (avoids duplicate frames from per-frame `-ss` fast seek).
## Key design decisions
| Decision | Why |
|----------|-----|
| In-memory compositing (not FFmpeg overlay) | Eliminates ~2400 process spawns + temp files per render. Pure pixel math is 10x faster. |
| 16-bit PNG intermediate | Raw `-f rawvideo` loses color metadata, causing moiré artifacts. PNG is self-describing. |
| sRGB→HLG LUT (256 entries) | DOM content is sRGB. Without conversion, it appears orange-shifted in HLG stream. |
| Native HDR detection before extraction | `extractAllVideoFrames` converts SDR→HDR. Pre-extraction probe identifies original HDR sources so only truly-HDR videos get native extraction. |
## Files changed
| File | What changed |
|------|-------------|
| `packages/engine/src/utils/alphaBlit.ts` | **NEW** — PNG decode, sRGB→HDR LUT, alpha compositing (14 tests) |
| `packages/engine/src/services/screenshotService.ts` | Transparent background CDP, `captureAlphaPng()` |
| `packages/engine/src/services/videoFrameInjector.ts` | `hideVideoElements()` / `showVideoElements()` |
| `packages/engine/src/services/streamingEncoder.ts` | Input color space tags for rgb48le |
| `packages/producer/src/services/renderOrchestrator.ts` | Two-pass HDR capture loop, native HDR detection |
## How to test
Render a composition with an HDR video background and text overlays. Both should be visible — HDR video at full quality, text crisp with correct colors (not orange-shifted).
## Stack position
**3 of 6** — Stacked on #265 (HDR output pipeline). This is the foundation for all layered compositing that follows.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Adds the ability to render HDR video output (H.265 10-bit, BT.2020) from HyperFrames compositions. When the renderer detects HDR source video, it automatically switches to the HDR output pipeline — no flags needed.
## What it does
- **Auto-detection** — Probes each video source with `ffprobe`. If any has bt2020/PQ/HLG color metadata, the output switches to H.265 10-bit with correct color tags. SDR-only compositions are unaffected (H.264, bt709).
- **HLG pass-through** — Native HLG pixels from FFmpeg extraction are piped directly to the encoder without conversion. This avoids brightness loss from HLG→linear→PQ conversion (which requires an OOTF system gamma we can't reliably apply).
- **Encoder HDR support** — Both chunk and streaming encoders accept HDR presets: `libx265`, `yuv420p10le`, BT.2020 color primaries, `hvc1` codec tag (required for Apple playback).
- **WebGPU HDR capture (gated)** — A complete WebGPU float16 readback pipeline is implemented and tested but gated behind headed Chrome (headless doesn't expose WebGPU). Ready for future use with WebGPU canvas content.
- **HDR utilities** — `detectTransfer()` (PQ vs HLG), `getHdrEncoderColorParams()`, `analyzeCompositionHdr()`. 15 unit tests.
## Key design decisions
| Decision | Why |
|----------|-----|
| No `--hdr` flag | SDR content encoded as HDR causes orange shift in browsers. Auto-detect eliminates this. |
| HLG pass-through (not HLG→PQ) | Conversion loses brightness without OOTF. Pass-through matches source exactly. |
| `hvc1` codec tag | Apple QuickTime requires `hvc1` (not `hev1`) for HEVC playback. |
| 1-hour streaming timeout | HDR capture at ~6fps needs more time than the default 10-minute FFmpeg timeout. |
## Files changed
| File | What changed |
|------|-------------|
| `packages/engine/src/utils/hdr.ts` | **NEW** — HDR detection, transfer types, encoder params (15 tests) |
| `packages/engine/src/services/hdrCapture.ts` | **NEW** — WebGPU readback, HLG conversion, PQ encode |
| `packages/engine/src/services/streamingEncoder.ts` | HDR presets, raw rgb48le input, color tags |
| `packages/engine/src/services/chunkEncoder.ts` | HDR presets, conditional color tags |
| `packages/producer/src/services/renderOrchestrator.ts` | Auto-detection loop, HDR pass-through capture path |
## How to test
Render a composition with an HDR video source. The output should be H.265 10-bit with HDR metadata visible in `ffprobe` (bt2020, arib-std-b67 or smpte2084). Plays correctly in QuickTime and on HDR displays.
## Stack position
**2 of 6** — Stacked on #258 (SDR/HDR normalization). Provides the encoder infrastructure that phases 1-5 build on.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
* fix(cli): use 'where' instead of 'which' on Windows for FFmpeg and browser detection
- findFFmpeg() now uses 'where ffmpeg' on Windows, 'which ffmpeg' on Unix
- whichBinary() now uses 'where' on Windows, 'which' on Unix
Fixes FFmpeg detection failure on Windows where 'which' command doesn't exist.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(cli): handle multi-line output from Windows 'where' command
Windows 'where' can return multiple paths (one per line) when there
are multiple matches on PATH. Take only the first non-empty line.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(cli): extend Windows 'where' fix to whisper, tts, and clipboard modules
- whisper/manager.ts: whichBinary() now uses 'where' on Windows
- tts/synthesize.ts: findPython() now uses 'where' on Windows
- utils/clipboard.ts: detectProvider() now uses 'where' on Windows
All functions handle multi-line output from 'where' command.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Capture pipeline:
- agentPromptGenerator now writes AGENTS.md + CLAUDE.md (drop legacy
.cursorrules), matching the dual-file convention already used by the
_shared templates in hyperframes init. AGENTS.md is picked up natively by
Cursor, Codex, Gemini CLI, Windsurf, Aider, and Jules; CLAUDE.md covers
Claude Code. Both files share the same content — a capture data inventory
that points agents to the website-to-hyperframes skill.
website-to-hyperframes skill refinements (derived from 8-site regression test):
- Drop slash-command phrasing throughout SKILL.md and step-6-build.md so the
skill works identically across Claude Code (slash), Cursor (auto-discover
by description), and other agents.
- Remove stale HANDOFF.md references from SKILL.md step-7 summary and
reference table — matches the intent of the prior step-7 cleanup.
- step-5-vo: specify narration.txt filename convention (pronunciation-
substituted spoken text; distinct from SCRIPT.md the creative doc).
- step-6 self-review adds three rules derived from actual lint warnings
observed across the 8 regression runs:
- Every <template> root needs data-start + data-duration (catches
root_composition_missing_data_start/duration, seen in 4/8 runs).
- Caption exits need a hard tl.set kill after tl.to(opacity:0), or
per-word karaoke tweens can leave captions stuck on screen
(caption_exit_missing_hard_kill).
- No duplicate media nodes with identical src + start + duration, or
the compiler discovers them twice (duplicate_media_discovery_risk).
Housekeeping:
- .gitignore: add cursor-tests/, basecamp-video/, projects/, videos/ —
local regression-test scratch dirs that should never be committed.
- Remove two broken symlinks from .claude/skills/ that pointed to paths
which never existed in the repo (.claude/skills/ is already gitignored).
Made-with: Cursor
Capture improvements:
- Font weights via document.fonts API + DOM sampling (variable font detection)
- Section background-image extraction (no more false #FFFFFF on hero photos)
- Detected libraries surfaced in CLAUDE.md brand summary
- Structured visible-text.txt with [tag] prefixes, cookie/nav noise filtered
- tokens.json cleaned: removed images/paragraphs/icons (duplicated elsewhere),
filtered sections to heading-only, trimmed cssVariables to design-relevant
- Removed redundant scroll pass in htmlExtractor (2-5s faster per capture)
- Font cap at 20 families, Placeholder/Fallback fonts filtered
CLAUDE.md rewrite:
- Removed prescriptive tone ("use exact strings" → "rephrase freely")
- Removed fluff sections (How to Create, DESIGN.md warning, Example Prompts,
Source Patterns)
- asset-descriptions.md promoted to first data row
- Removed assets-catalog.json from inventory
Skill fixes:
- Dead shader refs → point to packages/shader-transitions/README.md
- Google Fonts import in techniques.md → local @font-face placeholder
- Added Stripe DESIGN.md as light-brand example
- Removed HANDOFF.md generation from step-7
- Updated step-1 for new font weight + visible-text formats
* fix(core): drive adapter seeks when composition has no GSAP timeline
renderSeek returned early when deps.getTimeline() was null, skipping the
onDeterministicSeek call that drives all frame adapters (CSS, WAAPI,
Lottie, Three.js). That meant compositions using any non-GSAP animation
primitive froze on their initial frame during capture.
Now we still quantize the seek time and fire onDeterministicSeek even
without a timeline, so each adapter gets a chance to advance.
GSAP compositions are unaffected — timeline-driven seek still takes the
same path it did before.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(producer): auto-fallback screenshot capture for raf and iframes
Co-Authored-By: Codex <codex@openai.com>
* test(producer): add render compatibility regression fixtures
Co-Authored-By: Codex <codex@openai.com>
* fix(core): scrub CSS animations via WAAPI currentTime
Co-Authored-By: Codex <codex@openai.com>
* test(producer): cover css keyframe renders
Co-Authored-By: Codex <codex@openai.com>
* fix(producer): propagate virtual time into iframe documents
Co-Authored-By: Codex <codex@openai.com>
* test(producer): refresh iframe docker golden
Co-Authored-By: Codex <codex@openai.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Codex <codex@openai.com>
## Summary
Fixes#316 — `hyperframes skills` (and `npx skills add heygen-com/hyperframes`) fails with:
```
■ Failed to clone repository
fatal: active \`post-checkout\` hook found during \`git clone\`
└ Installation failed
```
## Root cause
Two layers stacked:
1. **Git 2.45+ refuses to execute hooks during `git clone` by default.** The opt-in is `GIT_CLONE_PROTECTION_ACTIVE=0` — the env-var name is intentionally explicit about the trade-off.
2. **Users who ran `git lfs install` globally have a post-checkout hook registered at `core.hooksPath`.** When the upstream `skills` CLI shells out to `git clone` to fetch a repo's `skills/` directory, git detects the user's LFS hook and aborts.
The check fires for **any repo**, regardless of whether the cloned repo uses LFS itself — it's protection against the user's own hooks, not the repo's content. Users who have git-lfs installed (very common) hit this for every clone the `skills` CLI does.
## The fix
`hyperframes skills` wraps `npx skills add`. The wrapper now sets `GIT_CLONE_PROTECTION_ACTIVE=0` on the spawned child's env via a single helper (`gitCloneFriendlyEnv`) with a docstring at the call site explaining exactly why. The rest of `process.env` is preserved — proxy settings, extra CA certs, locale, etc. stay untouched.
## What this fix doesn't do (deliberately)
This is the **code-path-we-own** fix. The deeper root cause is that the upstream `skills` CLI (vercel-labs/skills) should set this env var when it shells out to `git clone`. That would fix the bug for every user invoking `skills` directly — not just those who route through our wrapper. An upstream issue should be opened separately; not landing it as part of this PR.
## Users who call `npx skills add` directly
Documented in the new troubleshooting subsection: set the env var manually.
```bash
GIT_CLONE_PROTECTION_ACTIVE=0 npx skills add heygen-com/hyperframes
```
## Tests
`packages/cli/src/commands/skills.test.ts` — 2 cases:
- Every spawned child has `GIT_CLONE_PROTECTION_ACTIVE=0`
- The rest of `process.env` is preserved (not a wiped env)
Uses `vi.mock` on `node:child_process` because ESM doesn't allow live-module `vi.spyOn` on re-exported bindings.
## Docs
`docs/packages/cli.mdx` — new **Troubleshooting** subsection under the `skills` command. Explains both the automatic fix (`hyperframes skills` users are already covered) and the manual workaround (`npx skills add …` users).
## Closes
- #316
* fix(producer): external assets work on Windows (GH #321)
Two Unix-only assumptions in the external-asset pipeline caused every
absolute path on Windows to be rejected as "unsafe" at render time:
1. Containment checks used `child.startsWith(parent + "/")`. On Windows
the separator is `\`, so the predicate is always false unless the
paths are equal — every external asset tripped the safety guard in
`renderOrchestrator.ts`. The reporter saw:
[Render] Skipping external asset with unsafe path:
hf-ext/D:\coder\reactGin\hyperframes\reading\assets\segment_001.wav
Fix: use `path.relative()` through a shared helper
`isPathInside(child, parent)` that normalises separators per-platform
and correctly rejects siblings whose names start with the parent
(e.g. `/foo/bar-sibling` is NOT inside `/foo/bar`).
2. The external-asset key was built as `"hf-ext/" + absPath.replace(/^\//, "")`.
A Windows absolute path (`D:\coder\...`) became
`"hf-ext/D:\\coder\\..."` — and because Node's `path.join` treats a
drive-letter prefix as absolute, `join(compileDir, key)` silently
escaped `compileDir`. Fix: `toExternalAssetKey()` strips the drive
colon and normalises to forward slashes, producing
`hf-ext/D/coder/...` — a pure relative path that `path.join` cannot
promote to absolute on any OS.
Both helpers live in `packages/producer/src/utils/paths.ts` and are
exercised by 14 unit tests covering Unix paths, Windows drive-letter
paths, mixed separators, sibling-prefix confusion, and `..` traversal.
Docs: new "External assets" section in `docs/packages/producer.mdx`
describes detection, sanitised keys, and the cross-platform containment
invariant.
Closes#321.
* fix(producer): address review on #324 — UNC + integration test
Addresses the non-blocking observations from the PR #324 staff review
(https://github.com/heygen-com/hyperframes/pull/324#issuecomment):
1. UNC and extended-length Windows paths.
`toExternalAssetKey` now handles:
- `\\?\D:\very\long\path\clip.mp4` (extended-length) → `hf-ext/D/very/long/path/clip.mp4`
- `\\server\share\file.wav` (plain UNC) → `hf-ext/unc/server/share/file.wav`
- `\\?\UNC\server\share\file.wav` (extended-length UNC) → `hf-ext/unc/server/share/file.wav`
The UNC-collapsed form keeps the server boundary so two different
servers exposing the same share/file name cannot collide under one
relative key. Previously both edge cases silently produced keys with
stray `?` or `:` characters that downstream `isPathInside` rejected —
not a security hole, but a silent drop of user assets.
2. Short-circuit on already-sanitised input.
`toExternalAssetKey("hf-ext/…")` now returns its input unchanged
instead of prepending `hf-ext/` a second time. Makes the helper
genuinely idempotent, which is what the unit test claimed all along.
Renamed the test accordingly.
3. JSDoc caller contract.
`toExternalAssetKey` now documents that it expects canonicalised
input (`path.resolve`'d upstream) and does not strip `..`
components. `isPathInside` at copy time is still the defensive
backstop — called out explicitly in the doc so future callers read
the contract before the code.
4. End-to-end integration test.
`renderOrchestrator.test.ts` gains two seam tests that run the full
external-asset pipeline — build the sanitised key, populate an
`externalAssets` map, invoke `writeCompiledArtifacts`, and assert
both the success path (the file lands under `<compileDir>/hf-ext/…`)
and the escape-rejection path (a malicious `hf-ext/../../etc/passwd`
key does NOT materialise above `compileDir`). `writeCompiledArtifacts`
is exported for the test seam with a clear JSDoc disclaimer that
it's not part of the public API.
22 tests pass across `paths.test.ts` (17) and `renderOrchestrator.test.ts` (5).
Out of scope for this follow-up (tracked as follow-ups):
- Centralising every `startsWith("/")` absolute-path check into a
shared helper across htmlCompiler / audioExtractor / audioMixer /
videoFrameExtractor. Mentioned in the review; touches 5 files and
deserves its own PR.
- Windows CI runner.
The "FFmpeg not found" hint was hardcoded to `sudo apt install ffmpeg`
for any non-macOS platform — Windows users would see an apt command that
doesn't exist on their system, and Red Hat / Arch users got the wrong
package manager too.
`getFFmpegInstallHint()` already exists in browser/ffmpeg.ts (and is
already used by render.ts) and handles darwin / linux / win32 correctly.
Use it here too.
Also rewrite checkFFprobe:
- it previously used `which ffprobe` which is not available on Windows
(cmd uses `where`), so on Windows the check always reported "Not
found" even when ffprobe was on PATH
- run `ffprobe -version` directly instead, which works cross-platform
whenever ffprobe is resolvable on PATH, and surfaces the version
string in the same style as the FFmpeg check
Chrome's "Failed to load resource" message text does not include the failing
URL — it's only on msg.location().url. The previous filter in frameCapture.ts
only checked msg.text(), so every font 404 (e.g. Google Fonts <link> tags
in sandboxed render environments) fell through to the "[non-blocking]"
prefix instead of being suppressed.
Extract the classifier into isFontResourceError() and match against both
text and location.url, and extend the extension match to .ttf/.otf. Adds
a unit test covering the URL-in-location, URL-in-text, and non-font cases.
This is a targeted fix for the render-output noise that PR #311 attempted
to address by adding a ~120-entry SYSTEM_FONTS skip list. That approach
silently shadowed existing FONT_ALIASES (arial→inter, helvetica→inter,
courier new→jetbrains-mono, segoe ui→roboto, etc.) and changed render
output on Linux fleets that don't have those fonts installed. Fixing the
console-log filter here suppresses the noise without changing any font
resolution behavior.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stacked on top of #307. Fixes three mobile UX bugs that made the studio unusable on iPhone Safari — discovered while testing the audio-ownership work from #307 on a physical device.
## Bugs fixed
### 1. Untappable Play button / bottom controls
`#root` was set to `100vh`. iOS Safari reports `100vh` as the **largest** viewport (toolbar hidden) and never shrinks it — so with the toolbar visible, the bottom of the layout sits under it. The Play button + timecode were fully occluded.
### 2. Scrubber not draggable by touch
The seek bar had only `onMouseDown`. Mouse events don't fire for touches on iOS Safari, so nothing responded. You could tap to jump but not drag.
### 3. Safari's horizontal swipe hijacked scrubber drags
Even when the seek bar caught `pointerdown`, `touch-action: manipulation` still let Safari consume horizontal edge-swipes for back-navigation — dragging the scrubber left was impossible.
## What changed
| File | Fix |
|---|---|
| `packages/studio/src/styles/studio.css` | `#root` → `height: 100dvh` with `100vh` fallback. Dynamic viewport height shrinks when the iOS toolbar is visible, so the bottom of `#root` lines up with the visible area. |
| `packages/studio/src/App.tsx` | Two `h-screen` containers → `h-full` so nested children fill the now-dynamic parent instead of asserting `100vh` and overflowing. |
| `packages/studio/index.html` | Added `viewport-fit=cover` so iOS exposes real `env(safe-area-inset-bottom)` values. |
| `packages/studio/src/player/components/PlayerControls.tsx` | Controls row gets `padding-bottom: calc(0.5rem + env(safe-area-inset-bottom))` so it clears the landscape home indicator. Scrubber replaced `onMouseDown` with `onPointerDown` + `setPointerCapture`, plus `touch-action: none` so Safari doesn't hijack horizontal swipes. Added `pointercancel` + window-level `pointerup` fallbacks. |
All desktop code paths are unchanged: `100dvh` falls back to `100vh`, `env(safe-area-inset-bottom)` is `0` off-iOS, Pointer Events subsume Mouse Events on desktop.
## Verified live
Via the `cloudflared` tunnel I ran during review on the factory-series-c-video project:
- iPhone Safari, portrait: Play button now fully visible and tappable. Bottom controls sit just above the URL bar.
- iPhone Safari, landscape: controls clear the home indicator.
- Finger-drag the scrubber left and right: tracks the touch smoothly, finger can leave the 6 px bar height without losing the drag.
- Desktop click-to-seek and click-drag: still work.
- Arrow-key seeking: still works.
## Stacked dependency
Base is `fix/player-audio-ownership-review` (PR #307). Once #307 merges, rebase this branch onto `main` — the changes are fully independent; the stacking is just to avoid waiting on the review for #307 before shipping pure UX wins.
## Test plan
- [x] `tsc --noEmit` on `packages/studio` — clean
- [x] `bun run --filter @hyperframes/studio build` — clean
- [x] Live repro on iPhone Safari via cloudflared tunnel: Play button tappable, scrubber drags with touch
- [ ] Android Chrome sanity pass before release (same Pointer Events code path, but worth eye-balling)
Follow-up to PR #298 addressing @jrusso1020's review. Each item below maps to a point in his comment.
## Significant
### 1\. Drift threshold 150 ms → 50 ms
_mirrorParentMediaTime_ was too loose for lip-synced talking-head content. ITU-R BT.1359 puts A/V perceptibility at ±45 ms; 150 ms sat well inside the "unacceptable" zone. Dropped to 50 ms, extracted as a static constant for clarity.
**Verified live on factory-series-c-video (agent-browser):** steady-state offset under parent ownership sampled five times over 400 ms = `[35.7, 33.5, 31.2, 27.2, 36.9]` ms — below the perceptibility floor. Before this PR the same measurement could drift up to 150 ms before correction.
### 2\. Dynamic sub-composition media proxies
Under parent ownership, a sub-composition that attaches a new `<audio data-start>` mid-playback was correctly silenced in the iframe (sticky `outputMuted`) but had no parent-frame counterpart to play → silent hole in the audio track.
Added a `MutationObserver` on the iframe body watching for `audio[data-start]` / `video[data-start]` additions. New elements are adopted through the same `_adoptIframeMedia` helper the initial scan uses, and if parent ownership is already active the new proxy gets its `currentTime` mirrored and `play()` called immediately (gated on `!this._paused`). Observer disconnects on iframe reload + component disconnect.
### 3\. `bridgeMuted` sticky in `syncRuntimeMedia`
The asymmetry James flagged: `outputMuted` was sticky per-tick, `bridgeMuted` was one-shot via `onSetMuted`. A sub-composition activating after a user mute would briefly play at author volume before the next bridge message.
`syncRuntimeMedia` now accepts `userMuted` and the per-clip loop uses a single combined `shouldMute` gate. One invariant, two inputs.
### 4\. Reset `_audioOwner` on iframe reload
The latch never cleared. On composition switch the player would stay in `parent` ownership against a fresh runtime that hadn't received `set-media-output-muted` and whose autoplay-blocked latch was clean — a brief double-audio window until the next `NotAllowedError` re-promoted (idempotently).
`_onIframeLoad` now resets `_audioOwner = "runtime"`, pauses any parent proxies, and disconnects the old MutationObserver before a fresh one attaches to the new document. If the player had been in `parent` ownership, a corresponding `audioownershipchange` event fires with `reason: "iframe-reload"`.
## Worth addressing
### 5\. Promotion → observable event + reason
Promotion was invisible. Added `CustomEvent("audioownershipchange", { detail: { owner, reason } })` fired on every owner transition. `reason` is either `"autoplay-blocked"` (promote → parent) or `"iframe-reload"` (reset → runtime). Gives host apps an SLO-ready signal for "% of sessions in parent ownership" without exposing internal state.
**Verified live:** dispatching a synthetic `media-autoplay-blocked` in the live studio produced `{ owner: "parent", reason: "autoplay-blocked" }` on the web component exactly once.
### 6\. Parent proxy play() rejection → `playbackerror` event
Previously swallowed silently. Now re-emitted as `CustomEvent("playbackerror", { detail: { source: "parent-proxy", error } })` so embedding apps can recover or fall back.
### 7\. Mobile verification on real hardware
Tested with a tunnel in a real iOS device.
## Test gaps (from review)
- `userMuted` stickiness (mirror of the existing `outputMuted` test).
- **OR invariant** between `outputMuted` and `userMuted` — explicit test that setting one false while the other is true keeps `el.muted === true`.
- **Contract pin:** `syncRuntimeMedia` fires `onAutoplayBlocked` on **every** rejection (no internal dedupe) — so a future refactor can't quietly move the latch and break the caller's posting logic.
- **Caller-side latch pattern:** a 5-rejection simulation with the init.ts-style wrapper posts exactly once.
- **`audioownershipchange`** **dispatch** on promotion + once per transition (no duplicate on idempotent re-promote).
- **Mid-playback promotion:** `_paused = false` at flip time fires `_playParentMedia` immediately.
- **`playbackerror`** **surface** on parent proxy rejection with the right `source` tag.
## Minor
- One-line comment on `_promoteToParentProxy` explaining the `postMessage` async race (the mute lands after ~one message-loop tick; the autoplay gate that triggered promotion keeps the iframe rejecting `play()` during that window, so the double-play bug doesn't reappear).
## What's good (from the review)
Kept as-is — noted for posterity:
- `muted` vs `volume` framing (orthogonal channels).
- Probing reality via `NotAllowedError` instead of `matchMedia('(pointer: coarse)')` / UA sniffing.
- Two orthogonal mute channels.
- Backwards compat (new actions / messages safely ignored by either side).
## Test results
- `packages/core/src/runtime/media.test.ts` — **42 tests pass** (+4 new: `userMuted` sticky, OR invariant, fires-every-rejection, caller-latch dedupe)
- `packages/core/src/runtime/bridge.test.ts` — **15 tests pass**
- `packages/player/src/hyperframes-player.test.ts` — **26 tests pass** (+3 new: `audioownershipchange` dispatch, mid-playback promotion, `playbackerror` surface)
- Typecheck green on `core` + `player`
- `tsup` build green on `core` / `player` / `cli`
- Live factory-series-c-video repro via agent-browser: runtime ownership still zero `volumechange` thrash, zero `PARENT.play()` calls; parent ownership measures 27–37 ms steady-state drift, well inside the 50 ms threshold.
## Test plan
- [x] Unit tests (83 total across touched files)
- [x] Typecheck clean
- [x] Build clean
- [x] Live studio repro on factory-series-c-video: runtime path unchanged, parent path drift tightened
- [x] `audioownershipchange` event fires with correct detail on synthetic autoplay block
- [x] Physical iOS / Android device verification (unchanged since #298)
Closes#309. Full credit to @gigadeniga for the diagnosis — the root cause + proposed fix in that issue are exactly what landed here.
## The bug
\`npx hyperframes preview\` failed deterministically on Crostini (ChromeOS Linux) with \`Ports 3002–3101 are all in use\`, even when nothing was actually listening on any of them.
## Why
\`testPortOnAllHosts\` ran four probes in parallel:
\`\`\`ts
const hosts = ["127.0.0.1", "0.0.0.0", "::1", "::"];
const results = await Promise.all(hosts.map((h) => isPortAvailableOnHost(port, h)));
\`\`\`
Each probe binds a socket and then calls \`server.close()\`. Close is async — the socket stays open until its callback fires on the next event-loop tick. While it's open, the wildcard binds (\`0.0.0.0\`, \`::\`) that include the loopback address race the still-open loopback socket and return \`EADDRINUSE\` spuriously. On Crostini this happens 100% of the time; other Linux configs hit it intermittently; macOS is less predictable. Net effect: every port in the 100-port scan range appears busy and the preview refuses to start.
Reproduces on any Linux box with the standalone snippet from the issue:
\`\`\`
127.0.0.1: OK
0.0.0.0: EADDRINUSE ← false positive
::1: OK
::: EADDRINUSE ← false positive
\`\`\`
## Fix
Serialize the probes. Each socket is fully closed before the next opens, eliminating the race window entirely.
\`\`\`ts
for (const host of hosts) {
const available = await isPortAvailableOnHost(port, host);
if (!available) return false;
}
return true;
\`\`\`
Kept the four-host check rather than collapsing to just \`0.0.0.0\` + \`::\` — the multi-host coverage is load-bearing for the devbox / SSH-forwarding case where a port is free on loopback but held on the wildcard. Sequentializing is the smaller, less-behaviourally-affecting fix.
## Regression tests
\`packages/cli/src/server/portUtils.test.ts\` — three cases binding real sockets, no mocks:
- **Returns true for a genuinely free port** — directly reproduces the Crostini bug; would fail on Linux against the parallel implementation.
- **Returns false when the port is occupied on \`0.0.0.0\`** — confirms the multi-host check still catches the devbox scenario.
- **Releases each probe socket before the next run** — two back-to-back calls for the same free port both return true, pinning the sequential contract against future refactors that might try to reparallelize for perf.
## Test plan
- [x] \`bunx vitest run packages/cli/src/server/portUtils.test.ts\` — 3/3 pass
- [x] Full CLI suite — 109/109 pass
- [x] \`tsc --noEmit\` clean
## Notes
- Independent of any version bump; ship whenever.
- Probing 4 hosts serially adds at most ~tens of milliseconds per port on the scan (binds are very fast on loopback). The worst-case cost shows up when the first port in the range is free — previously 1 parallel round-trip, now 4 sequential — and it's imperceptible (\`preview\` bind is a one-time startup cost, not a hot path).
## Summary
Fixes the double-voice issue in studio preview where narration plays twice with a drifting offset (measured 23ms → 80ms over a 28s clip).
## Root cause
Two audio pipelines were playing the same source in parallel:
1. The iframe runtime played `<audio data-start>` elements via `syncRuntimeMedia` — the intended path.
2. `<hyperframes-player>` also created parent-frame `<audio>` copies on iframe load and auto-played them in response to every runtime `state` message.
The existing `_muteIframeMedia` tried to silence the iframe copies via `el.volume = 0`, but `syncRuntimeMedia` re-asserts `el.volume` from `data-volume` every tick, so the mute never held. Studio seeks went through `__player.seek()`, which only updated the iframe timeline; parent copies kept their stale `currentTime` and drift compounded across seeks.
Confirmed via agent-browser instrumentation on `factory-series-c-video`:
- 6 `volumechange` events per play cycle (mute-fight signature)
- Both copies audible at `volume=1`, offset growing 23ms → 80ms
- Every seek widened the drift further
PR #295 (v0.4.2) actually **made it audible** — before that, parent copies 404'd on the wrong URL and played silently. Fixing the URL exposed the latent double-playback.
## Fix
Explicit single-owner audio ownership between `<hyperframes-player>` and the runtime.
- **Default ownership is `runtime`**: iframe drives audible playback; parent proxies stay paused and inert. Matches every desktop / studio code path. No parent `play()`, no `volumechange` thrash.
- **On `NotAllowedError`** from the runtime's `play()` attempt (autoplay-gated iframes), the runtime posts `media-autoplay-blocked` once. The player promotes to `parent` ownership: sends `set-media-output-muted: true` to the runtime, starts parent proxies, mirrors `currentTime` from state messages with a 150ms correction threshold.
Two orthogonal mute channels replace the volume fight:
| Channel | Purpose |
|---|---|
| `set-muted` | User's mute preference (existing, unchanged) |
| `set-media-output-muted` | Internal ownership handoff (new) |
`syncRuntimeMedia` now accepts `outputMuted` and asserts `el.muted = true` per active tick — sticky against sub-composition media that arrives mid-playback. Uses native `muted` (orthogonal to `volume`) so no other code path can clobber it.
## Why this shape
- **Single owner, explicit transition.** No races, no tug-of-war.
- **Probes reality, not device class.** We flip on an actual `NotAllowedError`, not on `matchMedia('(pointer: coarse)')` or user-agent sniffing.
- **Uses `muted` instead of abusing `volume`.** `muted` is orthogonal to `volume`; `syncRuntimeMedia` doesn't write to it; author / user settings stay intact.
- **Parent proxies become a thin mirror.** Under parent ownership, their `currentTime` is slaved to the iframe timeline via state messages — no independent drift.
- **Backwards compatible.** Old runtimes without the new bridge action ignore the message; old players without the new message just get the previous behavior.
- **Capture engine unaffected** — it bypasses both DOM pipelines and muxes audio from source files.
## Files changed
- `packages/core/src/runtime/types.ts` — `set-media-output-muted` action + `media-autoplay-blocked` outbound message types.
- `packages/core/src/runtime/state.ts` — `mediaOutputMuted` + `mediaAutoplayBlockedPosted` fields.
- `packages/core/src/runtime/bridge.ts` — route new action to `onSetMediaOutputMuted`.
- `packages/core/src/runtime/media.ts` — `outputMuted` param asserts `el.muted = true` per tick; `NotAllowedError` detection fires `onAutoplayBlocked`.
- `packages/core/src/runtime/init.ts` — wire new bridge handler; coordinate with `set-muted`; post `media-autoplay-blocked` once per session.
- `packages/player/src/hyperframes-player.ts` — `_audioOwner` state; delete `_muteIframeMedia`; `_promoteToParentProxy`; mirror parent `currentTime`; gate all parent play/pause/seek on ownership.
## Verified end-to-end with agent-browser on `factory-series-c-video`
**Runtime ownership (default — desktop studio):**
| | Before | After |
|---|---|---|
| `PARENT.play()` calls per play cycle | 1 | **0** |
| iframe `volumechange` events | 6 | **0** |
| Audible streams | 2 (drifting) | **1 (iframe)** |
**Parent ownership (simulated autoplay block — direct message):**
| | Value |
|---|---|
| iframe audio | `muted=true`, `volume=1` (untouched) |
| parent audio | `muted=false`, `volume=1`, audible |
| Parent ↔ iframe `currentTime` offset | ~6 ms steady state |
| Offset > 150 ms | corrected by mirror sync |
**Mobile path simulated with iPhone 14 emulation + injected `NotAllowedError` from iframe `<audio>.play()`:**
Event timeline captured via agent-browser instrumentation:
```
t=0.0 ms IFRAME.play() called ← runtime attempts playback
t=0.4 ms IFRAME.play() REJECTED: NotAllowedError ← simulated mobile gate
t=0.4 ms →IFRAME bridge set-media-output-muted=true ← player promotes
t=0.6 ms PARENT.play() called ← parent proxy starts
t=0.8 ms ←IFRAME msg media-autoplay-blocked ← runtime signal
t=1.0 ms PARENT.play() resolved ← audible
t=1.3 ms IFRAME muted=true, volume=1 ← iframe silenced via native muted
```
Steady state at t=4 s under promoted parent ownership:
| Element | currentTime | paused | volume | muted |
|---|---|---|---|---|
| Parent audio | 4.060 s | false | 1.0 | **false** (audible) |
| Iframe audio | 4.068 s | false | 1.0 | **true** (silent) |
**Offset: 8 ms**, single audible stream, orthogonal mute channel respected.
## Test plan
- [x] `bunx vitest run` under `packages/core` — **467 / 467 pass** (incl. 4 new `media.test.ts` + 2 new `bridge.test.ts`)
- [x] `bunx vitest run` under `packages/player` — **23 / 23 pass** (3 rewrites for new contract, 2 new for promotion flow)
- [x] `bun run build` — all packages green
- [x] Fresh preview + browser repro on `factory-series-c-video`:
- [x] Runtime ownership: single audio stream, no drift
- [x] Parent ownership promotion via direct `media-autoplay-blocked` message: iframe muted, parent audible
- [x] iPhone 14 emulation + injected `NotAllowedError`: full promotion chain verified in ~1 s, 8 ms steady-state offset
- [x] No `volumechange` thrash in either ownership mode
- [x] One round of QA on a physical iOS / Android device before release — exercises real `NotAllowedError` path (expected behavior identical to simulation above)