When the host doesn't have a usable GPU (CI containers, eval rigs without
GPU passthrough, dev VMs), Chrome's hardware-mode WebGL flags
(`--use-gl=egl/metal/d3d11`) silently leave WebGL unavailable —
`getContext("webgl")` returns null, three.js' WebGLRenderer dies, the
canvas stays black. Surfaced today by Abhay's c2v-eval failing on a
docker render of an hf bundle that uses three.js + a custom fragment
shader.
The fix that's been there: `--use-gl=angle --use-angle=swiftshader` (CPU
software WebGL, ~5-50× slower but pixel-identical). The engine already
exposed `browserGpuMode: "software"` for this. The gap was discovery —
users had to know to pass `--no-browser-gpu` on no-GPU hosts.
This change adds `browserGpuMode: "auto"` (now the CLI default for local
renders): on first launch in the process, probe Chrome with hardware
args, check `canvas.getContext("webgl") !== null`, cache the result.
~1-2 s on first render, free on every subsequent render in the same
worker. Hardware GPUs keep their fast path; no-GPU hosts get SwiftShader
without ceremony.
Behaviour matrix:
- No flag, no env, local → "auto" (NEW default)
- `--browser-gpu` → "hardware" (force; errors if no GPU)
- `--no-browser-gpu` → "software" (force SwiftShader)
- `PRODUCER_BROWSER_GPU_MODE` → "hardware" / "software" / "auto" / unset
- Docker mode → forced "software" (unchanged)
Engine-config default stays "software" (conservative for embedders); the
"auto" default lives in the CLI's `resolveBrowserGpuForCli` so producer
embedders aren't surprised by a probe-on-launch.
Also adds `--enable-unsafe-swiftshader` to the software flag set —
Chrome 120+ deprecated implicit SwiftShader fallback and emits a
deprecation warning unless the flag is set explicitly. Despite the
"unsafe" name this is exactly the pre-deprecation behaviour; the rename
is about Chrome's threat model on the open web, not about the rendering
itself.
Verification:
- Engine 535/535 + CLI 256/256 (incl. new probe tests + tri-state CLI test)
- Empirical: probe on this no-GPU devbox returns "software" in 240 ms,
cached 0 ms on subsequent calls
- Format / lint / typecheck clean across all packages
Refs the Abhay/Slack thread on c2v-eval rendering without a GPU node.
The add command's tag fallback (e.g. `hyperframes add html-in-canvas`)
uses the same 24h cached manifest as single-item lookups. When new items
are added to the registry, the stale cache returns an incomplete item
list, causing tag resolution to find zero matches.
Pass skipCache: true in the tag fallback path so it always fetches the
latest manifest from the registry.
* docs: group VFX blocks under HTML-in-Canvas, captions under Captions in sidebar
* docs: add HTML-in-Canvas guide, Chrome flag disclaimer, remove captions
- Add docs/guides/html-in-canvas.mdx — comprehensive guide covering the
API, feature detection, re-capture patterns, and catalog blocks
- Add Chrome flag Warning banner to every HTML-in-Canvas block page
- Remove captions blocks from registry (will ship separately)
- Add html-in-canvas guide to docs navigation (top of Guides section)
* feat: update liquid glass, portal, shatter with HyperFrames branding
Replace generic placeholder content with HyperFrames-themed text:
- Liquid Glass: 'Ship videos 10x faster' with stats and gradient text
- Portal: 'Write HTML / Render Video' with HyperFrames nav
- Shatter: 'HTML is Video' with render speed/file size metrics
Re-rendered and uploaded preview videos to S3.
Merges main's refactored capture (CaptureSceneOptions, forceVisible,
stabilizeTransformedBoxShadows, foreignObjectRendering fallback) with
our HTML-in-Canvas drawElementImage capture path. The native capture
tries first and falls back to html2canvas on failure.
hyperframes add now auto-detects whether the argument is a single
block name or a tag. No --tag flag needed:
hyperframes add html-in-canvas # installs all 7 html-in-canvas blocks
hyperframes add captions # installs all 5 caption blocks
hyperframes add vfx-shatter # installs one block
When the name doesn't match a registry item, the CLI falls back to
tag-based resolution and bulk-installs all matching blocks.
Also fixes tag assignments:
- VFX blocks tagged "html-in-canvas" (not "vfx")
- Caption blocks tagged "captions" only (no html-in-canvas)
All 12 blocks now share the 'html-in-canvas' tag, enabling:
hyperframes add --tag html-in-canvas
This installs all VFX + caption blocks in one command.
Install all registry blocks matching a tag in one command:
hyperframes add --tag vfx # installs all 7 VFX blocks
hyperframes add --tag captions # installs all 5 caption blocks
The resolver loads each item's full manifest to check tags, then
installs matching blocks sequentially. Failed items are skipped
with a warning so one broken block doesn't abort the batch.
Also supports JSON output for CI: hyperframes add --tag vfx --json
Add 12 new registry blocks across two new categories:
**VFX Blocks (7):**
- vfx-text-cursor: dramatic text reveal with cursor glow and chromatic shadows
- vfx-liquid-background: organic liquid simulation with vertex displacement
- vfx-iphone-device: real GLTF iPhone + MacBook with live HTML screens
- vfx-magnetic: magnetic field particle visualization
- vfx-portal: dimension breach portal transition
- vfx-liquid-glass: Voronoi glass fracture with parallax reveal
- vfx-shatter: glass shatter with physics-driven fragments
**Caption Blocks (5):**
- captions-slam: bold uppercase scale-pop with back.out overshoot
- captions-karaoke: word-by-word color reveal on frosted glass pill
- captions-minimal: clean Netflix-subtitle style fade
- captions-bounce: playful elastic bounce with colorful word pills
- captions-cinematic: elegant slow fade with letter-spacing animation
All blocks include registry-item.json and are registered in registry.json.
VFX blocks use experimental stability, caption blocks use stable.
* fix(runtime): clear play guard after hard seek to prevent audio desync on scrub
When scrubbing the timeline during playback, syncRuntimeMedia detects
the offset jump and hard-seeks the media element. But the in-flight
play() guard (playRequested WeakSet) from the previous play() call
prevented the next sync tick from re-issuing play() — leaving the
element paused at the new position for 50-150ms while the GSAP timeline
continued advancing. This caused audible audio desync after every scrub.
Fix: clear playRequested on the element after a hard seek so the very
next sync tick can re-issue play().
Also adds a lint rule (video_audio_double_source) that catches
compositions where an unmuted <video> and a separate <audio> point to
the same source — a pattern that causes double playback at runtime.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(runtime): detect failed seeks past MP3 buffer and force full fetch
Root cause: streaming MP3 with preload="metadata" only buffers the first
~15 seconds. Seeking past the buffered range silently fails — currentTime
stays at 0 while the timeline advances, causing permanent audio desync
that only a page refresh fixes.
Three changes:
1. Move preload="auto" enforcement to run for ALL active elements on
every sync tick (not just during play). This catches elements whose
preload was overridden after init.ts set it.
2. After a hard seek, check if currentTime actually reached the target.
If not (drift > 0.5s), call load() once to force the browser to
fully fetch the media and build a complete seek index.
3. Clear the load-retry guard when the clip leaves its active window
so re-entry can retry if needed.
Reproduced on hyperframes.dev Hermes launch video: vo.mp3 buffered to
15.96s, seeking to 20s failed silently. bg-music.wav (fully buffered)
was unaffected.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per CodeQL's `js/bad-tag-filter` recommendation, replace the regex-based
`<script>` body extraction with a `parseHTML` + `querySelectorAll`
walk. The rule explicitly says "use a parser library" — and linkedom
is already imported in this file, so the diff is small.
This eliminates the regex entirely, so the rule can no longer fire on
this site (instead of chasing whitespace / case / trailing-content
edge cases one at a time).
CodeQL still flagged `</script\s*>` as too narrow — the rule wants
tolerance for `</script\t\n bar>` (HTML parser treats trailing content
in a close tag as part of the tag). Switched to `</script[^>]*>` for
full coverage.
The bundler still always emits the canonical `</script>`; this is
test-side hardening, not a runtime fix.
CodeQL's `js/bad-tag-filter` rule flagged `</script>` as too strict —
`</script >` (with whitespace before `>`) is valid HTML and would slip
past the matcher. Changed to `</script\s*>` for full defense-in-depth.
The bundler always emits the canonical form, so no real-traffic miss —
this is hardening the test's parse-loop, not fixing a downstream bug.
Addresses CodeQL alert on #641.
CodeQL flagged the inline `<script>...</script>` regex as case-sensitive,
which would miss `<SCRIPT>` tags. The bundler always emits lowercase, so
this is a defense-in-depth fix matching the `/i` flag already used by the
sibling regexes in this file (lines 37 & 75).
Addresses CodeQL review on #641.
Per @vai-bot's review on hf#641:
Important #1: dead `src=""` substitution sites
=============================================
Now that `bundleToSingleHtml` inlines the runtime IIFE by default, the empty
`src=""` placeholder is never emitted in the no-env-var path — the 5 downstream
substitution sites that grep for `src=""` were dead.
Two of them (studio dev server + studio vite preview) genuinely WANT the
placeholder so they can hot-reload a local /api/runtime.js endpoint without
re-inlining ~150 KB on every composition edit. Three of them (CLI validate,
snapshot, layout) were just doing the same inlining the bundler already does.
Resolution:
- Add a `runtime: "inline" | "placeholder"` option to `BundleOptions`. Default
is "inline" (matches the self-contained-bundle promise the function name
makes). The two studio surfaces explicitly pass `{ runtime: "placeholder" }`
to opt in.
- studioServer.ts + studio/vite.config.ts: pass the option, keep their
existing string-replace logic unchanged.
- validate.ts + snapshot.ts + layout.ts: delete the now-redundant runtime
substitution code (regex never matches the new inlined-runtime shape).
Important #2: joinJsChunks ASI hazard
======================================
The new helper appended `;` to chunks not already ending in `;` and joined
on `\n`. If a chunk ended with a `// line comment`, the appended semicolon
was eaten by the comment, leaving the next chunk's first statement attached
to the previous chunk's last expression — exactly the ASI hazard the helper
exists to prevent.
Fix: append `\n;` instead of `;` for chunks not already terminated. The
newline closes the line comment, the standalone `;` becomes the statement
separator. For typical chunks (already ending in `;`), output is unchanged
— still clean `\n`-joined chunks with no bare-semicolon lines.
Also added a trailing `;` to `wrapScopedCompositionScript`'s IIFE close
(`})()` → `})();`) so composition scripts join cleanly without falling
through to the `\n;` fallback.
New test: regression guard at the chunk boundary verifies every inline
script body in the bundle parses cleanly via esbuild even when a source JS
file ends with a line comment.
Verification
============
- `bun run --filter @hyperframes/core test` — 653/653 pass
- `bun run --filter @hyperframes/cli test` — 243/243 pass
- `bun run --filter @hyperframes/{core,cli,studio} typecheck` — clean
- `bunx oxfmt --check` + `bunx oxlint` on all touched files — clean
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three issues in `bundleToSingleHtml` reported via Abhay's LLM-based code-validity
eval against the bundled output. Each is independently small; they share a single
PR because they're all artifacts of the bundler-output shape.
1. Empty `src=""` runtime placeholder (real bug)
`htmlBundler.ts:injectInterceptor` emitted
`<script data-hyperframes-preview-runtime="1" src=""></script>`
when no `HYPERFRAME_RUNTIME_URL` was configured. Empty `src` resolves to the
page URL itself; Chrome flags this as an infinite-fetch hazard. Three other
consumers (studioServer, validate, snapshot) post-process the placeholder to
substitute either a real URL or an inlined body — `bundleToSingleHtml` did
not, so the bundle wasn't actually self-contained despite the function name.
Fix: when no URL is configured, inline the runtime IIFE directly via
`getHyperframeRuntimeScript()`. Otherwise emit `src=…` as before.
2. Bare-semicolon lines between joined JS chunks (cosmetic)
Three sites used `chunks.join("\n;\n")` (body-script coalesce, local JS,
composition scripts) which produced a lone `;` on its own line between
chunks. Valid JS but a code smell. Replace with a `joinJsChunks()` helper
that ensures each chunk ends in `;` and joins on `\n`.
3. Empty `catch (_err) {}` in compositionScoping.ts (lint-noisy)
The `_err` underscore prefix signals "intentionally swallowed" but bundle-time
linters often don't honor that convention. Replaced with `catch { /* ... */ }`
(no binding, explanatory comment) — same behavior, no rule fires.
Tests: 2 new regression guards (runtime-not-empty-src, no-bare-semi) plus
existing tests updated to reflect the new inlined-runtime shape (the previous
"runtime block must not contain getElementById" assertion no longer holds
because the inlined body itself uses getElementById; replaced with a more
specific "author script not merged into runtime tag" check).
Issue #4 from the original report (Unterminated string at line 1111 col 18,
char 65497) was not directly reproducible after applying these fixes — esbuild
parses all 4 inline scripts in the rebundled output cleanly. The unterminated-
string symptom was likely a downstream artifact of the bare-semicolon joining
or the empty-src placeholder confusing the lint tool. If the original symptom
persists on a clean re-run against the fixed bundle, will open a follow-up PR
with a focused repro.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Extract applyMask helper from postprocess and add 5 unit tests pinning
the contract this PR is selling: fg.alpha + bg.alpha === 255 per pixel,
RGB triples byte-identical between fg and bg, and bg=null path leaves
the bg buffer untouched. Without these, a future postprocess change
(mask threshold, premultiplied alpha, gamma) could silently break the
inverse-alpha relationship and the existing plumbing tests would all
still pass.
- Add stdin 'error' listener inside spawnFfmpeg. If either encoder dies
mid-render, Node emits an unhandled error on the dead writable on the
next .write() and crashes the CLI before waitForExit's reject path
can surface the encoder's stderr tail. Doubled encoder count = doubled
failure surface, so this is worth pinning down.
- Tighten stdio param to a 3-tuple so an accidental 1-element array fails
at type-check.
- Sharpen backpressure comment: write→true means "highWaterMark not
exceeded," not "libuv flushed." Reuse-without-corruption is safe only
because session.process is slow enough that libuv drains in between.
Addresses review on PR #637.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both `composition_file_too_large` and `timeline_track_too_dense` previously
said "Agents produce better results when large scenes are split into smaller
sub-compositions." The audience-flavored framing ("Agents produce better
results") doesn't tell a reader (agent or human) WHY smaller is better.
Reframe to concrete properties of smaller compositions: easier to read,
iterate on, and diff. The fixHint already covers the inspect/revise/validate
detail; the message now leads with a tight reason.
Per Abhay in #C0ACCNHLG3U:
> "an agent reading 'Agents produce better results' sounds weird. We should
> give the agent an actual reason why smaller is better for them."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Emit an inverse-alpha background plate alongside the cutout in a single
inference pass. Same source RGB, alpha = 255 − mask. Dual-encoder pipeline
runs in parallel; both outputs share the same --quality preset.
This is a hole-cut plate (subject region transparent), not an inpainted
clean plate — composite something opaque under it to fill the hole.
Docs and skill cover when each is the right tool.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Replace the flat 30s upload timeout with a size-adaptive calculation: `max(120s, bytes / 500KB/s)`
- Metadata requests (presigned URL, complete) keep the original 30s timeout
- Companion to the backend change removing the 64 MB upload limit in experiment-framework
## Context
With the backend size limit removed, large projects (78 MB+) need proportionally longer to upload. A 78 MB project now gets ~164s, a 500 MB project ~17 min. The old 30s timeout would abort any upload over ~15 MB on a typical connection.
## Test plan
- [x] All 4 existing vitest tests pass
- [x] Build succeeds, no type errors
- [x] Lint + format pass (oxlint + oxfmt)
- [x] Timeout values verified for 10/78/200/500/1000 MB archives
## Problem
The Blue Sweater intro HyperFrames project was only available as a standalone exported project zip. It was not installable from the public registry or visible in the Catalog Showcases group.
## What this fixes
Adds `blue-sweater-intro-video` as a registry block with its composition, avatar image, and sound mix asset. The block is exposed through the generated catalog page, `docs/public/catalog-index.json`, and the Showcases navigation.
The manifest and generated catalog page credit the creator as [Joe Sai](https://x.com/_blue_sweater_).
## Root cause
Catalog-visible blocks are driven by `registry/registry.json`, each block's `registry-item.json`, generated docs/catalog files, and CDN-hosted preview media. The exported project had a valid standalone composition, but it had not been converted into that registry/catalog contract or uploaded to the docs preview CDN.
## Verification
### Local checks
- `bun install`
- `bun run build`
- `bunx tsx scripts/generate-catalog-pages.ts`
- `bun run generate:catalog-previews -- --only blue-sweater-intro-video`
- `bun packages/cli/src/cli.ts add blue-sweater-intro-video --dir /tmp/hf-blue-sweater-install-test --no-clipboard --json` against a locally served registry
- `bun packages/cli/src/cli.ts lint /tmp/hf-blue-sweater-install-test` returned 0 errors and 3 static GSAP overlap warnings from the supplied timeline/parser path
- `bun packages/cli/src/cli.ts validate /tmp/hf-blue-sweater-install-test --timeout 5000` returned 0 runtime errors and 0 warnings, with contrast audit warnings only
- `bun packages/cli/src/cli.ts inspect /tmp/hf-blue-sweater-install-test --at 0.5,2.5,5.5,9.8,11.2 --json` returned 0 layout issues
- `bun packages/cli/src/cli.ts render /tmp/hf-blue-sweater-install-test --output /tmp/hf-blue-sweater-install-test/blue-sweater-intro-video-render.mp4 --fps 24 --quality draft --workers 3`
- `ffprobe` reported the installed render duration as `12.000000`
- `bunx oxfmt --check registry/registry.json registry/blocks/blue-sweater-intro-video/registry-item.json registry/blocks/blue-sweater-intro-video/blue-sweater-intro-video.html docs/docs.json docs/public/catalog-index.json docs/catalog/blocks/blue-sweater-intro-video.mdx`
- `git diff --check`
- `bunx vitest run packages/cli/src/commands/add.test.ts packages/core/src/registry/types.test.ts`
### Browser verification
- Started a real local HyperFrames preview for the installed test project.
- Used `agent-browser` to open `http://localhost:5198/api/projects/hf-blue-sweater-install-test/preview` at 1920x1080.
- Verified the runtime registered `install-test` and `blue-sweater-intro-video` timelines.
- Sought the block to the final card and verified `@_blue_sweater_` and the following state were visible.
- Recorded an `agent-browser`-driven full animation pass; `ffprobe` confirmed a 1920x1080 WebM with 110 video frames.
- Checked the fresh `agent-browser` session for page errors after the direct preview flow: `errors: []`.
- Used `agent-browser` to load an HTML page with the exact generated CDN `video`/`poster` URLs; the browser reported `readyState: 4`, `videoWidth: 1920`, `videoHeight: 1080`, and `paused: false`.
### CDN upload
Uploaded the generated preview media with AWS CLI to the existing docs image bucket path:
- `s3://heygen-public/hyperframes-oss/docs/images/catalog/blocks/blue-sweater-intro-video.mp4`
- `s3://heygen-public/hyperframes-oss/docs/images/catalog/blocks/blue-sweater-intro-video.png`
Verified both public CDN URLs return `HTTP 200` with correct content type and immutable cache headers:
- `https://static.heygen.ai/hyperframes-oss/docs/images/catalog/blocks/blue-sweater-intro-video.mp4` (`video/mp4`)
- `https://static.heygen.ai/hyperframes-oss/docs/images/catalog/blocks/blue-sweater-intro-video.png` (`image/png`)
## Notes
- Local-only browser proof artifacts:
- `/tmp/hf-blue-sweater-browser-proof/fresh-final-card.png`
- `/tmp/hf-blue-sweater-browser-proof/fresh-browser-flow.webm`
- `/tmp/hf-blue-sweater-cdn-check.png`
- Local-only installed render artifact:
- `/tmp/hf-blue-sweater-install-test/blue-sweater-intro-video-render.mp4`
Four regression tests (font-variant-numeric, many-cuts, missing-host-comp-id,
variables-prod) failed on this PR with `Unable to parse PSNR output at <last
checkpoint>s`. Root cause: the harness derived all 100 checkpoints from the
*rendered* video's container duration, then asked ffmpeg's PSNR filter to
compare the same frame index from both videos.
The encoder changes earlier in this PR add `-avoid_negative_ts make_zero` to
the mux step. With AAC audio that shifts the first audio sample to t=0
instead of the encoder-delay offset, extending reported container duration
by ~20ms without changing video frame count. For the four failing tests,
the i=99 checkpoint then landed on a frame index that exists in the rendered
video but not in the snapshot baseline (e.g. round(2.98998 * 24) = 72 in a
72-frame baseline). ffmpeg's PSNR filter ran on zero matched frames and
emitted no `average:` line, so the parser threw.
Fix: probe both videos and use min(rendered, snapshot) duration when
spreading checkpoints. This is the correct semantics for symmetric PSNR
comparison anyway — both videos must have a frame at every sampled time.
The change is local to the harness; no encoder behavior changes, no
baselines regenerated.
Other regression tests with audio (chat, sub-composition-video,
vignelli-stacking) passed because their checkpoint-99 frame index landed
inside the baseline's frame range with several frames of slack. The four
failing tests had round-number durations where a 20ms drift was enough to
push the last checkpoint past `nb_frames - 1`.
The model removes background from any video with a person — we tested
with avatars because they were convenient, but anyone can bring a
talking-head clip, presenter footage, vlog, etc. Replace avatar-specific
filenames (avatar.mp4 / brandon.mp4) with neutral subject.mp4 (or
presenter.mp4 in the text-behind-subject example) and rephrase
copy that read as if avatars were the only use case.
Touches docs/guides/remove-background.mdx, hyperframes-media SKILL.md,
and hyperframes/patterns.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- engine/chunkEncoder, engine/streamingEncoder: extend `-bf 0` to GPU h264
paths (nvenc, qsv, vaapi) and `-b_strategy 0` for qsv so GPU-encoded
outputs avoid negative-DTS freezes too — not just SW libx264.
- engine/videoFrameExtractor: detect mid-path traversal (e.g.
`assets/../../foo.mp4`) by normalizing first and re-anchoring at the
project root. Adds a regression test.
- engine/videoFrameExtractor: dedupe stderr "src not resolvable" warnings
by `video.src` so a comp with N broken sources logs once, not N times.
- engine/videoFrameExtractor.test: drop dynamic `require("node:fs")`,
use ES `import { writeFileSync } from "node:fs"`.
- engine/ffprobe: extract `readTagCI` helper for case-insensitive ffprobe
tag reads (will recur for other libavformat-versioned sidecar tags).
- cli/background-removal/pipeline: collapse Quality / QUALITIES /
QUALITY_CRF / DEFAULT_QUALITY / isQuality surface using
`Quality = keyof typeof QUALITY_CRF`.
- producer/renderOrchestrator: replace `v.src.startsWith("/")` with
`isAbsolute(v.src)` in the HDR probe path so Windows absolute paths
(`C:\...`) aren't treated as relative — matches the audioMixer guard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tag-based alpha detection (alpha_mode / ALPHA_MODE / pix_fmt yuva*) is
fundamentally brittle. Failure modes seen in the wild:
- case-sensitivity across ffmpeg versions (alpha_mode vs ALPHA_MODE)
- older muxers that omit the sidecar tag entirely
- mp4-as-webm rewraps that drop the tag
- ffprobe reporting yuv420p for VP9-with-alpha because the alpha plane
lives in a Matroska BlockAdditional sidecar, not the main pix_fmt
Each of those silently strips alpha at extraction time. The bug doesn't
surface until the rendered output is missing layers — frustrating to debug,
silent in stdout. The previous case-insensitive fix patched one of the
failure modes; this commit removes the class.
The robust alternative is codec-based: any bitstream that CAN carry alpha
(VP9, VP8, ProRes 4444) gets the alpha-aware decoder and PNG output by
default, regardless of what the tag says. The cost is a small file-size
increase on opaque VP9/VP8 sources (cached PNGs vs JPGs); the benefit is
no class of silent alpha loss from tag misdetection.
- Adds codecMayHaveAlpha() + decoderForCodec() helpers and exports them.
- Updates extractVideoFramesRange to force libvpx-vp9 / libvpx for VP9 / VP8
unconditionally (was: only when metadata.hasAlpha).
- Updates resolveFrameFormat to default to PNG for any alpha-capable codec
(was: only when metadata.hasAlpha).
- +4 unit tests covering the codec table.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Locks in the case-insensitive behavior alongside the existing alpha_mode
(lowercase) test. If either path regresses, the producer would silently
extract alpha-having webms as opaque JPGs and the injected <img> overlays
would cover every element below them on the z-stack — a bug that doesn't
surface in the studio preview, only in production renders.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Newer libavformat builds write the VP9-alpha sidecar tag as 'ALPHA_MODE'
(uppercase); older builds write 'alpha_mode'. ffprobe.ts only checked the
lowercase form, so files produced by recent ffmpeg encoders (including the
output of 'hyperframes remove-background' itself) were misclassified as
having no alpha channel. Knock-on effect: the producer extracted them as
JPGs (no alpha), the injected <img> overlays were fully opaque rectangles,
and any element below them on the z-stack (text, captions, other layers)
silently disappeared from the rendered output — even though the studio
preview rendered the same composition correctly via native <video> playback.
Symptom in our repro: a text-behind-subject composition showed the
headline correctly in studio preview but the production render covered
the headline entirely with the opaque avatar image.
Fix: read videoStream.tags.alpha_mode OR videoStream.tags.ALPHA_MODE.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skill (hyperframes-cli): three-pattern table (cutout-over-different-scene
vs over-its-own-source vs over-different-take) + the two non-obvious rules
(wrap video in non-timed div for opacity control, both videos data-start=0
for sync). Skill (hyperframes/patterns): worked text-behind-subject example.
Docs: --quality flag, compositing pitfalls section, quality preset table.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A <video src='../assets/foo.mp4'> inside a sub-composition silently dropped
from extraction; the rendered output froze on the first decoded frame for
the entire clip, with no error in stdout.
Root cause: browser URL resolver clamps '..' at origin root (studio preview
loads fine), but path.join(projectDir, '../assets/foo.mp4') normalizes to
parent-of-project/assets/foo.mp4, which usually doesn't exist. existsSync
returns false, extraction is skipped, no frame lookup is built, the
per-frame injector has nothing to swap, and the <video> element's first
decoded frame paints every screenshot.
- Adds resolveProjectRelativeSrc in videoFrameExtractor that mirrors browser
clamping (literal join first, then leading '..' stripped).
- Surfaces a loud stderr warning when the resolver misses.
- Mirrors fix in audioMixer.ts (same bug for <audio src='../'>) and
renderOrchestrator HDR probe loop.
- +6 regression tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three related render robustness fixes:
1. frameCapture.ts: bump videos-ready check from `readyState >= 1`
(HAVE_METADATA — only dimensions known) to `>= 2` (HAVE_CURRENT_DATA —
first frame is rasterized). Without this, when two `<video>` elements
with different codecs (h264 mp4 + VP9 webm) decode at different rates,
the faster one passes readiness while the slower one still hasn't
painted, producing a black "first frame" for the slower clip.
2. chunkEncoder.ts (libx264 path) + streamingEncoder.ts: disable B-frames
for h264 (`-bf 0`). Standard libx264 with B-frames produces negative
DTS at stream start (the first B-frame's decode order is "before" the
first I-frame's presentation time). VS Code preview, several browser
<video> implementations, and some HW decoders freeze on the first
frame and only audio plays. -bf 0 makes PTS == DTS at every frame,
eliminating the issue at the source. Quality cost is ~5–10% larger
files at the same CRF — worthwhile for "the file plays everywhere".
3. chunkEncoder.ts (encoder + mux paths): add `-avoid_negative_ts make_zero`
as belt-and-suspenders against negative DTS sneaking back in via
`-c:v copy` mux passes when audio/video PTS bases differ.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- inference.ts: force `.toColourspace("b-w")` on the resized mask. Sharp upcasts
the 1-channel raw input to RGB-interleaved during resize, so `fullMask[i]`
was reading R,G,B,R,G,B... of pixels 0..691199 instead of the alpha for all
2,073,600 pixels. Visible symptom: horizontal scanline alpha artifact in
every transparent webm — the avatar appeared semi-transparent throughout.
- pipeline.ts: add BT.709 + limited-range colorspace tags so Chrome's YUV→RGB
matches the source mp4 (without these, ffmpeg's default RGB→YUV is BT.601
and skin tones drift visibly when the cutout is overlaid on its source).
- pipeline.ts: add Quality preset type ("fast"/"balanced"/"best" → CRF 30/18/12).
Default raised from CRF 30 → 18 ("balanced") so the most common pattern
(text-behind-subject) works out of the box without visible doubling.
- remove-background.ts: wire `--quality` flag with validation, +2 examples.
- Tests: BT.709 tags present, quality preset → CRF mapping, default is balanced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Review on PR #619 caught two places that still pointed transcribe/tts
at hyperframes-cli — directly undercutting the description-trigger
goal of the split:
- skills/hyperframes/SKILL.md description ended with "For CLI commands
(init, lint, preview, render, transcribe, tts) see the
hyperframes-cli skill." Now splits the redirect: dev-loop commands
(init, lint, inspect, preview, render) → hyperframes-cli; asset
preprocessing (tts, transcribe, remove-background) →
hyperframes-media.
- packages/cli/src/templates/_shared/CLAUDE.md is the skills table
baked into every project bootstrapped by `hyperframes init`. Its
hyperframes-cli row still listed transcribe/tts. Trimmed to the
dev-loop commands and added a hyperframes-media row beside it, so
new projects pick up the correct mapping.
Also caught by greppping for stale skill lists:
- .codex-plugin/plugin.json longDescription bundled transcribe/tts
into "use the CLI for ...". Split into "use the CLI for the dev
loop (init/preview/render), preprocess assets
(tts/transcribe/remove-background)" so the Codex plugin store
surface matches reality.
Confirmed `npx hyperframes skills` shells out to `npx skills add
heygen-com/hyperframes --all` (packages/cli/src/commands/skills.ts),
so the skill list is read dynamically from the repo and picks up
hyperframes-media without code changes.
Code review found the new hyperframes-media skill was parallel
content with skills/hyperframes/references/tts.md and the "Whisper
Model Guide" section of transcript-guide.md — same voice table, same
.en-translates-non-English warning, same TTS→transcribe chain in
both places. Plus some scope creep in hyperframes-media (audio/video
HTML snippets that duplicate the canonical track docs in
hyperframes/SKILL.md:265+).
Consolidation:
- hyperframes-media is now the single source of truth for CLI
invocation, voice selection, multilingual phonemization, whisper
model selection, and the .en gotcha. Picked up the multilingual
prefix decoding from the deleted tts.md.
- skills/hyperframes/references/tts.md deleted; the bullet in
hyperframes/SKILL.md is removed (no replacement — agents land on
hyperframes-media via its own description).
- skills/hyperframes/references/transcript-guide.md keeps only the
caption-side concerns: input-format table, mandatory quality
check, cleaning JS, external-API import path, and the
"if no transcript exists" flow. The intro bash recipe and Whisper
Model Guide section both moved to hyperframes-media. Top of the
file now points to hyperframes-media for CLI/model details.
Other tightening in hyperframes-media:
- Dropped WHAT-narration filler and the inline <audio>/<video> HTML
snippets — they duplicate the canonical track-attribute docs in
hyperframes/SKILL.md.
- Added the `id` field (`w0`, `w1`, ...) to the transcript output
shape — the actual Word interface in
packages/cli/src/whisper/normalize.ts includes it (optional for
backwards compat), used by caption override logic.
- Compressed the TTS → Transcribe → Captions chain section.
Net: hyperframes-media 147 → 136 lines, transcript-guide.md 152 →
106 lines, tts.md gone (-75 lines).
James
