mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-02 20:18:35 +00:00
Vance Ingalls
a21a62b574
## Summary Compositions with HDR video AND DOM overlays (text, graphics, SDR video) couldn't render both correctly — either HDR data was lost (Chrome captures sRGB only) or DOM overlays were missing (FFmpeg pass-through skips Chrome). This PR adds in-memory alpha compositing that combines both. ## What it does **Per-frame two-pass capture:** 1. **DOM pass** — Chrome screenshots the page with a transparent background (CDP alpha). HDR videos are hidden, leaving transparent holes where they go. 2. **HDR pass** — Pre-extracted native HLG/PQ frames (16-bit PNG from FFmpeg) are read from disk. 3. **Composite** — DOM pixels (sRGB RGBA8) are alpha-composited over HDR pixels (rgb48le) in Node.js memory, with sRGB→HLG/PQ conversion via a 256-entry lookup table. **Key components:** - `decodePng()` / `decodePngToRgb48le()` — Pure Node.js PNG decoders (no native dependencies). Support all 5 PNG filter types. - `blitRgba8OverRgb48le()` — Alpha composite with per-pixel sRGB→HDR LUT conversion. Fast paths for alpha=0 (skip) and alpha=255 (overwrite). - `initTransparentBackground()` + `captureAlphaPng()` — Split CDP transparent background setup (once) from per-frame screenshot capture (eliminates 2 CDP round-trips per frame). - Single-pass FFmpeg extraction — All HDR frames extracted in one sequential FFmpeg run (avoids duplicate frames from per-frame `-ss` fast seek). ## Key design decisions | Decision | Why | |----------|-----| | In-memory compositing (not FFmpeg overlay) | Eliminates ~2400 process spawns + temp files per render. Pure pixel math is 10x faster. | | 16-bit PNG intermediate | Raw `-f rawvideo` loses color metadata, causing moiré artifacts. PNG is self-describing. | | sRGB→HLG LUT (256 entries) | DOM content is sRGB. Without conversion, it appears orange-shifted in HLG stream. | | Native HDR detection before extraction | `extractAllVideoFrames` converts SDR→HDR. Pre-extraction probe identifies original HDR sources so only truly-HDR videos get native extraction. | ## Files changed | File | What changed | |------|-------------| | `packages/engine/src/utils/alphaBlit.ts` | **NEW** — PNG decode, sRGB→HDR LUT, alpha compositing (14 tests) | | `packages/engine/src/services/screenshotService.ts` | Transparent background CDP, `captureAlphaPng()` | | `packages/engine/src/services/videoFrameInjector.ts` | `hideVideoElements()` / `showVideoElements()` | | `packages/engine/src/services/streamingEncoder.ts` | Input color space tags for rgb48le | | `packages/producer/src/services/renderOrchestrator.ts` | Two-pass HDR capture loop, native HDR detection | ## How to test Render a composition with an HDR video background and text overlays. Both should be visible — HDR video at full quality, text crisp with correct colors (not orange-shifted). ## Stack position **3 of 6** — Stacked on #265 (HDR output pipeline). This is the foundation for all layered compositing that follows. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
93 lines
1.6 KiB
Plaintext
93 lines
1.6 KiB
Plaintext
# Dependencies
|
|
node_modules/
|
|
|
|
# Build output
|
|
dist/
|
|
*.tsbuildinfo
|
|
|
|
# Environment
|
|
.env
|
|
.env.local
|
|
.env.*.local
|
|
|
|
# OS
|
|
.DS_Store
|
|
Thumbs.db
|
|
|
|
# Docs media — served from CDN, not committed.
|
|
# Regenerate with scripts/generate-{catalog,template}-previews.ts then upload
|
|
# with `bun run upload:docs-images`. Add explicit negations below for any
|
|
# non-generated assets (logos, svgs) that should stay in the repo.
|
|
docs/images/
|
|
|
|
# IDE
|
|
.vscode/
|
|
.idea/
|
|
*.swp
|
|
*.swo
|
|
|
|
# Test
|
|
coverage/
|
|
.debug/
|
|
|
|
# Producer regression test failures (generated debugging artifacts)
|
|
packages/producer/tests/*/failures/
|
|
|
|
# Rendered output (not test fixtures — those use git LFS)
|
|
output/
|
|
renders/
|
|
!packages/producer/tests/*/output/
|
|
|
|
# Composition source media (large binaries)
|
|
compositions/**/*.mp4
|
|
compositions/**/*.mov
|
|
compositions/**/*.MOV
|
|
|
|
# npm pack artifacts
|
|
*.tgz
|
|
|
|
# Logs
|
|
*.log
|
|
npm-debug.log*
|
|
|
|
# Temporary
|
|
tmp/
|
|
.tmp/
|
|
|
|
# Generated files
|
|
packages/producer/src/services/fontData.generated.ts
|
|
|
|
# Test artifacts
|
|
my-video/
|
|
examples/
|
|
packages/studio/data/
|
|
.desloppify/
|
|
.worktrees/
|
|
|
|
# Playwright MCP browser cache
|
|
.playwright-mcp/
|
|
|
|
# Installed skills (user-specific)
|
|
.agents/
|
|
.claude/skills/
|
|
skills-lock.json
|
|
|
|
# Skills from other PRs (not managed here)
|
|
skills/hyperframes-animation-map/
|
|
skills/hyperframes-contrast/
|
|
|
|
# Capture outputs
|
|
captures/
|
|
# Legacy test captures at repo root (use captures/ instead)
|
|
*-capture/
|
|
*-demo/
|
|
*-ad/
|
|
*-tour/
|
|
*-brand/
|
|
|
|
# Local regression-test and video project scratch directories.
|
|
# These hold per-site captures, compositions, narration audio, rendered MP4s.
|
|
cursor-tests/
|
|
basecamp-video/
|
|
launch-video*/
|