Files
hyperframes/docs/packages/engine.mdx
T
Vance Ingalls 7a695dd296 fix(cli): forward --hdr through Docker render + HDR docs
The --hdr flag was silently dropped when combined with --docker because
renderDocker did not include it in the args passed to the in-container
hyperframes render invocation. Forwarding it produces the same HDR10 MP4
(HEVC, yuv420p10le, BT.2020 PQ, mastering display + max-cll) as a local
render — verified end-to-end with the new hdr-pq regression fixture.

This PR also fills in the documentation gaps from the HDR feature stack
(#288, #289, #290, #265, #268), which each added a slice of HDR support
without pulling it together into a single guide.

Code change:
- packages/cli/src/commands/render.ts: refactor renderDocker to use a new
  pure helper buildDockerRunArgs (packages/cli/src/utils/dockerRunArgs.ts)
  that explicitly threads every render flag — including --hdr, --gpu,
  --quiet — into the docker run argv. Snapshot-tested in dockerRunArgs.test.ts
  so any future flag drop fails loudly.

Docs:
- New docs/guides/hdr.mdx — top-level HDR guide: quickstart, how detection
  works, source media requirements (HDR video + 16-bit HDR PNG), output
  format constraints, ffprobe verification, Docker rendering, limitations,
  common pitfalls
- docs/guides/common-mistakes.mdx — "Expected HDR but got SDR" entry; --hdr
  is detection, not force; notes Docker now supports it
- docs/guides/rendering.mdx — --hdr row in flags table; HDR card under
  Next Steps
- docs/packages/cli.mdx — expanded --hdr description with link to the guide
- docs/packages/producer.mdx — HDR Output section under RenderConfig
- docs/packages/engine.mdx — HDR APIs section (color-space utilities +
  WebGPU readback runtime); now compiles because DEFAULT_HDR10_MASTERING
  and HdrMasteringMetadata are re-exported from packages/engine/src/index.ts
- docs/docs.json — register the new HDR guide in nav

Review feedback from PR #346:
- packages/engine/src/index.ts — export DEFAULT_HDR10_MASTERING and
  HdrMasteringMetadata so the engine docs example compiles
- docs/guides/hdr.mdx — drop fabricated "animated HDR images log a warning
  and use first frame" claim (not implemented); clarify that HDR <img> is
  16-bit PNG only and other formats fall back to SDR; broaden the yt-dlp
  example to bestvideo[dynamic_range=HDR]+bestaudio so AV1-HDR streams
  are also picked up; qualify the Docker render benchmark claim
- packages/producer/src/services/renderOrchestrator.ts — bump the HDR + non-mp4
  fallback from log.info to log.warn and make the message actionable

Regression test (hdr-pq):
- packages/producer/tests/hdr-regression — delete the broken sub-tree.
  The old hdr-pq composition relied on a CDN shader-transitions library
  and window.__hf injection with scenes defaulting to opacity:0, so when
  the library failed to load most of the render was invisible. The fixture
  was also nested two directories deep, so discoverTestSuites (which only
  scans direct children of tests/) never picked it up.
- packages/producer/tests/hdr-pq — new top-level fixture: 5s of an HDR10
  HEVC Main10 BT.2020 PQ source clip with a static SDR DOM overlay. Source
  is a 5s excerpt (1:18–1:23) from https://youtu.be/56hEFqjKG0s, downloaded
  via yt-dlp and re-encoded to HEVC Main10 / 1920x1080 / 30fps with HDR10
  mastering metadata. Self-contained: no external CDNs, no custom fonts,
  no animation libraries. NOTICE.md captures attribution.
- packages/producer/src/regression-harness.ts — add optional hdr boolean
  to TestMetadata.renderConfig, validate it, and forward it to
  createRenderJob so HDR fixtures actually run through the HDR pipeline.
- .github/workflows/regression.yml — exclude hdr from the fast shard and
  add a dedicated hdr shard so the new fixture runs on CI.
- .gitattributes — LFS-track packages/producer/tests/*/src/*.mp4 so source
  clips don't bloat the repo.
- packages/engine/src/services/frameCapture.ts — inject a no-op __name
  shim into every page via evaluateOnNewDocument. tsx transforms the
  engine source on the fly with esbuild's keepNames, which wraps every
  named function in __name(fn, "name"). When page.evaluate() serializes
  a callback those calls would crash with "__name is not defined" because
  the helper only exists in Node. The shim makes the engine work whether
  it's imported from compiled dist or from source via tsx.
2026-04-19 21:17:30 -07:00

416 lines
14 KiB
Plaintext

---
title: "@hyperframes/engine"
description: "Seekable page-to-video capture engine using Chrome's BeginFrame API."
---
The engine package provides the low-level video capture pipeline: it loads an HTML page in headless Chrome, seeks to each frame independently, and captures pixel buffers using Chrome's `HeadlessExperimental.beginFrame` API. This is the layer that makes Hyperframes rendering deterministic.
```bash
npm install @hyperframes/engine
```
## When to Use
<Warning>
**Most users should NOT use the engine directly.** Use the [CLI](/packages/cli) (`npx hyperframes render`) or the [producer](/packages/producer) package instead — they handle runtime injection, audio mixing, and encoding for you.
</Warning>
**Use `@hyperframes/engine` when you need to:**
- Build a custom rendering pipeline with full control over frame capture
- Integrate Hyperframes capture into an existing video processing system
- Capture individual frames (e.g., for thumbnails or sprite sheets) without encoding to video
- Implement a custom encoding backend (not FFmpeg)
**Use a different package if you want to:**
- Render an HTML composition to a finished MP4 or WebM — use the [producer](/packages/producer) or [CLI](/packages/cli)
- Preview compositions in the browser — use the [CLI](/packages/cli) or [studio](/packages/studio)
- Lint or parse composition HTML — use [core](/packages/core)
## How It Works
The engine implements a **seek-and-capture** loop that is fundamentally different from screen recording:
<Steps>
<Step title="Launch headless Chrome">
The engine starts `chrome-headless-shell`, a minimal headless Chrome binary optimized for programmatic control via the Chrome DevTools Protocol (CDP).
</Step>
<Step title="Load the composition">
Your HTML composition is loaded into a browser page. The Hyperframes runtime is injected to manage timeline seeking.
</Step>
<Step title="Seek to each frame">
For every frame in the video (e.g., 900 frames for a 30-second video at 30fps), the engine calls `renderSeek(time)` to advance the composition to the exact timestamp. No wall clock is involved — each frame is independently positioned.
</Step>
<Step title="Capture via BeginFrame">
Chrome's `HeadlessExperimental.beginFrame` API captures the compositor output as a pixel buffer. This produces pixel-perfect frames without any screen recording artifacts.
</Step>
<Step title="Hand off frames">
Captured frame buffers are passed to a consumer — typically FFmpeg (via the producer) for encoding into MP4, but you can provide your own consumer.
</Step>
</Steps>
This approach guarantees [deterministic rendering](/concepts/determinism): the same HTML always produces the identical video, regardless of system load or timing.
## Configuration
```typescript
import { resolveConfig, DEFAULT_CONFIG } from '@hyperframes/engine';
import type { EngineConfig } from '@hyperframes/engine';
// Use defaults
const config = DEFAULT_CONFIG;
// Or resolve with overrides
const config = resolveConfig({
// ... custom options
});
```
### Quality Presets
| Preset | Use Case | Speed |
|--------|----------|-------|
| `draft` | Fast iteration during development | Fastest |
| `standard` | Production renders with good quality/speed balance | Moderate |
| `high` | Final delivery, maximum quality | Slowest |
### FPS Options
| FPS | Use Case |
|-----|----------|
| `24` | Cinematic look, smaller file size |
| `30` | Standard web video, good balance |
| `60` | Smooth motion, UI animations, screen recordings |
## Programmatic Usage
The engine uses a session-based API for frame capture:
```typescript
import {
createCaptureSession,
initializeSession,
captureFrame,
captureFrameToBuffer,
getCompositionDuration,
closeCaptureSession,
} from '@hyperframes/engine';
// 1. Create a capture session
const session = await createCaptureSession({ fps: 30, width: 1920, height: 1080 });
// 2. Initialize with a composition
await initializeSession(session, './my-video/index.html');
// 3. Get the total duration
const duration = getCompositionDuration(session);
// 4. Capture frames
const totalFrames = Math.ceil(duration * 30);
for (let i = 0; i < totalFrames; i++) {
// Capture to disk
const result = await captureFrame(session, i);
// result.path, result.captureTimeMs
// Or capture to buffer (in-memory)
const bufResult = await captureFrameToBuffer(session, i);
// bufResult.buffer, bufResult.captureTimeMs
}
// 5. Clean up
await closeCaptureSession(session);
```
### Browser Management
```typescript
import {
acquireBrowser,
releaseBrowser,
resolveHeadlessShellPath,
buildChromeArgs,
} from '@hyperframes/engine';
// Acquire a browser instance (creates or reuses from pool)
const browser = await acquireBrowser();
// Get the Chrome binary path
const chromePath = await resolveHeadlessShellPath();
// Release when done
await releaseBrowser(browser);
```
### Encoding
The engine includes FFmpeg encoding utilities with support for MP4 (h264) and WebM (VP9 with alpha):
```typescript
import {
encodeFramesFromDir,
muxVideoWithAudio,
applyFaststart,
detectGpuEncoder,
getEncoderPreset,
ENCODER_PRESETS,
} from '@hyperframes/engine';
// Get format-aware encoder settings
const mp4Preset = getEncoderPreset('standard', 'mp4');
// { codec: "h264", pixelFormat: "yuv420p", preset: "medium", quality: 23 }
const webmPreset = getEncoderPreset('standard', 'webm');
// { codec: "vp9", pixelFormat: "yuva420p", preset: "good", quality: 23 }
// Encode captured frames to video
await encodeFramesFromDir(framesDir, 'frame_%06d.png', outputPath, {
fps: 30,
...webmPreset,
});
// Mix video with audio (uses Opus for WebM, AAC for MP4)
await muxVideoWithAudio(videoPath, audioPath, outputPath);
// Apply MP4 faststart for streaming (no-op for WebM)
await applyFaststart(inputPath, outputPath);
// Detect GPU encoding support
const gpu = await detectGpuEncoder();
// gpu: "nvenc" | "videotoolbox" | "vaapi" | null
```
#### WebM with VP9 Alpha
When encoding for transparency, use `format: "webm"` with `getEncoderPreset()`. This configures:
- **VP9 codec** (`libvpx-vp9`) with alpha-capable `yuva420p` pixel format
- **`-auto-alt-ref 0`** and **`alpha_mode=1`** metadata for proper alpha encoding
- **`-row-mt 1`** for multi-threaded VP9 encoding
- **Opus audio** in the mux step (instead of AAC for MP4)
### Streaming Encoder
For memory-efficient encoding without writing frames to disk:
```typescript
import { spawnStreamingEncoder } from '@hyperframes/engine';
const encoder = await spawnStreamingEncoder({
outputPath: './output.mp4',
fps: 30,
width: 1920,
height: 1080,
});
// Feed frames directly to encoder
encoder.writeFrame(frameBuffer);
// ...
const result = await encoder.finalize();
```
### Video Frame Extraction
Extract frames from source video files for injection into the browser:
```typescript
import {
parseVideoElements,
extractAllVideoFrames,
getFrameAtTime,
createFrameLookupTable,
FrameLookupTable,
} from '@hyperframes/engine';
// Parse video elements from HTML
const videos = parseVideoElements(html);
// Extract all frames from a video
const frames = await extractAllVideoFrames(videoPath, { fps: 30 });
// Create a lookup table for fast frame access
const lookup = createFrameLookupTable(frames);
const frame = lookup.getFrameAtTime(5.0);
```
### Audio Processing
```typescript
import { parseAudioElements, processCompositionAudio } from '@hyperframes/engine';
// Parse audio elements from HTML
const audioElements = parseAudioElements(html);
// Process and mix all audio tracks
const mixResult = await processCompositionAudio({ audioElements, duration, fps });
```
### Parallel Rendering
```typescript
import {
calculateOptimalWorkers,
distributeFrames,
executeParallelCapture,
getSystemResources,
} from '@hyperframes/engine';
// Check system resources
const resources = getSystemResources();
// Calculate optimal worker count
const workers = calculateOptimalWorkers(totalFrames);
// Distribute frames across workers
const tasks = distributeFrames(totalFrames, workers);
// Execute parallel capture
const results = await executeParallelCapture(tasks);
```
### File Server
Serve composition files over HTTP for the browser to load:
```typescript
import { createFileServer } from '@hyperframes/engine';
const server = await createFileServer({ root: './my-video', port: 0 });
// server.url, server.port
// ... use server.url as the composition URL
await server.close();
```
## HDR APIs
The engine exports two layers of HDR support: **color-space utilities** that classify sources and configure the FFmpeg encoder, and a **WebGPU readback runtime** for capturing CSS-animated DOM directly into HDR.
For end-to-end HDR rendering (HDR video and image sources composited into an HDR10 MP4) use the [producer](/packages/producer) or the CLI's `--hdr` flag — see [HDR Rendering](/guides/hdr). The APIs below are for custom integrations.
### Color space utilities
```typescript
import {
isHdrColorSpace,
detectTransfer,
analyzeCompositionHdr,
getHdrEncoderColorParams,
DEFAULT_HDR10_MASTERING,
} from '@hyperframes/engine';
import type { HdrTransfer, HdrEncoderColorParams, HdrMasteringMetadata } from '@hyperframes/engine';
// Classify a single source from its ffprobe color space
isHdrColorSpace(colorSpace); // boolean — true for BT.2020 / PQ / HLG
detectTransfer(colorSpace); // 'pq' | 'hlg' (gate on isHdrColorSpace first)
// Pick the dominant transfer across many sources
analyzeCompositionHdr([cs1, cs2]); // { hasHdr, dominantTransfer: 'pq' | 'hlg' | null }
// Build the FFmpeg color params + HDR10 static metadata for x265
const params = getHdrEncoderColorParams('pq');
// {
// colorPrimaries: 'bt2020',
// colorTrc: 'smpte2084',
// colorspace: 'bt2020nc',
// pixelFormat: 'yuv420p10le',
// x265ColorParams: 'colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:master-display=...:max-cll=1000,400',
// mastering: { masterDisplay: '...', maxCll: '1000,400' },
// }
```
`getHdrEncoderColorParams` always includes both color tagging *and* the HDR10 static metadata (mastering display + content light level). Without that metadata, downstream players treat the file as SDR BT.2020 and tone-map incorrectly. Pass a custom `HdrMasteringMetadata` if you have measured per-content values; otherwise the conservative `DEFAULT_HDR10_MASTERING` defaults match how most HDR10 grading suites tag content.
### WebGPU HDR DOM capture
For capturing CSS-animated DOM directly into HDR (no FFmpeg source involved), the engine exposes a separate WebGPU pipeline:
```typescript
import {
launchHdrBrowser,
buildHdrChromeArgs,
initHdrReadback,
uploadAndReadbackHdrFrame,
float16ToPqRgb,
} from '@hyperframes/engine';
// Launch headed Chrome with WebGPU enabled
const { browser, page } = await launchHdrBrowser({ width: 1920, height: 1080 });
// Inject the WebGPU readback runtime
const ok = await initHdrReadback(page, 1920, 1080);
// For each frame: upload float16 pixels, read back float16 RGBA
const { rgba16, bytesPerRow } = await uploadAndReadbackHdrFrame(page, float16Base64);
// Convert linear float16 → PQ-encoded 16-bit RGB suitable for piping into ffmpeg/x265
const pqRgb = float16ToPqRgb(rgba16, width, height, bytesPerRow);
```
<Warning>
This path requires **headed Chrome with `--enable-unsafe-webgpu`** — WebGPU is unavailable in `chrome-headless-shell`. It is *not* used by the default `--hdr` render pipeline (which extracts HDR pixels from sources via FFmpeg and composites in Node). Use it only for advanced custom pipelines that need CSS animations driving HDR pixel output.
</Warning>
## The `window.__hf` Protocol
The engine communicates with the browser page via the `window.__hf` protocol. Any page that implements this protocol can be captured by the engine — you are not limited to Hyperframes compositions.
```typescript
// The page must expose this on window.__hf
interface HfProtocol {
duration: number; // Total duration in seconds
seek(time: number): void; // Seek to a specific time
media?: HfMediaElement[]; // Optional media element declarations
}
interface HfMediaElement {
elementId: string; // DOM element ID
src: string; // Media source URL
startTime: number; // Start time on timeline
endTime: number; // End time on timeline
mediaOffset?: number; // Playback offset in source
volume?: number; // Volume (0-1)
hasAudio?: boolean; // Whether element has audio
}
```
## Key Concepts
### BeginFrame Rendering
Traditional screen capture records at wall-clock speed — if your system is under load, frames get dropped. The engine uses Chrome's `HeadlessExperimental.beginFrame` to explicitly advance the compositor, producing each frame on demand. This means:
- **No dropped frames** — every frame is captured
- **No timing dependency** — a 60-second video does not take 60 seconds to capture
- **Pixel-perfect output** — the compositor produces the exact pixels it would display
For more on how this enables deterministic output, see [Deterministic Rendering](/concepts/determinism).
### Seek Contract
The engine relies on the Hyperframes runtime's `renderSeek(time)` function. When called, `renderSeek`:
1. Pauses all GSAP timelines
2. Seeks every timeline to the exact timestamp
3. Updates all media elements (video, audio) to match
4. Mounts/unmounts clips based on their `data-start` and `data-duration`
This contract is what makes frame-by-frame capture possible — each frame is a complete, independent snapshot of the composition at that point in time.
### Chrome Requirements
The engine requires `chrome-headless-shell`, which is included when you install the package. It uses a pinned Chrome version to ensure consistent rendering across environments. For fully deterministic output (including fonts), use Docker mode via the [producer](/packages/producer).
## Related Packages
<CardGroup cols={2}>
<Card title="Producer" icon="film" href="/packages/producer">
Wraps the engine with runtime injection, FFmpeg encoding, and audio mixing for complete MP4 output.
</Card>
<Card title="Core" icon="cube" href="/packages/core">
Provides the types, runtime, and linter that the engine depends on.
</Card>
<Card title="CLI" icon="terminal" href="/packages/cli">
The easiest way to render — calls the producer (and engine) under the hood.
</Card>
<Card title="Studio" icon="palette" href="/packages/studio">
Visual editor for building compositions before rendering them with the engine.
</Card>
</CardGroup>