mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-15 22:08:03 +00:00
The --hdr flag was silently dropped when combined with --docker because renderDocker did not include it in the args passed to the in-container hyperframes render invocation. Forwarding it produces the same HDR10 MP4 (HEVC, yuv420p10le, BT.2020 PQ, mastering display + max-cll) as a local render — verified end-to-end with the new hdr-pq regression fixture. This PR also fills in the documentation gaps from the HDR feature stack (#288, #289, #290, #265, #268), which each added a slice of HDR support without pulling it together into a single guide. Code change: - packages/cli/src/commands/render.ts: refactor renderDocker to use a new pure helper buildDockerRunArgs (packages/cli/src/utils/dockerRunArgs.ts) that explicitly threads every render flag — including --hdr, --gpu, --quiet — into the docker run argv. Snapshot-tested in dockerRunArgs.test.ts so any future flag drop fails loudly. Docs: - New docs/guides/hdr.mdx — top-level HDR guide: quickstart, how detection works, source media requirements (HDR video + 16-bit HDR PNG), output format constraints, ffprobe verification, Docker rendering, limitations, common pitfalls - docs/guides/common-mistakes.mdx — "Expected HDR but got SDR" entry; --hdr is detection, not force; notes Docker now supports it - docs/guides/rendering.mdx — --hdr row in flags table; HDR card under Next Steps - docs/packages/cli.mdx — expanded --hdr description with link to the guide - docs/packages/producer.mdx — HDR Output section under RenderConfig - docs/packages/engine.mdx — HDR APIs section (color-space utilities + WebGPU readback runtime); now compiles because DEFAULT_HDR10_MASTERING and HdrMasteringMetadata are re-exported from packages/engine/src/index.ts - docs/docs.json — register the new HDR guide in nav Review feedback from PR #346: - packages/engine/src/index.ts — export DEFAULT_HDR10_MASTERING and HdrMasteringMetadata so the engine docs example compiles - docs/guides/hdr.mdx — drop fabricated "animated HDR images log a warning and use first frame" claim (not implemented); clarify that HDR <img> is 16-bit PNG only and other formats fall back to SDR; broaden the yt-dlp example to bestvideo[dynamic_range=HDR]+bestaudio so AV1-HDR streams are also picked up; qualify the Docker render benchmark claim - packages/producer/src/services/renderOrchestrator.ts — bump the HDR + non-mp4 fallback from log.info to log.warn and make the message actionable Regression test (hdr-pq): - packages/producer/tests/hdr-regression — delete the broken sub-tree. The old hdr-pq composition relied on a CDN shader-transitions library and window.__hf injection with scenes defaulting to opacity:0, so when the library failed to load most of the render was invisible. The fixture was also nested two directories deep, so discoverTestSuites (which only scans direct children of tests/) never picked it up. - packages/producer/tests/hdr-pq — new top-level fixture: 5s of an HDR10 HEVC Main10 BT.2020 PQ source clip with a static SDR DOM overlay. Source is a 5s excerpt (1:18–1:23) from https://youtu.be/56hEFqjKG0s, downloaded via yt-dlp and re-encoded to HEVC Main10 / 1920x1080 / 30fps with HDR10 mastering metadata. Self-contained: no external CDNs, no custom fonts, no animation libraries. NOTICE.md captures attribution. - packages/producer/src/regression-harness.ts — add optional hdr boolean to TestMetadata.renderConfig, validate it, and forward it to createRenderJob so HDR fixtures actually run through the HDR pipeline. - .github/workflows/regression.yml — exclude hdr from the fast shard and add a dedicated hdr shard so the new fixture runs on CI. - .gitattributes — LFS-track packages/producer/tests/*/src/*.mp4 so source clips don't bloat the repo. - packages/engine/src/services/frameCapture.ts — inject a no-op __name shim into every page via evaluateOnNewDocument. tsx transforms the engine source on the fly with esbuild's keepNames, which wraps every named function in __name(fn, "name"). When page.evaluate() serializes a callback those calls would crash with "__name is not defined" because the helper only exists in Node. The shim makes the engine work whether it's imported from compiled dist or from source via tsx.
416 lines
14 KiB
Plaintext
416 lines
14 KiB
Plaintext
---
|
|
title: "@hyperframes/engine"
|
|
description: "Seekable page-to-video capture engine using Chrome's BeginFrame API."
|
|
---
|
|
|
|
The engine package provides the low-level video capture pipeline: it loads an HTML page in headless Chrome, seeks to each frame independently, and captures pixel buffers using Chrome's `HeadlessExperimental.beginFrame` API. This is the layer that makes Hyperframes rendering deterministic.
|
|
|
|
```bash
|
|
npm install @hyperframes/engine
|
|
```
|
|
|
|
## When to Use
|
|
|
|
<Warning>
|
|
**Most users should NOT use the engine directly.** Use the [CLI](/packages/cli) (`npx hyperframes render`) or the [producer](/packages/producer) package instead — they handle runtime injection, audio mixing, and encoding for you.
|
|
</Warning>
|
|
|
|
**Use `@hyperframes/engine` when you need to:**
|
|
- Build a custom rendering pipeline with full control over frame capture
|
|
- Integrate Hyperframes capture into an existing video processing system
|
|
- Capture individual frames (e.g., for thumbnails or sprite sheets) without encoding to video
|
|
- Implement a custom encoding backend (not FFmpeg)
|
|
|
|
**Use a different package if you want to:**
|
|
- Render an HTML composition to a finished MP4 or WebM — use the [producer](/packages/producer) or [CLI](/packages/cli)
|
|
- Preview compositions in the browser — use the [CLI](/packages/cli) or [studio](/packages/studio)
|
|
- Lint or parse composition HTML — use [core](/packages/core)
|
|
|
|
## How It Works
|
|
|
|
The engine implements a **seek-and-capture** loop that is fundamentally different from screen recording:
|
|
|
|
<Steps>
|
|
<Step title="Launch headless Chrome">
|
|
The engine starts `chrome-headless-shell`, a minimal headless Chrome binary optimized for programmatic control via the Chrome DevTools Protocol (CDP).
|
|
</Step>
|
|
<Step title="Load the composition">
|
|
Your HTML composition is loaded into a browser page. The Hyperframes runtime is injected to manage timeline seeking.
|
|
</Step>
|
|
<Step title="Seek to each frame">
|
|
For every frame in the video (e.g., 900 frames for a 30-second video at 30fps), the engine calls `renderSeek(time)` to advance the composition to the exact timestamp. No wall clock is involved — each frame is independently positioned.
|
|
</Step>
|
|
<Step title="Capture via BeginFrame">
|
|
Chrome's `HeadlessExperimental.beginFrame` API captures the compositor output as a pixel buffer. This produces pixel-perfect frames without any screen recording artifacts.
|
|
</Step>
|
|
<Step title="Hand off frames">
|
|
Captured frame buffers are passed to a consumer — typically FFmpeg (via the producer) for encoding into MP4, but you can provide your own consumer.
|
|
</Step>
|
|
</Steps>
|
|
|
|
This approach guarantees [deterministic rendering](/concepts/determinism): the same HTML always produces the identical video, regardless of system load or timing.
|
|
|
|
## Configuration
|
|
|
|
```typescript
|
|
import { resolveConfig, DEFAULT_CONFIG } from '@hyperframes/engine';
|
|
import type { EngineConfig } from '@hyperframes/engine';
|
|
|
|
// Use defaults
|
|
const config = DEFAULT_CONFIG;
|
|
|
|
// Or resolve with overrides
|
|
const config = resolveConfig({
|
|
// ... custom options
|
|
});
|
|
```
|
|
|
|
### Quality Presets
|
|
|
|
| Preset | Use Case | Speed |
|
|
|--------|----------|-------|
|
|
| `draft` | Fast iteration during development | Fastest |
|
|
| `standard` | Production renders with good quality/speed balance | Moderate |
|
|
| `high` | Final delivery, maximum quality | Slowest |
|
|
|
|
### FPS Options
|
|
|
|
| FPS | Use Case |
|
|
|-----|----------|
|
|
| `24` | Cinematic look, smaller file size |
|
|
| `30` | Standard web video, good balance |
|
|
| `60` | Smooth motion, UI animations, screen recordings |
|
|
|
|
## Programmatic Usage
|
|
|
|
The engine uses a session-based API for frame capture:
|
|
|
|
```typescript
|
|
import {
|
|
createCaptureSession,
|
|
initializeSession,
|
|
captureFrame,
|
|
captureFrameToBuffer,
|
|
getCompositionDuration,
|
|
closeCaptureSession,
|
|
} from '@hyperframes/engine';
|
|
|
|
// 1. Create a capture session
|
|
const session = await createCaptureSession({ fps: 30, width: 1920, height: 1080 });
|
|
|
|
// 2. Initialize with a composition
|
|
await initializeSession(session, './my-video/index.html');
|
|
|
|
// 3. Get the total duration
|
|
const duration = getCompositionDuration(session);
|
|
|
|
// 4. Capture frames
|
|
const totalFrames = Math.ceil(duration * 30);
|
|
for (let i = 0; i < totalFrames; i++) {
|
|
// Capture to disk
|
|
const result = await captureFrame(session, i);
|
|
// result.path, result.captureTimeMs
|
|
|
|
// Or capture to buffer (in-memory)
|
|
const bufResult = await captureFrameToBuffer(session, i);
|
|
// bufResult.buffer, bufResult.captureTimeMs
|
|
}
|
|
|
|
// 5. Clean up
|
|
await closeCaptureSession(session);
|
|
```
|
|
|
|
### Browser Management
|
|
|
|
```typescript
|
|
import {
|
|
acquireBrowser,
|
|
releaseBrowser,
|
|
resolveHeadlessShellPath,
|
|
buildChromeArgs,
|
|
} from '@hyperframes/engine';
|
|
|
|
// Acquire a browser instance (creates or reuses from pool)
|
|
const browser = await acquireBrowser();
|
|
|
|
// Get the Chrome binary path
|
|
const chromePath = await resolveHeadlessShellPath();
|
|
|
|
// Release when done
|
|
await releaseBrowser(browser);
|
|
```
|
|
|
|
### Encoding
|
|
|
|
The engine includes FFmpeg encoding utilities with support for MP4 (h264) and WebM (VP9 with alpha):
|
|
|
|
```typescript
|
|
import {
|
|
encodeFramesFromDir,
|
|
muxVideoWithAudio,
|
|
applyFaststart,
|
|
detectGpuEncoder,
|
|
getEncoderPreset,
|
|
ENCODER_PRESETS,
|
|
} from '@hyperframes/engine';
|
|
|
|
// Get format-aware encoder settings
|
|
const mp4Preset = getEncoderPreset('standard', 'mp4');
|
|
// { codec: "h264", pixelFormat: "yuv420p", preset: "medium", quality: 23 }
|
|
|
|
const webmPreset = getEncoderPreset('standard', 'webm');
|
|
// { codec: "vp9", pixelFormat: "yuva420p", preset: "good", quality: 23 }
|
|
|
|
// Encode captured frames to video
|
|
await encodeFramesFromDir(framesDir, 'frame_%06d.png', outputPath, {
|
|
fps: 30,
|
|
...webmPreset,
|
|
});
|
|
|
|
// Mix video with audio (uses Opus for WebM, AAC for MP4)
|
|
await muxVideoWithAudio(videoPath, audioPath, outputPath);
|
|
|
|
// Apply MP4 faststart for streaming (no-op for WebM)
|
|
await applyFaststart(inputPath, outputPath);
|
|
|
|
// Detect GPU encoding support
|
|
const gpu = await detectGpuEncoder();
|
|
// gpu: "nvenc" | "videotoolbox" | "vaapi" | null
|
|
```
|
|
|
|
#### WebM with VP9 Alpha
|
|
|
|
When encoding for transparency, use `format: "webm"` with `getEncoderPreset()`. This configures:
|
|
- **VP9 codec** (`libvpx-vp9`) with alpha-capable `yuva420p` pixel format
|
|
- **`-auto-alt-ref 0`** and **`alpha_mode=1`** metadata for proper alpha encoding
|
|
- **`-row-mt 1`** for multi-threaded VP9 encoding
|
|
- **Opus audio** in the mux step (instead of AAC for MP4)
|
|
|
|
### Streaming Encoder
|
|
|
|
For memory-efficient encoding without writing frames to disk:
|
|
|
|
```typescript
|
|
import { spawnStreamingEncoder } from '@hyperframes/engine';
|
|
|
|
const encoder = await spawnStreamingEncoder({
|
|
outputPath: './output.mp4',
|
|
fps: 30,
|
|
width: 1920,
|
|
height: 1080,
|
|
});
|
|
|
|
// Feed frames directly to encoder
|
|
encoder.writeFrame(frameBuffer);
|
|
// ...
|
|
const result = await encoder.finalize();
|
|
```
|
|
|
|
### Video Frame Extraction
|
|
|
|
Extract frames from source video files for injection into the browser:
|
|
|
|
```typescript
|
|
import {
|
|
parseVideoElements,
|
|
extractAllVideoFrames,
|
|
getFrameAtTime,
|
|
createFrameLookupTable,
|
|
FrameLookupTable,
|
|
} from '@hyperframes/engine';
|
|
|
|
// Parse video elements from HTML
|
|
const videos = parseVideoElements(html);
|
|
|
|
// Extract all frames from a video
|
|
const frames = await extractAllVideoFrames(videoPath, { fps: 30 });
|
|
|
|
// Create a lookup table for fast frame access
|
|
const lookup = createFrameLookupTable(frames);
|
|
const frame = lookup.getFrameAtTime(5.0);
|
|
```
|
|
|
|
### Audio Processing
|
|
|
|
```typescript
|
|
import { parseAudioElements, processCompositionAudio } from '@hyperframes/engine';
|
|
|
|
// Parse audio elements from HTML
|
|
const audioElements = parseAudioElements(html);
|
|
|
|
// Process and mix all audio tracks
|
|
const mixResult = await processCompositionAudio({ audioElements, duration, fps });
|
|
```
|
|
|
|
### Parallel Rendering
|
|
|
|
```typescript
|
|
import {
|
|
calculateOptimalWorkers,
|
|
distributeFrames,
|
|
executeParallelCapture,
|
|
getSystemResources,
|
|
} from '@hyperframes/engine';
|
|
|
|
// Check system resources
|
|
const resources = getSystemResources();
|
|
|
|
// Calculate optimal worker count
|
|
const workers = calculateOptimalWorkers(totalFrames);
|
|
|
|
// Distribute frames across workers
|
|
const tasks = distributeFrames(totalFrames, workers);
|
|
|
|
// Execute parallel capture
|
|
const results = await executeParallelCapture(tasks);
|
|
```
|
|
|
|
### File Server
|
|
|
|
Serve composition files over HTTP for the browser to load:
|
|
|
|
```typescript
|
|
import { createFileServer } from '@hyperframes/engine';
|
|
|
|
const server = await createFileServer({ root: './my-video', port: 0 });
|
|
// server.url, server.port
|
|
// ... use server.url as the composition URL
|
|
await server.close();
|
|
```
|
|
|
|
## HDR APIs
|
|
|
|
The engine exports two layers of HDR support: **color-space utilities** that classify sources and configure the FFmpeg encoder, and a **WebGPU readback runtime** for capturing CSS-animated DOM directly into HDR.
|
|
|
|
For end-to-end HDR rendering (HDR video and image sources composited into an HDR10 MP4) use the [producer](/packages/producer) or the CLI's `--hdr` flag — see [HDR Rendering](/guides/hdr). The APIs below are for custom integrations.
|
|
|
|
### Color space utilities
|
|
|
|
```typescript
|
|
import {
|
|
isHdrColorSpace,
|
|
detectTransfer,
|
|
analyzeCompositionHdr,
|
|
getHdrEncoderColorParams,
|
|
DEFAULT_HDR10_MASTERING,
|
|
} from '@hyperframes/engine';
|
|
import type { HdrTransfer, HdrEncoderColorParams, HdrMasteringMetadata } from '@hyperframes/engine';
|
|
|
|
// Classify a single source from its ffprobe color space
|
|
isHdrColorSpace(colorSpace); // boolean — true for BT.2020 / PQ / HLG
|
|
detectTransfer(colorSpace); // 'pq' | 'hlg' (gate on isHdrColorSpace first)
|
|
|
|
// Pick the dominant transfer across many sources
|
|
analyzeCompositionHdr([cs1, cs2]); // { hasHdr, dominantTransfer: 'pq' | 'hlg' | null }
|
|
|
|
// Build the FFmpeg color params + HDR10 static metadata for x265
|
|
const params = getHdrEncoderColorParams('pq');
|
|
// {
|
|
// colorPrimaries: 'bt2020',
|
|
// colorTrc: 'smpte2084',
|
|
// colorspace: 'bt2020nc',
|
|
// pixelFormat: 'yuv420p10le',
|
|
// x265ColorParams: 'colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:master-display=...:max-cll=1000,400',
|
|
// mastering: { masterDisplay: '...', maxCll: '1000,400' },
|
|
// }
|
|
```
|
|
|
|
`getHdrEncoderColorParams` always includes both color tagging *and* the HDR10 static metadata (mastering display + content light level). Without that metadata, downstream players treat the file as SDR BT.2020 and tone-map incorrectly. Pass a custom `HdrMasteringMetadata` if you have measured per-content values; otherwise the conservative `DEFAULT_HDR10_MASTERING` defaults match how most HDR10 grading suites tag content.
|
|
|
|
### WebGPU HDR DOM capture
|
|
|
|
For capturing CSS-animated DOM directly into HDR (no FFmpeg source involved), the engine exposes a separate WebGPU pipeline:
|
|
|
|
```typescript
|
|
import {
|
|
launchHdrBrowser,
|
|
buildHdrChromeArgs,
|
|
initHdrReadback,
|
|
uploadAndReadbackHdrFrame,
|
|
float16ToPqRgb,
|
|
} from '@hyperframes/engine';
|
|
|
|
// Launch headed Chrome with WebGPU enabled
|
|
const { browser, page } = await launchHdrBrowser({ width: 1920, height: 1080 });
|
|
|
|
// Inject the WebGPU readback runtime
|
|
const ok = await initHdrReadback(page, 1920, 1080);
|
|
|
|
// For each frame: upload float16 pixels, read back float16 RGBA
|
|
const { rgba16, bytesPerRow } = await uploadAndReadbackHdrFrame(page, float16Base64);
|
|
|
|
// Convert linear float16 → PQ-encoded 16-bit RGB suitable for piping into ffmpeg/x265
|
|
const pqRgb = float16ToPqRgb(rgba16, width, height, bytesPerRow);
|
|
```
|
|
|
|
<Warning>
|
|
This path requires **headed Chrome with `--enable-unsafe-webgpu`** — WebGPU is unavailable in `chrome-headless-shell`. It is *not* used by the default `--hdr` render pipeline (which extracts HDR pixels from sources via FFmpeg and composites in Node). Use it only for advanced custom pipelines that need CSS animations driving HDR pixel output.
|
|
</Warning>
|
|
|
|
## The `window.__hf` Protocol
|
|
|
|
The engine communicates with the browser page via the `window.__hf` protocol. Any page that implements this protocol can be captured by the engine — you are not limited to Hyperframes compositions.
|
|
|
|
```typescript
|
|
// The page must expose this on window.__hf
|
|
interface HfProtocol {
|
|
duration: number; // Total duration in seconds
|
|
seek(time: number): void; // Seek to a specific time
|
|
media?: HfMediaElement[]; // Optional media element declarations
|
|
}
|
|
|
|
interface HfMediaElement {
|
|
elementId: string; // DOM element ID
|
|
src: string; // Media source URL
|
|
startTime: number; // Start time on timeline
|
|
endTime: number; // End time on timeline
|
|
mediaOffset?: number; // Playback offset in source
|
|
volume?: number; // Volume (0-1)
|
|
hasAudio?: boolean; // Whether element has audio
|
|
}
|
|
```
|
|
|
|
## Key Concepts
|
|
|
|
### BeginFrame Rendering
|
|
|
|
Traditional screen capture records at wall-clock speed — if your system is under load, frames get dropped. The engine uses Chrome's `HeadlessExperimental.beginFrame` to explicitly advance the compositor, producing each frame on demand. This means:
|
|
|
|
- **No dropped frames** — every frame is captured
|
|
- **No timing dependency** — a 60-second video does not take 60 seconds to capture
|
|
- **Pixel-perfect output** — the compositor produces the exact pixels it would display
|
|
|
|
For more on how this enables deterministic output, see [Deterministic Rendering](/concepts/determinism).
|
|
|
|
### Seek Contract
|
|
|
|
The engine relies on the Hyperframes runtime's `renderSeek(time)` function. When called, `renderSeek`:
|
|
|
|
1. Pauses all GSAP timelines
|
|
2. Seeks every timeline to the exact timestamp
|
|
3. Updates all media elements (video, audio) to match
|
|
4. Mounts/unmounts clips based on their `data-start` and `data-duration`
|
|
|
|
This contract is what makes frame-by-frame capture possible — each frame is a complete, independent snapshot of the composition at that point in time.
|
|
|
|
### Chrome Requirements
|
|
|
|
The engine requires `chrome-headless-shell`, which is included when you install the package. It uses a pinned Chrome version to ensure consistent rendering across environments. For fully deterministic output (including fonts), use Docker mode via the [producer](/packages/producer).
|
|
|
|
## Related Packages
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Producer" icon="film" href="/packages/producer">
|
|
Wraps the engine with runtime injection, FFmpeg encoding, and audio mixing for complete MP4 output.
|
|
</Card>
|
|
<Card title="Core" icon="cube" href="/packages/core">
|
|
Provides the types, runtime, and linter that the engine depends on.
|
|
</Card>
|
|
<Card title="CLI" icon="terminal" href="/packages/cli">
|
|
The easiest way to render — calls the producer (and engine) under the hood.
|
|
</Card>
|
|
<Card title="Studio" icon="palette" href="/packages/studio">
|
|
Visual editor for building compositions before rendering them with the engine.
|
|
</Card>
|
|
</CardGroup>
|