mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
## Problem HyperFrames already had `--gpu`, but that flag only controlled FFmpeg hardware encoding. The browser capture path still forced Chrome/WebGL through SwiftShader software GL via `--use-angle=swiftshader`, so WebGL-heavy local renders could leave the biggest bottleneck on the CPU path. That made the existing flag naming easy to misread: `--gpu` sounded like it accelerated the whole render, but it did not change the browser frame-capture backend. ## What this fixes - Enables host browser GPU acceleration automatically for local CLI renders. - Adds `--no-browser-gpu` as the local opt-out for software Chrome/WebGL capture. - Keeps `--browser-gpu` as an explicit local browser-GPU request. - Adds `browserGpuMode: "software" | "hardware"` to engine config, with `PRODUCER_BROWSER_GPU_MODE` env support for lower-level producer users. - Keeps Docker browser capture on the deterministic software path. - Maps hardware browser GPU mode to platform-native Chrome backends: - macOS: Metal-backed ANGLE - Windows: D3D11-backed ANGLE - Linux: EGL - Blocks explicit `--browser-gpu --docker` with a clear error because Docker browser GPU passthrough is not cross-platform. - Clarifies docs so `--gpu` means FFmpeg encoder GPU and browser GPU means Chrome/WebGL capture GPU. - Keeps encoder backend selection auto-detected from FFmpeg capabilities: - NVIDIA: NVENC - macOS: VideoToolbox - Linux: VAAPI - Intel: QSV ## Why two flags There are two separate GPU surfaces in the render pipeline: 1. Browser GPU controls Chrome frame capture. - Affects WebGL, canvas, CSS rendering, compositing, and screenshot capture inside the browser. - This is enabled automatically for local CLI renders. - Use `--no-browser-gpu` when you want the software browser baseline. 2. `--gpu` controls FFmpeg video encoding. - Affects the final encode step after frames have already been captured. - The concrete encoder is auto-detected from the host FFmpeg build and hardware. - It can be faster for some machines/codecs, but it is not equivalent to browser rendering acceleration. The controls stay independent because users may want: - `hyperframes render` for the fast local default with browser GPU capture. - `hyperframes render --no-browser-gpu` for the software-browser local baseline. - `hyperframes render --gpu` for browser GPU capture plus hardware FFmpeg encoding. - `hyperframes render --no-browser-gpu --gpu` for software browser capture plus hardware FFmpeg encoding. - `hyperframes render --docker` for deterministic browser capture. ## Why `--gpu` does not imply browser GPU Keeping `--gpu` scoped to FFmpeg encoding avoids a semantic break and keeps the risk profile explicit: - `--gpu` already means encoder acceleration. Expanding it to also change Chrome capture would silently alter behavior for users who only wanted hardware encoding. - Browser GPU and encoder GPU have different portability. Encoder GPU can work in Docker when the host exposes the right devices; browser GPU passthrough is not cross-platform, so this PR intentionally blocks explicit `--browser-gpu --docker`. - The Apple presentation benchmark shows why the controls should stay separate: browser GPU capture was the useful improvement, while macOS VideoToolbox via `--gpu` was slower and produced larger output for this `standard` H.264 run. If HyperFrames later wants a single umbrella acceleration control, it should be explicit, for example `--acceleration browser|encoder|all` or `--gpu=browser|encoder|all`, rather than changing the meaning of the existing boolean `--gpu`. ## Root cause `buildChromeArgs()` always injected `--use-gl=angle --use-angle=swiftshader`. `disableGpu` only appended `--disable-gpu`; it did not provide a hardware-GPU mode. That made the public `--gpu` flag look broader than it was, because render capture stayed software-backed even when encoder GPU was requested. ## Verification ### Local checks - `bun install` - `bun run build:hyperframes-runtime` - `bun run --filter @hyperframes/engine test src/config.test.ts src/services/browserManager.test.ts` - `bun run --filter @hyperframes/cli test src/utils/dockerRunArgs.test.ts src/commands/render.test.ts` - `bun run --filter @hyperframes/cli typecheck` - `bun run --filter @hyperframes/engine typecheck` - `bun run --filter @hyperframes/producer typecheck` - `cd packages/producer && bunx vitest run src/services/renderOrchestrator.test.ts` - `bunx oxlint packages/cli/src/commands/render.ts packages/cli/src/commands/render.test.ts packages/cli/src/utils/dockerRunArgs.ts packages/cli/src/utils/dockerRunArgs.test.ts packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/browserManager.ts packages/engine/src/services/browserManager.test.ts packages/producer/src/services/renderOrchestrator.test.ts` - `bunx oxfmt --check ...` on changed source/docs files - `git diff --check` - `bun packages/cli/src/cli.ts render --help | rg -n "browser-gpu|no-browser-gpu|GPU"` - `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --output /tmp/hf-auto-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict` - Render plan prints `GPU: browser GPU (auto)`. - `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --no-browser-gpu --output /tmp/hf-software-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict` - Render plan does not print browser GPU. - `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --docker --browser-gpu --output /tmp/should-not-render.mp4` - Exits 1 with `Browser GPU is local-only`. - `buildDockerRunArgs()` regression coverage asserts Docker container args include `--no-browser-gpu`, preventing nested container renders from re-enabling browser GPU through the local CLI default. - `resolveBrowserGpuForCli()` regression coverage asserts `PRODUCER_BROWSER_GPU_MODE=software` opts out when no CLI browser-GPU flag is supplied, while explicit `--browser-gpu` / `--no-browser-gpu` still win. - `ffmpeg -v error -i /tmp/hf-auto-browser-gpu-smoke.mp4 -f null -` - `ffmpeg -v error -i /tmp/hf-software-browser-gpu-smoke.mp4 -f null -` - `ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of json /tmp/hf-browser-gpu-smoke.mp4` -> H.264, 1920x1080, 24fps, 5.0s ### Apple presentation benchmark Rendered `/Users/miguel07code/Downloads/apple-presentation.zip` as supplied after extracting to `/tmp/hf-apple-profile/apple-presentation`. Fixed settings: - 1920x1080 - 30fps - `standard` quality - 4240 frames - 141.32s duration - 8-worker cap; render auto-calibration used 6 capture workers - macOS host detected FFmpeg GPU encoder: `videotoolbox` | Mode | Equivalent flags after this PR | Wall time | vs software-browser baseline | Speed | Capture | Encode | Output | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Software browser + CPU encode | `--no-browser-gpu` | 120.77s | baseline | 1.17x | 97.87s | 10.04s | 8.38MB | | Browser GPU + CPU encode | default local render | 70.10s | 42.0% faster | 2.02x | 50.72s | 9.91s | 8.39MB | | Software browser + encoder GPU | `--no-browser-gpu --gpu` | 133.16s | 10.3% slower | 1.06x | 103.58s | 18.31s | 25.43MB | | Browser GPU + encoder GPU | `--gpu` | 74.12s | 38.6% faster | 1.91x | 46.69s | 17.93s | 25.45MB | Result: browser GPU capture is the meaningful improvement for this WebGL/browser-capture-heavy presentation. VideoToolbox encoding was slower and produced larger files for this current `standard` H.264 path, so `--gpu` should stay separate and opt-in. Why `--gpu` plus browser GPU was slower than browser GPU alone: the combined run captured about 4.0s faster than browser GPU alone, but VideoToolbox encoding was about 8.0s slower than CPU x264 encoding, so the encode loss outweighed the capture gain. ### VideoToolbox flag check I also isolated the encode stage against the already-captured Apple frames to check whether macOS GPU encoding only needed special flags. `ffmpeg -h encoder=h264_videotoolbox` does not expose a CRF/CQ-style quality option like x264. It exposes bitrate-oriented and VideoToolbox-specific options such as `-b:v`, `-realtime`, `-profile`, `-coder`, `-prio_speed`, `-power_efficient`, and `-allow_sw`. That means our current `-q:v` mapping is not equivalent to x264 CRF and can produce very different bitrate/size behavior. Measured full-frame encode variants on this host: | VideoToolbox variant | Encode wall time | Output size | Bitrate | | --- | ---: | ---: | ---: | | Current `-q:v 64 -allow_sw 1` | 18.76s | 25.31MB | 1.43 Mbps | | Current without `-allow_sw 1` | 18.21s | 25.31MB | 1.43 Mbps | | `-b:v 500k -maxrate 750k -bufsize 1000k -profile high -coder cabac -realtime 1 -prio_speed 1 -power_efficient 0` | 20.58s | 7.42MB | 0.42 Mbps | | Same with `-b:v 1500k` | 20.84s | 16.70MB | 0.95 Mbps | | `-b:v 500k -profile baseline -coder cavlc -realtime 1 -prio_speed 1 -power_efficient 0` | 18.11s | 8.94MB | 0.51 Mbps | Conclusion: VideoToolbox can be made size/bitrate-predictable with explicit `--video-bitrate`, but the tested speed-oriented flags did not make it faster than CPU x264 wall time for this render. That reinforces keeping `--gpu` encoder acceleration explicit and separate from browser GPU capture. Artifacts from the local benchmark: - `/tmp/hf-apple-profile/results/cpu.mp4` - `/tmp/hf-apple-profile/results/browser-gpu.mp4` - `/tmp/hf-apple-profile/results/encoder-gpu.mp4` - `/tmp/hf-apple-profile/results/full-gpu.mp4` - `/tmp/hf-apple-profile/results/summary.json` All four benchmark MP4s completed `ffprobe` and full `ffmpeg -f null` decode checks. ### Pixel comparison Compared decoded MP4 output between software-browser and browser-GPU renders: - Apple presentation: - 4240 frames compared - 636 exact matching decoded frame hashes - 3604 different decoded frame hashes - Average PSNR: 57.79 dB - `css-spinner-render-compat` clean fixture: - 120 frames compared - 0 exact matching decoded frame hashes - Average PSNR: 61.57 dB Interpretation: browser GPU output is not strict hash/pixel-identical to the software-browser path after lossy H.264 encode, but the measured deltas are visually tiny. Above 50 dB PSNR is typically visually indistinguishable for normal video review. Use `--no-browser-gpu` or Docker when strict cross-run/cross-machine reproducibility matters more than local speed. ### Browser verification - Started HyperFrames Studio preview for `packages/producer/tests/css-spinner-render-compat/src`. - Used `agent-browser` to open `http://localhost:5191#project/src` and verify the composition loaded in Studio. - Screenshots: - `/tmp/hf-gpu-browser-proof/preview-loaded.png` - `/tmp/hf-gpu-browser-proof/preview-playing.png` - `/tmp/hf-gpu-browser-proof/preview-frame-60.png` - Agent-browser recordings: - `/tmp/hf-gpu-browser-proof/preview-playback.webm` - `/tmp/hf-gpu-browser-proof/preview-seek.webm` ## Notes - Browser GPU is enabled automatically for local CLI renders and disabled in Docker. - `--no-browser-gpu` is the opt-out for software Chrome/WebGL capture. - `--gpu` remains encoder-only and opt-in. - The Apple presentation zip has existing lint errors around unmanaged nested videos and imperative media `play()` calls. The benchmark still compares the same supplied source across modes, but it should not be treated as a clean deterministic-composition fixture.
255 lines
10 KiB
TypeScript
255 lines
10 KiB
TypeScript
/**
|
|
* Engine Configuration
|
|
*
|
|
* Typed configuration for the rendering pipeline. Replaces the PRODUCER_*
|
|
* env var sprawl with a structured interface. Env vars still work as
|
|
* fallbacks for backward compatibility during migration.
|
|
*/
|
|
|
|
/**
|
|
* Full engine configuration. All fields are wired through the config
|
|
* object; env vars serve as backward-compatible fallbacks resolved
|
|
* in `resolveConfig()`.
|
|
*/
|
|
export interface EngineConfig {
|
|
// ── Rendering ────────────────────────────────────────────────────────
|
|
fps: 24 | 30 | 60;
|
|
quality: "draft" | "standard" | "high";
|
|
format: "jpeg" | "png";
|
|
jpegQuality: number;
|
|
|
|
// ── Parallelism ──────────────────────────────────────────────────────
|
|
/** Max worker count. "auto" uses CPU-based heuristic. */
|
|
concurrency: number | "auto";
|
|
/** CPU cores allocated per worker. */
|
|
coresPerWorker: number;
|
|
/** Minimum frames before parallel workers are used. */
|
|
minParallelFrames: number;
|
|
/** Frame count threshold for "large render" heuristics. */
|
|
largeRenderThreshold: number;
|
|
|
|
// ── Browser ──────────────────────────────────────────────────────────
|
|
chromePath?: string;
|
|
disableGpu: boolean;
|
|
/**
|
|
* Chrome/WebGL rendering backend. "software" keeps the existing SwiftShader
|
|
* path for reproducible output; "hardware" lets Chrome use the host GPU.
|
|
*/
|
|
browserGpuMode: "software" | "hardware";
|
|
enableBrowserPool: boolean;
|
|
browserTimeout: number;
|
|
protocolTimeout: number;
|
|
/** Expected Chromium major version (optional validation). */
|
|
expectedChromiumMajor?: number;
|
|
/** Force screenshot capture mode (skip BeginFrame even on Linux). */
|
|
forceScreenshot: boolean;
|
|
|
|
// ── Encoding ─────────────────────────────────────────────────────────
|
|
enableChunkedEncode: boolean;
|
|
chunkSizeFrames: number;
|
|
enableStreamingEncode: boolean;
|
|
|
|
// ── FFmpeg timeouts ──────────────────────────────────────────────────
|
|
/** Timeout for FFmpeg frame encoding (ms). Default: 600_000 */
|
|
ffmpegEncodeTimeout: number;
|
|
/** Timeout for FFmpeg mux/faststart processes (ms). Default: 300_000 */
|
|
ffmpegProcessTimeout: number;
|
|
/** Timeout for FFmpeg streaming encode (ms). Default: 600_000 */
|
|
ffmpegStreamingTimeout: number;
|
|
|
|
// ── HDR ──────────────────────────────────────────────────────────────
|
|
/** HDR output transfer function. false = SDR output (default). */
|
|
hdr: { transfer: "hlg" | "pq" } | false;
|
|
/** Auto-detect HDR from video sources when hdr is not explicitly set. */
|
|
hdrAutoDetect: boolean;
|
|
|
|
// ── Media ────────────────────────────────────────────────────────────
|
|
audioGain: number;
|
|
frameDataUriCacheLimit: number;
|
|
|
|
// ── Timeouts ─────────────────────────────────────────────────────────
|
|
playerReadyTimeout: number;
|
|
renderReadyTimeout: number;
|
|
|
|
// ── Runtime ──────────────────────────────────────────────────────────
|
|
/** Verify Hyperframe runtime SHA256 checksums. */
|
|
verifyRuntime: boolean;
|
|
/** Custom manifest path for Hyperframe runtime. */
|
|
runtimeManifestPath?: string;
|
|
|
|
// ── Cache ────────────────────────────────────────────────────────────
|
|
/**
|
|
* Directory where the content-addressed extraction cache persists frame
|
|
* bundles keyed on (path, mtime, size, mediaStart, duration, fps, format).
|
|
* Undefined disables caching — extraction runs into the render's workDir
|
|
* and cleanup removes it when the render ends, preserving the pre-cache
|
|
* behaviour.
|
|
*
|
|
* **Single-writer.** The cache is not safe for concurrent renders pointing
|
|
* at the same directory. A `.hf-complete` sentinel prevents another render
|
|
* from serving an entry that hasn't finished extracting, but individual
|
|
* frame files are written non-atomically — a second render reading during
|
|
* the write window can observe a truncated frame. Give each concurrent
|
|
* render pipeline its own `extractCacheDir`, or gate with an external mutex.
|
|
*
|
|
* **Network filesystems.** `mtime` resolution on NFS/SMB mounts can be
|
|
* coarser than expected (seconds rather than nanoseconds), which may
|
|
* produce spurious cache hits if a source file is overwritten within the
|
|
* same mtime tick. Local filesystems are the intended deployment target.
|
|
*
|
|
* Env fallback: `HYPERFRAMES_EXTRACT_CACHE_DIR`.
|
|
*/
|
|
extractCacheDir?: string;
|
|
|
|
// ── Debug ────────────────────────────────────────────────────────────
|
|
debug: boolean;
|
|
}
|
|
|
|
/** Default configuration — sensible for Hyperframes compositions. */
|
|
export const DEFAULT_CONFIG: EngineConfig = {
|
|
fps: 30,
|
|
quality: "standard",
|
|
format: "jpeg",
|
|
jpegQuality: 80,
|
|
|
|
concurrency: "auto",
|
|
coresPerWorker: 2.5,
|
|
minParallelFrames: 120,
|
|
largeRenderThreshold: 1000,
|
|
|
|
disableGpu: false,
|
|
browserGpuMode: "software",
|
|
enableBrowserPool: false,
|
|
browserTimeout: 120_000,
|
|
protocolTimeout: 300_000,
|
|
forceScreenshot: false,
|
|
|
|
enableChunkedEncode: false,
|
|
chunkSizeFrames: 360,
|
|
enableStreamingEncode: false,
|
|
|
|
ffmpegEncodeTimeout: 600_000,
|
|
ffmpegProcessTimeout: 300_000,
|
|
ffmpegStreamingTimeout: 600_000,
|
|
|
|
hdr: false,
|
|
hdrAutoDetect: true,
|
|
|
|
audioGain: 1,
|
|
frameDataUriCacheLimit: 256,
|
|
|
|
playerReadyTimeout: 45_000,
|
|
renderReadyTimeout: 15_000,
|
|
|
|
verifyRuntime: true,
|
|
|
|
debug: false,
|
|
};
|
|
|
|
/**
|
|
* Resolve configuration by merging: defaults ← env vars ← explicit overrides.
|
|
* Env vars provide backward compatibility during migration; explicit config
|
|
* takes precedence over everything.
|
|
*/
|
|
export function resolveConfig(overrides?: Partial<EngineConfig>): EngineConfig {
|
|
const env = (key: string): string | undefined => process.env[key];
|
|
const envNum = (key: string, fallback: number): number => {
|
|
const raw = env(key);
|
|
if (raw === undefined || raw === "") return fallback;
|
|
const n = Number(raw);
|
|
return Number.isFinite(n) ? n : fallback;
|
|
};
|
|
const envBool = (key: string, fallback: boolean): boolean => {
|
|
const raw = env(key);
|
|
if (raw === undefined) return fallback;
|
|
return raw === "true";
|
|
};
|
|
const envBrowserGpuMode = (): EngineConfig["browserGpuMode"] => {
|
|
const raw = env("PRODUCER_BROWSER_GPU_MODE");
|
|
if (raw === "hardware" || raw === "software") return raw;
|
|
return DEFAULT_CONFIG.browserGpuMode;
|
|
};
|
|
|
|
// Env-var layer (backward compat)
|
|
const fromEnv: Partial<EngineConfig> = {
|
|
concurrency: env("PRODUCER_MAX_WORKERS") ? Number(env("PRODUCER_MAX_WORKERS")) : undefined,
|
|
coresPerWorker: envNum("PRODUCER_CORES_PER_WORKER", DEFAULT_CONFIG.coresPerWorker),
|
|
minParallelFrames: envNum("PRODUCER_MIN_PARALLEL_FRAMES", DEFAULT_CONFIG.minParallelFrames),
|
|
largeRenderThreshold: envNum(
|
|
"PRODUCER_LARGE_RENDER_THRESHOLD",
|
|
DEFAULT_CONFIG.largeRenderThreshold,
|
|
),
|
|
|
|
chromePath: env("PRODUCER_HEADLESS_SHELL_PATH"),
|
|
disableGpu: envBool("PRODUCER_DISABLE_GPU", DEFAULT_CONFIG.disableGpu),
|
|
browserGpuMode: envBrowserGpuMode(),
|
|
enableBrowserPool: envBool("PRODUCER_ENABLE_BROWSER_POOL", DEFAULT_CONFIG.enableBrowserPool),
|
|
browserTimeout: envNum("PRODUCER_PUPPETEER_LAUNCH_TIMEOUT_MS", DEFAULT_CONFIG.browserTimeout),
|
|
protocolTimeout: envNum(
|
|
"PRODUCER_PUPPETEER_PROTOCOL_TIMEOUT_MS",
|
|
DEFAULT_CONFIG.protocolTimeout,
|
|
),
|
|
expectedChromiumMajor: env("PRODUCER_EXPECTED_CHROMIUM_MAJOR")
|
|
? Number(env("PRODUCER_EXPECTED_CHROMIUM_MAJOR"))
|
|
: undefined,
|
|
|
|
forceScreenshot: envBool("PRODUCER_FORCE_SCREENSHOT", DEFAULT_CONFIG.forceScreenshot),
|
|
|
|
enableChunkedEncode: envBool(
|
|
"PRODUCER_ENABLE_CHUNKED_ENCODE",
|
|
DEFAULT_CONFIG.enableChunkedEncode,
|
|
),
|
|
chunkSizeFrames: Math.max(
|
|
120,
|
|
envNum("PRODUCER_CHUNK_SIZE_FRAMES", DEFAULT_CONFIG.chunkSizeFrames),
|
|
),
|
|
enableStreamingEncode: envBool(
|
|
"PRODUCER_ENABLE_STREAMING_ENCODE",
|
|
DEFAULT_CONFIG.enableStreamingEncode,
|
|
),
|
|
|
|
ffmpegEncodeTimeout: envNum("FFMPEG_ENCODE_TIMEOUT_MS", DEFAULT_CONFIG.ffmpegEncodeTimeout),
|
|
ffmpegProcessTimeout: envNum("FFMPEG_PROCESS_TIMEOUT_MS", DEFAULT_CONFIG.ffmpegProcessTimeout),
|
|
ffmpegStreamingTimeout: envNum(
|
|
"FFMPEG_STREAMING_TIMEOUT_MS",
|
|
DEFAULT_CONFIG.ffmpegStreamingTimeout,
|
|
),
|
|
|
|
hdr: (() => {
|
|
const raw = env("PRODUCER_HDR_TRANSFER");
|
|
if (raw === "hlg" || raw === "pq") return { transfer: raw };
|
|
return false;
|
|
})(),
|
|
hdrAutoDetect: envBool("PRODUCER_HDR_AUTO_DETECT", DEFAULT_CONFIG.hdrAutoDetect),
|
|
|
|
audioGain: envNum("PRODUCER_AUDIO_GAIN", DEFAULT_CONFIG.audioGain),
|
|
frameDataUriCacheLimit: Math.max(
|
|
32,
|
|
envNum("PRODUCER_FRAME_DATA_URI_CACHE_LIMIT", DEFAULT_CONFIG.frameDataUriCacheLimit),
|
|
),
|
|
|
|
playerReadyTimeout: envNum(
|
|
"PRODUCER_PLAYER_READY_TIMEOUT_MS",
|
|
DEFAULT_CONFIG.playerReadyTimeout,
|
|
),
|
|
renderReadyTimeout: envNum(
|
|
"PRODUCER_RENDER_READY_TIMEOUT_MS",
|
|
DEFAULT_CONFIG.renderReadyTimeout,
|
|
),
|
|
|
|
verifyRuntime: env("PRODUCER_VERIFY_HYPERFRAME_RUNTIME") !== "false",
|
|
runtimeManifestPath: env("PRODUCER_HYPERFRAME_MANIFEST_PATH"),
|
|
|
|
extractCacheDir: env("HYPERFRAMES_EXTRACT_CACHE_DIR"),
|
|
};
|
|
|
|
// Remove undefined values so they don't override defaults
|
|
const cleanEnv = Object.fromEntries(Object.entries(fromEnv).filter(([, v]) => v !== undefined));
|
|
|
|
return {
|
|
...DEFAULT_CONFIG,
|
|
...cleanEnv,
|
|
...overrides,
|
|
};
|
|
}
|