feat: add browser GPU render mode (#571)

## Problem

HyperFrames already had `--gpu`, but that flag only controlled FFmpeg hardware encoding. The browser capture path still forced Chrome/WebGL through SwiftShader software GL via `--use-angle=swiftshader`, so WebGL-heavy local renders could leave the biggest bottleneck on the CPU path.

That made the existing flag naming easy to misread: `--gpu` sounded like it accelerated the whole render, but it did not change the browser frame-capture backend.

## What this fixes

- Enables host browser GPU acceleration automatically for local CLI renders.
- Adds `--no-browser-gpu` as the local opt-out for software Chrome/WebGL capture.
- Keeps `--browser-gpu` as an explicit local browser-GPU request.
- Adds `browserGpuMode: "software" | "hardware"` to engine config, with `PRODUCER_BROWSER_GPU_MODE` env support for lower-level producer users.
- Keeps Docker browser capture on the deterministic software path.
- Maps hardware browser GPU mode to platform-native Chrome backends:
  - macOS: Metal-backed ANGLE
  - Windows: D3D11-backed ANGLE
  - Linux: EGL
- Blocks explicit `--browser-gpu --docker` with a clear error because Docker browser GPU passthrough is not cross-platform.
- Clarifies docs so `--gpu` means FFmpeg encoder GPU and browser GPU means Chrome/WebGL capture GPU.
- Keeps encoder backend selection auto-detected from FFmpeg capabilities:
  - NVIDIA: NVENC
  - macOS: VideoToolbox
  - Linux: VAAPI
  - Intel: QSV

## Why two flags

There are two separate GPU surfaces in the render pipeline:

1. Browser GPU controls Chrome frame capture.
   - Affects WebGL, canvas, CSS rendering, compositing, and screenshot capture inside the browser.
   - This is enabled automatically for local CLI renders.
   - Use `--no-browser-gpu` when you want the software browser baseline.

2. `--gpu` controls FFmpeg video encoding.
   - Affects the final encode step after frames have already been captured.
   - The concrete encoder is auto-detected from the host FFmpeg build and hardware.
   - It can be faster for some machines/codecs, but it is not equivalent to browser rendering acceleration.

The controls stay independent because users may want:

- `hyperframes render` for the fast local default with browser GPU capture.
- `hyperframes render --no-browser-gpu` for the software-browser local baseline.
- `hyperframes render --gpu` for browser GPU capture plus hardware FFmpeg encoding.
- `hyperframes render --no-browser-gpu --gpu` for software browser capture plus hardware FFmpeg encoding.
- `hyperframes render --docker` for deterministic browser capture.

## Why `--gpu` does not imply browser GPU

Keeping `--gpu` scoped to FFmpeg encoding avoids a semantic break and keeps the risk profile explicit:

- `--gpu` already means encoder acceleration. Expanding it to also change Chrome capture would silently alter behavior for users who only wanted hardware encoding.
- Browser GPU and encoder GPU have different portability. Encoder GPU can work in Docker when the host exposes the right devices; browser GPU passthrough is not cross-platform, so this PR intentionally blocks explicit `--browser-gpu --docker`.
- The Apple presentation benchmark shows why the controls should stay separate: browser GPU capture was the useful improvement, while macOS VideoToolbox via `--gpu` was slower and produced larger output for this `standard` H.264 run.

If HyperFrames later wants a single umbrella acceleration control, it should be explicit, for example `--acceleration browser|encoder|all` or `--gpu=browser|encoder|all`, rather than changing the meaning of the existing boolean `--gpu`.

## Root cause

`buildChromeArgs()` always injected `--use-gl=angle --use-angle=swiftshader`. `disableGpu` only appended `--disable-gpu`; it did not provide a hardware-GPU mode. That made the public `--gpu` flag look broader than it was, because render capture stayed software-backed even when encoder GPU was requested.

## Verification

### Local checks

- `bun install`
- `bun run build:hyperframes-runtime`
- `bun run --filter @hyperframes/engine test src/config.test.ts src/services/browserManager.test.ts`
- `bun run --filter @hyperframes/cli test src/utils/dockerRunArgs.test.ts src/commands/render.test.ts`
- `bun run --filter @hyperframes/cli typecheck`
- `bun run --filter @hyperframes/engine typecheck`
- `bun run --filter @hyperframes/producer typecheck`
- `cd packages/producer && bunx vitest run src/services/renderOrchestrator.test.ts`
- `bunx oxlint packages/cli/src/commands/render.ts packages/cli/src/commands/render.test.ts packages/cli/src/utils/dockerRunArgs.ts packages/cli/src/utils/dockerRunArgs.test.ts packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/browserManager.ts packages/engine/src/services/browserManager.test.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `bunx oxfmt --check ...` on changed source/docs files
- `git diff --check`
- `bun packages/cli/src/cli.ts render --help | rg -n "browser-gpu|no-browser-gpu|GPU"`
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --output /tmp/hf-auto-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict`
  - Render plan prints `GPU: browser GPU (auto)`.
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --no-browser-gpu --output /tmp/hf-software-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict`
  - Render plan does not print browser GPU.
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --docker --browser-gpu --output /tmp/should-not-render.mp4`
  - Exits 1 with `Browser GPU is local-only`.
- `buildDockerRunArgs()` regression coverage asserts Docker container args include `--no-browser-gpu`, preventing nested container renders from re-enabling browser GPU through the local CLI default.
- `resolveBrowserGpuForCli()` regression coverage asserts `PRODUCER_BROWSER_GPU_MODE=software` opts out when no CLI browser-GPU flag is supplied, while explicit `--browser-gpu` / `--no-browser-gpu` still win.
- `ffmpeg -v error -i /tmp/hf-auto-browser-gpu-smoke.mp4 -f null -`
- `ffmpeg -v error -i /tmp/hf-software-browser-gpu-smoke.mp4 -f null -`
- `ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of json /tmp/hf-browser-gpu-smoke.mp4` -> H.264, 1920x1080, 24fps, 5.0s

### Apple presentation benchmark

Rendered `/Users/miguel07code/Downloads/apple-presentation.zip` as supplied after extracting to `/tmp/hf-apple-profile/apple-presentation`.

Fixed settings:

- 1920x1080
- 30fps
- `standard` quality
- 4240 frames
- 141.32s duration
- 8-worker cap; render auto-calibration used 6 capture workers
- macOS host detected FFmpeg GPU encoder: `videotoolbox`

| Mode | Equivalent flags after this PR | Wall time | vs software-browser baseline | Speed | Capture | Encode | Output |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Software browser + CPU encode | `--no-browser-gpu` | 120.77s | baseline | 1.17x | 97.87s | 10.04s | 8.38MB |
| Browser GPU + CPU encode | default local render | 70.10s | 42.0% faster | 2.02x | 50.72s | 9.91s | 8.39MB |
| Software browser + encoder GPU | `--no-browser-gpu --gpu` | 133.16s | 10.3% slower | 1.06x | 103.58s | 18.31s | 25.43MB |
| Browser GPU + encoder GPU | `--gpu` | 74.12s | 38.6% faster | 1.91x | 46.69s | 17.93s | 25.45MB |

Result: browser GPU capture is the meaningful improvement for this WebGL/browser-capture-heavy presentation. VideoToolbox encoding was slower and produced larger files for this current `standard` H.264 path, so `--gpu` should stay separate and opt-in.

Why `--gpu` plus browser GPU was slower than browser GPU alone: the combined run captured about 4.0s faster than browser GPU alone, but VideoToolbox encoding was about 8.0s slower than CPU x264 encoding, so the encode loss outweighed the capture gain.

### VideoToolbox flag check

I also isolated the encode stage against the already-captured Apple frames to check whether macOS GPU encoding only needed special flags.

`ffmpeg -h encoder=h264_videotoolbox` does not expose a CRF/CQ-style quality option like x264. It exposes bitrate-oriented and VideoToolbox-specific options such as `-b:v`, `-realtime`, `-profile`, `-coder`, `-prio_speed`, `-power_efficient`, and `-allow_sw`. That means our current `-q:v` mapping is not equivalent to x264 CRF and can produce very different bitrate/size behavior.

Measured full-frame encode variants on this host:

| VideoToolbox variant | Encode wall time | Output size | Bitrate |
| --- | ---: | ---: | ---: |
| Current `-q:v 64 -allow_sw 1` | 18.76s | 25.31MB | 1.43 Mbps |
| Current without `-allow_sw 1` | 18.21s | 25.31MB | 1.43 Mbps |
| `-b:v 500k -maxrate 750k -bufsize 1000k -profile high -coder cabac -realtime 1 -prio_speed 1 -power_efficient 0` | 20.58s | 7.42MB | 0.42 Mbps |
| Same with `-b:v 1500k` | 20.84s | 16.70MB | 0.95 Mbps |
| `-b:v 500k -profile baseline -coder cavlc -realtime 1 -prio_speed 1 -power_efficient 0` | 18.11s | 8.94MB | 0.51 Mbps |

Conclusion: VideoToolbox can be made size/bitrate-predictable with explicit `--video-bitrate`, but the tested speed-oriented flags did not make it faster than CPU x264 wall time for this render. That reinforces keeping `--gpu` encoder acceleration explicit and separate from browser GPU capture.

Artifacts from the local benchmark:

- `/tmp/hf-apple-profile/results/cpu.mp4`
- `/tmp/hf-apple-profile/results/browser-gpu.mp4`
- `/tmp/hf-apple-profile/results/encoder-gpu.mp4`
- `/tmp/hf-apple-profile/results/full-gpu.mp4`
- `/tmp/hf-apple-profile/results/summary.json`

All four benchmark MP4s completed `ffprobe` and full `ffmpeg -f null` decode checks.

### Pixel comparison

Compared decoded MP4 output between software-browser and browser-GPU renders:

- Apple presentation:
  - 4240 frames compared
  - 636 exact matching decoded frame hashes
  - 3604 different decoded frame hashes
  - Average PSNR: 57.79 dB
- `css-spinner-render-compat` clean fixture:
  - 120 frames compared
  - 0 exact matching decoded frame hashes
  - Average PSNR: 61.57 dB

Interpretation: browser GPU output is not strict hash/pixel-identical to the software-browser path after lossy H.264 encode, but the measured deltas are visually tiny. Above 50 dB PSNR is typically visually indistinguishable for normal video review. Use `--no-browser-gpu` or Docker when strict cross-run/cross-machine reproducibility matters more than local speed.

### Browser verification

- Started HyperFrames Studio preview for `packages/producer/tests/css-spinner-render-compat/src`.
- Used `agent-browser` to open `http://localhost:5191#project/src` and verify the composition loaded in Studio.
- Screenshots:
  - `/tmp/hf-gpu-browser-proof/preview-loaded.png`
  - `/tmp/hf-gpu-browser-proof/preview-playing.png`
  - `/tmp/hf-gpu-browser-proof/preview-frame-60.png`
- Agent-browser recordings:
  - `/tmp/hf-gpu-browser-proof/preview-playback.webm`
  - `/tmp/hf-gpu-browser-proof/preview-seek.webm`

## Notes

- Browser GPU is enabled automatically for local CLI renders and disabled in Docker.
- `--no-browser-gpu` is the opt-out for software Chrome/WebGL capture.
- `--gpu` remains encoder-only and opt-in.
- The Apple presentation zip has existing lint errors around unmanaged nested videos and imperative media `play()` calls. The benchmark still compares the same supplied source across modes, but it should not be treated as a clean deterministic-composition fixture.
This commit is contained in:
Miguel Ángel
2026-04-30 06:46:14 +02:00
committed by GitHub
parent 4ab304e22f
commit 395fb9c084
16 changed files with 321 additions and 18 deletions
+102
View File
@@ -0,0 +1,102 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
const producerState = vi.hoisted(() => ({
createdJobs: [] as Array<Record<string, unknown>>,
resolveConfigCalls: [] as Array<Record<string, unknown>>,
}));
vi.mock("../utils/producer.js", () => ({
loadProducer: vi.fn(async () => ({
resolveConfig: vi.fn((overrides: Record<string, unknown>) => {
producerState.resolveConfigCalls.push(overrides);
return { ...overrides, resolved: true };
}),
createRenderJob: vi.fn((config: Record<string, unknown>) => {
producerState.createdJobs.push(config);
return { config, progress: 100 };
}),
executeRenderJob: vi.fn(async () => undefined),
})),
}));
vi.mock("../telemetry/events.js", () => ({
trackRenderComplete: vi.fn(),
trackRenderError: vi.fn(),
}));
describe("renderLocal browser GPU config", () => {
const savedEnv = new Map<string, string | undefined>();
function setEnv(key: string, value: string) {
savedEnv.set(key, process.env[key]);
process.env[key] = value;
}
beforeEach(() => {
producerState.createdJobs = [];
producerState.resolveConfigCalls = [];
savedEnv.clear();
});
afterEach(() => {
for (const [key, value] of savedEnv) {
if (value === undefined) {
delete process.env[key];
} else {
process.env[key] = value;
}
}
vi.clearAllMocks();
});
it("passes an explicit software override for --no-browser-gpu even when env requests hardware", async () => {
setEnv("PRODUCER_BROWSER_GPU_MODE", "hardware");
const { renderLocal } = await import("./render.js");
await renderLocal("/tmp/project", "/tmp/out.mp4", {
fps: 30,
quality: "standard",
format: "mp4",
gpu: false,
browserGpu: false,
hdrMode: "auto",
quiet: true,
});
expect(producerState.resolveConfigCalls).toContainEqual({ browserGpuMode: "software" });
expect(producerState.createdJobs[0]?.producerConfig).toMatchObject({
browserGpuMode: "software",
resolved: true,
});
});
it("passes an explicit hardware override for default local browser GPU", async () => {
const { renderLocal } = await import("./render.js");
await renderLocal("/tmp/project", "/tmp/out.mp4", {
fps: 30,
quality: "standard",
format: "mp4",
gpu: false,
browserGpu: true,
hdrMode: "auto",
quiet: true,
});
expect(producerState.resolveConfigCalls).toContainEqual({ browserGpuMode: "hardware" });
expect(producerState.createdJobs[0]?.producerConfig).toMatchObject({
browserGpuMode: "hardware",
resolved: true,
});
});
it("resolves browser GPU from CLI flags, Docker mode, and env fallback", async () => {
const { resolveBrowserGpuForCli } = await import("./render.js");
expect(resolveBrowserGpuForCli(false, undefined, undefined)).toBe(true);
expect(resolveBrowserGpuForCli(false, undefined, "hardware")).toBe(true);
expect(resolveBrowserGpuForCli(false, undefined, "software")).toBe(false);
expect(resolveBrowserGpuForCli(false, true, "software")).toBe(true);
expect(resolveBrowserGpuForCli(false, false, "hardware")).toBe(false);
expect(resolveBrowserGpuForCli(true, undefined, "hardware")).toBe(false);
});
});
+43 -1
View File
@@ -9,6 +9,7 @@ export const examples: Example[] = [
["High quality at 60fps", "hyperframes render --fps 60 --quality high --output hd.mp4"],
["Deterministic render via Docker", "hyperframes render --docker --output deterministic.mp4"],
["Parallel rendering with 6 workers", "hyperframes render --workers 6 --output fast.mp4"],
["Opt out of browser GPU render", "hyperframes render --no-browser-gpu --output cpu.mp4"],
["HDR output (auto-detected)", "hyperframes render --output hdr-output.mp4"],
];
import { cpus, freemem, tmpdir } from "node:os";
@@ -99,6 +100,11 @@ export default defineCommand({
description: "Target video bitrate such as 10M. Mutually exclusive with --crf.",
},
gpu: { type: "boolean", description: "Use GPU encoding", default: false },
"browser-gpu": {
type: "boolean",
description:
"Use host GPU acceleration for Chrome/WebGL capture. Enabled by default for local renders; use --no-browser-gpu to opt out.",
},
quiet: {
type: "boolean",
description: "Suppress verbose output",
@@ -186,6 +192,8 @@ export default defineCommand({
const useDocker = args.docker ?? false;
const useGpu = args.gpu ?? false;
const browserGpuArg = args["browser-gpu"];
const useBrowserGpu = resolveBrowserGpuForCli(useDocker, browserGpuArg);
const quiet = args.quiet ?? false;
const strictAll = args["strict-all"] ?? false;
const strictErrors = (args.strict ?? false) || strictAll;
@@ -197,6 +205,15 @@ export default defineCommand({
process.exit(1);
}
if (useDocker && browserGpuArg === true) {
errorBox(
"Browser GPU is local-only",
"--browser-gpu uses the host Chrome GPU backend. Docker mode keeps browser rendering deterministic and does not expose a cross-platform Chrome GPU backend.",
"Run without --docker, or use --gpu for Docker GPU encoding where your Docker host supports GPU passthrough.",
);
process.exit(1);
}
let crf: number | undefined;
if (crfRaw != null) {
const parsed = Number(crfRaw);
@@ -227,6 +244,13 @@ export default defineCommand({
c.dim(" \u2192 " + outputPath),
);
console.log(c.dim(" " + fps + "fps \u00B7 " + quality + " \u00B7 " + workerLabel));
if (useGpu || useBrowserGpu) {
const gpuModes = [
useGpu ? "encoder GPU" : null,
useBrowserGpu ? "browser GPU (auto)" : null,
].filter(Boolean);
console.log(c.dim(" GPU: " + gpuModes.join(" + ")));
}
console.log("");
}
@@ -312,6 +336,7 @@ export default defineCommand({
format,
workers,
gpu: useGpu,
browserGpu: useBrowserGpu,
hdrMode: args.sdr ? "force-sdr" : args.hdr ? "force-hdr" : "auto",
crf,
videoBitrate,
@@ -324,6 +349,7 @@ export default defineCommand({
format,
workers,
gpu: useGpu,
browserGpu: useBrowserGpu,
hdrMode: args.sdr ? "force-sdr" : args.hdr ? "force-hdr" : "auto",
crf,
videoBitrate,
@@ -340,6 +366,7 @@ interface RenderOptions {
format: "mp4" | "webm" | "mov";
workers?: number;
gpu: boolean;
browserGpu: boolean;
hdrMode: "auto" | "force-hdr" | "force-sdr";
crf?: number;
videoBitrate?: string;
@@ -347,6 +374,17 @@ interface RenderOptions {
browserPath?: string;
}
export function resolveBrowserGpuForCli(
useDocker: boolean,
browserGpuArg: boolean | undefined,
envMode = process.env.PRODUCER_BROWSER_GPU_MODE,
): boolean {
if (useDocker) return false;
if (browserGpuArg !== undefined) return browserGpuArg;
if (envMode === "software") return false;
return true;
}
const DOCKER_IMAGE_PREFIX = "hyperframes-renderer";
function dockerImageTag(version: string): string {
@@ -464,6 +502,7 @@ async function renderDocker(
format: options.format,
workers: options.workers,
gpu: options.gpu,
browserGpu: options.browserGpu,
hdrMode: options.hdrMode,
crf: options.crf,
videoBitrate: options.videoBitrate,
@@ -508,7 +547,7 @@ async function renderDocker(
printRenderComplete(outputPath, elapsed, options.quiet);
}
async function renderLocal(
export async function renderLocal(
projectDir: string,
outputPath: string,
options: RenderOptions,
@@ -530,6 +569,9 @@ async function renderLocal(
format: options.format,
workers: options.workers,
useGpu: options.gpu,
producerConfig: producer.resolveConfig({
browserGpuMode: options.browserGpu ? "hardware" : "software",
}),
hdrMode: options.hdrMode,
crf: options.crf,
videoBitrate: options.videoBitrate,
+4 -1
View File
@@ -19,11 +19,14 @@ Requires: Docker installed and running.
- `-w, --workers` — Parallel workers 1-8 (default: auto)
- `--crf` — Override encoder CRF (mutually exclusive with `--video-bitrate`)
- `--video-bitrate` — Target video bitrate such as `10M` (mutually exclusive with `--crf`)
- `--gpu` — Use GPU encoding (NVENC, VideoToolbox, VAAPI)
- `--gpu` — Use GPU encoding (NVENC, VideoToolbox, VAAPI, QSV)
- `--browser-gpu` / `--no-browser-gpu` — Use or opt out of host GPU acceleration for local Chrome/WebGL capture (enabled by default for local renders, disabled in Docker)
- `-o, --output` — Custom output path
## Tips
- Use `draft` quality for fast previews during development
- Local renders use browser GPU capture automatically; use `--no-browser-gpu` to compare against the software-browser path
- Use `--gpu` when a local render also benefits from hardware FFmpeg encoding
- Use `npx hyperframes benchmark` to find optimal settings
- 4 workers is usually the sweet spot for most compositions
@@ -6,6 +6,7 @@ const BASE: DockerRenderOptions = {
quality: "standard",
format: "mp4",
gpu: false,
browserGpu: false,
hdrMode: "auto",
crf: undefined,
videoBitrate: undefined,
@@ -42,6 +43,7 @@ describe("buildDockerRunArgs", () => {
"standard",
"--format",
"mp4",
"--no-browser-gpu",
]
`);
});
@@ -91,6 +93,7 @@ describe("buildDockerRunArgs", () => {
"18",
"--quiet",
"--gpu",
"--no-browser-gpu",
"--hdr",
]
`);
@@ -139,6 +142,11 @@ describe("buildDockerRunArgs", () => {
expect(on).toContain("--gpu");
});
it("forces software browser capture inside Docker", () => {
const args = buildDockerRunArgs({ ...FIXED_INPUT, options: BASE });
expect(args).toContain("--no-browser-gpu");
});
it("forwards every renderer-shaped option (regression tripwire for silent drops)", () => {
const args = buildDockerRunArgs({
...FIXED_INPUT,
@@ -148,6 +156,7 @@ describe("buildDockerRunArgs", () => {
format: "webm",
workers: 8,
gpu: true,
browserGpu: false,
hdrMode: "force-hdr",
crf: 16,
videoBitrate: undefined,
@@ -165,6 +174,7 @@ describe("buildDockerRunArgs", () => {
expect(args).toContain("16");
expect(args).toContain("--quiet");
expect(args).toContain("--gpu");
expect(args).toContain("--no-browser-gpu");
expect(args).toContain("--hdr");
});
+2
View File
@@ -24,6 +24,7 @@ export interface DockerRenderOptions {
format: "mp4" | "webm" | "mov";
workers?: number;
gpu: boolean;
browserGpu: boolean;
hdrMode: "auto" | "force-hdr" | "force-sdr";
crf?: number;
videoBitrate?: string;
@@ -59,6 +60,7 @@ export function buildDockerRunArgs(input: DockerRunArgsInput): string[] {
...(options.videoBitrate ? ["--video-bitrate", options.videoBitrate] : []),
...(options.quiet ? ["--quiet"] : []),
...(options.gpu ? ["--gpu"] : []),
...(options.browserGpu ? [] : ["--no-browser-gpu"]),
...(options.hdrMode === "force-hdr" ? ["--hdr"] : []),
...(options.hdrMode === "force-sdr" ? ["--sdr"] : []),
];