diff --git a/docs/packages/cli.mdx b/docs/packages/cli.mdx index 98a30dd4e2..0df3029feb 100644 --- a/docs/packages/cli.mdx +++ b/docs/packages/cli.mdx @@ -233,7 +233,7 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_ ### `render` - Render a composition to MP4: + Render a composition to MP4 or WebM: ```bash # Local mode (fast iteration) @@ -242,6 +242,9 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_ # Docker mode (deterministic output) npx hyperframes render --docker --output output.mp4 + # WebM with transparency (for overlays, captions, lower thirds) + npx hyperframes render --format webm --output overlay.webm + # With options npx hyperframes render --output output.mp4 --fps 60 --quality high ``` @@ -249,6 +252,7 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_ | Flag | Values | Default | Description | |------|--------|---------|-------------| | `--output` | path | `renders/.mp4` | Output file path | + | `--format` | mp4, webm | mp4 | Output format (WebM renders with transparency) | | `--fps` | 24, 30, 60 | 30 | Frames per second | | `--quality` | draft, standard, high | standard | Encoding quality preset | | `--workers` | 1-8 | 4 | Parallel render workers | @@ -256,6 +260,23 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_ | `--docker` | — | off | Use Docker for [deterministic rendering](/concepts/determinism) | | `--quiet` | — | off | Suppress verbose output | + #### WebM with Transparency + + Use `--format webm` to render compositions with a transparent background. This produces VP9 video with alpha channel in a WebM container — the standard format for overlayable video. + + ```bash + # Render a caption overlay with transparent background + npx hyperframes render --format webm --output captions.webm + + # Overlay on another video with FFmpeg + ffmpeg -c:v libvpx-vp9 -i captions.webm -i background.mp4 \ + -filter_complex "[1:v][0:v]overlay=0:0" -y composited.mp4 + ``` + + + For transparency to work, your composition's HTML should use `background: transparent` on the root elements. WebM renders use PNG frame capture (instead of JPEG) to preserve the alpha channel. + + See [Rendering](/guides/rendering) for all options and modes. ### `benchmark` diff --git a/docs/packages/engine.mdx b/docs/packages/engine.mdx index 9306b13364..9da65133fd 100644 --- a/docs/packages/engine.mdx +++ b/docs/packages/engine.mdx @@ -22,7 +22,7 @@ npm install @hyperframes/engine - Implement a custom encoding backend (not FFmpeg) **Use a different package if you want to:** -- Render an HTML composition to a finished MP4 — use the [producer](/packages/producer) or [CLI](/packages/cli) +- Render an HTML composition to a finished MP4 or WebM — use the [producer](/packages/producer) or [CLI](/packages/cli) - Preview compositions in the browser — use the [CLI](/packages/cli) or [studio](/packages/studio) - Lint or parse composition HTML — use [core](/packages/core) @@ -142,7 +142,7 @@ await releaseBrowser(browser); ### Encoding -The engine includes FFmpeg encoding utilities: +The engine includes FFmpeg encoding utilities with support for MP4 (h264) and WebM (VP9 with alpha): ```typescript import { @@ -150,23 +150,42 @@ import { muxVideoWithAudio, applyFaststart, detectGpuEncoder, + getEncoderPreset, ENCODER_PRESETS, } from '@hyperframes/engine'; +// Get format-aware encoder settings +const mp4Preset = getEncoderPreset('standard', 'mp4'); +// { codec: "h264", pixelFormat: "yuv420p", preset: "medium", quality: 23 } + +const webmPreset = getEncoderPreset('standard', 'webm'); +// { codec: "vp9", pixelFormat: "yuva420p", preset: "good", quality: 23 } + +// Encode captured frames to video +await encodeFramesFromDir(framesDir, 'frame_%06d.png', outputPath, { + fps: 30, + ...webmPreset, +}); + +// Mix video with audio (uses Opus for WebM, AAC for MP4) +await muxVideoWithAudio(videoPath, audioPath, outputPath); + +// Apply MP4 faststart for streaming (no-op for WebM) +await applyFaststart(inputPath, outputPath); + // Detect GPU encoding support const gpu = await detectGpuEncoder(); // gpu: "nvenc" | "videotoolbox" | "vaapi" | null - -// Encode captured frames to video -await encodeFramesFromDir({ framesDir, outputPath, fps: 30, quality: 'standard' }); - -// Mix video with audio tracks -await muxVideoWithAudio({ videoPath, audioTracks, outputPath }); - -// Apply MP4 faststart for streaming -await applyFaststart(outputPath); ``` +#### WebM with VP9 Alpha + +When encoding for transparency, use `format: "webm"` with `getEncoderPreset()`. This configures: +- **VP9 codec** (`libvpx-vp9`) with alpha-capable `yuva420p` pixel format +- **`-auto-alt-ref 0`** and **`alpha_mode=1`** metadata for proper alpha encoding +- **`-row-mt 1`** for multi-threaded VP9 encoding +- **Opus audio** in the mux step (instead of AAC for MP4) + ### Streaming Encoder For memory-efficient encoding without writing frames to disk: diff --git a/docs/packages/producer.mdx b/docs/packages/producer.mdx index 141abc0c11..19e882a19a 100644 --- a/docs/packages/producer.mdx +++ b/docs/packages/producer.mdx @@ -3,7 +3,7 @@ title: "@hyperframes/producer" description: "Full HTML-to-video rendering pipeline with encoding, audio mixing, and Docker support." --- -The producer package combines the [engine's](/packages/engine) frame capture with FFmpeg encoding to deliver a complete HTML-to-MP4 rendering pipeline. It handles runtime injection, readiness gates, audio mixing, and optional Docker-based deterministic rendering. +The producer package combines the [engine's](/packages/engine) frame capture with FFmpeg encoding to deliver a complete HTML-to-video rendering pipeline. It supports MP4 (h264) and WebM (VP9 with alpha transparency), and handles runtime injection, readiness gates, audio mixing, and optional Docker-based deterministic rendering. ```bash npm install @hyperframes/producer @@ -12,7 +12,7 @@ npm install @hyperframes/producer ## When to Use **Use `@hyperframes/producer` when you need to:** -- Render compositions to MP4 programmatically from Node.js (e.g., in a backend service or CI pipeline) +- Render compositions to MP4 or WebM programmatically from Node.js (e.g., in a backend service or CI pipeline) - Build a custom rendering service with fine-grained control over the pipeline - Run visual regression tests against golden baselines - Benchmark render performance across different configurations @@ -44,8 +44,8 @@ The producer orchestrates the full render pipeline: Uses the [engine's](/packages/engine) BeginFrame pipeline to capture each frame as a pixel buffer. - - Pipes frame buffers into FFmpeg with the selected quality preset and encoding settings. + + Pipes frame buffers into FFmpeg with the selected quality preset. MP4 uses h264; WebM uses VP9 with alpha transparency support. Extracts audio from video clips and audio elements, applies `data-volume` and `data-media-start` offsets, and mixes them into the final MP4. @@ -77,12 +77,33 @@ import type { RenderConfig } from '@hyperframes/producer'; const config: RenderConfig = { fps: 30, // 24, 30, or 60 quality: 'standard', // 'draft', 'standard', or 'high' + format: 'mp4', // 'mp4' or 'webm' (WebM renders with transparency) workers: 4, // Parallel render workers (1-8) useGpu: false, // GPU-accelerated encoding debug: false, // Debug logging }; ``` +#### WebM with Transparency + +Set `format: 'webm'` to render with a transparent background using VP9 alpha: + +```typescript +const job = createRenderJob({ + fps: 30, + quality: 'standard', + format: 'webm', +}); + +await executeRenderJob(job, './my-overlay', './overlay.webm'); +``` + +When `format: 'webm'`: +- Frames are captured as PNG (preserves alpha channel) +- Chrome's page background is set to transparent via CDP +- FFmpeg encodes with VP9 + `yuva420p` pixel format +- Audio is encoded as Opus (instead of AAC for MP4) + ### Progress Callbacks ```typescript diff --git a/packages/cli/src/commands/render.ts b/packages/cli/src/commands/render.ts index 369dfc2910..805e01d640 100644 --- a/packages/cli/src/commands/render.ts +++ b/packages/cli/src/commands/render.ts @@ -10,14 +10,29 @@ import { trackRenderComplete, trackRenderError } from "../telemetry/events.js"; const VALID_FPS = new Set([24, 30, 60]); const VALID_QUALITY = new Set(["draft", "standard", "high"]); +const VALID_FORMAT = new Set(["mp4", "webm"]); export default defineCommand({ - meta: { name: "render", description: "Render a composition to MP4" }, + meta: { + name: "render", + description: `Render a composition to MP4 or WebM + +Examples: + hyperframes render --output output.mp4 + hyperframes render --format webm --output overlay.webm # transparent WebM + hyperframes render --fps 60 --quality high --output hd.mp4 + hyperframes render --docker --output deterministic.mp4`, + }, args: { dir: { type: "positional", description: "Project directory", required: false }, output: { type: "string", description: "Output path (default: renders/.mp4)" }, fps: { type: "string", description: "Frame rate: 24, 30, 60", default: "30" }, quality: { type: "string", description: "Quality: draft, standard, high", default: "standard" }, + format: { + type: "string", + description: "Output format: mp4, webm (WebM renders with transparency)", + default: "mp4", + }, workers: { type: "string", description: "Parallel workers 1-8" }, docker: { type: "boolean", description: "Use Docker for deterministic render", default: false }, gpu: { type: "boolean", description: "Use GPU encoding", default: false }, @@ -43,6 +58,14 @@ export default defineCommand({ } const quality = qualityRaw as "draft" | "standard" | "high"; + // ── Validate format ───────────────────────────────────────────────── + const formatRaw = args.format ?? "mp4"; + if (!VALID_FORMAT.has(formatRaw)) { + errorBox("Invalid format", `Got "${formatRaw}". Must be mp4 or webm.`); + process.exit(1); + } + const format = formatRaw as "mp4" | "webm"; + // ── Validate workers ────────────────────────────────────────────────── let workers: number | undefined; if (args.workers != null) { @@ -56,7 +79,10 @@ export default defineCommand({ // ── Resolve output path ─────────────────────────────────────────────── const rendersDir = resolve("renders"); - const outputPath = args.output ? resolve(args.output) : join(rendersDir, `${project.name}.mp4`); + const ext = format === "webm" ? ".webm" : ".mp4"; + const outputPath = args.output + ? resolve(args.output) + : join(rendersDir, `${project.name}${ext}`); // Ensure output directory exists const outputDir = dirname(outputPath); @@ -129,11 +155,19 @@ export default defineCommand({ // ── Render ──────────────────────────────────────────────────────────── if (useDocker) { - await renderDocker(project.dir, outputPath, { fps, quality, workers, gpu: useGpu, quiet }); + await renderDocker(project.dir, outputPath, { + fps, + quality, + format, + workers, + gpu: useGpu, + quiet, + }); } else { await renderLocal(project.dir, outputPath, { fps, quality, + format, workers, gpu: useGpu, quiet, @@ -146,6 +180,7 @@ export default defineCommand({ interface RenderOptions { fps: 24 | 30 | 60; quality: "draft" | "standard" | "high"; + format: "mp4" | "webm"; workers?: number; gpu: boolean; quiet: boolean; @@ -164,6 +199,7 @@ async function renderDocker( const job = producer.createRenderJob({ fps: options.fps, quality: options.quality, + format: options.format, workers: options.workers, useGpu: options.gpu, }); @@ -206,6 +242,7 @@ async function renderLocal( const job = producer.createRenderJob({ fps: options.fps, quality: options.quality, + format: options.format, workers: options.workers, useGpu: options.gpu, }); diff --git a/packages/engine/src/index.ts b/packages/engine/src/index.ts index a91b6d3c4b..a29347b1b3 100644 --- a/packages/engine/src/index.ts +++ b/packages/engine/src/index.ts @@ -88,6 +88,7 @@ export { applyFaststart, detectGpuEncoder, ENCODER_PRESETS, + getEncoderPreset, type GpuEncoder, } from "./services/chunkEncoder.js"; export type { EncoderOptions, EncodeResult, MuxResult } from "./services/chunkEncoder.types.js"; diff --git a/packages/engine/src/services/browserManager.ts b/packages/engine/src/services/browserManager.ts index d12ad7fde9..1262f1c75d 100644 --- a/packages/engine/src/services/browserManager.ts +++ b/packages/engine/src/services/browserManager.ts @@ -108,11 +108,7 @@ export async function acquireBrowser( } else { // Screenshot mode with renderSeek: works on all platforms. captureMode = "screenshot"; - if (headlessShell && !isLinux) { - executablePath = headlessShell; - } else { - executablePath = undefined; // Puppeteer uses its bundled Chromium - } + executablePath = headlessShell ?? undefined; } const ppt = await getPuppeteer(); diff --git a/packages/engine/src/services/chunkEncoder.test.ts b/packages/engine/src/services/chunkEncoder.test.ts index 1628292c2e..28c7142006 100644 --- a/packages/engine/src/services/chunkEncoder.test.ts +++ b/packages/engine/src/services/chunkEncoder.test.ts @@ -1,5 +1,5 @@ import { describe, it, expect } from "vitest"; -import { ENCODER_PRESETS } from "./chunkEncoder.js"; +import { ENCODER_PRESETS, getEncoderPreset } from "./chunkEncoder.js"; describe("ENCODER_PRESETS", () => { it("has draft, standard, and high presets", () => { @@ -25,3 +25,40 @@ describe("ENCODER_PRESETS", () => { expect(ENCODER_PRESETS.standard.quality).toBeLessThan(ENCODER_PRESETS.draft.quality); }); }); + +describe("getEncoderPreset", () => { + it("returns h264 with yuv420p for mp4 format", () => { + const preset = getEncoderPreset("standard", "mp4"); + expect(preset.codec).toBe("h264"); + expect(preset.pixelFormat).toBe("yuv420p"); + }); + + it("returns vp9 with yuva420p for webm format", () => { + const preset = getEncoderPreset("standard", "webm"); + expect(preset.codec).toBe("vp9"); + expect(preset.pixelFormat).toBe("yuva420p"); + }); + + it("maps draft ultrafast to vp9 realtime deadline", () => { + const preset = getEncoderPreset("draft", "webm"); + expect(preset.preset).toBe("realtime"); + expect(preset.codec).toBe("vp9"); + }); + + it("maps standard/high to vp9 good deadline", () => { + expect(getEncoderPreset("standard", "webm").preset).toBe("good"); + expect(getEncoderPreset("high", "webm").preset).toBe("good"); + }); + + it("preserves quality values across formats", () => { + for (const q of ["draft", "standard", "high"] as const) { + expect(getEncoderPreset(q, "webm").quality).toBe(ENCODER_PRESETS[q].quality); + } + }); + + it("defaults to mp4 when format is omitted", () => { + const preset = getEncoderPreset("standard"); + expect(preset.codec).toBe("h264"); + expect(preset.pixelFormat).toBe("yuv420p"); + }); +}); diff --git a/packages/engine/src/services/chunkEncoder.ts b/packages/engine/src/services/chunkEncoder.ts index 273704fb09..34286efd7f 100644 --- a/packages/engine/src/services/chunkEncoder.ts +++ b/packages/engine/src/services/chunkEncoder.ts @@ -6,7 +6,7 @@ */ import { spawn } from "child_process"; -import { existsSync, mkdirSync, readdirSync, statSync, writeFileSync } from "fs"; +import { copyFileSync, existsSync, mkdirSync, readdirSync, statSync, writeFileSync } from "fs"; import { join, dirname } from "path"; import { DEFAULT_CONFIG, type EngineConfig } from "../config.js"; import { type GpuEncoder, getCachedGpuEncoder, getGpuEncoderName } from "../utils/gpuEncoder.js"; @@ -21,6 +21,26 @@ export const ENCODER_PRESETS = { high: { preset: "slow", quality: 18, codec: "h264" as const }, }; +/** + * Get encoder preset for a given quality and output format. + * WebM uses VP9 with alpha-capable pixel format; MP4 uses h264. + */ +export function getEncoderPreset( + quality: "draft" | "standard" | "high", + format: "mp4" | "webm" = "mp4", +): { preset: string; quality: number; codec: "h264" | "vp9"; pixelFormat: string } { + const base = ENCODER_PRESETS[quality]; + if (format === "webm") { + return { + preset: base.preset === "ultrafast" ? "realtime" : "good", + quality: base.quality, + codec: "vp9", + pixelFormat: "yuva420p", + }; + } + return { ...base, pixelFormat: "yuv420p" }; +} + // Re-export GPU utilities so existing consumers that import from chunkEncoder still work. export { detectGpuEncoder, type GpuEncoder } from "../utils/gpuEncoder.js"; @@ -83,6 +103,11 @@ function buildEncoderArgs( } else if (codec === "vp9") { args.push("-c:v", "libvpx-vp9", "-b:v", bitrate || "0", "-crf", String(quality)); args.push("-deadline", preset === "ultrafast" ? "realtime" : "good"); + args.push("-row-mt", "1"); + if (pixelFormat === "yuva420p") { + args.push("-auto-alt-ref", "0"); + args.push("-metadata:s:v:0", "alpha_mode=1"); + } } else if (codec === "prores") { args.push("-c:v", "prores_ks", "-profile:v", preset, "-vendor", "apl0"); return [...args, "-y", outputPath]; @@ -243,7 +268,8 @@ export async function encodeFramesChunkedConcat( } const startNumber = i * chunkSize; const framesInChunk = Math.min(chunkSize, files.length - startNumber); - const chunkPath = join(chunkDir, `chunk_${String(i).padStart(4, "0")}.mp4`); + const ext = outputPath.endsWith(".webm") ? ".webm" : ".mp4"; + const chunkPath = join(chunkDir, `chunk_${String(i).padStart(4, "0")}${ext}`); const inputPath = join(framesDir, framePattern); const inputArgs = [ "-framerate", @@ -347,23 +373,15 @@ export async function muxVideoWithAudio( const outputDir = dirname(outputPath); if (!existsSync(outputDir)) mkdirSync(outputDir, { recursive: true }); - const args = [ - "-i", - videoPath, - "-i", - audioPath, - "-c:v", - "copy", - "-c:a", - "aac", - "-b:a", - "192k", - "-shortest", - "-movflags", - "+faststart", - "-y", - outputPath, - ]; + const isWebm = outputPath.endsWith(".webm"); + const args = ["-i", videoPath, "-i", audioPath, "-c:v", "copy"]; + + if (isWebm) { + args.push("-c:a", "libopus", "-b:a", "128k"); + } else { + args.push("-c:a", "aac", "-b:a", "192k", "-movflags", "+faststart"); + } + args.push("-shortest", "-y", outputPath); const processTimeout = config?.ffmpegProcessTimeout ?? DEFAULT_CONFIG.ffmpegProcessTimeout; const result = await runFfmpeg(args, { signal, timeout: processTimeout }); @@ -394,6 +412,11 @@ export async function applyFaststart( signal?: AbortSignal, config?: Partial>, ): Promise { + // faststart is MP4-only (moves moov atom to file start for streaming) + if (outputPath.endsWith(".webm")) { + if (inputPath !== outputPath) copyFileSync(inputPath, outputPath); + return { success: true, outputPath, durationMs: 0 }; + } const args = ["-i", inputPath, "-c", "copy", "-movflags", "+faststart", "-y", outputPath]; const processTimeout = config?.ffmpegProcessTimeout ?? DEFAULT_CONFIG.ffmpegProcessTimeout; diff --git a/packages/engine/src/services/frameCapture.ts b/packages/engine/src/services/frameCapture.ts index 54b755c15c..60a14253b3 100644 --- a/packages/engine/src/services/frameCapture.ts +++ b/packages/engine/src/services/frameCapture.ts @@ -107,6 +107,16 @@ export async function createCaptureSession( }; await page.setViewport(viewport); + // For PNG capture (used by WebM/transparency), make the page background transparent + // so Chrome's screenshot captures alpha channel data. Must use the same CDP session + // that the screenshot service uses (getCdpSession caches per page). + if (options.format === "png") { + const cdp = await getCdpSession(page); + await cdp.send("Emulation.setDefaultBackgroundColorOverride", { + color: { r: 0, g: 0, b: 0, a: 0 }, + }); + } + return { browser, page, diff --git a/packages/engine/src/services/screenshotService.ts b/packages/engine/src/services/screenshotService.ts index 300eea7b47..90539d5e76 100644 --- a/packages/engine/src/services/screenshotService.ts +++ b/packages/engine/src/services/screenshotService.ts @@ -104,6 +104,8 @@ export async function pageScreenshotCapture(page: Page, options: CaptureOptions) const result = await client.send("Page.captureScreenshot", { format, quality: format === "jpeg" ? (options.quality ?? 80) : undefined, + fromSurface: true, + captureBeyondViewport: false, optimizeForSpeed: true, }); return Buffer.from(result.data, "base64"); diff --git a/packages/engine/src/services/streamingEncoder.ts b/packages/engine/src/services/streamingEncoder.ts index 0a3e9fe651..7a99fa1155 100644 --- a/packages/engine/src/services/streamingEncoder.ts +++ b/packages/engine/src/services/streamingEncoder.ts @@ -172,6 +172,11 @@ function buildStreamingArgs( } else if (codec === "vp9") { args.push("-c:v", "libvpx-vp9", "-b:v", bitrate || "0", "-crf", String(quality)); args.push("-deadline", preset === "ultrafast" ? "realtime" : "good"); + args.push("-row-mt", "1"); + if (pixelFormat === "yuva420p") { + args.push("-auto-alt-ref", "0"); + args.push("-metadata:s:v:0", "alpha_mode=1"); + } } else if (codec === "prores") { args.push("-c:v", "prores_ks", "-profile:v", preset, "-vendor", "apl0"); return [...args, "-y", outputPath]; diff --git a/packages/producer/src/regression-harness.ts b/packages/producer/src/regression-harness.ts index 41acc51ee5..9161e2c34f 100644 --- a/packages/producer/src/regression-harness.ts +++ b/packages/producer/src/regression-harness.ts @@ -31,6 +31,7 @@ type TestMetadata = { maxAudioLagWindows: number; renderConfig: { fps: 24 | 30 | 60; + format?: "mp4" | "webm"; // Optional: defaults to "mp4" workers?: number; // Optional: auto-calculates if omitted }; }; @@ -145,6 +146,9 @@ function validateMetadata(meta: unknown): TestMetadata { if (![24, 30, 60].includes(rc.fps as number)) { throw new Error("meta.json: 'renderConfig.fps' must be 24, 30, or 60"); } + if (rc.format !== undefined && rc.format !== "mp4" && rc.format !== "webm") { + throw new Error("meta.json: 'renderConfig.format' must be 'mp4' or 'webm' (or omit for mp4)"); + } if (rc.workers !== undefined) { if (typeof rc.workers !== "number" || rc.workers < 1) { throw new Error("meta.json: 'renderConfig.workers' must be >= 1 (or omit to auto-calculate)"); @@ -306,28 +310,37 @@ function psnrAtCheckpoint( } function extractMonoPcm16(videoPath: string): Int16Array { - const { stdout } = runFfmpeg( - [ - "-hide_banner", - "-loglevel", - "error", - "-i", - videoPath, - "-vn", - "-ac", - "1", - "-ar", - "16000", - "-f", - "s16le", - "-", - ], - `Audio extraction (${videoPath})`, - ); - if (stdout.byteLength < 2) { + try { + const { stdout } = runFfmpeg( + [ + "-hide_banner", + "-loglevel", + "error", + "-i", + videoPath, + "-vn", + "-ac", + "1", + "-ar", + "16000", + "-f", + "s16le", + "-", + ], + `Audio extraction (${videoPath})`, + ); + if (stdout.byteLength < 2) { + return new Int16Array(0); + } + return new Int16Array(stdout.buffer, stdout.byteOffset, Math.floor(stdout.byteLength / 2)); + } catch (err) { + // No audio stream (e.g., WebM without audio) — log but don't fail + const msg = err instanceof Error ? err.message : String(err); + if (!msg.includes("does not contain any stream")) { + logPretty(`Audio extraction warning: ${msg.slice(0, 200)}`, "⚠️"); + } return new Int16Array(0); } - return new Int16Array(stdout.buffer, stdout.byteOffset, Math.floor(stdout.byteLength / 2)); } function buildRmsEnvelope(samples: Int16Array, windowSize = 2048, hopSize = 1024): number[] { @@ -535,12 +548,14 @@ async function runTestSuite( mkdirSync(tempRoot, { recursive: true }); const tempDownloadDir = join(tempRoot, "downloads"); - const renderedOutputPath = join(tempRoot, "output.mp4"); + const outputFormat = suite.meta.renderConfig.format ?? "mp4"; + const videoExt = outputFormat === "webm" ? ".webm" : ".mp4"; + const renderedOutputPath = join(tempRoot, `output${videoExt}`); // Snapshot files stored in test's output/ directory const snapshotDir = join(suite.dir, "output"); const snapshotCompiledPath = join(snapshotDir, "compiled.html"); - const snapshotVideoPath = join(snapshotDir, "output.mp4"); + const snapshotVideoPath = join(snapshotDir, `output${videoExt}`); console.log(JSON.stringify({ event: "test_start", suite: suite.id, name: suite.meta.name })); logPretty(`Running test: ${suite.meta.name}`, "🧪"); @@ -626,6 +641,7 @@ async function runTestSuite( const job = createRenderJob({ fps: suite.meta.renderConfig.fps, quality: "high", // Always use max quality for tests + format: outputFormat, workers: suite.meta.renderConfig.workers, useGpu: false, debug: false, @@ -643,7 +659,11 @@ async function runTestSuite( } copyFileSync(renderedOutputPath, snapshotVideoPath); console.log( - JSON.stringify({ event: "snapshot_updated", suite: suite.id, file: "output/output.mp4" }), + JSON.stringify({ + event: "snapshot_updated", + suite: suite.id, + file: `output/output${videoExt}`, + }), ); result.visual = { passed: true, failedFrames: 0, checkpoints: [] }; result.audio = { passed: true, correlation: 1, lagWindows: 0 }; diff --git a/packages/producer/src/services/renderOrchestrator.ts b/packages/producer/src/services/renderOrchestrator.ts index f28b75e82d..5fefde788a 100644 --- a/packages/producer/src/services/renderOrchestrator.ts +++ b/packages/producer/src/services/renderOrchestrator.ts @@ -44,7 +44,7 @@ import { encodeFramesChunkedConcat, muxVideoWithAudio, applyFaststart, - ENCODER_PRESETS, + getEncoderPreset, processCompositionAudio, type AudioElement, calculateOptimalWorkers, @@ -99,6 +99,8 @@ export type RenderStatus = export interface RenderConfig { fps: 24 | 30 | 60; quality: "draft" | "standard" | "high"; + /** Output container format. WebM uses VP9+alpha for transparency. */ + format?: "mp4" | "webm"; workers?: number; useGpu?: boolean; debug?: boolean; @@ -354,7 +356,13 @@ export async function executeRenderJob( let restoreLogger: (() => void) | null = null; const perfStages: Record = {}; const perfOutputPath = join(workDir, "perf-summary.json"); - const cfg = job.config.producerConfig ?? resolveConfig(); + const cfg = { ...(job.config.producerConfig ?? resolveConfig()) }; + const outputFormat = (job.config.format ?? "mp4") as "mp4" | "webm"; + const isWebm = outputFormat === "webm"; + // WebM/transparency requires screenshot mode — beginFrame doesn't support alpha channel + if (isWebm) { + cfg.forceScreenshot = true; + } const enableChunkedEncode = cfg.enableChunkedEncode; const chunkedEncodeSize = cfg.chunkSizeFrames; const enableStreamingEncode = cfg.enableStreamingEncode; @@ -458,8 +466,8 @@ export async function executeRenderJob( width, height, fps: job.config.fps, - format: "jpeg", - quality: 80, + format: isWebm ? "png" : "jpeg", + quality: isWebm ? undefined : 80, }; probeSession = await createCaptureSession( fileServer.url, @@ -702,14 +710,15 @@ export async function executeRenderJob( width, height, fps: job.config.fps, - format: "jpeg", - quality: job.config.quality === "draft" ? 80 : 95, + format: isWebm ? "png" : "jpeg", + quality: isWebm ? undefined : job.config.quality === "draft" ? 80 : 95, }; const workerCount = calculateOptimalWorkers(job.totalFrames!, job.config.workers, cfg); - const videoOnlyPath = join(workDir, "video-only.mp4"); - const preset = ENCODER_PRESETS[job.config.quality]; + const videoExt = isWebm ? ".webm" : ".mp4"; + const videoOnlyPath = join(workDir, `video-only${videoExt}`); + const preset = getEncoderPreset(job.config.quality, outputFormat); job.framesRendered = 0; @@ -727,6 +736,7 @@ export async function executeRenderJob( codec: preset.codec, preset: preset.preset, quality: preset.quality, + pixelFormat: preset.pixelFormat, useGpu: job.config.useGpu, imageFormat: captureOptions.format || "jpeg", }, @@ -940,36 +950,32 @@ export async function executeRenderJob( const stage5Start = Date.now(); updateJobStatus(job, "encoding", "Encoding video", 75, onProgress); + const frameExt = isWebm ? "png" : "jpg"; + const framePattern = `frame_%06d.${frameExt}`; + const encoderOpts = { + fps: job.config.fps, + width, + height, + codec: preset.codec, + preset: preset.preset, + quality: preset.quality, + pixelFormat: preset.pixelFormat, + useGpu: job.config.useGpu, + }; const encodeResult = enableChunkedEncode ? await encodeFramesChunkedConcat( framesDir, - "frame_%06d.jpg", + framePattern, videoOnlyPath, - { - fps: job.config.fps, - width, - height, - codec: preset.codec, - preset: preset.preset, - quality: preset.quality, - useGpu: job.config.useGpu, - }, + encoderOpts, chunkedEncodeSize, abortSignal, ) : await encodeFramesFromDir( framesDir, - "frame_%06d.jpg", + framePattern, videoOnlyPath, - { - fps: job.config.fps, - width, - height, - codec: preset.codec, - preset: preset.preset, - quality: preset.quality, - useGpu: job.config.useGpu, - }, + encoderOpts, abortSignal, ); assertNotAborted(); @@ -1060,7 +1066,7 @@ export async function executeRenderJob( if (job.config.debug) { // Copy output MP4 into debug dir for easy access if (existsSync(outputPath)) { - const debugOutput = join(workDir, "output.mp4"); + const debugOutput = join(workDir, isWebm ? "output.webm" : "output.mp4"); copyFileSync(outputPath, debugOutput); } } else { diff --git a/packages/producer/tests/webm-transparency/meta.json b/packages/producer/tests/webm-transparency/meta.json new file mode 100644 index 0000000000..709f10927d --- /dev/null +++ b/packages/producer/tests/webm-transparency/meta.json @@ -0,0 +1,14 @@ +{ + "name": "webm-transparency", + "description": "WebM VP9 render with transparent background (alpha channel). Validates the full WebM pipeline: PNG capture, VP9 encoding with yuva420p, and alpha_mode=1 metadata.", + "tags": ["webm", "transparency", "fast"], + "minPsnr": 30, + "maxFrameFailures": 2, + "minAudioCorrelation": 0.0, + "maxAudioLagWindows": 120, + "renderConfig": { + "fps": 30, + "format": "webm", + "workers": 1 + } +} diff --git a/packages/producer/tests/webm-transparency/output/compiled.html b/packages/producer/tests/webm-transparency/output/compiled.html new file mode 100644 index 0000000000..253f5fdd1f --- /dev/null +++ b/packages/producer/tests/webm-transparency/output/compiled.html @@ -0,0 +1,49 @@ + + + + + + + +
+ +
+ + +
Overlay Test
+ + + +
+ + diff --git a/packages/producer/tests/webm-transparency/output/output.webm b/packages/producer/tests/webm-transparency/output/output.webm new file mode 100644 index 0000000000..7332d394fc Binary files /dev/null and b/packages/producer/tests/webm-transparency/output/output.webm differ diff --git a/packages/producer/tests/webm-transparency/src/index.html b/packages/producer/tests/webm-transparency/src/index.html new file mode 100644 index 0000000000..fc5ed22b46 --- /dev/null +++ b/packages/producer/tests/webm-transparency/src/index.html @@ -0,0 +1,71 @@ + + + + + + + +
+ +
+ + +
Overlay Test
+ + + +
+ +