mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-20 12:25:51 +00:00
Merge pull request #81 from heygen-com/feat/webm-transparency
feat(render): WebM output with VP9 alpha transparency
This commit is contained in:
+22
-1
@@ -233,7 +233,7 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_
|
||||
<Tab title="Build">
|
||||
### `render`
|
||||
|
||||
Render a composition to MP4:
|
||||
Render a composition to MP4 or WebM:
|
||||
|
||||
```bash
|
||||
# Local mode (fast iteration)
|
||||
@@ -242,6 +242,9 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_
|
||||
# Docker mode (deterministic output)
|
||||
npx hyperframes render --docker --output output.mp4
|
||||
|
||||
# WebM with transparency (for overlays, captions, lower thirds)
|
||||
npx hyperframes render --format webm --output overlay.webm
|
||||
|
||||
# With options
|
||||
npx hyperframes render --output output.mp4 --fps 60 --quality high
|
||||
```
|
||||
@@ -249,6 +252,7 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_
|
||||
| Flag | Values | Default | Description |
|
||||
|------|--------|---------|-------------|
|
||||
| `--output` | path | `renders/<name>.mp4` | Output file path |
|
||||
| `--format` | mp4, webm | mp4 | Output format (WebM renders with transparency) |
|
||||
| `--fps` | 24, 30, 60 | 30 | Frames per second |
|
||||
| `--quality` | draft, standard, high | standard | Encoding quality preset |
|
||||
| `--workers` | 1-8 | 4 | Parallel render workers |
|
||||
@@ -256,6 +260,23 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_
|
||||
| `--docker` | — | off | Use Docker for [deterministic rendering](/concepts/determinism) |
|
||||
| `--quiet` | — | off | Suppress verbose output |
|
||||
|
||||
#### WebM with Transparency
|
||||
|
||||
Use `--format webm` to render compositions with a transparent background. This produces VP9 video with alpha channel in a WebM container — the standard format for overlayable video.
|
||||
|
||||
```bash
|
||||
# Render a caption overlay with transparent background
|
||||
npx hyperframes render --format webm --output captions.webm
|
||||
|
||||
# Overlay on another video with FFmpeg
|
||||
ffmpeg -c:v libvpx-vp9 -i captions.webm -i background.mp4 \
|
||||
-filter_complex "[1:v][0:v]overlay=0:0" -y composited.mp4
|
||||
```
|
||||
|
||||
<Tip>
|
||||
For transparency to work, your composition's HTML should use `background: transparent` on the root elements. WebM renders use PNG frame capture (instead of JPEG) to preserve the alpha channel.
|
||||
</Tip>
|
||||
|
||||
See [Rendering](/guides/rendering) for all options and modes.
|
||||
|
||||
### `benchmark`
|
||||
|
||||
+30
-11
@@ -22,7 +22,7 @@ npm install @hyperframes/engine
|
||||
- Implement a custom encoding backend (not FFmpeg)
|
||||
|
||||
**Use a different package if you want to:**
|
||||
- Render an HTML composition to a finished MP4 — use the [producer](/packages/producer) or [CLI](/packages/cli)
|
||||
- Render an HTML composition to a finished MP4 or WebM — use the [producer](/packages/producer) or [CLI](/packages/cli)
|
||||
- Preview compositions in the browser — use the [CLI](/packages/cli) or [studio](/packages/studio)
|
||||
- Lint or parse composition HTML — use [core](/packages/core)
|
||||
|
||||
@@ -142,7 +142,7 @@ await releaseBrowser(browser);
|
||||
|
||||
### Encoding
|
||||
|
||||
The engine includes FFmpeg encoding utilities:
|
||||
The engine includes FFmpeg encoding utilities with support for MP4 (h264) and WebM (VP9 with alpha):
|
||||
|
||||
```typescript
|
||||
import {
|
||||
@@ -150,23 +150,42 @@ import {
|
||||
muxVideoWithAudio,
|
||||
applyFaststart,
|
||||
detectGpuEncoder,
|
||||
getEncoderPreset,
|
||||
ENCODER_PRESETS,
|
||||
} from '@hyperframes/engine';
|
||||
|
||||
// Get format-aware encoder settings
|
||||
const mp4Preset = getEncoderPreset('standard', 'mp4');
|
||||
// { codec: "h264", pixelFormat: "yuv420p", preset: "medium", quality: 23 }
|
||||
|
||||
const webmPreset = getEncoderPreset('standard', 'webm');
|
||||
// { codec: "vp9", pixelFormat: "yuva420p", preset: "good", quality: 23 }
|
||||
|
||||
// Encode captured frames to video
|
||||
await encodeFramesFromDir(framesDir, 'frame_%06d.png', outputPath, {
|
||||
fps: 30,
|
||||
...webmPreset,
|
||||
});
|
||||
|
||||
// Mix video with audio (uses Opus for WebM, AAC for MP4)
|
||||
await muxVideoWithAudio(videoPath, audioPath, outputPath);
|
||||
|
||||
// Apply MP4 faststart for streaming (no-op for WebM)
|
||||
await applyFaststart(inputPath, outputPath);
|
||||
|
||||
// Detect GPU encoding support
|
||||
const gpu = await detectGpuEncoder();
|
||||
// gpu: "nvenc" | "videotoolbox" | "vaapi" | null
|
||||
|
||||
// Encode captured frames to video
|
||||
await encodeFramesFromDir({ framesDir, outputPath, fps: 30, quality: 'standard' });
|
||||
|
||||
// Mix video with audio tracks
|
||||
await muxVideoWithAudio({ videoPath, audioTracks, outputPath });
|
||||
|
||||
// Apply MP4 faststart for streaming
|
||||
await applyFaststart(outputPath);
|
||||
```
|
||||
|
||||
#### WebM with VP9 Alpha
|
||||
|
||||
When encoding for transparency, use `format: "webm"` with `getEncoderPreset()`. This configures:
|
||||
- **VP9 codec** (`libvpx-vp9`) with alpha-capable `yuva420p` pixel format
|
||||
- **`-auto-alt-ref 0`** and **`alpha_mode=1`** metadata for proper alpha encoding
|
||||
- **`-row-mt 1`** for multi-threaded VP9 encoding
|
||||
- **Opus audio** in the mux step (instead of AAC for MP4)
|
||||
|
||||
### Streaming Encoder
|
||||
|
||||
For memory-efficient encoding without writing frames to disk:
|
||||
|
||||
@@ -3,7 +3,7 @@ title: "@hyperframes/producer"
|
||||
description: "Full HTML-to-video rendering pipeline with encoding, audio mixing, and Docker support."
|
||||
---
|
||||
|
||||
The producer package combines the [engine's](/packages/engine) frame capture with FFmpeg encoding to deliver a complete HTML-to-MP4 rendering pipeline. It handles runtime injection, readiness gates, audio mixing, and optional Docker-based deterministic rendering.
|
||||
The producer package combines the [engine's](/packages/engine) frame capture with FFmpeg encoding to deliver a complete HTML-to-video rendering pipeline. It supports MP4 (h264) and WebM (VP9 with alpha transparency), and handles runtime injection, readiness gates, audio mixing, and optional Docker-based deterministic rendering.
|
||||
|
||||
```bash
|
||||
npm install @hyperframes/producer
|
||||
@@ -12,7 +12,7 @@ npm install @hyperframes/producer
|
||||
## When to Use
|
||||
|
||||
**Use `@hyperframes/producer` when you need to:**
|
||||
- Render compositions to MP4 programmatically from Node.js (e.g., in a backend service or CI pipeline)
|
||||
- Render compositions to MP4 or WebM programmatically from Node.js (e.g., in a backend service or CI pipeline)
|
||||
- Build a custom rendering service with fine-grained control over the pipeline
|
||||
- Run visual regression tests against golden baselines
|
||||
- Benchmark render performance across different configurations
|
||||
@@ -44,8 +44,8 @@ The producer orchestrates the full render pipeline:
|
||||
<Step title="Capture frames via the engine">
|
||||
Uses the [engine's](/packages/engine) BeginFrame pipeline to capture each frame as a pixel buffer.
|
||||
</Step>
|
||||
<Step title="Encode to MP4 via FFmpeg">
|
||||
Pipes frame buffers into FFmpeg with the selected quality preset and encoding settings.
|
||||
<Step title="Encode to MP4 or WebM via FFmpeg">
|
||||
Pipes frame buffers into FFmpeg with the selected quality preset. MP4 uses h264; WebM uses VP9 with alpha transparency support.
|
||||
</Step>
|
||||
<Step title="Mix audio tracks">
|
||||
Extracts audio from video clips and audio elements, applies `data-volume` and `data-media-start` offsets, and mixes them into the final MP4.
|
||||
@@ -77,12 +77,33 @@ import type { RenderConfig } from '@hyperframes/producer';
|
||||
const config: RenderConfig = {
|
||||
fps: 30, // 24, 30, or 60
|
||||
quality: 'standard', // 'draft', 'standard', or 'high'
|
||||
format: 'mp4', // 'mp4' or 'webm' (WebM renders with transparency)
|
||||
workers: 4, // Parallel render workers (1-8)
|
||||
useGpu: false, // GPU-accelerated encoding
|
||||
debug: false, // Debug logging
|
||||
};
|
||||
```
|
||||
|
||||
#### WebM with Transparency
|
||||
|
||||
Set `format: 'webm'` to render with a transparent background using VP9 alpha:
|
||||
|
||||
```typescript
|
||||
const job = createRenderJob({
|
||||
fps: 30,
|
||||
quality: 'standard',
|
||||
format: 'webm',
|
||||
});
|
||||
|
||||
await executeRenderJob(job, './my-overlay', './overlay.webm');
|
||||
```
|
||||
|
||||
When `format: 'webm'`:
|
||||
- Frames are captured as PNG (preserves alpha channel)
|
||||
- Chrome's page background is set to transparent via CDP
|
||||
- FFmpeg encodes with VP9 + `yuva420p` pixel format
|
||||
- Audio is encoded as Opus (instead of AAC for MP4)
|
||||
|
||||
### Progress Callbacks
|
||||
|
||||
```typescript
|
||||
|
||||
@@ -10,14 +10,29 @@ import { trackRenderComplete, trackRenderError } from "../telemetry/events.js";
|
||||
|
||||
const VALID_FPS = new Set([24, 30, 60]);
|
||||
const VALID_QUALITY = new Set(["draft", "standard", "high"]);
|
||||
const VALID_FORMAT = new Set(["mp4", "webm"]);
|
||||
|
||||
export default defineCommand({
|
||||
meta: { name: "render", description: "Render a composition to MP4" },
|
||||
meta: {
|
||||
name: "render",
|
||||
description: `Render a composition to MP4 or WebM
|
||||
|
||||
Examples:
|
||||
hyperframes render --output output.mp4
|
||||
hyperframes render --format webm --output overlay.webm # transparent WebM
|
||||
hyperframes render --fps 60 --quality high --output hd.mp4
|
||||
hyperframes render --docker --output deterministic.mp4`,
|
||||
},
|
||||
args: {
|
||||
dir: { type: "positional", description: "Project directory", required: false },
|
||||
output: { type: "string", description: "Output path (default: renders/<name>.mp4)" },
|
||||
fps: { type: "string", description: "Frame rate: 24, 30, 60", default: "30" },
|
||||
quality: { type: "string", description: "Quality: draft, standard, high", default: "standard" },
|
||||
format: {
|
||||
type: "string",
|
||||
description: "Output format: mp4, webm (WebM renders with transparency)",
|
||||
default: "mp4",
|
||||
},
|
||||
workers: { type: "string", description: "Parallel workers 1-8" },
|
||||
docker: { type: "boolean", description: "Use Docker for deterministic render", default: false },
|
||||
gpu: { type: "boolean", description: "Use GPU encoding", default: false },
|
||||
@@ -43,6 +58,14 @@ export default defineCommand({
|
||||
}
|
||||
const quality = qualityRaw as "draft" | "standard" | "high";
|
||||
|
||||
// ── Validate format ─────────────────────────────────────────────────
|
||||
const formatRaw = args.format ?? "mp4";
|
||||
if (!VALID_FORMAT.has(formatRaw)) {
|
||||
errorBox("Invalid format", `Got "${formatRaw}". Must be mp4 or webm.`);
|
||||
process.exit(1);
|
||||
}
|
||||
const format = formatRaw as "mp4" | "webm";
|
||||
|
||||
// ── Validate workers ──────────────────────────────────────────────────
|
||||
let workers: number | undefined;
|
||||
if (args.workers != null) {
|
||||
@@ -56,7 +79,10 @@ export default defineCommand({
|
||||
|
||||
// ── Resolve output path ───────────────────────────────────────────────
|
||||
const rendersDir = resolve("renders");
|
||||
const outputPath = args.output ? resolve(args.output) : join(rendersDir, `${project.name}.mp4`);
|
||||
const ext = format === "webm" ? ".webm" : ".mp4";
|
||||
const outputPath = args.output
|
||||
? resolve(args.output)
|
||||
: join(rendersDir, `${project.name}${ext}`);
|
||||
|
||||
// Ensure output directory exists
|
||||
const outputDir = dirname(outputPath);
|
||||
@@ -129,11 +155,19 @@ export default defineCommand({
|
||||
|
||||
// ── Render ────────────────────────────────────────────────────────────
|
||||
if (useDocker) {
|
||||
await renderDocker(project.dir, outputPath, { fps, quality, workers, gpu: useGpu, quiet });
|
||||
await renderDocker(project.dir, outputPath, {
|
||||
fps,
|
||||
quality,
|
||||
format,
|
||||
workers,
|
||||
gpu: useGpu,
|
||||
quiet,
|
||||
});
|
||||
} else {
|
||||
await renderLocal(project.dir, outputPath, {
|
||||
fps,
|
||||
quality,
|
||||
format,
|
||||
workers,
|
||||
gpu: useGpu,
|
||||
quiet,
|
||||
@@ -146,6 +180,7 @@ export default defineCommand({
|
||||
interface RenderOptions {
|
||||
fps: 24 | 30 | 60;
|
||||
quality: "draft" | "standard" | "high";
|
||||
format: "mp4" | "webm";
|
||||
workers?: number;
|
||||
gpu: boolean;
|
||||
quiet: boolean;
|
||||
@@ -164,6 +199,7 @@ async function renderDocker(
|
||||
const job = producer.createRenderJob({
|
||||
fps: options.fps,
|
||||
quality: options.quality,
|
||||
format: options.format,
|
||||
workers: options.workers,
|
||||
useGpu: options.gpu,
|
||||
});
|
||||
@@ -206,6 +242,7 @@ async function renderLocal(
|
||||
const job = producer.createRenderJob({
|
||||
fps: options.fps,
|
||||
quality: options.quality,
|
||||
format: options.format,
|
||||
workers: options.workers,
|
||||
useGpu: options.gpu,
|
||||
});
|
||||
|
||||
@@ -88,6 +88,7 @@ export {
|
||||
applyFaststart,
|
||||
detectGpuEncoder,
|
||||
ENCODER_PRESETS,
|
||||
getEncoderPreset,
|
||||
type GpuEncoder,
|
||||
} from "./services/chunkEncoder.js";
|
||||
export type { EncoderOptions, EncodeResult, MuxResult } from "./services/chunkEncoder.types.js";
|
||||
|
||||
@@ -108,11 +108,7 @@ export async function acquireBrowser(
|
||||
} else {
|
||||
// Screenshot mode with renderSeek: works on all platforms.
|
||||
captureMode = "screenshot";
|
||||
if (headlessShell && !isLinux) {
|
||||
executablePath = headlessShell;
|
||||
} else {
|
||||
executablePath = undefined; // Puppeteer uses its bundled Chromium
|
||||
}
|
||||
executablePath = headlessShell ?? undefined;
|
||||
}
|
||||
|
||||
const ppt = await getPuppeteer();
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { ENCODER_PRESETS } from "./chunkEncoder.js";
|
||||
import { ENCODER_PRESETS, getEncoderPreset } from "./chunkEncoder.js";
|
||||
|
||||
describe("ENCODER_PRESETS", () => {
|
||||
it("has draft, standard, and high presets", () => {
|
||||
@@ -25,3 +25,40 @@ describe("ENCODER_PRESETS", () => {
|
||||
expect(ENCODER_PRESETS.standard.quality).toBeLessThan(ENCODER_PRESETS.draft.quality);
|
||||
});
|
||||
});
|
||||
|
||||
describe("getEncoderPreset", () => {
|
||||
it("returns h264 with yuv420p for mp4 format", () => {
|
||||
const preset = getEncoderPreset("standard", "mp4");
|
||||
expect(preset.codec).toBe("h264");
|
||||
expect(preset.pixelFormat).toBe("yuv420p");
|
||||
});
|
||||
|
||||
it("returns vp9 with yuva420p for webm format", () => {
|
||||
const preset = getEncoderPreset("standard", "webm");
|
||||
expect(preset.codec).toBe("vp9");
|
||||
expect(preset.pixelFormat).toBe("yuva420p");
|
||||
});
|
||||
|
||||
it("maps draft ultrafast to vp9 realtime deadline", () => {
|
||||
const preset = getEncoderPreset("draft", "webm");
|
||||
expect(preset.preset).toBe("realtime");
|
||||
expect(preset.codec).toBe("vp9");
|
||||
});
|
||||
|
||||
it("maps standard/high to vp9 good deadline", () => {
|
||||
expect(getEncoderPreset("standard", "webm").preset).toBe("good");
|
||||
expect(getEncoderPreset("high", "webm").preset).toBe("good");
|
||||
});
|
||||
|
||||
it("preserves quality values across formats", () => {
|
||||
for (const q of ["draft", "standard", "high"] as const) {
|
||||
expect(getEncoderPreset(q, "webm").quality).toBe(ENCODER_PRESETS[q].quality);
|
||||
}
|
||||
});
|
||||
|
||||
it("defaults to mp4 when format is omitted", () => {
|
||||
const preset = getEncoderPreset("standard");
|
||||
expect(preset.codec).toBe("h264");
|
||||
expect(preset.pixelFormat).toBe("yuv420p");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
*/
|
||||
|
||||
import { spawn } from "child_process";
|
||||
import { existsSync, mkdirSync, readdirSync, statSync, writeFileSync } from "fs";
|
||||
import { copyFileSync, existsSync, mkdirSync, readdirSync, statSync, writeFileSync } from "fs";
|
||||
import { join, dirname } from "path";
|
||||
import { DEFAULT_CONFIG, type EngineConfig } from "../config.js";
|
||||
import { type GpuEncoder, getCachedGpuEncoder, getGpuEncoderName } from "../utils/gpuEncoder.js";
|
||||
@@ -21,6 +21,26 @@ export const ENCODER_PRESETS = {
|
||||
high: { preset: "slow", quality: 18, codec: "h264" as const },
|
||||
};
|
||||
|
||||
/**
|
||||
* Get encoder preset for a given quality and output format.
|
||||
* WebM uses VP9 with alpha-capable pixel format; MP4 uses h264.
|
||||
*/
|
||||
export function getEncoderPreset(
|
||||
quality: "draft" | "standard" | "high",
|
||||
format: "mp4" | "webm" = "mp4",
|
||||
): { preset: string; quality: number; codec: "h264" | "vp9"; pixelFormat: string } {
|
||||
const base = ENCODER_PRESETS[quality];
|
||||
if (format === "webm") {
|
||||
return {
|
||||
preset: base.preset === "ultrafast" ? "realtime" : "good",
|
||||
quality: base.quality,
|
||||
codec: "vp9",
|
||||
pixelFormat: "yuva420p",
|
||||
};
|
||||
}
|
||||
return { ...base, pixelFormat: "yuv420p" };
|
||||
}
|
||||
|
||||
// Re-export GPU utilities so existing consumers that import from chunkEncoder still work.
|
||||
export { detectGpuEncoder, type GpuEncoder } from "../utils/gpuEncoder.js";
|
||||
|
||||
@@ -83,6 +103,11 @@ function buildEncoderArgs(
|
||||
} else if (codec === "vp9") {
|
||||
args.push("-c:v", "libvpx-vp9", "-b:v", bitrate || "0", "-crf", String(quality));
|
||||
args.push("-deadline", preset === "ultrafast" ? "realtime" : "good");
|
||||
args.push("-row-mt", "1");
|
||||
if (pixelFormat === "yuva420p") {
|
||||
args.push("-auto-alt-ref", "0");
|
||||
args.push("-metadata:s:v:0", "alpha_mode=1");
|
||||
}
|
||||
} else if (codec === "prores") {
|
||||
args.push("-c:v", "prores_ks", "-profile:v", preset, "-vendor", "apl0");
|
||||
return [...args, "-y", outputPath];
|
||||
@@ -243,7 +268,8 @@ export async function encodeFramesChunkedConcat(
|
||||
}
|
||||
const startNumber = i * chunkSize;
|
||||
const framesInChunk = Math.min(chunkSize, files.length - startNumber);
|
||||
const chunkPath = join(chunkDir, `chunk_${String(i).padStart(4, "0")}.mp4`);
|
||||
const ext = outputPath.endsWith(".webm") ? ".webm" : ".mp4";
|
||||
const chunkPath = join(chunkDir, `chunk_${String(i).padStart(4, "0")}${ext}`);
|
||||
const inputPath = join(framesDir, framePattern);
|
||||
const inputArgs = [
|
||||
"-framerate",
|
||||
@@ -347,23 +373,15 @@ export async function muxVideoWithAudio(
|
||||
const outputDir = dirname(outputPath);
|
||||
if (!existsSync(outputDir)) mkdirSync(outputDir, { recursive: true });
|
||||
|
||||
const args = [
|
||||
"-i",
|
||||
videoPath,
|
||||
"-i",
|
||||
audioPath,
|
||||
"-c:v",
|
||||
"copy",
|
||||
"-c:a",
|
||||
"aac",
|
||||
"-b:a",
|
||||
"192k",
|
||||
"-shortest",
|
||||
"-movflags",
|
||||
"+faststart",
|
||||
"-y",
|
||||
outputPath,
|
||||
];
|
||||
const isWebm = outputPath.endsWith(".webm");
|
||||
const args = ["-i", videoPath, "-i", audioPath, "-c:v", "copy"];
|
||||
|
||||
if (isWebm) {
|
||||
args.push("-c:a", "libopus", "-b:a", "128k");
|
||||
} else {
|
||||
args.push("-c:a", "aac", "-b:a", "192k", "-movflags", "+faststart");
|
||||
}
|
||||
args.push("-shortest", "-y", outputPath);
|
||||
|
||||
const processTimeout = config?.ffmpegProcessTimeout ?? DEFAULT_CONFIG.ffmpegProcessTimeout;
|
||||
const result = await runFfmpeg(args, { signal, timeout: processTimeout });
|
||||
@@ -394,6 +412,11 @@ export async function applyFaststart(
|
||||
signal?: AbortSignal,
|
||||
config?: Partial<Pick<EngineConfig, "ffmpegProcessTimeout">>,
|
||||
): Promise<MuxResult> {
|
||||
// faststart is MP4-only (moves moov atom to file start for streaming)
|
||||
if (outputPath.endsWith(".webm")) {
|
||||
if (inputPath !== outputPath) copyFileSync(inputPath, outputPath);
|
||||
return { success: true, outputPath, durationMs: 0 };
|
||||
}
|
||||
const args = ["-i", inputPath, "-c", "copy", "-movflags", "+faststart", "-y", outputPath];
|
||||
|
||||
const processTimeout = config?.ffmpegProcessTimeout ?? DEFAULT_CONFIG.ffmpegProcessTimeout;
|
||||
|
||||
@@ -107,6 +107,16 @@ export async function createCaptureSession(
|
||||
};
|
||||
await page.setViewport(viewport);
|
||||
|
||||
// For PNG capture (used by WebM/transparency), make the page background transparent
|
||||
// so Chrome's screenshot captures alpha channel data. Must use the same CDP session
|
||||
// that the screenshot service uses (getCdpSession caches per page).
|
||||
if (options.format === "png") {
|
||||
const cdp = await getCdpSession(page);
|
||||
await cdp.send("Emulation.setDefaultBackgroundColorOverride", {
|
||||
color: { r: 0, g: 0, b: 0, a: 0 },
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
browser,
|
||||
page,
|
||||
|
||||
@@ -104,6 +104,8 @@ export async function pageScreenshotCapture(page: Page, options: CaptureOptions)
|
||||
const result = await client.send("Page.captureScreenshot", {
|
||||
format,
|
||||
quality: format === "jpeg" ? (options.quality ?? 80) : undefined,
|
||||
fromSurface: true,
|
||||
captureBeyondViewport: false,
|
||||
optimizeForSpeed: true,
|
||||
});
|
||||
return Buffer.from(result.data, "base64");
|
||||
|
||||
@@ -172,6 +172,11 @@ function buildStreamingArgs(
|
||||
} else if (codec === "vp9") {
|
||||
args.push("-c:v", "libvpx-vp9", "-b:v", bitrate || "0", "-crf", String(quality));
|
||||
args.push("-deadline", preset === "ultrafast" ? "realtime" : "good");
|
||||
args.push("-row-mt", "1");
|
||||
if (pixelFormat === "yuva420p") {
|
||||
args.push("-auto-alt-ref", "0");
|
||||
args.push("-metadata:s:v:0", "alpha_mode=1");
|
||||
}
|
||||
} else if (codec === "prores") {
|
||||
args.push("-c:v", "prores_ks", "-profile:v", preset, "-vendor", "apl0");
|
||||
return [...args, "-y", outputPath];
|
||||
|
||||
@@ -31,6 +31,7 @@ type TestMetadata = {
|
||||
maxAudioLagWindows: number;
|
||||
renderConfig: {
|
||||
fps: 24 | 30 | 60;
|
||||
format?: "mp4" | "webm"; // Optional: defaults to "mp4"
|
||||
workers?: number; // Optional: auto-calculates if omitted
|
||||
};
|
||||
};
|
||||
@@ -145,6 +146,9 @@ function validateMetadata(meta: unknown): TestMetadata {
|
||||
if (![24, 30, 60].includes(rc.fps as number)) {
|
||||
throw new Error("meta.json: 'renderConfig.fps' must be 24, 30, or 60");
|
||||
}
|
||||
if (rc.format !== undefined && rc.format !== "mp4" && rc.format !== "webm") {
|
||||
throw new Error("meta.json: 'renderConfig.format' must be 'mp4' or 'webm' (or omit for mp4)");
|
||||
}
|
||||
if (rc.workers !== undefined) {
|
||||
if (typeof rc.workers !== "number" || rc.workers < 1) {
|
||||
throw new Error("meta.json: 'renderConfig.workers' must be >= 1 (or omit to auto-calculate)");
|
||||
@@ -306,28 +310,37 @@ function psnrAtCheckpoint(
|
||||
}
|
||||
|
||||
function extractMonoPcm16(videoPath: string): Int16Array {
|
||||
const { stdout } = runFfmpeg(
|
||||
[
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-i",
|
||||
videoPath,
|
||||
"-vn",
|
||||
"-ac",
|
||||
"1",
|
||||
"-ar",
|
||||
"16000",
|
||||
"-f",
|
||||
"s16le",
|
||||
"-",
|
||||
],
|
||||
`Audio extraction (${videoPath})`,
|
||||
);
|
||||
if (stdout.byteLength < 2) {
|
||||
try {
|
||||
const { stdout } = runFfmpeg(
|
||||
[
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-i",
|
||||
videoPath,
|
||||
"-vn",
|
||||
"-ac",
|
||||
"1",
|
||||
"-ar",
|
||||
"16000",
|
||||
"-f",
|
||||
"s16le",
|
||||
"-",
|
||||
],
|
||||
`Audio extraction (${videoPath})`,
|
||||
);
|
||||
if (stdout.byteLength < 2) {
|
||||
return new Int16Array(0);
|
||||
}
|
||||
return new Int16Array(stdout.buffer, stdout.byteOffset, Math.floor(stdout.byteLength / 2));
|
||||
} catch (err) {
|
||||
// No audio stream (e.g., WebM without audio) — log but don't fail
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
if (!msg.includes("does not contain any stream")) {
|
||||
logPretty(`Audio extraction warning: ${msg.slice(0, 200)}`, "⚠️");
|
||||
}
|
||||
return new Int16Array(0);
|
||||
}
|
||||
return new Int16Array(stdout.buffer, stdout.byteOffset, Math.floor(stdout.byteLength / 2));
|
||||
}
|
||||
|
||||
function buildRmsEnvelope(samples: Int16Array, windowSize = 2048, hopSize = 1024): number[] {
|
||||
@@ -535,12 +548,14 @@ async function runTestSuite(
|
||||
mkdirSync(tempRoot, { recursive: true });
|
||||
|
||||
const tempDownloadDir = join(tempRoot, "downloads");
|
||||
const renderedOutputPath = join(tempRoot, "output.mp4");
|
||||
const outputFormat = suite.meta.renderConfig.format ?? "mp4";
|
||||
const videoExt = outputFormat === "webm" ? ".webm" : ".mp4";
|
||||
const renderedOutputPath = join(tempRoot, `output${videoExt}`);
|
||||
|
||||
// Snapshot files stored in test's output/ directory
|
||||
const snapshotDir = join(suite.dir, "output");
|
||||
const snapshotCompiledPath = join(snapshotDir, "compiled.html");
|
||||
const snapshotVideoPath = join(snapshotDir, "output.mp4");
|
||||
const snapshotVideoPath = join(snapshotDir, `output${videoExt}`);
|
||||
|
||||
console.log(JSON.stringify({ event: "test_start", suite: suite.id, name: suite.meta.name }));
|
||||
logPretty(`Running test: ${suite.meta.name}`, "🧪");
|
||||
@@ -626,6 +641,7 @@ async function runTestSuite(
|
||||
const job = createRenderJob({
|
||||
fps: suite.meta.renderConfig.fps,
|
||||
quality: "high", // Always use max quality for tests
|
||||
format: outputFormat,
|
||||
workers: suite.meta.renderConfig.workers,
|
||||
useGpu: false,
|
||||
debug: false,
|
||||
@@ -643,7 +659,11 @@ async function runTestSuite(
|
||||
}
|
||||
copyFileSync(renderedOutputPath, snapshotVideoPath);
|
||||
console.log(
|
||||
JSON.stringify({ event: "snapshot_updated", suite: suite.id, file: "output/output.mp4" }),
|
||||
JSON.stringify({
|
||||
event: "snapshot_updated",
|
||||
suite: suite.id,
|
||||
file: `output/output${videoExt}`,
|
||||
}),
|
||||
);
|
||||
result.visual = { passed: true, failedFrames: 0, checkpoints: [] };
|
||||
result.audio = { passed: true, correlation: 1, lagWindows: 0 };
|
||||
|
||||
@@ -44,7 +44,7 @@ import {
|
||||
encodeFramesChunkedConcat,
|
||||
muxVideoWithAudio,
|
||||
applyFaststart,
|
||||
ENCODER_PRESETS,
|
||||
getEncoderPreset,
|
||||
processCompositionAudio,
|
||||
type AudioElement,
|
||||
calculateOptimalWorkers,
|
||||
@@ -99,6 +99,8 @@ export type RenderStatus =
|
||||
export interface RenderConfig {
|
||||
fps: 24 | 30 | 60;
|
||||
quality: "draft" | "standard" | "high";
|
||||
/** Output container format. WebM uses VP9+alpha for transparency. */
|
||||
format?: "mp4" | "webm";
|
||||
workers?: number;
|
||||
useGpu?: boolean;
|
||||
debug?: boolean;
|
||||
@@ -354,7 +356,13 @@ export async function executeRenderJob(
|
||||
let restoreLogger: (() => void) | null = null;
|
||||
const perfStages: Record<string, number> = {};
|
||||
const perfOutputPath = join(workDir, "perf-summary.json");
|
||||
const cfg = job.config.producerConfig ?? resolveConfig();
|
||||
const cfg = { ...(job.config.producerConfig ?? resolveConfig()) };
|
||||
const outputFormat = (job.config.format ?? "mp4") as "mp4" | "webm";
|
||||
const isWebm = outputFormat === "webm";
|
||||
// WebM/transparency requires screenshot mode — beginFrame doesn't support alpha channel
|
||||
if (isWebm) {
|
||||
cfg.forceScreenshot = true;
|
||||
}
|
||||
const enableChunkedEncode = cfg.enableChunkedEncode;
|
||||
const chunkedEncodeSize = cfg.chunkSizeFrames;
|
||||
const enableStreamingEncode = cfg.enableStreamingEncode;
|
||||
@@ -458,8 +466,8 @@ export async function executeRenderJob(
|
||||
width,
|
||||
height,
|
||||
fps: job.config.fps,
|
||||
format: "jpeg",
|
||||
quality: 80,
|
||||
format: isWebm ? "png" : "jpeg",
|
||||
quality: isWebm ? undefined : 80,
|
||||
};
|
||||
probeSession = await createCaptureSession(
|
||||
fileServer.url,
|
||||
@@ -702,14 +710,15 @@ export async function executeRenderJob(
|
||||
width,
|
||||
height,
|
||||
fps: job.config.fps,
|
||||
format: "jpeg",
|
||||
quality: job.config.quality === "draft" ? 80 : 95,
|
||||
format: isWebm ? "png" : "jpeg",
|
||||
quality: isWebm ? undefined : job.config.quality === "draft" ? 80 : 95,
|
||||
};
|
||||
|
||||
const workerCount = calculateOptimalWorkers(job.totalFrames!, job.config.workers, cfg);
|
||||
|
||||
const videoOnlyPath = join(workDir, "video-only.mp4");
|
||||
const preset = ENCODER_PRESETS[job.config.quality];
|
||||
const videoExt = isWebm ? ".webm" : ".mp4";
|
||||
const videoOnlyPath = join(workDir, `video-only${videoExt}`);
|
||||
const preset = getEncoderPreset(job.config.quality, outputFormat);
|
||||
|
||||
job.framesRendered = 0;
|
||||
|
||||
@@ -727,6 +736,7 @@ export async function executeRenderJob(
|
||||
codec: preset.codec,
|
||||
preset: preset.preset,
|
||||
quality: preset.quality,
|
||||
pixelFormat: preset.pixelFormat,
|
||||
useGpu: job.config.useGpu,
|
||||
imageFormat: captureOptions.format || "jpeg",
|
||||
},
|
||||
@@ -940,36 +950,32 @@ export async function executeRenderJob(
|
||||
const stage5Start = Date.now();
|
||||
updateJobStatus(job, "encoding", "Encoding video", 75, onProgress);
|
||||
|
||||
const frameExt = isWebm ? "png" : "jpg";
|
||||
const framePattern = `frame_%06d.${frameExt}`;
|
||||
const encoderOpts = {
|
||||
fps: job.config.fps,
|
||||
width,
|
||||
height,
|
||||
codec: preset.codec,
|
||||
preset: preset.preset,
|
||||
quality: preset.quality,
|
||||
pixelFormat: preset.pixelFormat,
|
||||
useGpu: job.config.useGpu,
|
||||
};
|
||||
const encodeResult = enableChunkedEncode
|
||||
? await encodeFramesChunkedConcat(
|
||||
framesDir,
|
||||
"frame_%06d.jpg",
|
||||
framePattern,
|
||||
videoOnlyPath,
|
||||
{
|
||||
fps: job.config.fps,
|
||||
width,
|
||||
height,
|
||||
codec: preset.codec,
|
||||
preset: preset.preset,
|
||||
quality: preset.quality,
|
||||
useGpu: job.config.useGpu,
|
||||
},
|
||||
encoderOpts,
|
||||
chunkedEncodeSize,
|
||||
abortSignal,
|
||||
)
|
||||
: await encodeFramesFromDir(
|
||||
framesDir,
|
||||
"frame_%06d.jpg",
|
||||
framePattern,
|
||||
videoOnlyPath,
|
||||
{
|
||||
fps: job.config.fps,
|
||||
width,
|
||||
height,
|
||||
codec: preset.codec,
|
||||
preset: preset.preset,
|
||||
quality: preset.quality,
|
||||
useGpu: job.config.useGpu,
|
||||
},
|
||||
encoderOpts,
|
||||
abortSignal,
|
||||
);
|
||||
assertNotAborted();
|
||||
@@ -1060,7 +1066,7 @@ export async function executeRenderJob(
|
||||
if (job.config.debug) {
|
||||
// Copy output MP4 into debug dir for easy access
|
||||
if (existsSync(outputPath)) {
|
||||
const debugOutput = join(workDir, "output.mp4");
|
||||
const debugOutput = join(workDir, isWebm ? "output.webm" : "output.mp4");
|
||||
copyFileSync(outputPath, debugOutput);
|
||||
}
|
||||
} else {
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"name": "webm-transparency",
|
||||
"description": "WebM VP9 render with transparent background (alpha channel). Validates the full WebM pipeline: PNG capture, VP9 encoding with yuva420p, and alpha_mode=1 metadata.",
|
||||
"tags": ["webm", "transparency", "fast"],
|
||||
"minPsnr": 30,
|
||||
"maxFrameFailures": 2,
|
||||
"minAudioCorrelation": 0.0,
|
||||
"maxAudioLagWindows": 120,
|
||||
"renderConfig": {
|
||||
"fps": 30,
|
||||
"format": "webm",
|
||||
"workers": 1
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<style>
|
||||
* { margin: 0; padding: 0; }
|
||||
html, body { background: transparent; overflow: hidden; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root" data-composition-id="main" data-start="0" data-duration="2" data-width="640" data-height="360" style="width: 640px; height: 360px; background: transparent; position: relative">
|
||||
<!-- Semi-transparent red box that animates across the frame -->
|
||||
<div id="box" data-start="0" data-duration="2" data-track-index="0" style="
|
||||
width: 120px;
|
||||
height: 120px;
|
||||
background: rgba(255, 0, 0, 0.7);
|
||||
border-radius: 16px;
|
||||
position: absolute;
|
||||
top: 120px;
|
||||
left: 50px;
|
||||
"></div>
|
||||
|
||||
<!-- White text with transparent background -->
|
||||
<div id="label" data-start="0.5" data-duration="1.5" data-track-index="1" style="
|
||||
font-family: sans-serif;
|
||||
font-size: 32px;
|
||||
font-weight: bold;
|
||||
color: white;
|
||||
text-shadow: 0 2px 8px rgba(0,0,0,0.5);
|
||||
position: absolute;
|
||||
top: 280px;
|
||||
left: 50px;
|
||||
opacity: 0;
|
||||
">Overlay Test</div>
|
||||
|
||||
<script src="https://cdn.jsdelivr.net/npm/gsap@3/dist/gsap.min.js"></script>
|
||||
|
||||
</div>
|
||||
<script>window.__timelines = window.__timelines || {};
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
|
||||
// Animate box sliding right
|
||||
tl.to("#box", { left: 470, duration: 2, ease: "power2.inOut" }, 0);
|
||||
|
||||
// Fade in label
|
||||
tl.to("#label", { opacity: 1, duration: 0.5, ease: "power2.out" }, 0.5);
|
||||
|
||||
window.__timelines["main"] = tl;</script></body>
|
||||
</html>
|
||||
Binary file not shown.
@@ -0,0 +1,71 @@
|
||||
<!doctype html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<style>
|
||||
* { margin: 0; padding: 0; }
|
||||
html, body { background: transparent; overflow: hidden; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div
|
||||
id="root"
|
||||
data-composition-id="main"
|
||||
data-start="0"
|
||||
data-duration="2"
|
||||
data-width="640"
|
||||
data-height="360"
|
||||
style="width: 640px; height: 360px; background: transparent; position: relative"
|
||||
>
|
||||
<!-- Semi-transparent red box that animates across the frame -->
|
||||
<div
|
||||
id="box"
|
||||
data-start="0"
|
||||
data-duration="2"
|
||||
data-track-index="0"
|
||||
style="
|
||||
width: 120px;
|
||||
height: 120px;
|
||||
background: rgba(255, 0, 0, 0.7);
|
||||
border-radius: 16px;
|
||||
position: absolute;
|
||||
top: 120px;
|
||||
left: 50px;
|
||||
"
|
||||
></div>
|
||||
|
||||
<!-- White text with transparent background -->
|
||||
<div
|
||||
id="label"
|
||||
data-start="0.5"
|
||||
data-duration="1.5"
|
||||
data-track-index="1"
|
||||
style="
|
||||
font-family: sans-serif;
|
||||
font-size: 32px;
|
||||
font-weight: bold;
|
||||
color: white;
|
||||
text-shadow: 0 2px 8px rgba(0,0,0,0.5);
|
||||
position: absolute;
|
||||
top: 280px;
|
||||
left: 50px;
|
||||
opacity: 0;
|
||||
"
|
||||
>Overlay Test</div>
|
||||
|
||||
<script src="https://cdn.jsdelivr.net/npm/gsap@3/dist/gsap.min.js"></script>
|
||||
<script>
|
||||
window.__timelines = window.__timelines || {};
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
|
||||
// Animate box sliding right
|
||||
tl.to("#box", { left: 470, duration: 2, ease: "power2.inOut" }, 0);
|
||||
|
||||
// Fade in label
|
||||
tl.to("#label", { opacity: 1, duration: 0.5, ease: "power2.out" }, 0.5);
|
||||
|
||||
window.__timelines["main"] = tl;
|
||||
</script>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user