From 6d2569c6bbc869acd7a175509d0c5813e6fd45a1 Mon Sep 17 00:00:00 2001 From: James Russo Date: Tue, 19 May 2026 03:13:30 -0400 Subject: [PATCH] test(producer): add webm-vp9 distributed regression fixture (#952) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(producer): enable webm in distributed mode via concat-copy PR 8.2 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). Wires libvpx-vp9 webm through the distributed pipeline now that PR 8.1 proved concat-copy works. Architectural decision: Path A (concat-copy) — based on PR 8.1's smoke test result (9/9 tests pass for both yuv420p and yuva420p VP9 streams). The simpler architecture wins; no re-encode in assemble, no encode- parallelism loss. Changes: - plan.ts: - DistributedRenderConfig.format and PlanResult.format now include "webm" — type-level acceptance matches the runtime gate. - rejectUnsupportedDistributedFormat() no longer trips on webm. HDR mp4 remains the only refused configuration. - resolveEncoderTriple() returns libvpx-vp9-software + yuva420p + preset="good" for format="webm". yuva420p preserves alpha — the format's main reason for existing for web delivery. - codec= remains rejected for non-mp4 formats (mov is always ProRes 4444; webm is always libvpx-vp9). The error message lists all four distributed-supported formats. - FormatNotSupportedInDistributedError docstring updated to reflect the new reality (only HDR is unsupported). - freezePlan.ts: LockedRenderConfig.encoder gains "libvpx-vp9-software". Mirrors libx265-software / prores-software / png-sequence in shape; the chunk worker reads this discriminant to decide encode args. - renderChunk.ts: drops the now-incorrect cast that excluded webm from buildSyntheticRenderJob's format input; tightens the preset-format cast to include webm. - assemble.ts: docstring + comment updates. The mp4/mov concat-copy path is format-agnostic — webm uses the exact same code (applyFaststart is a no-op for webm via the existing chunkEncoder.ts gate; muxVideoWithAudio already routes webm to libopus audio). - planFormatBanlist.test.ts: webm-rejection tests removed; replaced with "accepts webm" tests + a HDR+webm combo test that verifies HDR is the trip regardless of format. - plan.test.ts: new describe block pins the webm wiring contract: format="webm" produces an encoder=libvpx-vp9-software / pixelFormat=yuva420p planDir with closedGop=true and gopSize=chunkSize. - webm-concat-copy.test.ts (smoke): extended with a yuva420p variant that proves the alpha pixel format the distributed pipeline actually emits also round-trips through concat-copy. 9/9 tests pass locally. §8 format support matrix in DISTRIBUTED-RENDERING-PLAN.md is intentionally left unchanged at this PR — it flips to ✓ in PR 8.4 once the end-to-end fixture (PR 8.3) is green. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(producer): include webm in plan-time needsAlpha + strengthen alpha smoke PR review feedback from Miguel and Vai on #951 caught a real bug: `plan.ts`'s `needsAlpha` disjunction excluded `"webm"`, so the plan stage froze `forceScreenshot: false` into the `LockedRenderConfig` even though distributed webm uses `yuva420p`. Every chunk worker captured opaque RGB via BeginFrame (which doesn't preserve alpha on Linux headless-shell), and libvpx-vp9 encoded uniformly-opaque alpha that the encoder then dropped — producing un-keyable webm. Two changes: 1. **plan.ts**: include `"webm"` in `needsAlpha`. Matches the in-process renderer's logic at `renderOrchestrator.ts:1469` (`const needsAlpha = isWebm || isMov || isPngSequence`); the two sites must stay in sync since the distributed pipeline's PSNR regression compares against the in-process baseline. 2. **Smoke test (yuva420p describe)**: source frames now use a real alpha gradient (`geq=a='X*255/W'` on top of `testsrc2`) instead of `testsrc2 + format=rgba` which was uniformly opaque. The decode- pix_fmt assertion is dropped (ffprobe reports `yuv420p` for VP9-with-alpha because the alpha lives in a Matroska `BlockAdditional` sidecar) and replaced with two stronger checks: - `TAG:ALPHA_MODE=1` is present on the stream — proves the encoder was actually configured for alpha - alpha plane variance after `-c:v libvpx-vp9 -i ... -pix_fmt rgba -vf extractplanes=a,signalstats` — proves the alpha sub-stream round-trips through concat-copy with spatially-varying content, not uniform/dropped alpha - decode-test gate is now exit-code-only (was `exitCode || stderr` which would flake on chatty ffmpeg `-v error` builds emitting non-fatal DTS/container notes) These checks would have caught the `needsAlpha` bug before review. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(aws-lambda): widen narrow format types to include webm CI on PR #951 was failing at typecheck/build because the producer's `DistributedRenderConfig.format` widened to include webm in this PR but the aws-lambda package's narrow `"mp4" | "mov" | "png-sequence"` type literals in `events.ts`, `handler.ts`, and `validateConfig.ts` hadn't kept up. `renderToLambda.ts:87` passed `config.format` (now including webm) into a parameter typed against the narrow union, producing TS2345. This widening originally landed in PR #952 (test fixture PR) but needs to be atomic with the producer's widening here to keep each PR independently typecheck-clean. Also refactor `formatExtension` from a switch dispatch to a `Record` lookup. Adding the webm case tipped the switch's CRAP to the 30.0 fallow threshold; the lookup table drops cyclomatic from 5 to 1 with the same compile-time exhaustiveness guarantee (TS errors on missing entries when `DistributedFormat` adds a new format). The runtime `_exhaustive: never` throw was only protecting against a string slipping past TS; `validateConfig.ts`'s `ALLOWED_FORMATS` already gates untrusted input at the SDK boundary. Co-Authored-By: Claude Opus 4.7 (1M context) * test(producer): add webm-vp9 distributed regression fixture PR 8.3 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). End-to-end regression coverage for the webm distributed path PRs 8.1 and 8.2 wired up. Adds packages/producer/tests/distributed/webm-vp9/ matching the mp4-h264-sdr fixture pattern: a 2-second composition (60 frames @ 30fps) with text, a crossfade across the frame-30 chunk seam, and a continuous icon rotation — exercises chunk-boundary continuity for both display contents and VP9 closed-GOP alpha encoding. `chunkSize: 15` produces 4 chunks so 3 seams are tested, and the crossfade straddles the middle seam to surface alpha-plane discontinuities introduced by alt-ref drift. Baseline regenerated inside Dockerfile.test via `bun run --cwd packages/producer docker:test:update webm-vp9`. Runs in: - in-process mode: byte-identical match against baseline ✓ - distributed-simulated mode: PSNR 56.88-63.49 dB across 100 checkpoints, well above the 30 dB threshold ✓ Wiring updates required to let webm flow through the harness: - regression-harness-distributed.ts: - checkDistributedSupport() no longer rejects webm. HDR mp4 + NTSC fps + non-{24,30,60} fps remain rejected. - RunDistributedSimulatedInput.format widened to include webm. - Docstring + comments updated. - regression-harness-distributed.test.ts: webm-rejection test replaced with "accepts format=webm" test. - regression-harness.ts: the now-incorrect format cast at the distributed-input call site is dropped; comment about why webm was excluded is replaced with "webm is now distributed-supported". - regression-harness-lambda-local-types.ts: RunLambdaLocalInput.format widened to include webm so lambda-local mode can also exercise webm fixtures end-to-end. - aws-lambda webm support (Path A through the Lambda handler): - formatExtension.ts: DistributedFormat gains "webm" → ".webm" case. - events.ts: RenderChunkEvent / AssembleEvent / PlanLambdaResult Format widened to include webm. - sdk/validateConfig.ts: ALLOWED_FORMATS gains "webm". - handler.ts: downloadChunkObjects format param widened. The Lambda handler delegates to the producer's assemble() primitive which PR 8.2 already taught to handle webm (concat-copy + applyFaststart no-op + muxVideoWithAudio with libopus); no Lambda-side rendering changes are needed beyond the type/validation surfaces above. Co-Authored-By: Claude Opus 4.7 (1M context) * docs(aws-lambda): drop stale webm rejection from validateConfig docblock PR #952 review nit (Miguel): the validateConfig.ts file-header comment still claimed the SDK rejects webm, but the runtime check no longer does (ALLOWED_FORMATS now includes 'webm'). Update the docblock to reflect that only force-hdr remains an SDK-side rejection. Co-Authored-By: Claude Opus 4.7 (1M context) * ci(regression): add webm-vp9 to shard-3 + refactor formatExtension Three follow-ups bundled together (Vai's review feedback on PR #952 plus the fallow audit finding that surfaced when the webm case was added): 1. **Wire webm-vp9 into CI regression.** The fixture was added in this PR but never appeared in any `.github/workflows/regression.yml` shard's args allowlist, so the regression harness's positional-args gate skipped it in CI. Append `webm-vp9` to shard-3 (which already carries `mp4-h264-sdr` + `webm-transparency`) so the fixture runs. 2. **Fix stale "four hard gates" prose in checkDistributedSupport docstring.** Earlier in the stack I removed the webm bullet but didn't update the count. Two gates remain (fps + hdr). 3. **Refactor `formatExtension` from switch to lookup table.** Adding the webm case made the switch dispatch's CRAP score hit 30.0 (cyclomatic = 5, plus the function's small body). Replaced with a `Record` lookup, which: - drops cyclomatic from 5 → 1, - keeps exhaustiveness enforcement at compile time (TS errors if a new format gets added to `DistributedFormat` without a matching key in the Record literal), - drops the runtime `_exhaustive: never` throw, which was only guarding against an arbitrary string slipping past TS — a caller-side concern, not this function's job. The function now reads as a table lookup, which matches what it actually does, and the fallow audit now reports zero new complexity findings (down from 1). Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: Claude Opus 4.7 (1M context) --- .github/workflows/regression.yml | 2 +- packages/aws-lambda/src/sdk/validateConfig.ts | 5 +- .../regression-harness-distributed.test.ts | 7 +- .../src/regression-harness-distributed.ts | 19 +- .../regression-harness-lambda-local-types.ts | 2 +- packages/producer/src/regression-harness.ts | 17 +- .../tests/distributed/webm-vp9/meta.json | 17 ++ .../distributed/webm-vp9/output/compiled.html | 165 ++++++++++++++++++ .../distributed/webm-vp9/output/output.webm | 3 + .../tests/distributed/webm-vp9/src/index.html | 131 ++++++++++++++ 10 files changed, 335 insertions(+), 33 deletions(-) create mode 100644 packages/producer/tests/distributed/webm-vp9/meta.json create mode 100644 packages/producer/tests/distributed/webm-vp9/output/compiled.html create mode 100644 packages/producer/tests/distributed/webm-vp9/output/output.webm create mode 100644 packages/producer/tests/distributed/webm-vp9/src/index.html diff --git a/.github/workflows/regression.yml b/.github/workflows/regression.yml index 62e2bc503..e077fd951 100644 --- a/.github/workflows/regression.yml +++ b/.github/workflows/regression.yml @@ -64,7 +64,7 @@ jobs: - shard: shard-2 args: "style-15-prod hdr-hlg-regression style-1-prod many-cuts vfr-screen-recording render-symlinked-assets" - shard: shard-3 - args: "style-7-prod style-8-prod style-10-prod css-spinner-render-compat webm-transparency mp4-h264-sdr" + args: "style-7-prod style-8-prod style-10-prod css-spinner-render-compat webm-transparency mp4-h264-sdr webm-vp9" - shard: shard-4 args: "style-16-prod style-9-prod style-17-prod iframe-render-compat variables-prod mp4-h265-sdr" - shard: shard-5 diff --git a/packages/aws-lambda/src/sdk/validateConfig.ts b/packages/aws-lambda/src/sdk/validateConfig.ts index f959bcab1..dcda6c002 100644 --- a/packages/aws-lambda/src/sdk/validateConfig.ts +++ b/packages/aws-lambda/src/sdk/validateConfig.ts @@ -12,8 +12,9 @@ * * The check is deliberately narrow — it covers the *shape* errors any * caller could have surfaced with `tsc` if they passed a literal, plus - * the documented `webm`/`force-hdr` rejections from §5.3 of the - * distributed-rendering plan. Anything deeper (font availability, plan + * the `force-hdr` rejection (HDR mp4 isn't supported in distributed + * mode). webm was previously rejected here too; v0.7+ supports it via + * closed-GOP concat-copy. Anything deeper (font availability, plan * size cap, GPU mode at runtime) needs the actual planner. */ diff --git a/packages/producer/src/regression-harness-distributed.test.ts b/packages/producer/src/regression-harness-distributed.test.ts index a036aba36..6de6de6be 100644 --- a/packages/producer/src/regression-harness-distributed.test.ts +++ b/packages/producer/src/regression-harness-distributed.test.ts @@ -70,12 +70,9 @@ describe("checkDistributedSupport()", () => { } }); - it("rejects format=webm", () => { + it("accepts format=webm (distributed-supported via closed-GOP concat-copy)", () => { const result = checkDistributedSupport({ fps: { num: 30, den: 1 }, format: "webm" }); - expect(result.supported).toBe(false); - if (!result.supported) { - expect(result.reason).toMatch(/webm/); - } + expect(result.supported).toBe(true); }); it("rejects hdr=true", () => { diff --git a/packages/producer/src/regression-harness-distributed.ts b/packages/producer/src/regression-harness-distributed.ts index 449bced29..5283b31f1 100644 --- a/packages/producer/src/regression-harness-distributed.ts +++ b/packages/producer/src/regression-harness-distributed.ts @@ -25,9 +25,9 @@ * capture jitter, so the harness can't use it as a per-test gate. * * Not every fixture can run in distributed-simulated mode. Distributed mode - * refuses webm, HDR mp4, NTSC framerates, and non-{24,30,60} fps at plan - * time. Fixtures that don't meet the constraints are skipped — the harness - * logs the reason and the fixture is treated as "passed (skipped)" in + * refuses HDR mp4, NTSC framerates, and non-{24,30,60} fps at plan time. + * Fixtures that don't meet the constraints are skipped — the harness logs + * the reason and the fixture is treated as "passed (skipped)" in * distributed-simulated mode. */ @@ -67,15 +67,13 @@ export type DistributedSupportResult = { supported: true } | { supported: false; /** * Decide whether a fixture's `renderConfig` is one the distributed pipeline - * can actually run. The four hard gates: + * can actually run. Two hard gates: * * - fps must be `{ num: 24|30|60, den: 1 }`. `DistributedRenderConfig.fps` * accepts only the three integer values, and rationals like * `{ num: 30000, den: 1001 }` (NTSC) trip the type system at the call * site. We surface this gate in code rather than only in TS so the * harness can skip the fixture cleanly instead of throwing. - * - format must not be `webm`. `plan()` refuses webm with - * `FORMAT_NOT_SUPPORTED_IN_DISTRIBUTED`. * - hdr must not be `true`. Distributed mode is SDR-only at v1. * * Callers that want the structured reason can read it off the returned @@ -99,13 +97,6 @@ export function checkDistributedSupport(renderConfig: { reason: `fps ${fpsNum} not in {24, 30, 60} (DistributedRenderConfig.fps is a closed set)`, }; } - const format = renderConfig.format ?? "mp4"; - if (format === "webm") { - return { - supported: false, - reason: "format=webm refused in distributed mode (VP9+matroska concat-copy is unstable)", - }; - } if (renderConfig.hdr === true) { return { supported: false, @@ -129,7 +120,7 @@ export interface RunDistributedSimulatedInput { renderedOutputPath: string; /** From the fixture's renderConfig — must pass `checkDistributedSupport`. */ fps: 24 | 30 | 60; - format: "mp4" | "mov" | "png-sequence"; + format: "mp4" | "mov" | "png-sequence" | "webm"; /** * Codec for `format: "mp4"`. Defaults to `"h264"`; pass `"h265"` to * exercise the libx265 closed-GOP path. Ignored for non-mp4 formats — diff --git a/packages/producer/src/regression-harness-lambda-local-types.ts b/packages/producer/src/regression-harness-lambda-local-types.ts index bb021951a..54d11c25f 100644 --- a/packages/producer/src/regression-harness-lambda-local-types.ts +++ b/packages/producer/src/regression-harness-lambda-local-types.ts @@ -26,7 +26,7 @@ export interface RunLambdaLocalInput { */ width: number; height: number; - format: "mp4" | "mov" | "png-sequence"; + format: "mp4" | "mov" | "png-sequence" | "webm"; codec?: "h264" | "h265"; chunkSize?: number; maxParallelChunks?: number; diff --git a/packages/producer/src/regression-harness.ts b/packages/producer/src/regression-harness.ts index b3cef543a..19d5b8f3e 100644 --- a/packages/producer/src/regression-harness.ts +++ b/packages/producer/src/regression-harness.ts @@ -94,8 +94,8 @@ type TestMetadata = { * single video file — the harness branches its comparison logic * accordingly (per-frame byte equality instead of PSNR). `"mov"` and * `"webm"` are encoded video containers that share the PSNR path with - * `"mp4"`. `"webm"` is rejected by the distributed pipeline at plan - * time; the in-process renderer accepts it. + * `"mp4"`. Distributed mode supports all four — webm goes through + * libvpx-vp9 with closed-GOP concat-copy. */ format?: "mp4" | "webm" | "mov" | "png-sequence"; /** @@ -163,7 +163,7 @@ type TestResult = { passed: boolean; /** * Set when `--mode=distributed-simulated` skips a fixture that the - * distributed pipeline can't run (webm, HDR, NTSC fps, fps∉{24,30,60}). + * distributed pipeline can't run (HDR, NTSC fps, fps∉{24,30,60}). * `passed` is `true` for skipped fixtures — skipping is a clean outcome, * not a failure — but the summary distinguishes them. */ @@ -939,19 +939,16 @@ async function runTestSuite( result.skipped = { reason: support.reason }; return result; } - // `checkDistributedSupport` already narrowed fps to {24,30,60} and - // rejected webm; the cast surfaces that guarantee to TS. + // `checkDistributedSupport` already narrowed fps to {24,30,60}; the + // cast surfaces that guarantee to TS. webm is now distributed- + // supported via closed-GOP concat-copy, so the format passes through. const fpsNum = suite.meta.renderConfig.fps.num as 24 | 30 | 60; const distributedInput = { projectDir: tempSrcDir, tempRoot, renderedOutputPath, fps: fpsNum, - // `runDistributedSimulatedRender` / `runLambdaLocalRender`'s - // `format` parameter accepts the distributed-supported set; - // the harness type allows `"webm"` too but - // `checkDistributedSupport` rejected that above. Narrow. - format: outputFormat as "mp4" | "mov" | "png-sequence", + format: outputFormat, codec: suite.meta.renderConfig.codec, chunkSize: suite.meta.renderConfig.chunkSize, maxParallelChunks: suite.meta.renderConfig.maxParallelChunks, diff --git a/packages/producer/tests/distributed/webm-vp9/meta.json b/packages/producer/tests/distributed/webm-vp9/meta.json new file mode 100644 index 000000000..5f67e58f5 --- /dev/null +++ b/packages/producer/tests/distributed/webm-vp9/meta.json @@ -0,0 +1,17 @@ +{ + "name": "Distributed: webm VP9", + "description": "60-frame composition (2s @ 30fps) with text and a small rotating SVG icon, rendered to webm (VP9 + yuva420p). renderConfig.format=webm routes the distributed pipeline through libvpx-vp9 with closed-GOP keyint params (-g N -keyint_min N -auto-alt-ref 0 -cpu-used 2) so per-chunk VP9 output can be losslessly stitched with `ffmpeg -f concat -c copy`. The in-process baseline renders to webm too (codec=vp9, pixelFormat=yuva420p), so the harness's PSNR comparison measures 'libvpx-vp9 chunked + concat' against 'libvpx-vp9 single-pass'. Closed-GOP forces more keyframes than open-GOP, which inflates per-chunk bitrate at constant CRF; PSNR threshold is set at 30 dB to absorb the resulting cross-mode drift without masking gross regressions.", + "tags": ["distributed", "webm", "vp9", "sdr"], + + "minPsnr": 30, + "maxFrameFailures": 0, + + "minAudioCorrelation": 0.9, + "maxAudioLagWindows": 120, + + "renderConfig": { + "fps": 30, + "format": "webm", + "chunkSize": 15 + } +} diff --git a/packages/producer/tests/distributed/webm-vp9/output/compiled.html b/packages/producer/tests/distributed/webm-vp9/output/compiled.html new file mode 100644 index 000000000..c2f95979e --- /dev/null +++ b/packages/producer/tests/distributed/webm-vp9/output/compiled.html @@ -0,0 +1,165 @@ + + + + + + webm VP9 distributed fixture + + + + +
+
+
CHUNK
+
PHASE ONE
+
+
+
CHUNK
+
PHASE TWO
+
+ + + + + + +
+ + + + diff --git a/packages/producer/tests/distributed/webm-vp9/output/output.webm b/packages/producer/tests/distributed/webm-vp9/output/output.webm new file mode 100644 index 000000000..d541643af --- /dev/null +++ b/packages/producer/tests/distributed/webm-vp9/output/output.webm @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b98efde6993524eaec419b2a7d3c37f33dcc84602bbd89cfcb1eea41cc125506 +size 78674 diff --git a/packages/producer/tests/distributed/webm-vp9/src/index.html b/packages/producer/tests/distributed/webm-vp9/src/index.html new file mode 100644 index 000000000..e73c0db59 --- /dev/null +++ b/packages/producer/tests/distributed/webm-vp9/src/index.html @@ -0,0 +1,131 @@ + + + + + + webm VP9 distributed fixture + + + + +
+
+
CHUNK
+
PHASE ONE
+
+
+
CHUNK
+
PHASE TWO
+
+ + + + + + +
+ + + +