docs(lambda): document webm support + simplify-review fixes (#953)

* docs(lambda): document webm support in distributed mode PR 8.4 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). User-facing docs catch up with the shipped capability. Updates docs/deploy/migrating-to-hyperframes-lambda.mdx: - "Output format" row in the migration table now lists `webm` alongside mp4 / mov / png-sequence with a note that webm uses libvpx-vp9 + closed-GOP concat-copy. HDR mp4 remains the only refused format. - "No webm distributed" caveat replaced with "webm uses closed-GOP VP9" explainer covering the encoder args (`-g <chunkSize>`, `-keyint_min <chunkSize>`, `-auto-alt-ref 0`, `-cpu-used 2`), why alt-ref disable is load-bearing, and that the output preserves alpha via yuva420p with Opus audio. - Migration checklist no longer asks adopters to filter out webm compositions; only HDR-dependent renders need to stay on the previous framework. aws-lambda.mdx doesn't currently call out webm as unsupported (only HDR in the v1 surface list), so it gets no copy edits beyond the migration guide. The internal planning doc (DISTRIBUTED-RENDERING-PLAN.md §7.2, §8, §12 — kept outside the repo) gets matching updates: format support matrix flipped ✓, v1.5 backlog #1 marked shipped, HDR promoted to the new top item, and the rev-12 → rev-13 status line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: address simplify-review findings on webm stack Folds in cleanups identified by a multi-agent code-review pass over the 4-PR webm-distributed stack: - plan.ts: `resolveEncoderTriple()` webm case now calls `getEncoderPreset(quality, "webm")` for its preset string instead of hardcoding "good". The hardcode was wrong for `quality: "draft"` (`getEncoderPreset` returns "realtime" for that tier) — would have silently overridden the draft → realtime mapping for distributed webm renders. - chunkEncoder.ts: trim the new VP9 closed-GOP comment block from ~18 lines of WHY narration down to the 6 lines that actually explain why (alt-ref + cpu-used drift). Match the alpha branch's idempotent-push comment to the same standard. - chunkEncoder.test.ts: drop the duplicate WHY comment that restated the implementation comment in plain words. - webm-concat-copy.test.ts: rewrite the file-header docstring to describe the contract being tested instead of the PR-8.1-gating history; strip "PR 8.2 / Path A / Path B" references from error messages (they belong in PR bodies, not in test output). Consolidate the yuva420p alpha smoke into a single `it()` block (was a full 4-test describe with duplicated setup) — the yuv420p block already covers the probe/decode/frame-count contract; the alpha smoke only needs to prove the alpha args don't break concat-copy. - plan.test.ts: drop the "PR 8.1 proved the contract" comment. - webm-vp9 fixture: drop the aspirational "Other webm-with-audio fixtures cover the mux path separately when added" sentence (no other fixtures exist). Regenerated the baseline via `docker:test:update webm-vp9` to reflect the updated comment. - migrating-to-hyperframes-lambda.mdx: add a paragraph about distributed webm's perf cost — ~10-25% larger files at constant CRF due to forced keyframes, and slower per-chunk encode due to `-cpu-used 2` being more conservative than the libvpx default. All unit tests + the webm-vp9 distributed-simulated regression still pass after these changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): accept --format=webm in `hyperframes lambda render` The CLI's `lambda render` subcommand's FORMATS allowlist and the `RenderArgs.format` type still narrowed to `mp4 | mov | png-sequence`, so even though the producer + aws-lambda packages now support webm end-to-end, the CLI surface rejected it with `--format must be mp4|mov| png-sequence`. Add webm to both spots and update the --help description. Surfaced during real-AWS deploy prep — the local lambda-local / distributed-simulated tests didn't go through the CLI so the gap went unnoticed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(producer): font cache writes to /tmp on Lambda (read-only \$HOME) The deterministic Google Fonts cache was rooted at `\$HOME/.cache/hyperframes/fonts`, which fails on AWS Lambda — the runtime's `\$HOME` resolves to a `/home/sbx_*` directory tree that's read-only. `mkdirSync(..., { recursive: true })` can't create that path and the plan stage trips with `ENOENT: no such file or directory, mkdir '/home/sbx_user1051/.cache/hyperframes/fonts/space-mono'` on every Lambda render that pulls a Google Font (i.e. every distributed fixture using `@import url("https://fonts.googleapis.com/...")`). Detect Lambda via `\$AWS_LAMBDA_FUNCTION_NAME` and route the cache to `tmpdir()/hyperframes/fonts` in that case. Lambda's `/tmp` survives across invocations on a warm container, so cache hit rate is the same as non-Lambda runs. Also honor an explicit `\$HYPERFRAMES_FONT_CACHE_DIR` override for adopters who want a different location regardless of the runtime. Surfaced while verifying webm distributed end-to-end on real AWS — the same bug affects mp4 fixtures using Google Fonts; webm just happened to be the one I tried first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: extract DistributedFormat type + trim font-cache resolver Second simplify-review pass on the webm stack flagged two cleanups: 1. **`DistributedFormat` type duplicated 10 times.** Every file in the distributed pipeline carried its own copy of `"mp4" | "mov" | "png-sequence" | "webm"` — adding a new format meant a 10-place edit with no compile-time guarantee they stayed in sync. Extract a single source of truth in `packages/producer/src/services/distributed/shared.ts`, re-export from `@hyperframes/producer/distributed` and `@hyperframes/aws-lambda/sdk`, and have all callers pull from there. The aws-lambda `ALLOWED_FORMATS` runtime tuple and the CLI's `FORMATS` tuple now both use `satisfies readonly DistributedFormat[]` so the compiler enforces the runtime allowlist stays in sync with the type. 2. **`deterministicFonts.ts` font-cache resolver was over-commented.** Trim the 7-line block to 4 lines (drop the aspirational "and other read-only-FS execution environments" — only Lambda is detected — and the warm-container `/tmp` persistence narration — anyone reading already knows Lambda /tmp semantics). Collapse the two-step `if (explicit && explicit.length > 0)` into a single nullish-coalesce expression now that the empty-string defensive check is gone (`process.env.X` is `string | undefined`, no third shape to guard against). Out-of-scope skips (called out by the agents, deferred): - In-process `RenderConfig.format` and the in-process CLI's `render.ts` format union still carry their own inline copies. The union happens to coincide today but they're separate concerns — leaving them alone limits this PR's blast radius. - `fontCacheDir(slug)` / `resolveFontCacheRoot()` naming asymmetry flagged as taste; skipping. - Pre-existing redundant `existsSync` before `mkdirSync({ recursive: true })` in `fontCacheDir` — out of scope. All tests + typecheck still pass. Lambda render still works end-to-end (no functional changes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(lambda): drop plan-doc reference from migration checklist PR review feedback: source/docs should not mention the distributed-rendering planning doc. Tighten the migration checklist sentence to describe the webm path directly rather than referencing the doc's version label. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(producer): split resolveEncoderTriple into mp4 + non-mp4 helpers CI Fallow audit on PR #953 flagged `resolveEncoderTriple` at CRAP 31.6 — the function interleaved (a) mp4 codec validation + dispatch, (b) the non-mp4 codec-rejection throw, and (c) per-format dispatch. Splitting into `resolveMp4EncoderTriple` + `resolveNonMp4EncoderTriple` drops the top-level function's cyclomatic complexity below the threshold while preserving every error message and code path. Behavior unchanged. Also extracts an `EncoderTriple` type alias so the three functions share the return shape declaratively rather than repeating it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-09-10 22:20:14 +00:00 · 2026-05-19 04:11:26 -04:00
parent 6d2569c6bb
commit 5d264e146c
25 changed files with 337 additions and 351 deletions
@@ -70,6 +70,11 @@ export {
 // ── Assemble (Activity C) ───────────────────────────────────────────────────
 export { assemble, type AssembleResult } from "./services/distributed/assemble.js";

+// ── Format union ────────────────────────────────────────────────────────────
+// Canonical output-format type. The aws-lambda package re-exports it so
+// CLI / adopter SDKs can derive runtime allowlists from one source.
+export type { DistributedFormat } from "./services/distributed/shared.js";
+
 // ── Plan-time shared types from `freezePlan` ───────────────────────────────
 // Re-exported so adopters that deserialize a planDir's `meta/encoder.json`
 // or `meta/chunks.json` see the same shapes the producer wrote them as.
@@ -35,6 +35,7 @@ import { existsSync, mkdirSync } from "node:fs";
 import { join } from "node:path";
 import type { Fps } from "@hyperframes/core";
 import { assemble, plan, renderChunk } from "./distributed.js";
+import type { DistributedFormat } from "./services/distributed/shared.js";

 /**
 * Three-mode contract that backs `--mode=<value>` on the regression
@@ -81,7 +82,7 @@ export type DistributedSupportResult = { supported: true } | { supported: false;
 */
 export function checkDistributedSupport(renderConfig: {
  fps: Fps;
-  format?: "mp4" | "webm" | "mov" | "png-sequence";
+  format?: DistributedFormat;
  hdr?: boolean;
 }): DistributedSupportResult {
  if (renderConfig.fps.den !== 1) {
@@ -120,7 +121,7 @@ export interface RunDistributedSimulatedInput {
  renderedOutputPath: string;
  /** From the fixture's renderConfig — must pass `checkDistributedSupport`. */
  fps: 24 | 30 | 60;
-  format: "mp4" | "mov" | "png-sequence" | "webm";
+  format: DistributedFormat;
  /**
   * Codec for `format: "mp4"`. Defaults to `"h264"`; pass `"h265"` to
   * exercise the libx265 closed-GOP path. Ignored for non-mp4 formats —
@@ -9,6 +9,8 @@
 * type-check pass.
 */

+import type { DistributedFormat } from "./services/distributed/shared.js";
+
 /** Inputs for {@link runLambdaLocalRender}. Same contract as `runDistributedSimulatedRender`. */
 export interface RunLambdaLocalInput {
  projectDir: string;
@@ -26,7 +28,7 @@ export interface RunLambdaLocalInput {
   */
  width: number;
  height: number;
-  format: "mp4" | "mov" | "png-sequence" | "webm";
+  format: DistributedFormat;
  codec?: "h264" | "h265";
  chunkSize?: number;
  maxParallelChunks?: number;
@@ -51,6 +51,7 @@ import {
 // imports) into the program even though the tsconfig `exclude` list
 // nominally hides it. `tsx` resolves the path normally at runtime.
 import type { RunLambdaLocalRender } from "./regression-harness-lambda-local-types.js";
+import type { DistributedFormat } from "./services/distributed/shared.js";

 const LAMBDA_LOCAL_MODULE = "./regression-harness-lambda-local.js";

@@ -97,7 +98,7 @@ type TestMetadata = {
     * `"mp4"`. Distributed mode supports all four — webm goes through
     * libvpx-vp9 with closed-GOP concat-copy.
     */
-    format?: "mp4" | "webm" | "mov" | "png-sequence";
+    format?: DistributedFormat;
    /**
     * Codec selection for `format: "mp4"`, forwarded to
     * `DistributedRenderConfig.codec`. The in-process renderer doesn't take
@@ -1,5 +1,5 @@
 import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
-import { homedir } from "node:os";
+import { homedir, tmpdir } from "node:os";
 import { join } from "node:path";

 import { parseHTML } from "linkedom";
@@ -330,7 +330,19 @@ function warnUnresolvedFonts(unresolved: string[]): void {
 // Google Fonts on-demand fetch + local cache
 // ---------------------------------------------------------------------------

-const GOOGLE_FONTS_CACHE_DIR = join(homedir(), ".cache", "hyperframes", "fonts");
+// On AWS Lambda `$HOME` resolves to a `/home/sbx_*` tree that's
+// read-only; only `/tmp` is writable. Route the cache there when
+// running inside Lambda, and honor `HYPERFRAMES_FONT_CACHE_DIR` as
+// an explicit override for any environment.
+function resolveFontCacheRoot(): string {
+  return (
+    process.env.HYPERFRAMES_FONT_CACHE_DIR ??
+    (process.env.AWS_LAMBDA_FUNCTION_NAME
+      ? join(tmpdir(), "hyperframes", "fonts")
+      : join(homedir(), ".cache", "hyperframes", "fonts"))
+  );
+}
+const GOOGLE_FONTS_CACHE_DIR = resolveFontCacheRoot();

 // Chrome UA triggers woff2 responses from Google Fonts CSS API
 const WOFF2_USER_AGENT =
@@ -38,6 +38,7 @@ import { applyFaststart, muxVideoWithAudio, runFfmpeg } from "@hyperframes/engin
 import { defaultLogger, type ProducerLogger } from "../../logger.js";
 import { padOrTrimAudioToVideoFrameCount } from "../render/audioPadTrim.js";
 import type { ChunkSliceJson } from "../render/stages/freezePlan.js";
+import type { DistributedFormat } from "./shared.js";

 /**
 * Result of {@link assemble}. `fileSize` reflects the final file on disk
@@ -61,7 +62,7 @@ interface PlanJsonForAssemble {
    fpsDen: number;
    width: number;
    height: number;
-    format: "mp4" | "mov" | "png-sequence" | "webm";
+    format: DistributedFormat;
  };
 }

@@ -467,10 +467,9 @@ describe("plan() — webm format (distributed VP9)", () => {
  it(
    'maps `format: "webm"` to libvpx-vp9-software + yuva420p',
    async () => {
-      // Webm is distributed-supported via closed-GOP concat-copy (PR 8.1
-      // proved the contract; this test pins the plan-time encoder choice).
-      // yuva420p preserves the format's reason for existing — alpha video
-      // for web playback over colored backgrounds.
+      // Pins the plan-time encoder choice for webm: libvpx-vp9-software
+      // with yuva420p so the format's alpha-channel contract round-trips
+      // through chunked rendering.
      const planDir = join(runRoot, "plan-webm-vp9");
      mkdirSync(planDir, { recursive: true });
      const result = await plan(
@@ -36,7 +36,7 @@ import {
 } from "node:fs";
 import { join, relative, sep } from "node:path";
 import { type CanvasResolution } from "@hyperframes/core";
-import { type EngineConfig, resolveConfig } from "@hyperframes/engine";
+import { type EngineConfig, getEncoderPreset, resolveConfig } from "@hyperframes/engine";
 import { defaultLogger, type ProducerLogger } from "../../logger.js";
 import { runAudioStage } from "../render/stages/audioStage.js";
 import { runCompileStage } from "../render/stages/compileStage.js";
@@ -57,6 +57,7 @@ import { validateNoGpuEncode, validateNoSystemFonts } from "../render/planValida
 import { snapshotRuntimeEnv } from "../render/runtimeEnvSnapshot.js";
 import {
  buildSyntheticRenderJob,
+  type DistributedFormat,
  PLAN_VIDEOS_META_RELATIVE_PATH,
  type PlanVideosJson,
  readFfmpegVersion,
@@ -86,7 +87,7 @@ export interface DistributedRenderConfig {
   * `tests/distributed/_smoke/webm-concat-copy.test.ts` for the gating
   * experiment that proved the contract.
   */
-  format: "mp4" | "mov" | "png-sequence" | "webm";
+  format: DistributedFormat;
  /**
   * Codec selection for `format: "mp4"`. `"h264"` (the default) → libx264 +
   * yuv420p; `"h265"` → libx265 + yuv420p with closed-GOP keyint params
@@ -176,7 +177,7 @@ export interface PlanResult {
  fps: 24 | 30 | 60;
  width: number;
  height: number;
-  format: "mp4" | "mov" | "png-sequence" | "webm";
+  format: DistributedFormat;
  ffmpegVersion: string;
  producerVersion: string;
 }
@@ -522,28 +523,15 @@ function buildLockedRenderConfig(input: {
 * caller error immediately rather than producing a silently-wrong planDir
 * whose chunk worker would override the codec choice.
 */
-function resolveEncoderTriple(config: DistributedRenderConfig): {
+type EncoderTriple = {
  encoder: LockedRenderConfig["encoder"];
  pixelFormat: string;
  preset: string;
-} {
+};
+
+function resolveEncoderTriple(config: DistributedRenderConfig): EncoderTriple {
  if (config.format === "mp4") {
-    const codec = config.codec ?? "h264";
-    // Explicit unknown-codec throw rather than silent fall-through to h264.
-    // A JS caller building config from JSON who passes `codec: "h266"` or
-    // `codec: "H265"` (typo / wrong case) would otherwise produce h264
-    // output with no signal. The non-mp4-format branch below already throws
-    // for the symmetric "wrong combination" case — match that shape.
-    if (codec !== "h264" && codec !== "h265") {
-      throw new Error(
-        `[plan] DistributedRenderConfig.codec must be "h264" or "h265" for format="mp4"; ` +
-          `received ${JSON.stringify(codec)}. Omit codec to default to h264.`,
-      );
-    }
-    if (codec === "h265") {
-      return { encoder: "libx265-software", pixelFormat: "yuv420p", preset: "medium" };
-    }
-    return { encoder: "libx264-software", pixelFormat: "yuv420p", preset: "medium" };
+    return resolveMp4EncoderTriple(config.codec);
  }
  if (config.codec !== undefined) {
    throw new Error(
@@ -553,16 +541,46 @@ function resolveEncoderTriple(config: DistributedRenderConfig): {
        `libvpx-vp9, and png-sequence has no encoder.`,
    );
  }
-  if (config.format === "mov") {
+  return resolveNonMp4EncoderTriple(config.format, config.quality ?? "standard");
+}
+
+function resolveMp4EncoderTriple(codec: DistributedRenderConfig["codec"]): EncoderTriple {
+  const c = codec ?? "h264";
+  // Explicit unknown-codec throw rather than silent fall-through to h264.
+  // A JS caller building config from JSON who passes `codec: "h266"` or
+  // `codec: "H265"` (typo / wrong case) would otherwise produce h264
+  // output with no signal. The non-mp4-format branch already throws for
+  // the symmetric "wrong combination" case — match that shape.
+  if (c !== "h264" && c !== "h265") {
+    throw new Error(
+      `[plan] DistributedRenderConfig.codec must be "h264" or "h265" for format="mp4"; ` +
+        `received ${JSON.stringify(c)}. Omit codec to default to h264.`,
+    );
+  }
+  if (c === "h265") {
+    return { encoder: "libx265-software", pixelFormat: "yuv420p", preset: "medium" };
+  }
+  return { encoder: "libx264-software", pixelFormat: "yuv420p", preset: "medium" };
+}
+
+function resolveNonMp4EncoderTriple(
+  format: Exclude<DistributedFormat, "mp4">,
+  quality: "draft" | "standard" | "high",
+): EncoderTriple {
+  if (format === "mov") {
    return { encoder: "prores-software", pixelFormat: "yuva444p10le", preset: "4444" };
  }
-  if (config.format === "webm") {
-    // webm distributes via closed-GOP libvpx-vp9 + concat-copy. yuva420p
-    // matches the in-process renderer's webm pixel format (alpha-capable
-    // — the format's main reason for existing). `getEncoderPreset` in
-    // the engine returns "good" for non-draft quality tiers; that becomes
-    // libvpx-vp9's `-deadline good` at encode time.
-    return { encoder: "libvpx-vp9-software", pixelFormat: "yuva420p", preset: "good" };
+  if (format === "webm") {
+    // Defer to `getEncoderPreset` for the libvpx-vp9 preset string so the
+    // draft tier maps to `-deadline realtime` instead of `-deadline good`;
+    // hardcoding "good" here would silently override that mapping for
+    // `quality: "draft"`.
+    const enginePreset = getEncoderPreset(quality, "webm");
+    return {
+      encoder: "libvpx-vp9-software",
+      pixelFormat: enginePreset.pixelFormat,
+      preset: enginePreset.preset,
+    };
  }
  return { encoder: "png-sequence", pixelFormat: "rgba", preset: "lossless" };
 }
@@ -68,6 +68,7 @@ import { applyRuntimeEnvSnapshot } from "../render/runtimeEnvSnapshot.js";
 import { buildVirtualTimeShim, createFileServer, type FileServerHandle } from "../fileServer.js";
 import {
  buildSyntheticRenderJob,
+  type DistributedFormat,
  PLAN_VIDEOS_META_RELATIVE_PATH,
  type PlanVideosJson,
  readFfmpegVersion,
@@ -199,7 +200,7 @@ interface PlanJson {
    fpsDen: number;
    width: number;
    height: number;
-    format: "mp4" | "mov" | "png-sequence" | "webm";
+    format: DistributedFormat;
  };
  chunkCount: number;
  totalFrames: number;
@@ -14,6 +14,14 @@ import { type VideoElement, type VideoMetadata } from "@hyperframes/engine";
 import { type RenderConfig, type RenderJob, createRenderJob } from "../renderOrchestrator.js";
 import { defaultLogger, type ProducerLogger } from "../../logger.js";

+/**
+ * Output container formats the distributed pipeline supports end-to-end.
+ * Single source of truth for the format union — `plan()`, `renderChunk()`,
+ * `assemble()`, the aws-lambda handler, and the harness all derive from
+ * this type. Adding a new format starts here.
+ */
+export type DistributedFormat = "mp4" | "mov" | "png-sequence" | "webm";
+
 /**
 * Filename of the per-video extraction manifest written by `plan()` into
 * `<planDir>/meta/` and consumed by `renderChunk()` to rebuild the
@@ -32,6 +32,7 @@
 */

 import { createHash } from "node:crypto";
+import type { DistributedFormat } from "../../distributed/shared.js";

 /**
 * Schema-version prefix mixed into every digest. Bump the trailing version
@@ -71,7 +72,7 @@ export interface PlanDimensions {
  fpsDen: number;
  width: number;
  height: number;
-  format: "mp4" | "mov" | "png-sequence" | "webm";
+  format: DistributedFormat;
 }

 export interface PlanHashInput {