Files
hyperframes/packages/aws-lambda
James RussoandClaude Opus 4.7 21f5066832 feat(producer): enable webm in distributed mode via concat-copy (#951)
* feat(producer): enable webm in distributed mode via concat-copy

PR 8.2 of the WebM distributed-rendering plan (v1.5 backlog #1; see
DISTRIBUTED-RENDERING-PLAN.md §7.2). Wires libvpx-vp9 webm through the
distributed pipeline now that PR 8.1 proved concat-copy works.

Architectural decision: Path A (concat-copy) — based on PR 8.1's smoke
test result (9/9 tests pass for both yuv420p and yuva420p VP9 streams).
The simpler architecture wins; no re-encode in assemble, no encode-
parallelism loss.

Changes:

- plan.ts:
  - DistributedRenderConfig.format and PlanResult.format now include
    "webm" — type-level acceptance matches the runtime gate.
  - rejectUnsupportedDistributedFormat() no longer trips on webm. HDR
    mp4 remains the only refused configuration.
  - resolveEncoderTriple() returns libvpx-vp9-software + yuva420p +
    preset="good" for format="webm". yuva420p preserves alpha — the
    format's main reason for existing for web delivery.
  - codec= remains rejected for non-mp4 formats (mov is always ProRes
    4444; webm is always libvpx-vp9). The error message lists all four
    distributed-supported formats.
  - FormatNotSupportedInDistributedError docstring updated to reflect
    the new reality (only HDR is unsupported).

- freezePlan.ts: LockedRenderConfig.encoder gains "libvpx-vp9-software".
  Mirrors libx265-software / prores-software / png-sequence in shape;
  the chunk worker reads this discriminant to decide encode args.

- renderChunk.ts: drops the now-incorrect cast that excluded webm from
  buildSyntheticRenderJob's format input; tightens the preset-format
  cast to include webm.

- assemble.ts: docstring + comment updates. The mp4/mov concat-copy
  path is format-agnostic — webm uses the exact same code (applyFaststart
  is a no-op for webm via the existing chunkEncoder.ts gate;
  muxVideoWithAudio already routes webm to libopus audio).

- planFormatBanlist.test.ts: webm-rejection tests removed; replaced with
  "accepts webm" tests + a HDR+webm combo test that verifies HDR is the
  trip regardless of format.

- plan.test.ts: new describe block pins the webm wiring contract:
  format="webm" produces an encoder=libvpx-vp9-software /
  pixelFormat=yuva420p planDir with closedGop=true and gopSize=chunkSize.

- webm-concat-copy.test.ts (smoke): extended with a yuva420p variant
  that proves the alpha pixel format the distributed pipeline actually
  emits also round-trips through concat-copy. 9/9 tests pass locally.

§8 format support matrix in DISTRIBUTED-RENDERING-PLAN.md is intentionally
left unchanged at this PR — it flips to ✓ in PR 8.4 once the end-to-end
fixture (PR 8.3) is green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(producer): include webm in plan-time needsAlpha + strengthen alpha smoke

PR review feedback from Miguel and Vai on #951 caught a real bug:
`plan.ts`'s `needsAlpha` disjunction excluded `"webm"`, so the plan
stage froze `forceScreenshot: false` into the `LockedRenderConfig`
even though distributed webm uses `yuva420p`. Every chunk worker
captured opaque RGB via BeginFrame (which doesn't preserve alpha on
Linux headless-shell), and libvpx-vp9 encoded uniformly-opaque alpha
that the encoder then dropped — producing un-keyable webm.

Two changes:

1. **plan.ts**: include `"webm"` in `needsAlpha`. Matches the
   in-process renderer's logic at `renderOrchestrator.ts:1469`
   (`const needsAlpha = isWebm || isMov || isPngSequence`); the two
   sites must stay in sync since the distributed pipeline's PSNR
   regression compares against the in-process baseline.

2. **Smoke test (yuva420p describe)**: source frames now use a real
   alpha gradient (`geq=a='X*255/W'` on top of `testsrc2`) instead of
   `testsrc2 + format=rgba` which was uniformly opaque. The decode-
   pix_fmt assertion is dropped (ffprobe reports `yuv420p` for
   VP9-with-alpha because the alpha lives in a Matroska
   `BlockAdditional` sidecar) and replaced with two stronger checks:
   - `TAG:ALPHA_MODE=1` is present on the stream — proves the
     encoder was actually configured for alpha
   - alpha plane variance after `-c:v libvpx-vp9 -i ... -pix_fmt rgba
     -vf extractplanes=a,signalstats` — proves the alpha sub-stream
     round-trips through concat-copy with spatially-varying content,
     not uniform/dropped alpha
   - decode-test gate is now exit-code-only (was `exitCode || stderr`
     which would flake on chatty ffmpeg `-v error` builds emitting
     non-fatal DTS/container notes)

These checks would have caught the `needsAlpha` bug before review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(aws-lambda): widen narrow format types to include webm

CI on PR #951 was failing at typecheck/build because the producer's
`DistributedRenderConfig.format` widened to include webm in this PR
but the aws-lambda package's narrow `"mp4" | "mov" | "png-sequence"`
type literals in `events.ts`, `handler.ts`, and `validateConfig.ts`
hadn't kept up. `renderToLambda.ts:87` passed `config.format` (now
including webm) into a parameter typed against the narrow union,
producing TS2345.

This widening originally landed in PR #952 (test fixture PR) but
needs to be atomic with the producer's widening here to keep each
PR independently typecheck-clean.

Also refactor `formatExtension` from a switch dispatch to a
`Record<DistributedFormat, string>` lookup. Adding the webm case
tipped the switch's CRAP to the 30.0 fallow threshold; the lookup
table drops cyclomatic from 5 to 1 with the same compile-time
exhaustiveness guarantee (TS errors on missing entries when
`DistributedFormat` adds a new format). The runtime
`_exhaustive: never` throw was only protecting against a string
slipping past TS; `validateConfig.ts`'s `ALLOWED_FORMATS` already
gates untrusted input at the SDK boundary.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 02:46:21 -04:00
..
2026-05-19 02:33:31 +00:00

@hyperframes/aws-lambda

AWS Lambda adapter for HyperFrames distributed rendering. Ships three things together:

  1. The Lambda handler that wraps the OSS plan / renderChunk / assemble primitives behind a single dispatch boundary Step Functions can drive (src/handler.ts).
  2. A client-side SDKrenderToLambda, getRenderProgress, deploySite, plus validateDistributedRenderConfig and computeRenderCost (src/sdk/).
  3. An aws-cdk-lib L2 construct (HyperframesRenderStack) that provisions the same topology as examples/aws-lambda/template.yaml inside an adopter's own CDK app (src/cdk/).

The handler ZIP and the SAM template still drive a maintainer-run real-AWS smoke flow; the SDK + CDK are the supported public surface for adopters.

Architecture

┌──────────────────────────────────────────────────────────────────┐
│ Step Functions state machine                                     │
│   Plan → Map(N) RenderChunk → Assemble                           │
└──────────────────────────────────────────────────────────────────┘
                              │ dispatches by event.Action
                              ▼
┌──────────────────────────────────────────────────────────────────┐
│ One Lambda function (this package's `dist/handler.zip`)          │
│   handler.mjs                                                    │
│     ├─ Action="plan"        → @hyperframes/producer/distributed  │
│     ├─ Action="renderChunk" → @hyperframes/producer/distributed  │
│     └─ Action="assemble"    → @hyperframes/producer/distributed  │
│   bin/ffmpeg                — ffmpeg-static                      │
│   node_modules/@sparticuz/chromium/ — Lambda-optimised Chromium  │
└──────────────────────────────────────────────────────────────────┘
                              │ pure functions over local paths
                              ▼
┌──────────────────────────────────────────────────────────────────┐
│ S3 bucket — plan tarball + per-chunk outputs + final mp4         │
└──────────────────────────────────────────────────────────────────┘

The handler downloads inputs from S3 into /tmp, calls the OSS primitive, uploads outputs back to S3, and returns a small JSON result that fits inside Step Functions' history budget (under 200 bytes per chunk).

Chrome runtime

The package supports two Chromium sources:

Source Default Size When to pick it
@sparticuz/chromium yes ~70 MiB compressed Lambda. Decompresses into /tmp at runtime; the rest of the ecosystem already uses it for headless-Chrome-in-Lambda.
Bundled chrome-headless-shell no ~140 MiB Fallback. Used if @sparticuz/chromium ever drops HeadlessExperimental.beginFrame support.

Pick the source at build time:

bun run --cwd packages/aws-lambda build:zip
bun run --cwd packages/aws-lambda build:zip -- --source=chrome-headless-shell

The handler reads HYPERFRAMES_LAMBDA_CHROME_SOURCE at boot. The build script sets that env var via Lambda function configuration in examples/aws-lambda/template.yaml.

BeginFrame regression guard

HyperFrames' renderer drives Chrome via the CDP HeadlessExperimental.beginFrame command — same path the K8s deploy uses. The Lambda adapter assumes that @sparticuz/chromium's chrome-headless-shell build honours BeginFrame. To prove it (and re-prove it on every release), the package ships a Docker probe:

# Build the Lambda-like container and run the probe.
bun run --cwd packages/aws-lambda probe:beginframe:docker

The probe boots @sparticuz/chromium inside public.ecr.aws/lambda/nodejs:22 and asserts CDP beginFrame with screenshot: true returns a PNG buffer. Exit code 0 = green; non-zero = fall back to bundling chrome-headless-shell directly via --source=chrome-headless-shell.

Building the ZIP

bun install                                          # at the monorepo root
bun run --cwd packages/aws-lambda build:zip          # → packages/aws-lambda/dist/handler.zip
bun run --cwd packages/aws-lambda verify:zip-size    # CI gate

The build script bundles src/handler.ts via esbuild, stages @sparticuz/chromium and puppeteer-core under node_modules/, copies ffmpeg-static into bin/, and zips the result. The unzipped layout is designed to extract cleanly into Lambda's /var/task/.

verify:zip-size enforces:

  • Unzipped ≤ 248 MiB (in-house budget; Lambda hard ceiling is 250 MiB unzipped — AWS docs label this "250 MB" but use binary mebibytes)
  • Zipped ≤ 150 MiB (in-house budget; Lambda has no hard zipped cap for S3-deployed functions)

CI fails the PR if either is exceeded.

Running tests

bun run --cwd packages/aws-lambda test               # unit tests (no Chrome)
bun run --cwd packages/aws-lambda probe:beginframe   # local probe (Linux only)

Using the SDK

After deploying the stack (via the SAM template, CDK construct below, or your own CFN of choice), drive renders from Node:

import { deploySite, getRenderProgress, renderToLambda } from "@hyperframes/aws-lambda";

// One-time upload per project version.
const site = await deploySite({
  projectDir: "./my-composition",
  bucketName: "hyperframes-render-bucket",
});

// Start a render. Returns immediately — does NOT poll.
const handle = await renderToLambda({
  siteHandle: site,
  bucketName: site.bucketName,
  stateMachineArn: "arn:aws:states:us-east-1:123:stateMachine:hyperframes-render",
  config: {
    fps: 30,
    width: 1920,
    height: 1080,
    format: "mp4",
    chunkSize: 240,
    maxParallelChunks: 16,
    runtimeCap: "lambda",
  },
});

// Poll progress + cost on your own cadence.
const progress = await getRenderProgress({ executionArn: handle.executionArn });
console.log(progress.overallProgress, progress.costs.displayCost);
if (progress.status === "SUCCEEDED" && progress.outputFile) {
  console.log("Render landed at", progress.outputFile.s3Uri);
}

renderToLambda validates the config client-side via validateDistributedRenderConfig and throws a typed InvalidConfigError before the Step Functions execution starts, so shape errors surface synchronously instead of as opaque ExecutionFailed results.

getRenderProgress reports an approximate per-render cost (accruedSoFarUsd plus a formatted displayCost) derived from Lambda billed-duration × memory × the us-east-1 on-demand rate plus the Step Functions transition price. The math is documented in src/sdk/costAccounting.ts; numbers are best-effort and exclude S3 transfer.

Using the CDK construct

import { App, Stack } from "aws-cdk-lib";
import { HyperframesRenderStack } from "@hyperframes/aws-lambda/cdk";

const app = new App();
const stack = new Stack(app, "MyApp");
const render = new HyperframesRenderStack(stack, "Render", {
  // optional: reservedConcurrency: 8,
  // optional: lambdaMemoryMb: 10240,
  // optional: chromeSource: "sparticuz",
});

// Re-export so an adopter app can wire dashboards / SNS topics.
new CfnOutput(stack, "RenderBucketName", { value: render.bucket.bucketName });
new CfnOutput(stack, "StateMachineArn", { value: render.stateMachine.stateMachineArn });

aws-cdk-lib and constructs are optional peer dependencies: SDK-only consumers don't pull them at runtime. The construct itself imports from @hyperframes/aws-lambda/cdk.

What's still ahead

  • hyperframes lambda CLI (deploy / sites create / render / progress / destroy) — PR 6.5.
  • IAM bootstrap subcommand (policies role | user | validate) — PR 6.9.
  • Lambda-local regression harness (--mode=lambda-local) — PR 6.6.
  • Adopter-facing migration guide — PR 6.8.