Files
hyperframes/packages/aws-lambda/README.md
T
James RussoandClaude Opus 5.5 61d65e7177 fix(aws-lambda): match the shell to its own pin, ship only hashed files, refuse the shell ZIP in CDK and SAM (#8173)
* fix(aws-lambda): match the shell to its own pin, ship only hashed files, refuse the shell ZIP in CDK and SAM

Review follow-ups to the pinned shell route:
- build-zip --source=chrome-headless-shell accepts the folder only when its
  digest is pinned for this version, not for any pinned build.
- It leaves out the files the digest skips (debug.log and chrome_debug.log,
  which Chrome writes as it runs), so nothing unhashed ships.
- The CDK construct throws on chromeSource "chrome-headless-shell" and the
  SAM template allows only sparticuz, as lambda deploy already refused it:
  a shell ZIP cannot fit a function. The CHROME_PATH wiring for that ZIP is
  gone with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(aws-lambda): raise the packed size budget to 82,000 bytes

The shipped build-zip script now checks the shell against its own
version's pin and skips the files the digest leaves out. That takes the
tarball to 81,378 bytes in CI, over the 80,500 budget's 1% tolerance;
main was already within about 200 bytes of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:25:22 +00:00

250 lines
12 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# @hyperframes/aws-lambda
AWS Lambda adapter for HyperFrames distributed rendering. Ships three
things together:
1. The **Lambda handler** that wraps the OSS `plan` / `renderChunk` /
`assemble` primitives behind a single dispatch boundary Step Functions
can drive (`src/handler.ts`).
2. A **client-side SDK** — `renderToLambda`, `getRenderProgress`,
`deploySite`, plus `validateDistributedRenderConfig` and
`computeRenderCost` (`src/sdk/`).
3. An **`aws-cdk-lib` L2 construct** (`HyperframesRenderStack`) that
provisions the same topology as `examples/aws-lambda/template.yaml`
inside an adopter's own CDK app (`src/cdk/`).
The handler ZIP and the SAM template still drive a maintainer-run real-AWS
smoke flow; the SDK + CDK are the supported public surface for adopters.
## Architecture
```
┌──────────────────────────────────────────────────────────────────┐
│ Step Functions state machine │
│ Plan → Map(N) RenderChunk → Assemble │
└──────────────────────────────────────────────────────────────────┘
│ dispatches by event.Action
▼
┌──────────────────────────────────────────────────────────────────┐
│ One Lambda function (this package's `dist/handler.zip`) │
│ handler.mjs │
│ ├─ Action="plan" → @hyperframes/producer/distributed │
│ ├─ Action="renderChunk" → @hyperframes/producer/distributed │
│ └─ Action="assemble" → @hyperframes/producer/distributed │
│ bin/ffmpeg — ffmpeg-static │
│ node_modules/@sparticuz/chromium/ — Lambda-optimised Chromium │
└──────────────────────────────────────────────────────────────────┘
│ pure functions over local paths
▼
┌──────────────────────────────────────────────────────────────────┐
│ S3 bucket — Plan v2 manifest/blobs + chunks + output │
└──────────────────────────────────────────────────────────────────┘
```
The handler downloads inputs from S3 into `/tmp`, calls the OSS primitive,
uploads outputs back to S3, and returns a small JSON result that fits
inside Step Functions' history budget (under 200 bytes per chunk).
### Plan transport
Plan v2 is the only supported distributed plan transport.
`renderToLambda` sends `PlanProtocol: "v2"` without requiring a protocol
option:
```ts
await renderToLambda({
// ...bucket, state machine, project, and config...
});
```
V2 never overloads `PlanS3Uri`. The planner returns
`PlanV2ManifestS3Uri` and `PlanV2ArtifactS3Prefix`; chunk workers fetch
only manifest-selected chunk artifacts, while the assembler fetches its
own metadata and audio subset. Blobs are immutable SHA-256-addressed
objects, verified on upload and download, and the manifest is published
last. Unknown protocols and digest mismatches are terminal Step Functions
errors.
#### Upgrade order
This default changes application behavior and requires a coordinated
infrastructure upgrade. Before upgrading an application that calls
`renderToLambda`:
1. Pause new renders and let existing Step Functions executions drain.
2. Redeploy the Lambda handler and SAM template or CDK construct from the
same new package version.
3. Resume renders, then upgrade the application/SDK dependency.
A site is a `sites/<id>/files.json` file list that an older stack cannot
read. Before it uploads anything,
`renderToLambda` reads the state machine definition
(`states:DescribeStateMachine`) and fails with `STACK_TOO_OLD` when the
stack predates that format. If infrastructure cannot be redeployed first,
keep the application on its previous package version until the redeploy is
complete. Site ids from earlier versions keep working after the redeploy:
when `sites/<id>/files.json` is missing, the render uses that site's
`project.tar.gz`.
## Chrome runtime
The package supports two Chromium sources:
| Source | Default | Size | When to pick it |
| ------------------------------- | ------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------- |
| `@sparticuz/chromium` | yes | ~70 MiB compressed | Lambda. Decompresses into `/tmp` at runtime; the rest of the ecosystem already uses it for headless-Chrome-in-Lambda. |
| Bundled `chrome-headless-shell` | no | ~296 MB | Does not fit a ZIP function today. Kept for the case where `@sparticuz/chromium` drops `HeadlessExperimental.beginFrame`. |
Pick the source at build time:
```bash
bun run --cwd packages/aws-lambda build:zip
bun run --cwd packages/aws-lambda build:zip -- --source=chrome-headless-shell
```
The handler reads `HYPERFRAMES_LAMBDA_CHROME_SOURCE` at boot, and for
`chrome-headless-shell` the executable from `HYPERFRAMES_LAMBDA_CHROME_PATH`.
The build stages the repo's pinned shell build, and only when its folder
matches that build's pinned digest. It stages the whole folder (the
executable needs its ICU data and `.pak` files beside it), which is over
Lambda's 250 MiB unzipped limit for a ZIP function, so the build fails its
size gate and the x86 ZIP ships `@sparticuz/chromium`. For the same reason
`hyperframes lambda deploy`, `examples/aws-lambda/template.yaml` and the CDK
construct refuse `chrome-headless-shell` as the ZIP's Chrome source.
The arm64 video image (`image/Dockerfile`, `video` target) does not use
`@sparticuz/chromium`: it installs Chrome for Testing's linux-arm64
`chrome-headless-shell` folder, pinned by version and sha256, with the shared
libraries it needs, and sets both env vars itself. It keeps `@sparticuz/chromium`'s
fonts and `fonts.conf` from the handler ZIP, so text resolves to the same fonts as
on the x86 function; CI compares the two on a mixed-script page.
## BeginFrame regression guard
HyperFrames' renderer drives Chrome via the CDP
`HeadlessExperimental.beginFrame` command — same path the K8s deploy uses.
The Lambda adapter assumes that `@sparticuz/chromium`'s
chrome-headless-shell build honours BeginFrame. To prove it (and re-prove
it on every release), the package ships a Docker probe:
```bash
# Build the Lambda-like container and run the probe.
bun run --cwd packages/aws-lambda probe:beginframe:docker
```
The probe boots `@sparticuz/chromium` inside
`public.ecr.aws/lambda/nodejs:22` and asserts CDP `beginFrame` with
`screenshot: true` returns a PNG buffer. Exit code 0 = green; non-zero =
`@sparticuz/chromium` lost BeginFrame. `--source=chrome-headless-shell` does
not fit a ZIP function, so that function would move to a container image like
the arm64 video function.
## Building the ZIP
```bash
bun install # at the monorepo root
bun run --cwd packages/aws-lambda build:zip # → packages/aws-lambda/dist/handler.zip
bun run --cwd packages/aws-lambda verify:zip-size # CI gate
```
The build script bundles `src/handler.ts` via esbuild, stages
`@sparticuz/chromium` and `puppeteer-core` under `node_modules/`, copies
ffmpeg-static into `bin/`, and zips the result. The unzipped layout is
designed to extract cleanly into Lambda's `/var/task/`.
`verify:zip-size` enforces:
- Unzipped ≤ 248 MiB (in-house budget; Lambda hard ceiling is 250 MiB unzipped — AWS docs label this "250 MB" but use binary mebibytes)
- Zipped ≤ 150 MiB (in-house budget; Lambda has no hard zipped cap for S3-deployed functions)
The sparticuz ZIP is about 1.5 MiB under the unzipped budget, and the handler
bundles the producer, engine and core. So the `Lambda handler ZIP` workflow
builds it on every PR that touches them, and fails the PR if either is exceeded.
## Running tests
```bash
bun run --cwd packages/aws-lambda test # unit tests (no Chrome)
bun run --cwd packages/aws-lambda probe:beginframe # local probe (Linux only)
```
## Using the SDK
After deploying the stack (via the SAM template, CDK construct below, or
your own CFN of choice), drive renders from Node:
```ts
import { deploySite, getRenderProgress, renderToLambda } from "@hyperframes/aws-lambda";
// One-time upload per project version.
const site = await deploySite({
projectDir: "./my-composition",
bucketName: "hyperframes-render-bucket",
});
// Start a render. Returns immediately — does NOT poll.
const handle = await renderToLambda({
siteHandle: site,
bucketName: site.bucketName,
stateMachineArn: "arn:aws:states:us-east-1:123:stateMachine:hyperframes-render",
config: {
fps: 30,
width: 1920,
height: 1080,
format: "mp4",
chunkSize: 240,
maxParallelChunks: 16,
runtimeCap: "lambda",
},
});
// Poll progress + cost on your own cadence.
const progress = await getRenderProgress({ executionArn: handle.executionArn });
console.log(progress.overallProgress, progress.costs.displayCost);
if (progress.status === "SUCCEEDED" && progress.outputFile) {
console.log("Render landed at", progress.outputFile.s3Uri);
}
```
`renderToLambda` validates the config client-side via
`validateDistributedRenderConfig` and throws a typed `InvalidConfigError`
before the Step Functions execution starts, so shape errors surface
synchronously instead of as opaque `ExecutionFailed` results.
`getRenderProgress` reports an approximate per-render cost
(`accruedSoFarUsd` plus a formatted `displayCost`) derived from Lambda
billed-duration × memory × the us-east-1 on-demand rate plus the Step
Functions transition price. The math is documented in
`src/sdk/costAccounting.ts`; numbers are best-effort and exclude S3
transfer.
## Using the CDK construct
```ts
import { App, Stack } from "aws-cdk-lib";
import { HyperframesRenderStack } from "@hyperframes/aws-lambda/cdk";
const app = new App();
const stack = new Stack(app, "MyApp");
const render = new HyperframesRenderStack(stack, "Render", {
// optional: reservedConcurrency: 8,
// optional: lambdaMemoryMb: 10240,
// optional: chromeSource: "sparticuz",
});
// Re-export so an adopter app can wire dashboards / SNS topics.
new CfnOutput(stack, "RenderBucketName", { value: render.bucket.bucketName });
new CfnOutput(stack, "StateMachineArn", { value: render.stateMachine.stateMachineArn });
```
`aws-cdk-lib` and `constructs` are **optional peer dependencies**: SDK-only
consumers don't pull them at runtime. The construct itself imports from
`@hyperframes/aws-lambda/cdk`.
## What's still ahead
- `hyperframes lambda` CLI (deploy / sites create / render / progress / destroy) — PR 6.5.
- IAM bootstrap subcommand (`policies role | user | validate`) — PR 6.9.
- Lambda-local regression harness (`--mode=lambda-local`) — PR 6.6.
- Adopter-facing migration guide — PR 6.8.