mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-19 11:26:02 +00:00
* fix(aws-lambda): run chrome-headless-shell 158 in the arm64 video image The arm64 video image now installs Chrome for Testing's linux-arm64 chrome-headless-shell 158.0.8127.36, pinned by sha256 and fetched with curl -f, as the whole shell folder. The Lambda Node 22 base has none of the shared libraries the shell links and no fonts, so the image installs them from the base's repository at the same pinned release. The image sets HYPERFRAMES_LAMBDA_CHROME_SOURCE and HYPERFRAMES_LAMBDA_CHROME_PATH itself, and the nested video function no longer passes ChromeSource, which would override them. stage-arm64-image.ts stops fetching the sparticuz arm64 pack and drops the ZIP's x86 sparticuz Chrome instead. The x86 handler ZIP stays on @sparticuz/chromium, bumped from 148.0.0 to 153.0.0. It builds at 258,502,222 bytes unzipped, under Lambda's 262,144,000-byte limit and the 248 MiB in-house gate. Two fixes for --source=chrome-headless-shell: - build-zip.ts staged only the executable, which exits at launch without its ICU data and .pak files. It now stages the whole shell folder at chrome-headless-shell-linux64/. - Nothing set HYPERFRAMES_LAMBDA_CHROME_PATH, which src/chromium.ts requires for that source. The SAM template and the CDK stack now set it to the bundled shell when ChromeSource asks for it. Drops the unused @types/aws-lambda devDependency, which the dead-code audit reports whenever this package's manifest changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(publish): build the Lambda arm64 video image on an ARM runner The video stage now installs Chrome's system libraries with microdnf, so it runs code for arm64. The pack job built that image on ubuntu-latest with no emulation, which it could do only while the stage copied files. A new lambda-video-image job builds it natively on ubuntu-24.04-arm, from the handler zip whose sha256 the zip job reported, and reports the OCI tarball's sha256. Pack downloads that tarball, checks the sha256 against the job's, and pins it as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(aws-lambda): fit the comment ratchet and the package size budget chromium.ts loses the two doc lines on BUNDLED_SHELL_DIR and BUNDLED_SHELL_PATH; the names and the SAM template's comment already say what they are. @hyperframes/aws-lambda's unpacked budget goes from 290,508 to 298,000 bytes. The package unpacks to 295,324 bytes, up from the shared chrome-headless-shell staging module the ZIP and image scripts now import and from the README and docs that describe the two Chrome sources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
241 lines
11 KiB
Markdown
241 lines
11 KiB
Markdown
# @hyperframes/aws-lambda
|
||
|
||
AWS Lambda adapter for HyperFrames distributed rendering. Ships three
|
||
things together:
|
||
|
||
1. The **Lambda handler** that wraps the OSS `plan` / `renderChunk` /
|
||
`assemble` primitives behind a single dispatch boundary Step Functions
|
||
can drive (`src/handler.ts`).
|
||
2. A **client-side SDK** — `renderToLambda`, `getRenderProgress`,
|
||
`deploySite`, plus `validateDistributedRenderConfig` and
|
||
`computeRenderCost` (`src/sdk/`).
|
||
3. An **`aws-cdk-lib` L2 construct** (`HyperframesRenderStack`) that
|
||
provisions the same topology as `examples/aws-lambda/template.yaml`
|
||
inside an adopter's own CDK app (`src/cdk/`).
|
||
|
||
The handler ZIP and the SAM template still drive a maintainer-run real-AWS
|
||
smoke flow; the SDK + CDK are the supported public surface for adopters.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ Step Functions state machine │
|
||
│ Plan → Map(N) RenderChunk → Assemble │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
│ dispatches by event.Action
|
||
▼
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ One Lambda function (this package's `dist/handler.zip`) │
|
||
│ handler.mjs │
|
||
│ ├─ Action="plan" → @hyperframes/producer/distributed │
|
||
│ ├─ Action="renderChunk" → @hyperframes/producer/distributed │
|
||
│ └─ Action="assemble" → @hyperframes/producer/distributed │
|
||
│ bin/ffmpeg — ffmpeg-static │
|
||
│ node_modules/@sparticuz/chromium/ — Lambda-optimised Chromium │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
│ pure functions over local paths
|
||
▼
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ S3 bucket — Plan v2 manifest/blobs + chunks + output │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
The handler downloads inputs from S3 into `/tmp`, calls the OSS primitive,
|
||
uploads outputs back to S3, and returns a small JSON result that fits
|
||
inside Step Functions' history budget (under 200 bytes per chunk).
|
||
|
||
### Plan transport
|
||
|
||
Plan v2 is the only supported distributed plan transport.
|
||
`renderToLambda` sends `PlanProtocol: "v2"` without requiring a protocol
|
||
option:
|
||
|
||
```ts
|
||
await renderToLambda({
|
||
// ...bucket, state machine, project, and config...
|
||
});
|
||
```
|
||
|
||
V2 never overloads `PlanS3Uri`. The planner returns
|
||
`PlanV2ManifestS3Uri` and `PlanV2ArtifactS3Prefix`; chunk workers fetch
|
||
only manifest-selected chunk artifacts, while the assembler fetches its
|
||
own metadata and audio subset. Blobs are immutable SHA-256-addressed
|
||
objects, verified on upload and download, and the manifest is published
|
||
last. Unknown protocols and digest mismatches are terminal Step Functions
|
||
errors.
|
||
|
||
#### Upgrade order
|
||
|
||
This default changes application behavior and requires a coordinated
|
||
infrastructure upgrade. Before upgrading an application that calls
|
||
`renderToLambda`:
|
||
|
||
1. Pause new renders and let existing Step Functions executions drain.
|
||
2. Redeploy the Lambda handler and SAM template or CDK construct from the
|
||
same new package version.
|
||
3. Resume renders, then upgrade the application/SDK dependency.
|
||
|
||
A site is a `sites/<id>/files.json` file list that an older stack cannot
|
||
read. Before it uploads anything,
|
||
`renderToLambda` reads the state machine definition
|
||
(`states:DescribeStateMachine`) and fails with `STACK_TOO_OLD` when the
|
||
stack predates that format. If infrastructure cannot be redeployed first,
|
||
keep the application on its previous package version until the redeploy is
|
||
complete. Site ids from earlier versions keep working after the redeploy:
|
||
when `sites/<id>/files.json` is missing, the render uses that site's
|
||
`project.tar.gz`.
|
||
|
||
## Chrome runtime
|
||
|
||
The package supports two Chromium sources:
|
||
|
||
| Source | Default | Size | When to pick it |
|
||
| ------------------------------- | ------- | ------------------ | --------------------------------------------------------------------------------------------------------------------- |
|
||
| `@sparticuz/chromium` | yes | ~70 MiB compressed | Lambda. Decompresses into `/tmp` at runtime; the rest of the ecosystem already uses it for headless-Chrome-in-Lambda. |
|
||
| Bundled `chrome-headless-shell` | no | ~296 MB | Fallback. Used if `@sparticuz/chromium` ever drops `HeadlessExperimental.beginFrame` support. |
|
||
|
||
Pick the source at build time:
|
||
|
||
```bash
|
||
bun run --cwd packages/aws-lambda build:zip
|
||
bun run --cwd packages/aws-lambda build:zip -- --source=chrome-headless-shell
|
||
```
|
||
|
||
The handler reads `HYPERFRAMES_LAMBDA_CHROME_SOURCE` at boot, and for
|
||
`chrome-headless-shell` the executable from `HYPERFRAMES_LAMBDA_CHROME_PATH`.
|
||
`examples/aws-lambda/template.yaml` and the CDK construct set both from the
|
||
Chrome source. The build stages the whole shell folder (the executable needs
|
||
its ICU data and `.pak` files beside it), which is over Lambda's 250 MiB
|
||
unzipped limit for a ZIP function, so the x86 ZIP ships `@sparticuz/chromium`.
|
||
|
||
The arm64 video image (`image/Dockerfile`, `video` target) does not use
|
||
`@sparticuz/chromium`: it installs Chrome for Testing's linux-arm64
|
||
`chrome-headless-shell` folder, pinned by version and sha256, with the shared
|
||
libraries and fonts it needs, and sets both env vars itself.
|
||
|
||
## BeginFrame regression guard
|
||
|
||
HyperFrames' renderer drives Chrome via the CDP
|
||
`HeadlessExperimental.beginFrame` command — same path the K8s deploy uses.
|
||
The Lambda adapter assumes that `@sparticuz/chromium`'s
|
||
chrome-headless-shell build honours BeginFrame. To prove it (and re-prove
|
||
it on every release), the package ships a Docker probe:
|
||
|
||
```bash
|
||
# Build the Lambda-like container and run the probe.
|
||
bun run --cwd packages/aws-lambda probe:beginframe:docker
|
||
```
|
||
|
||
The probe boots `@sparticuz/chromium` inside
|
||
`public.ecr.aws/lambda/nodejs:22` and asserts CDP `beginFrame` with
|
||
`screenshot: true` returns a PNG buffer. Exit code 0 = green; non-zero =
|
||
fall back to bundling chrome-headless-shell directly via `--source=chrome-headless-shell`.
|
||
|
||
## Building the ZIP
|
||
|
||
```bash
|
||
bun install # at the monorepo root
|
||
bun run --cwd packages/aws-lambda build:zip # → packages/aws-lambda/dist/handler.zip
|
||
bun run --cwd packages/aws-lambda verify:zip-size # CI gate
|
||
```
|
||
|
||
The build script bundles `src/handler.ts` via esbuild, stages
|
||
`@sparticuz/chromium` and `puppeteer-core` under `node_modules/`, copies
|
||
ffmpeg-static into `bin/`, and zips the result. The unzipped layout is
|
||
designed to extract cleanly into Lambda's `/var/task/`.
|
||
|
||
`verify:zip-size` enforces:
|
||
|
||
- Unzipped ≤ 248 MiB (in-house budget; Lambda hard ceiling is 250 MiB unzipped — AWS docs label this "250 MB" but use binary mebibytes)
|
||
- Zipped ≤ 150 MiB (in-house budget; Lambda has no hard zipped cap for S3-deployed functions)
|
||
|
||
CI fails the PR if either is exceeded.
|
||
|
||
## Running tests
|
||
|
||
```bash
|
||
bun run --cwd packages/aws-lambda test # unit tests (no Chrome)
|
||
bun run --cwd packages/aws-lambda probe:beginframe # local probe (Linux only)
|
||
```
|
||
|
||
## Using the SDK
|
||
|
||
After deploying the stack (via the SAM template, CDK construct below, or
|
||
your own CFN of choice), drive renders from Node:
|
||
|
||
```ts
|
||
import { deploySite, getRenderProgress, renderToLambda } from "@hyperframes/aws-lambda";
|
||
|
||
// One-time upload per project version.
|
||
const site = await deploySite({
|
||
projectDir: "./my-composition",
|
||
bucketName: "hyperframes-render-bucket",
|
||
});
|
||
|
||
// Start a render. Returns immediately — does NOT poll.
|
||
const handle = await renderToLambda({
|
||
siteHandle: site,
|
||
bucketName: site.bucketName,
|
||
stateMachineArn: "arn:aws:states:us-east-1:123:stateMachine:hyperframes-render",
|
||
config: {
|
||
fps: 30,
|
||
width: 1920,
|
||
height: 1080,
|
||
format: "mp4",
|
||
chunkSize: 240,
|
||
maxParallelChunks: 16,
|
||
runtimeCap: "lambda",
|
||
},
|
||
});
|
||
|
||
// Poll progress + cost on your own cadence.
|
||
const progress = await getRenderProgress({ executionArn: handle.executionArn });
|
||
console.log(progress.overallProgress, progress.costs.displayCost);
|
||
if (progress.status === "SUCCEEDED" && progress.outputFile) {
|
||
console.log("Render landed at", progress.outputFile.s3Uri);
|
||
}
|
||
```
|
||
|
||
`renderToLambda` validates the config client-side via
|
||
`validateDistributedRenderConfig` and throws a typed `InvalidConfigError`
|
||
before the Step Functions execution starts, so shape errors surface
|
||
synchronously instead of as opaque `ExecutionFailed` results.
|
||
|
||
`getRenderProgress` reports an approximate per-render cost
|
||
(`accruedSoFarUsd` plus a formatted `displayCost`) derived from Lambda
|
||
billed-duration × memory × the us-east-1 on-demand rate plus the Step
|
||
Functions transition price. The math is documented in
|
||
`src/sdk/costAccounting.ts`; numbers are best-effort and exclude S3
|
||
transfer.
|
||
|
||
## Using the CDK construct
|
||
|
||
```ts
|
||
import { App, Stack } from "aws-cdk-lib";
|
||
import { HyperframesRenderStack } from "@hyperframes/aws-lambda/cdk";
|
||
|
||
const app = new App();
|
||
const stack = new Stack(app, "MyApp");
|
||
const render = new HyperframesRenderStack(stack, "Render", {
|
||
// optional: reservedConcurrency: 8,
|
||
// optional: lambdaMemoryMb: 10240,
|
||
// optional: chromeSource: "sparticuz",
|
||
});
|
||
|
||
// Re-export so an adopter app can wire dashboards / SNS topics.
|
||
new CfnOutput(stack, "RenderBucketName", { value: render.bucket.bucketName });
|
||
new CfnOutput(stack, "StateMachineArn", { value: render.stateMachine.stateMachineArn });
|
||
```
|
||
|
||
`aws-cdk-lib` and `constructs` are **optional peer dependencies**: SDK-only
|
||
consumers don't pull them at runtime. The construct itself imports from
|
||
`@hyperframes/aws-lambda/cdk`.
|
||
|
||
## What's still ahead
|
||
|
||
- `hyperframes lambda` CLI (deploy / sites create / render / progress / destroy) — PR 6.5.
|
||
- IAM bootstrap subcommand (`policies role | user | validate`) — PR 6.9.
|
||
- Lambda-local regression harness (`--mode=lambda-local`) — PR 6.6.
|
||
- Adopter-facing migration guide — PR 6.8.
|