mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-10-30 17:27:09 +00:00
* perf(aws-lambda): run chunks through a Distributed Map to lift the 40-chunk cap The inline Map runs at most 40 chunk renders at once. Both state machine definitions (CDK and SAM) now use a Distributed Map with STANDARD child executions, capped at the function's reserved concurrency so chunks past it queue instead of being throttled. Progress reads each chunk's child execution history so frames, cost and chunk errors still show up. * fix(aws-lambda): retry Lambda throttles for minutes and stop on oversized chunks A throttled task used to burn its four catch-all retries in about 30 s. It now gets its own jittered backoff of several minutes, because another render on the same function can still hold the reserved slots. Plan treats CHUNK_EXCEEDS_WORKER_TIMEOUT as terminal in both definitions. * chore(aws-lambda): trim stack comments to keep the comment share flat * fix(aws-lambda): keep render --wait working at hundreds of chunks Progress polls read the chunk Map Run's counts (one DescribeMapRun) instead of every child execution's history, take exact chunk frames and cost from the Map output once it exits, and read failed children only when the counts show a failure. The SDK client retries throttled calls up to 8 times with backoff. * feat(aws-lambda): size each render's concurrency to the account instead of a fixed 8 * fix(lambda): keep the concurrency SDK external in published modules * fix(cli): keep an existing stack's reserved concurrency on redeploy Redeploying without --concurrency sent ReservedConcurrency=-1, which removed the cap of 8 that earlier CLI versions reserved by default and stopped recording it in the state file. The override is now left out when the flag is omitted, so SAM keeps the stack's current value (a new stack still gets the template's unreserved default). The state file records the deployed value, and deploy prints the kept reservation. * fix(aws-lambda): keep progress working for callers without DescribeMapRun Callers whose IAM policy came from an earlier `lambda policies user` lack states:DescribeMapRun, so reading a running Distributed Map threw AccessDeniedException out of `lambda progress` and `render --wait` while the render itself went on to succeed. getRenderProgress now catches only that error around the map-run read, returns the parent snapshot (status, plan and assemble progress) and adds a warning to re-apply the policy, which both commands print. Other errors still throw. * fix(cli): let the documented deploy policy drop a reserved concurrency When a redeploy removes a function's reserved concurrency, CloudFormation calls lambda:DeleteFunctionConcurrency. The policy printed by `lambda policies role|user` granted only PutFunctionConcurrency, so the update failed and rolled back. Grant it, and pin it with states:DescribeMapRun in the required-actions test. * fix(aws-lambda): read smoke chunk hashes from the Distributed Map output With chunks in a Distributed Map, their task results live in child executions, so the smoke gate's chunk-hash check found 0 chunks in the parent history and failed every v1/v2 comparison. Read Chunks[] from the RenderChunks / RenderChunksV2 MapStateExited output instead, as getRenderProgress does. The stack this script deploys only has the Distributed Map, so the inline-Map path is gone. * fix(producer): cap a pinned chunk size at the worker concurrency, not 16 With --chunk-size or --target-chunk-frames and no --max-parallel-chunks, the plan still capped the render at 16 chunks however many workers the account allowed, so --chunk-size 30 on a 60 s render gave 16 chunks of 113 frames. The cap is now one wave across the worker concurrency, as the --max-parallel-chunks docs say; 16 remains when the concurrency is unknown. * feat(cli): remove a stack's reserved concurrency with --concurrency=none Since an omitted --concurrency now keeps the stack's reservation, there was no CLI way back to the unreserved default. --concurrency=none sends ReservedConcurrency=-1, the state file records the stack as unreserved and deploy says the reservation was removed. Positive N and an omitted flag behave as before; any other value is rejected instead of being silently ignored. * chore(aws-lambda): raise the package size budget for account-sized concurrency Measured with pnpm@10.17.1 pack after rebasing on main with the S3 assembly change: 67,763 packed / 244,685 unpacked; the budget is the check's own 5% headroom over that.
248 lines
14 KiB
Plaintext
248 lines
14 KiB
Plaintext
---
|
||
title: AWS Lambda
|
||
description: "Deploy distributed HyperFrames rendering to AWS Lambda and drive renders from a laptop or CI."
|
||
---
|
||
|
||
HyperFrames ships a first-class AWS Lambda deployment: one Lambda function fronts a Step Functions standard workflow that fans renders out across many parallel chunk workers, with intermediate artifacts in S3. End-to-end is three commands once your AWS credentials are configured.
|
||
|
||
```bash
|
||
hyperframes lambda deploy
|
||
hyperframes lambda render ./my-project --width 1920 --height 1080 --wait
|
||
hyperframes lambda destroy
|
||
```
|
||
|
||
Templates with [variables](/concepts/variables) work on the same Lambda stack — declare `data-composition-variables` on the composition, then pass values per render with `--variables` or fan a whole batch out with `lambda render-batch`. See the [Templates on Lambda](/deploy/templates-on-lambda) guide for the personalised-render pipeline (single render, batch from JSONL, programmatic SDK) and the 256 KiB Step Functions execution-input cap.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ Step Functions state machine │
|
||
│ Plan → Map(N) RenderChunk → Assemble │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
│ dispatches by event.Action
|
||
▼
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ One Lambda function (packages/aws-lambda/dist/handler.zip) │
|
||
│ handler.mjs │
|
||
│ ├─ Action="plan" → @hyperframes/producer/distributed │
|
||
│ ├─ Action="renderChunk" → @hyperframes/producer/distributed │
|
||
│ └─ Action="assemble" → @hyperframes/producer/distributed │
|
||
│ bin/ffmpeg — ffmpeg-static │
|
||
│ node_modules/@sparticuz/chromium/ — Lambda-optimised Chromium │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
│ pure functions over local paths
|
||
▼
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ S3 bucket — plan tarball + per-chunk outputs + final mp4 │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
The Lambda handler is a thin dispatch: parse the Step Functions event, download inputs from S3 into `/tmp`, call the OSS primitive from `@hyperframes/producer/distributed`, upload outputs back, return a small JSON result. Everything heavy — capture, encode, audio mix — happens inside the OSS primitives.
|
||
|
||
## Prerequisites
|
||
|
||
| Tool | Why | Install |
|
||
|------|-----|---------|
|
||
| AWS credentials | The CLI and the deploy step both call AWS APIs. | Env vars, `~/.aws/credentials`, SSO, or IMDS — any chain the AWS SDK for JavaScript v3 would resolve. |
|
||
| AWS SAM CLI | `hyperframes lambda deploy/destroy` shells out to `sam deploy`/`sam delete`. | [Install guide](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html) |
|
||
| `bun` | Used to build `packages/aws-lambda/dist/handler.zip` at deploy time. | `npm install -g bun` or [bun.sh](https://bun.sh) |
|
||
| Lambda adapter | The published CLI loads the AWS adapter only for Lambda commands. | `npm install -g @hyperframes/aws-lambda` alongside the CLI |
|
||
| HyperFrames repo checkout | `lambda deploy` builds the Lambda handler ZIP from source. Adopters who deploy outside a checkout can set `HYPERFRAMES_REPO_ROOT` to point at one. | `git clone https://github.com/heygen-com/hyperframes` |
|
||
|
||
## Three deployment paths
|
||
|
||
### Path 1 — `hyperframes lambda` CLI (recommended)
|
||
|
||
The CLI is a thin wrapper around the SAM template + the `@hyperframes/aws-lambda` SDK. For most adopters this is the right starting point.
|
||
|
||
```bash
|
||
hyperframes lambda deploy \
|
||
--stack-name=hyperframes-prod \
|
||
--region=us-east-1 \
|
||
--memory=10240
|
||
```
|
||
|
||
By default the stack reserves no concurrency. Each render reads the account's
|
||
unreserved Lambda concurrency when it plans, and runs up to 90% of it at once
|
||
(at most 256 chunks), leaving the rest for other functions in the account. The
|
||
plan logs the number it chose. Pass `--concurrency=N` to reserve `N` executions
|
||
for the function instead; every render then runs at most `N` chunks at once.
|
||
|
||
Upgrading: redeploying an existing stack without `--concurrency` keeps the
|
||
reservation it already has, and `deploy` prints it. Stacks deployed by earlier
|
||
CLI versions reserved 8 by default, so they stay at 8 until you pass
|
||
`--concurrency=N`. Pass `--concurrency=none` to remove the reservation so each
|
||
render sizes itself to the account.
|
||
|
||
After `deploy`, render anything with:
|
||
|
||
```bash
|
||
hyperframes lambda render ./my-project --width 1920 --height 1080 --wait
|
||
```
|
||
|
||
The `--wait` flag blocks and streams per-chunk progress + accrued cost; drop it to fire-and-forget, then poll with `hyperframes lambda progress <renderId>` on your own cadence.
|
||
|
||
See the [CLI reference](/packages/cli#hyperframes-lambda) for full flag documentation.
|
||
|
||
#### Pre-staging a project with `sites create`
|
||
|
||
Re-rendering the same project tree on every `lambda render` call re-tars and re-uploads it each time. For tight inner loops (CI smoke jobs, prompt iteration in a demo flow), pre-stage the project once and reuse the upload:
|
||
|
||
```bash
|
||
hyperframes lambda sites create ./my-project
|
||
# → Site ID: a1b2c3d4e5f6g7h8 (content-addressed)
|
||
|
||
hyperframes lambda render ./my-project --site-id=a1b2c3d4e5f6g7h8 \
|
||
--width 1920 --height 1080 --wait
|
||
```
|
||
|
||
The `siteId` is content-addressed via a SHA-256 of the project tree; re-running `sites create` on an unchanged tree skips the upload via a `HeadObject` short-circuit. Pass the same `--site-id` to as many `lambda render` calls as you like — they all reuse the one S3 PUT.
|
||
|
||
### Path 2 — Direct SAM deploy
|
||
|
||
If you want to read the CloudFormation before you deploy, or you need to customise the topology (extra alarms, SNS subscribers, KMS keys, …), invoke SAM directly against the template at `examples/aws-lambda/template.yaml`:
|
||
|
||
```bash
|
||
cd packages/aws-lambda
|
||
bun run build:zip # produces dist/handler.zip
|
||
cd ../../examples/aws-lambda
|
||
sam deploy \
|
||
--stack-name=hyperframes-prod \
|
||
--region=us-east-1 \
|
||
--resolve-s3 \
|
||
--capabilities CAPABILITY_IAM \
|
||
--no-confirm-changeset \
|
||
--parameter-overrides ChromeSource=sparticuz ReservedConcurrency=8
|
||
```
|
||
|
||
The template emits three CloudFormation outputs you'll need to invoke renders:
|
||
|
||
- `RenderBucketName` — S3 bucket for plan tarballs + per-chunk outputs + final renders.
|
||
- `RenderStateMachineArn` — the Step Functions standard workflow that orchestrates Plan → Map → Assemble.
|
||
- `RenderFunctionArn` — the single Lambda function the state machine dispatches against.
|
||
|
||
<Warning>
|
||
The SAM template's default for `ReservedConcurrency` is `-1` (unreserved), the same as the Path 1 CLI: each render runs up to 90% of the account's unreserved concurrency. Set it to bound how many chunks run at once.
|
||
</Warning>
|
||
|
||
### Path 3 — CDK construct
|
||
|
||
For users already running CDK, the `@hyperframes/aws-lambda` package exports a `HyperframesRenderStack` L2 construct that emits the same topology as the SAM template:
|
||
|
||
```ts
|
||
import { App, CfnOutput, Stack } from "aws-cdk-lib";
|
||
import { HyperframesRenderStack } from "@hyperframes/aws-lambda/cdk";
|
||
|
||
const app = new App();
|
||
const stack = new Stack(app, "MyApp");
|
||
const render = new HyperframesRenderStack(stack, "Render", {
|
||
projectName: "hyperframes",
|
||
lambdaMemoryMb: 10240,
|
||
reservedConcurrency: 8,
|
||
chromeSource: "sparticuz",
|
||
});
|
||
|
||
new CfnOutput(stack, "RenderBucketName", { value: render.bucket.bucketName });
|
||
new CfnOutput(stack, "StateMachineArn", { value: render.stateMachine.stateMachineArn });
|
||
```
|
||
|
||
`aws-cdk-lib` and `constructs` are declared as **optional peer dependencies** of `@hyperframes/aws-lambda`, so consumers who only need the SDK don't pay the CDK import cost.
|
||
|
||
The construct exposes `.bucket`, `.renderFunction`, and `.stateMachine` so you can wire dashboards, SNS topics, or other AWS resources alongside it without re-deriving ARNs.
|
||
|
||
## IAM permissions
|
||
|
||
The CLI ships a built-in IAM bootstrap to avoid the "User is not authorized to perform iam:CreateRole" first-deploy trap:
|
||
|
||
```bash
|
||
# Print an inline policy doc to attach to the IAM user that runs the CLI.
|
||
hyperframes lambda policies user
|
||
|
||
# Print { TrustRelationship, InlinePolicy } for a CloudFormation service role.
|
||
hyperframes lambda policies role
|
||
|
||
# Validate a checked-in policy still covers the CLI's needs (exit non-zero on missing).
|
||
hyperframes lambda policies validate ./infra/iam/hyperframes-deploy.json
|
||
```
|
||
|
||
The generated documents grant `Resource: "*"` for the CLI's required action set. After your first successful deploy you can narrow `Resource` to the deployed ARNs — predictable per the CloudFormation outputs above. Adopters running the CLI in CI typically check the policy doc into source control and run `policies validate` as a pre-deploy step to catch drift.
|
||
|
||
## Cost shape
|
||
|
||
Lambda renders are billed by GB-seconds (Lambda billed duration × configured memory) plus a tiny per-state-transition fee for Step Functions standard workflows. `hyperframes lambda progress` exposes the running tally:
|
||
|
||
```bash
|
||
hyperframes lambda progress my-render-id
|
||
# Status: SUCCEEDED
|
||
# Progress: 100%
|
||
# Frames: 480 / 480
|
||
# Lambdas: 5
|
||
# Cost: $0.0214 (Lambda $0.0210 + SFN $0.0004)
|
||
# Output: s3://hyperframes-renders/.../output.mp4
|
||
```
|
||
|
||
The cost number is an estimate: Lambda duration comes from the handler result,
|
||
and S3 transfer is not included. Use AWS billing data for actual spend.
|
||
|
||
## Troubleshooting
|
||
|
||
### `sam deploy` fails with "Stack already exists"
|
||
|
||
Pass the same `--stack-name` you used the first time. SAM is idempotent — re-running on an existing stack resolves to a no-op or an in-place update.
|
||
|
||
### `User is not authorized to perform iam:CreateRole`
|
||
|
||
The IAM credential running `lambda deploy` doesn't have permission to create the service role CloudFormation needs. Run `hyperframes lambda policies user` and attach the printed policy to your IAM user (or take the `policies role` output and have your admin create a deploy role).
|
||
|
||
### `Lambda function failed: PLAN_HASH_MISMATCH`
|
||
|
||
Step Functions invoked a `renderChunk` with a plan hash that didn't match the planDir on S3. Almost always means the producer version differs between the local `plan()` build and the deployed Lambda ZIP. Re-run `hyperframes lambda deploy` (which rebuilds the ZIP) and re-render.
|
||
|
||
### `Lambda function failed: BROWSER_GPU_NOT_SOFTWARE`
|
||
|
||
The handler launched Chromium but the runtime probe found a non-SwiftShader GL backend. Hardware GL is non-deterministic across chunk boundaries, so distributed renders refuse it at the runtime-image / launch-flags layer (not at the composition layer). Rebuild the handler ZIP and redeploy:
|
||
|
||
```bash
|
||
bun run --cwd packages/aws-lambda build:zip
|
||
hyperframes lambda deploy --stack-name=<your-stack>
|
||
```
|
||
|
||
The build pipeline pins `@sparticuz/chromium` + the Chrome flags (`--use-gl=swiftshader --use-angle=swiftshader`) so a fresh deploy almost always resolves this. If it persists, your stack's Lambda function is pointing at a stale handler ZIP from a previous deploy — `lambda deploy` always rebuilds, so re-running unsticks it.
|
||
|
||
### Render seems stuck at `RUNNING`
|
||
|
||
Most often a Lambda cold-start chain on a many-chunk render. The Map state's reserved concurrency caps how many chunks can run in parallel — if you set `--concurrency=4` and your render has 16 chunks, the state machine processes them in batches of 4. `hyperframes lambda progress <id>` shows how many invocations are in flight.
|
||
|
||
If progress doesn't advance for >10 minutes, check the Step Functions execution in the AWS console — failed Lambda invocations include the typed error name (`FONT_FETCH_FAILED`, `FFMPEG_VERSION_MISMATCH`, etc.) which short-circuits the state machine.
|
||
|
||
### Tearing down doesn't reclaim S3 storage
|
||
|
||
The render bucket is created with CloudFormation `Retain` on delete — `hyperframes lambda destroy` (or `sam delete`) tears the function + state machine down but the bucket survives. This is intentional: it protects final-rendered MP4s from being lost when you re-deploy. To fully reclaim storage, empty + delete the bucket via the AWS console / `aws s3 rb`.
|
||
|
||
## Current limits
|
||
|
||
- There is no completion webhook. Poll with `hyperframes lambda progress` or
|
||
watch the Step Functions execution.
|
||
- There is no Lambda-specific composition-discovery verb. Point
|
||
`lambda render` at the project directory containing `index.html`.
|
||
- Each `--region` is an independent stack. Cross-region failover is not built
|
||
in.
|
||
- The distributed AWS path is SDR-only.
|
||
- Each render chunk extracts only the video frames it shows, so the plan step
|
||
no longer grows with video length. On a 10,240 MB function, a 26-minute
|
||
576×1024 24 fps H.264 clip plans in about 20 seconds. Each chunk downloads
|
||
the whole source file of every video it shows. Variable-frame-rate and HDR
|
||
sources are the exception: the plan step still extracts every frame of those,
|
||
inside Lambda's 15-minute limit, in 2-minute segments, so
|
||
`FFMPEG_PROCESS_TIMEOUT_MS` applies to each segment, not to the whole clip.
|
||
- A chunk must also finish inside 15 minutes. For long renders, raise
|
||
`--max-parallel-chunks` (`maxParallelChunks` in the SDK) so each chunk covers
|
||
fewer frames.
|
||
|
||
## Related topics
|
||
|
||
- [Render variable-driven templates on Lambda](/deploy/templates-on-lambda)
|
||
- [Use the AWS Lambda package](/packages/aws-lambda)
|
||
- [Choose another rendering path](/deploy/overview)
|