mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-04 21:17:30 +00:00
6feb4bec7aa046a8956efd8a718db55423e2e2ca
31
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 James Russo
|
7f48a6894c |
feat!: remove Plan v1 distributed transport (#6970)
* feat!: remove Plan v1 distributed transport BREAKING CHANGE: Cloud adapters accept only Plan v2; producer v1 exports are removed. * ci: authorize Plan v1 cleanup deletions * docs: avoid citing generated smoke result file * fix(gcp): reject unsupported plan protocols in workflow * ci: preserve gcp package size reduction * fix: reject stale Plan v1 SDK options * chore: remove retired Plan parity scripts |
||
|
Miguel Ángel
|
51f9fa7c54 |
perf(aws-lambda): small-chunk cloud renders stitch their output in parallel (#6635)
* perf(aws-lambda): small-chunk cloud renders stitch their output in parallel When every chunk's payload is under S3's 5 MiB part minimum (typical for short 4K chunks), the assemble step could not copy any part server-side and pulled the whole output through one Lambda's network link. Those range-only parts now go to parallel invocations of the warm function that just rendered the chunks, about 32 MiB each and never more at once than the stack's reserved concurrency allows. The output bytes are unchanged. * fix(aws-lambda): a throttled stitch helper builds its parts in the assemble instead Another render can hold the stack's reserved concurrency while this one assembles, so a helper invocation may be refused. The assemble now builds that batch itself, logs it, and counts it in assemble_stages; the helper's error type is kept as the error name. Comments trimmed back to each file's share at the base. * chore(aws-lambda): raise the package size budget to the measured size plus headroom The parallel stitch helpers add about 1.7 KB packed and 9.9 KB unpacked; the new budget is the measured size times the checker's own 1.05 headroom. |
||
|
Miguel Ángel
|
7aac5c763b |
perf(aws-lambda): each chunk logs where its seconds went (#6391)
* perf(aws-lambda): each chunk logs where its seconds went The chunk_capture event now carries the handler's setup time and the renderer's own stage timings (plan check, browser boot, capture, encode, total), which the renderer already measured but the logs dropped. Lets a fleet run be split into start-up, capture and tail per chunk. * perf(aws-lambda): chunk logs also time the Chrome path resolution On the container image the compressed Chrome may be unpacked on every cold chunk; chromeMs separates that from the plan download in setupMs. |
||
|
Miguel Ángel
|
6a977e8db1 |
fix(producer): chunk capture round-robins only for callers that bill wall time (#6405)
* fix(producer): chunk capture round-robins only for callers that bill wall time
* fix(engine): worker, extraction and encoder sizing read the CPU quota, not the host core count
* Revert "fix(engine): worker, extraction and encoder sizing read the CPU quota, not the host core count"
This reverts commit
|
||
|
Miguel Ángel
|
fb2f7c1a80 |
perf(aws-lambda): warm the audio function while the plan runs (#6148)
* perf(aws-lambda): warm the audio function while the plan runs The audio task runs on the x86 render function, which is cold when the plan ran on the arm64 video function. A fire-and-forget invoke at the start of a render now wakes it, so the audio no longer outlasts the chunks. * perf(aws-lambda): warm the render function only on a stack with a video function An x86-only stack has no cold function to warm, so it takes the old path with no extra step. The CDK stack has no video function and is unchanged. * perf(aws-lambda): wrap a template comment * style(aws-lambda): format the warm-up template test |
||
|
Miguel Ángel
|
f5f884e33d |
fix(aws-lambda): make webm and mov audio in the separate audio task like mp4 (#6098)
* fix(aws-lambda): make webm and mov audio in the separate audio task like mp4 A WebM or MOV plan mixed and encoded its own audio, so on a stack whose plan runs on an arm64 function the audio came out of a different ffmpeg build than an x86 render's. The plan now leaves that audio to the separate audio task, as mp4 already does, and the assembler joins the chunks with that file. Only mp4 chunks stitch; cfr and png-sequence keep their audio in the plan. * test(aws-lambda): join mov chunks with the audio task's track too * refactor(producer): keep the separate-audio gate's one-line doc comment |
||
|
Miguel Ángel
|
cde21630e4 | perf(aws-lambda): render each chunk with as many capture browsers as the function can host (#6026) | ||
|
Miguel Ángel
|
dee8380d22 |
feat(aws-lambda): a cloud render mixes its audio beside its chunks (#5997)
* feat(aws-lambda): a cloud render mixes its audio beside its chunks * fix(aws-lambda): the audio task keeps a worker free and runs only when the plan left audio out * test(aws-lambda): the audio task's free worker holds for a reserved pool of 1 and 2 * refactor(aws-lambda): one helper points every handler at the bundled Chrome |
||
|
Miguel Ángel
|
992df8b203 |
perf(cloud): move project bytes once through shared content storage (#5772)
* perf(producer): distributed chunks download only the videos they show A project video under compiled/ that chunks extract their own frames from now goes only to the chunks that show it, like video-sources/ files. A chunk skips the readiness wait for a deferred video it does not show, so it no longer logs media_load_failed for a file it was never given. * fix(producer): cloud site uploads pack the files their id hashes, symlinks included * perf(aws-lambda): chunks read only the byte ranges of a video they need A chunk that extracts its own frames from a deferred project video now reads it from S3 by byte range through a short-lived pre-signed URL, instead of downloading and hashing the whole object first. S3 attests the object's stored SHA-256 before the URL is signed; every other artifact is still downloaded and verified as before. * perf(aws-lambda): upload project files once to a shared content-addressed area * chore(producer): keep planV2 comment share at the base level * fix(engine): verify the TLS certificate of https video inputs and never surface their signed query * fix(aws-lambda): renders re-upload a site's expired files instead of failing * perf(aws-lambda): the plan step probes project videos in place instead of downloading them * fix(engine): https video reads reconnect and time out, and redaction covers wrapped causes Signed-URL reads now pass -rw_timeout, -reconnect and -reconnect_delay_max (both Lambda builds accept them) and ffmpeg also -reconnect_on_network_error, so a transient S3 error or a stalled read fails fast or recovers instead of waiting out the process timeout. Query redaction now descends into error.cause. * fix(aws-lambda): share entry bundles and ratchet package budget * fix(aws-lambda): site ids uploaded before files.json still render A site id from an earlier sites create has only sites/<id>/project.tar.gz. renderToLambda now falls back to that tarball, which the plan step still reads, and when neither key exists it fails before starting with both keys named and the fix (run sites create again). * fix(aws-lambda): renders fail before upload on a stack too old to read sites A stack deployed before files.json sites untars the site file in its plan step and fails after four retries. renderToLambda now reads the state machine definition first and stops with STACK_TOO_OLD and the redeploy command when the plan task does not list SITE_MANIFEST_INVALID, which new SAM and CDK stacks now treat as terminal. The README's upgrade section drops the planProtocol v1 fallback, which hits the same failure. * docs(aws-lambda): document shared content storage and the upgrade path The deploy guide still described the tarball HeadObject short-circuit. It now covers the cas/sha256/ store and its 30-day lifecycle rule, the STACK_TOO_OLD, SITE_FILES_MISSING and SITE_NOT_FOUND errors, that old site ids keep working, and that every project uploads once more after the upgrade. A test keeps a troubleshooting entry for each pre-submit error. * fix(producer): chunks log each video they read by URL, without its query A chunk reading a video straight from the bucket left no trace in its log. It now logs one line per remote source with the path and the URL minus its query string, so signatures never reach the log. * fix(engine): ffmpeg errors never keep part of a signed URL's query A process's stderr tail was trimmed before its URLs were redacted, so a cut inside a signed URL left a query fragment the URL pattern no longer matched, and the fragment reached the error. The stderr tail is now redacted before every trim, and the redactor also strips a signing parameter cut off from its URL. * fix(aws-lambda): comments name the site manifest instead of a file path * fix(engine): stderr redaction survives chunk splits, and every ffmpeg spawn uses it The stderr tail was redacted on each chunk and then cut, so once a cut severed a long URL's scheme, the rest of its signature in the next chunk matched neither pattern. The tail is now kept raw with 2x slack and redacted only when the text is produced, then cut to size. The unused producer audioExtractor (its own spawn, unredacted stderr) is deleted and the parity harness's PCM decode goes through ManagedChildProcess; a test fails on any ffmpeg/ffprobe spawn in engine, producer or aws-lambda that does not. * chore(producer): record the audioExtractor deletion as intended * fix(engine): drop a signature fragment the stderr tail cut from its URL The raw tail is kept at twice the cap, but that cut can itself land inside a signature, and when redaction shrinks the rest below the cap the fragment survived. After a cut, the severed token is dropped before redaction. * refactor(engine): name the severed-token drop instead of commenting it * test(engine): a cut-off URL leaves nothing behind; aws-lambda size budget to measured * fix(producer): chunk URL-read tests use main's frame list; producer size budget to measured * chore(producer): size budget at the measured size, no headroom * chore(producer): size budget back to measured x1.05, the check's own rule |
||
|
Miguel Ángel
|
2a14cbc924 |
perf(aws-lambda): size distributed renders from account concurrency limits (#5813)
* perf(aws-lambda): run chunks through a Distributed Map to lift the 40-chunk cap The inline Map runs at most 40 chunk renders at once. Both state machine definitions (CDK and SAM) now use a Distributed Map with STANDARD child executions, capped at the function's reserved concurrency so chunks past it queue instead of being throttled. Progress reads each chunk's child execution history so frames, cost and chunk errors still show up. * fix(aws-lambda): retry Lambda throttles for minutes and stop on oversized chunks A throttled task used to burn its four catch-all retries in about 30 s. It now gets its own jittered backoff of several minutes, because another render on the same function can still hold the reserved slots. Plan treats CHUNK_EXCEEDS_WORKER_TIMEOUT as terminal in both definitions. * chore(aws-lambda): trim stack comments to keep the comment share flat * fix(aws-lambda): keep render --wait working at hundreds of chunks Progress polls read the chunk Map Run's counts (one DescribeMapRun) instead of every child execution's history, take exact chunk frames and cost from the Map output once it exits, and read failed children only when the counts show a failure. The SDK client retries throttled calls up to 8 times with backoff. * feat(aws-lambda): size each render's concurrency to the account instead of a fixed 8 * fix(lambda): keep the concurrency SDK external in published modules * fix(cli): keep an existing stack's reserved concurrency on redeploy Redeploying without --concurrency sent ReservedConcurrency=-1, which removed the cap of 8 that earlier CLI versions reserved by default and stopped recording it in the state file. The override is now left out when the flag is omitted, so SAM keeps the stack's current value (a new stack still gets the template's unreserved default). The state file records the deployed value, and deploy prints the kept reservation. * fix(aws-lambda): keep progress working for callers without DescribeMapRun Callers whose IAM policy came from an earlier `lambda policies user` lack states:DescribeMapRun, so reading a running Distributed Map threw AccessDeniedException out of `lambda progress` and `render --wait` while the render itself went on to succeed. getRenderProgress now catches only that error around the map-run read, returns the parent snapshot (status, plan and assemble progress) and adds a warning to re-apply the policy, which both commands print. Other errors still throw. * fix(cli): let the documented deploy policy drop a reserved concurrency When a redeploy removes a function's reserved concurrency, CloudFormation calls lambda:DeleteFunctionConcurrency. The policy printed by `lambda policies role|user` granted only PutFunctionConcurrency, so the update failed and rolled back. Grant it, and pin it with states:DescribeMapRun in the required-actions test. * fix(aws-lambda): read smoke chunk hashes from the Distributed Map output With chunks in a Distributed Map, their task results live in child executions, so the smoke gate's chunk-hash check found 0 chunks in the parent history and failed every v1/v2 comparison. Read Chunks[] from the RenderChunks / RenderChunksV2 MapStateExited output instead, as getRenderProgress does. The stack this script deploys only has the Distributed Map, so the inline-Map path is gone. * fix(producer): cap a pinned chunk size at the worker concurrency, not 16 With --chunk-size or --target-chunk-frames and no --max-parallel-chunks, the plan still capped the render at 16 chunks however many workers the account allowed, so --chunk-size 30 on a 60 s render gave 16 chunks of 113 frames. The cap is now one wave across the worker concurrency, as the --max-parallel-chunks docs say; 16 remains when the concurrency is unknown. * feat(cli): remove a stack's reserved concurrency with --concurrency=none Since an omitted --concurrency now keeps the stack's reservation, there was no CLI way back to the unreserved default. --concurrency=none sends ReservedConcurrency=-1, the state file records the stack as unreserved and deploy says the reservation was removed. Positive N and an omitted flag behave as before; any other value is rejected instead of being silently ignored. * chore(aws-lambda): raise the package size budget for account-sized concurrency Measured with pnpm@10.17.1 pack after rebasing on main with the S3 assembly change: 67,763 packed / 244,685 unpacked; the budget is the check's own 5% headroom over that. |
||
|
Miguel Ángel
|
70fbe32f8d |
perf(cloud): assemble chunk output with a single faststart index (#5812)
* perf(producer): assemble chunks that carry audio under one faststart index Reads each chunk's moov, merges the sample tables into one moov at the front, and writes the chunks' mdat payloads after it unchanged: the same bytes storage can copy into place. The assembler no longer needs the plan audio for such plans. * perf(aws-lambda): build the final mp4 inside S3 with its index up front The Lambda plan asks chunks to carry their audio. Assemble writes the faststart header as part 1 of a multipart upload and copies each chunk's mdat payload with UploadPartCopy, so the video bytes never pass through the function; any failure aborts the upload. * fix(engine): a chunk's first frame shows its own time, not the composition's opening frame * fix(producer): stitch silent AAC tails and clean failed outputs * fix(producer): open stitched chunks read-only with an explicit mode * chore(aws-lambda): raise the package size budget for S3-side MP4 assembly handler.js and index.js each bundle the new S3 stitch code (+5,550 bytes apiece) and s3Stitch.d.ts adds 970: +3,479 packed and +12,070 unpacked over main, measured with the CI pack. The budget rises by exactly that delta. * fix(aws-lambda): v1 plans give chunks their audio too, so both protocols match * perf(producer): plans whose chunks carry audio encode it once, at the frames' exact length * test(producer): terminate the plan audio ffprobe argv like every other probe |
||
|
Miguel Ángel
|
fb17f41196 |
perf(producer): overlap the plan step's audio work and hash files once (#5777)
* perf(producer): overlap the plan step's audio work and hash files once * refactor(producer): simplify the plan overlap and publish path |
||
|
Miguel Ángel
|
a498cf2032 |
perf(producer): assemble in one pass and keep its disk use under one output copy (#5768)
* perf(producer): assemble in one pass and keep its disk use under one output copy The distributed assembler joined the chunks into a full-size file, muxed the audio into a second one and rewrote that again for faststart, so local disk peaked near four copies of the video and a 30-minute 1080p render came close to Lambda's 10 GB /tmp. The join, audio mux and faststart now run as one ffmpeg pass over the concat demuxer; the audio pad/trim takes its frame count from the chunks. Video and audio packets are unchanged. Assemble and the Lambda assemble handler log per-stage timings. * refactor(producer): split assemble into plan check, cfr re-encode and final write Keeps assemble under the complexity gate and drops an orphaned doc comment in chunkEncoder. No behaviour change. |
||
|
Miguel Ángel
|
c69fd2c6ec |
perf(producer): size distributed chunks to the worker limit and concurrency (#5765)
* perf(producer): size distributed chunks to the worker limit and concurrency * chore(cli): drop a comment the field name already says |
||
|
James Russo
|
17a2a00ed5 |
feat(cloud): default distributed plans to v2 (#3311)
* feat(cloud): default distributed plans to v2 * fix(cloud): address plan v2 review feedback * fix(examples): document explicit v2 samples |
||
|
Miguel Ángel
|
dc4383113c |
fix(producer): mix audio into a container that can record encoder delay (#3200)
* fix(producer): mix audio into a container that can record encoder delay Every rendered composition's audio landed 1024 samples (21.33 ms at 48 kHz) after its authored `data-start`, against a frame-accurate video track. The mix is AAC-encoded, and AAC encoders emit ~1024 priming samples. The mix was written to a raw ADTS `.aac` file, which has nowhere to record that delay, so it decoded as real leading silence and every stage downstream preserved it faithfully. Measuring each intermediate localises it precisely: the source WAV is exact, the mixer's own output is already 21.33 ms late, and the pad/trim and mux stages inherit it unchanged. The filter graph itself is correct - run by hand to PCM it lands on the authored start. Switch the artifact to an MP4-family container, which stores the delay as an edit list that decoders strip. Same codec, same bitrate, so no size or quality change. The filename is a contract shared by three consumers - the mux input, the distributed plan artifact, and the PNG-sequence sidecar handed to users for NLE ingest - and its extension is what selects the muxer. Give it one owner in the engine rather than five literals, so those consumers cannot drift onto different containers. Note for reviewers: this renames the distributed plan's audio artifact, which is an on-disk contract between the plan writer and the assembler. Both move together here, but a plan written by an older build would not be found by a newer assembler. Flagging in case that mixed-version window matters for how these are deployed. * fix(cloud): read the plan audio artifact name from the producer contract The aws-lambda and gcp-cloud-run adapters each restated the plan's audio filename in five places, so renaming it in the producer left them looking for a file that is no longer written. CI caught it: the gcp dispatch test asserting a plan has no audio artifact started seeing one. Export the name from `@hyperframes/producer/distributed` and consume it in both adapters. This is the same failure the constant exists to prevent, one package boundary further out: a literal that drifts from the writer's is a silently missing audio track rather than a loud error, because both call sites only ever ask whether the file exists. * fix(cloud): accept a legacy plan's audio artifact name for one release Review raised a rolling-deploy window I had flagged but left undecided: `plan` and `assemble` are separate invocations bridged by object storage, so a pre-rollout planner can be paired with a post-rollout assembler. Both readers locate the artifact by existence alone, which makes that pairing a silently muted video rather than an error. That is reachable enough to be worth two lines, so reads now accept the old name while writes only ever emit the new one. Give the fallback one owner (`resolvePlanAudioPath` / `isPlanAudioArtifactPath`) rather than four call sites, marked for deletion one release out. Also fixes a hole in the first pass of this: the plan-v2 materializer matched either name but then joined the CURRENT one, so a legacy plan resolved to a path that was never written. It now joins the artifact's own name. Review nits in the same pass: correct the pad-branch docstring, which still described a concat-copy shape the pad branch stopped using when it moved to apad + re-encode, and fix the Windows fixture's stale `.aac` output extension so it cannot model a shape that reintroduces the priming delay. * test(producer): rebake the missing-host-comp-id golden without the audio delay The pinned reference was rendered before this branch, so it carries the 1024 sample encoder-priming delay in its audio. With the delay gone the correct audio now sits ahead of the reference and the harness's envelope correlation drops below its floor. Cross-correlating the old and new references at native 48 kHz gives a lag of exactly 1024 samples (21.33 ms) at a correlation of 0.99985: same audio, moved by exactly the amount this branch removes. Regenerated inside the CI container (Dockerfile.test, ffmpeg 5.1.9) rather than natively, so the reference matches the encoder CI will compare against - the container reproduced CI's failure to the digit (correlation 0.3938764027803616, lagWindows -12) before the rebake and passes at correlation 1.0 after it. Note for archaeology: the new reference is also 3 dB louder than the old one. That gap is not from this branch - `main` and this branch render the fixture at the same level - it is pre-existing drift the reference had accumulated, which a scale-invariant correlator could never see. The rebake absorbs it. Only output.mp4 is updated. `--update` also rewrites compiled.html, but that diff is embedded-font churn with no bearing on the comparison, which reports "Failed at compilation: 0" either way. * test(producer): rebake the variables-prod golden without the audio delay Same cause as the missing-host-comp-id rebake, caught by shard-8 once the earlier shard stopped failing and the rest of the matrix could run: this reference also carries the encoder-priming delay this branch removes. Reproduced in the CI container to the digit (correlation 0.42704173048439215, lagWindows -12), rebaked there, and it now passes at correlation 1.0. Worth recording: the shift here is 2048 samples (42.67 ms) at correlation 0.99983, exactly twice the 1024 of the other fixture. The delay compounds once per un-compensated AAC generation, and this fixture's audio needs its duration normalized, so it takes the pad/trim branch's re-encode and picks up a second frame of priming on top of the mixer's. So the pre-fix error was not a fixed 21 ms - it grew with the number of times the audio was re-encoded. All nine shards ran in that CI round with only this one failing, so the matrix has now covered every fixture against this change. |
||
|
Vance IngallsandClaude Opus 5
|
349c066a83 |
fix(producer): classify JSON error bodies as non-media sources too
A source that answers with a JSON error body still reached ffprobe and
produced `moov atom not found`. Replicate returns
`{"detail": "requested file not found"}` for a dead asset, and a gateway
in front of it can relay that body with a success status.
The sniff now treats `<`, `{`, or `[` as the opening byte of a text
document. No supported container starts with any of them, so this is the
same trade as before: three bytes instead of an allowlist that grows one
entry per payload shape observed in production.
Renamed accordingly, since the class now covers JSON as well as markup:
MARKUP_NOT_MEDIA -> NOT_MEDIA_PAYLOAD, MarkupNotMediaError ->
NotMediaPayloadError, markupPayload.ts -> notMediaPayload.ts. Registry
entries in the Lambda name map, the CDK and SAM plan lists, the Cloud Run
set, and SAFE_RENDER_ERROR_CODES move with it.
Also documents the reachability boundary on the error class: only a 2xx
response gets here. `downloadToTemp` rejects 404/410 as `http_not_found`
before writing a byte, and every ffprobe input is local because
videoFrameExtractor downloads http srcs first. So the shapes this
classifies are soft-404 and interstitial HTML, S3/CloudFront error
documents, and JSON API error bodies -- each served with a success
status. A genuine 404 surfaces as a download failure, not as this error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Vance IngallsandClaude Opus 5
|
f3689c1481 |
fix(producer): scope and harden the markup-payload sniff
Review follow-up on the STUDIO-5433 defense.
Correctness
- The sniff ran above the documented video/audio failure split, so an
<audio> src that resolved to an HTML payload aborted the whole render
instead of degrading to duration 0. It now runs inside the same try, so
video surfaces the typed error while audio still drops out, with a
warning naming the element.
- Raw fs errors (EISDIR on a directory src, EACCES, the existsSync->open
ENOENT race, EMFILE) escaped and failed the compile with an unclassified
error carrying an unredacted temp path. The sniff is now a classifier that
never throws: an unreadable file reports "not markup" and the real probe
produces the real error.
- Elements whose duration the compiler never resolves (a data-end video, a
looping audio) skipped the sniff entirely, so the original ffprobe error
still escaped, and looping audio was reported as owner "system" after
every frame had been captured. Video is now caught in the asset preflight,
which sees every local src regardless of authored timing; audio is
classified per-element in audioMixer as source/invalid_media/owner "user",
keeping audio failures non-fatal as they already were.
- Detection is a byte-level check for a leading "<" (BOM-, whitespace- and
NUL-tolerant, looped read) instead of a <!doctype|<html|<?xml string
prefix, which missed a NUL-prefixed payload, >256B of leading whitespace,
UTF-16-encoded HTML, and a prolog-less <svg. No supported container starts
with "<", so the allowlist no longer grows per payload shape.
- finally { await fh.close() } could replace the in-flight typed error with
the close error.
Routing and privacy
- MARKUP_NOT_MEDIA is now in SAFE_RENDER_ERROR_CODES, the Lambda terminal
name map, the CDK and SAM non-retryable plan lists, and the Cloud Run
non-retryable set, and the class carries owner/retryable. Previously the
API emitted errorCode: undefined and a deterministic authoring bug burned
the full distributed retry budget.
- The message no longer carries 32 raw payload bytes or the src.
redactTelemetryString preserves host and path for HTTP srcs, so
per-tenant CDN paths reached a message the server forwards to clients.
Correlation is a sha256 element fingerprint, matching
AssetMediaTypeMismatchError.
- The message names both causes (unresolved nested-composition URL, or an
HTML/XML error page served as 200) rather than misdiagnosing an S3 403
body as an authoring bug.
Tests
- Byte-level detection is unit-tested in engine: markup shapes, BOMs,
UTF-16, nine container signatures, unreadable inputs.
- Replaced the tautological assertions. The old checks for "html" in and
"moov" absent from a fixed message template could not fail for any input.
- New coverage for audio degradation, the audioMixer classification, the
preflight video/image/audio split, and the API error metadata.
- The sibling htmlCompiler.mediaType failure was a vitest-under-bun runner
mismatch, not a missing ffmpeg binary. It passes, including the 4-wide
probe-semaphore invariant the sniff now runs inside.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
James Russo
|
20f4f8f49c |
fix(producer): retry transient deterministic font fetches (#2865)
* fix(producer): retry transient deterministic font fetches * fix(producer): secure Lambda font cache directory |
||
|
James Russo
|
557d82b6a9 |
fix(producer): validate distributed video metadata (#2839)
## What - enforce a finite, validated `meta/videos.json` contract shared by Plan v1 and Plan v2 - preserve authored finite ends and source-derived trim-aware ends; bound any still-open end at the validated composition end - fail distributed planning when any declared video source did not extract instead of publishing a blank-capable plan - make the v1 chunk reader reject malformed/null video timing before frame injection - route deterministic video-source/metadata failures as non-retryable in AWS and GCP while retaining retries for transient extraction failures ## Why An open-ended video whose remote source could not be resolved retained `Infinity` through planning. Plan v2 correctly rejected that value, while Plan v1 serialized it as `null`; the v1 frame lookup could then suppress injected frames and silently produce incorrect output. The invariant belongs at the shared metadata boundary. Both protocols must receive identical finite timing, and unavailable sources must fail closed before plan publication. ## Test plan - [x] producer distributed planning, metadata, v1 chunk boundary, Plan v2 conversion/materialization, and public exports - [x] core runtime media semantics (authored slots, natural duration, looping, non-looping hold) - [x] engine video extraction and frame lookup - [x] AWS Lambda/CDK/SAM and GCP Cloud Run error normalization/retry classification - [x] producer, core, engine, AWS, and GCP typechecks/builds - [x] formatting, oxlint, tracked-artifact, fallow, and commit hooks - [x] exact incident composition replayed through the AWS Lambda handler's Lambda-local path in a Lambda-like container; Plan v1 and Plan v2 both fail closed as `VIDEO_SOURCE_UNRENDERABLE` during planning, before plan publication - [x] full PR CI, including all nine regression shards and Windows render/tests No production flags or deployment/release workflows are changed. |
||
|
James
|
96cafb47c6 | fix(producer): fallback distributed capture safely | ||
|
James
|
2a284a8e3a | fix(gcp): enforce effective BeginFrame capture | ||
|
James
|
09998789b5 | feat(aws-lambda): publish plan v2 directly to S3 | ||
|
James Russo
|
5bf61d6df0 |
feat(aws-lambda): support plan protocol v2 (#2789)
* feat(aws-lambda): support plan protocol v2 * fix(aws-lambda): align SAM v2 terminal errors |
||
|
Vance Ingalls
|
65888840fa |
fix(aws-lambda): validate event S3 URIs against render bucket (F-004) (#1213)
## Summary - Adds `validateEventS3Uris()`, called immediately after `unwrapEvent()` in the Lambda handler before any S3 I/O. - If `HYPERFRAMES_RENDER_BUCKET` env var is set, every S3 URI in the event (`ProjectS3Uri`, `PlanOutputS3Prefix`, `PlanS3Uri`, `ChunkOutputS3Prefix`, `ChunkS3Uris`, `AudioS3Uri`, `OutputS3Uri`) must resolve to that bucket. Mismatches throw `S3_URI_NOT_ALLOWED`. - Env var unset → validation skips (backwards-compatible; existing deployments without the var continue to work). - CDK stack (`HyperframesRenderStack`) auto-wires `HYPERFRAMES_RENDER_BUCKET: this.bucket.bucketName` so new deployments are protected without manual config. - `S3_URI_NOT_ALLOWED` added to all three `NON_RETRYABLE_*` lists in the Step Functions state machine so the state machine does not retry on this error. ## Security **F-004 MED** — The Lambda handler accepted S3 URIs from the event payload without verifying they targeted the function's own render bucket. An attacker who could inject a crafted Step Functions execution input could route `GetObject` / `PutObject` calls to arbitrary buckets in the same AWS account, potentially exfiltrating plan data or overwriting objects in unrelated buckets. ## Test plan - [x] `handler` rejects a `plan` event whose `ProjectS3Uri` targets a different bucket — `S3_URI_NOT_ALLOWED` thrown, zero S3 ops recorded - [x] `handler` rejects an `assemble` event with one cross-bucket chunk URI - [x] Validation is skipped when `HYPERFRAMES_RENDER_BUCKET` is unset (no regression for existing callers) - [x] All 12 handler unit tests pass |
||
|
Kiyeon Jeon
|
bc20b87b9a |
fix(aws-lambda): resolve Chromium for handlePlan before invoking probe
The producer's probe stage launches Chromium when Plan has to resolve browser-only data: root duration unknown, unresolved sub-compositions, or data-hf-auto-start media. Only handleRenderChunk was setting PRODUCER_HEADLESS_SHELL_PATH, so a cold Plan invocation launched puppeteer-core with no executablePath and failed before writing plan.tar.gz. Mirror the renderChunk env-var guard inside handlePlan so the bundled Sparticuz binary gets resolved on the first Plan invocation and reused on warm starts. The skipChromeResolution dep stays honored for SAM-local RIE smokes. A warm Lambda environment can mask this only if it previously served a renderChunk from another execution and left the env var sticky. Within a single Step Functions execution, Plan still runs before RenderChunks. A new dispatch test exercises the guard path by pre-seeding PRODUCER_HEADLESS_SHELL_PATH and asserting Plan does not overwrite it. |
||
|
James
|
71d1889da6 |
feat(distributed): add optional cfr flag for exact constant frame rate
Distributed-render output today uses -c:v copy through concat → mux → faststart, which means PTS timestamps from each chunk pass through unchanged. Container r_frame_rate is exact (#1040 + this PR's parent), but stream-level avg_frame_rate stays PTS-derived and can land on fractional rationals like 27648000/921677 over a 60s render. Same for sub-ms duration drift. This is the achievable bar within -c copy stream-copy concat. For most consumers (browser playback, YouTube, etc.) the difference is invisible. For downstream tools that strict-check avg_frame_rate or ms-precision duration (broadcast workflows, frame-accurate compositors, some third-party transcoders), it matters. Adds an opt-in cfr config flag (default false). When true, the assemble step's final pass re-encodes with -fps_mode cfr -r <fps> instead of -c copy, producing exact CFR output. Trade-off: ~2-5x the stitch time for a 60s 1080p clip; second-generation H.264 quality loss is negligible at -crf 18 but is non-zero. |
||
|
James RussoandClaude Opus 4.7
|
5d264e146c |
docs(lambda): document webm support + simplify-review fixes (#953)
* docs(lambda): document webm support in distributed mode PR 8.4 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). User-facing docs catch up with the shipped capability. Updates docs/deploy/migrating-to-hyperframes-lambda.mdx: - "Output format" row in the migration table now lists `webm` alongside mp4 / mov / png-sequence with a note that webm uses libvpx-vp9 + closed-GOP concat-copy. HDR mp4 remains the only refused format. - "No webm distributed" caveat replaced with "webm uses closed-GOP VP9" explainer covering the encoder args (`-g <chunkSize>`, `-keyint_min <chunkSize>`, `-auto-alt-ref 0`, `-cpu-used 2`), why alt-ref disable is load-bearing, and that the output preserves alpha via yuva420p with Opus audio. - Migration checklist no longer asks adopters to filter out webm compositions; only HDR-dependent renders need to stay on the previous framework. aws-lambda.mdx doesn't currently call out webm as unsupported (only HDR in the v1 surface list), so it gets no copy edits beyond the migration guide. The internal planning doc (DISTRIBUTED-RENDERING-PLAN.md §7.2, §8, §12 — kept outside the repo) gets matching updates: format support matrix flipped ✓, v1.5 backlog #1 marked shipped, HDR promoted to the new top item, and the rev-12 → rev-13 status line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: address simplify-review findings on webm stack Folds in cleanups identified by a multi-agent code-review pass over the 4-PR webm-distributed stack: - plan.ts: `resolveEncoderTriple()` webm case now calls `getEncoderPreset(quality, "webm")` for its preset string instead of hardcoding "good". The hardcode was wrong for `quality: "draft"` (`getEncoderPreset` returns "realtime" for that tier) — would have silently overridden the draft → realtime mapping for distributed webm renders. - chunkEncoder.ts: trim the new VP9 closed-GOP comment block from ~18 lines of WHY narration down to the 6 lines that actually explain why (alt-ref + cpu-used drift). Match the alpha branch's idempotent-push comment to the same standard. - chunkEncoder.test.ts: drop the duplicate WHY comment that restated the implementation comment in plain words. - webm-concat-copy.test.ts: rewrite the file-header docstring to describe the contract being tested instead of the PR-8.1-gating history; strip "PR 8.2 / Path A / Path B" references from error messages (they belong in PR bodies, not in test output). Consolidate the yuva420p alpha smoke into a single `it()` block (was a full 4-test describe with duplicated setup) — the yuv420p block already covers the probe/decode/frame-count contract; the alpha smoke only needs to prove the alpha args don't break concat-copy. - plan.test.ts: drop the "PR 8.1 proved the contract" comment. - webm-vp9 fixture: drop the aspirational "Other webm-with-audio fixtures cover the mux path separately when added" sentence (no other fixtures exist). Regenerated the baseline via `docker:test:update webm-vp9` to reflect the updated comment. - migrating-to-hyperframes-lambda.mdx: add a paragraph about distributed webm's perf cost — ~10-25% larger files at constant CRF due to forced keyframes, and slower per-chunk encode due to `-cpu-used 2` being more conservative than the libvpx default. All unit tests + the webm-vp9 distributed-simulated regression still pass after these changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): accept --format=webm in `hyperframes lambda render` The CLI's `lambda render` subcommand's FORMATS allowlist and the `RenderArgs.format` type still narrowed to `mp4 | mov | png-sequence`, so even though the producer + aws-lambda packages now support webm end-to-end, the CLI surface rejected it with `--format must be mp4|mov| png-sequence`. Add webm to both spots and update the --help description. Surfaced during real-AWS deploy prep — the local lambda-local / distributed-simulated tests didn't go through the CLI so the gap went unnoticed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(producer): font cache writes to /tmp on Lambda (read-only \$HOME) The deterministic Google Fonts cache was rooted at `\$HOME/.cache/hyperframes/fonts`, which fails on AWS Lambda — the runtime's `\$HOME` resolves to a `/home/sbx_*` directory tree that's read-only. `mkdirSync(..., { recursive: true })` can't create that path and the plan stage trips with `ENOENT: no such file or directory, mkdir '/home/sbx_user1051/.cache/hyperframes/fonts/space-mono'` on every Lambda render that pulls a Google Font (i.e. every distributed fixture using `@import url("https://fonts.googleapis.com/...")`). Detect Lambda via `\$AWS_LAMBDA_FUNCTION_NAME` and route the cache to `tmpdir()/hyperframes/fonts` in that case. Lambda's `/tmp` survives across invocations on a warm container, so cache hit rate is the same as non-Lambda runs. Also honor an explicit `\$HYPERFRAMES_FONT_CACHE_DIR` override for adopters who want a different location regardless of the runtime. Surfaced while verifying webm distributed end-to-end on real AWS — the same bug affects mp4 fixtures using Google Fonts; webm just happened to be the one I tried first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: extract DistributedFormat type + trim font-cache resolver Second simplify-review pass on the webm stack flagged two cleanups: 1. **`DistributedFormat` type duplicated 10 times.** Every file in the distributed pipeline carried its own copy of `"mp4" | "mov" | "png-sequence" | "webm"` — adding a new format meant a 10-place edit with no compile-time guarantee they stayed in sync. Extract a single source of truth in `packages/producer/src/services/distributed/shared.ts`, re-export from `@hyperframes/producer/distributed` and `@hyperframes/aws-lambda/sdk`, and have all callers pull from there. The aws-lambda `ALLOWED_FORMATS` runtime tuple and the CLI's `FORMATS` tuple now both use `satisfies readonly DistributedFormat[]` so the compiler enforces the runtime allowlist stays in sync with the type. 2. **`deterministicFonts.ts` font-cache resolver was over-commented.** Trim the 7-line block to 4 lines (drop the aspirational "and other read-only-FS execution environments" — only Lambda is detected — and the warm-container `/tmp` persistence narration — anyone reading already knows Lambda /tmp semantics). Collapse the two-step `if (explicit && explicit.length > 0)` into a single nullish-coalesce expression now that the empty-string defensive check is gone (`process.env.X` is `string | undefined`, no third shape to guard against). Out-of-scope skips (called out by the agents, deferred): - In-process `RenderConfig.format` and the in-process CLI's `render.ts` format union still carry their own inline copies. The union happens to coincide today but they're separate concerns — leaving them alone limits this PR's blast radius. - `fontCacheDir(slug)` / `resolveFontCacheRoot()` naming asymmetry flagged as taste; skipping. - Pre-existing redundant `existsSync` before `mkdirSync({ recursive: true })` in `fontCacheDir` — out of scope. All tests + typecheck still pass. Lambda render still works end-to-end (no functional changes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(lambda): drop plan-doc reference from migration checklist PR review feedback: source/docs should not mention the distributed-rendering planning doc. Tighten the migration checklist sentence to describe the webm path directly rather than referencing the doc's version label. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(producer): split resolveEncoderTriple into mp4 + non-mp4 helpers CI Fallow audit on PR #953 flagged `resolveEncoderTriple` at CRAP 31.6 — the function interleaved (a) mp4 codec validation + dispatch, (b) the non-mp4 codec-rejection throw, and (c) per-format dispatch. Splitting into `resolveMp4EncoderTriple` + `resolveNonMp4EncoderTriple` drops the top-level function's cyclomatic complexity below the threshold while preserving every error message and code path. Behavior unchanged. Also extracts an `EncoderTriple` type alias so the three functions share the return shape declaratively rather than repeating it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
James RussoandClaude Opus 4.7
|
21f5066832 |
feat(producer): enable webm in distributed mode via concat-copy (#951)
* feat(producer): enable webm in distributed mode via concat-copy PR 8.2 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). Wires libvpx-vp9 webm through the distributed pipeline now that PR 8.1 proved concat-copy works. Architectural decision: Path A (concat-copy) — based on PR 8.1's smoke test result (9/9 tests pass for both yuv420p and yuva420p VP9 streams). The simpler architecture wins; no re-encode in assemble, no encode- parallelism loss. Changes: - plan.ts: - DistributedRenderConfig.format and PlanResult.format now include "webm" — type-level acceptance matches the runtime gate. - rejectUnsupportedDistributedFormat() no longer trips on webm. HDR mp4 remains the only refused configuration. - resolveEncoderTriple() returns libvpx-vp9-software + yuva420p + preset="good" for format="webm". yuva420p preserves alpha — the format's main reason for existing for web delivery. - codec= remains rejected for non-mp4 formats (mov is always ProRes 4444; webm is always libvpx-vp9). The error message lists all four distributed-supported formats. - FormatNotSupportedInDistributedError docstring updated to reflect the new reality (only HDR is unsupported). - freezePlan.ts: LockedRenderConfig.encoder gains "libvpx-vp9-software". Mirrors libx265-software / prores-software / png-sequence in shape; the chunk worker reads this discriminant to decide encode args. - renderChunk.ts: drops the now-incorrect cast that excluded webm from buildSyntheticRenderJob's format input; tightens the preset-format cast to include webm. - assemble.ts: docstring + comment updates. The mp4/mov concat-copy path is format-agnostic — webm uses the exact same code (applyFaststart is a no-op for webm via the existing chunkEncoder.ts gate; muxVideoWithAudio already routes webm to libopus audio). - planFormatBanlist.test.ts: webm-rejection tests removed; replaced with "accepts webm" tests + a HDR+webm combo test that verifies HDR is the trip regardless of format. - plan.test.ts: new describe block pins the webm wiring contract: format="webm" produces an encoder=libvpx-vp9-software / pixelFormat=yuva420p planDir with closedGop=true and gopSize=chunkSize. - webm-concat-copy.test.ts (smoke): extended with a yuva420p variant that proves the alpha pixel format the distributed pipeline actually emits also round-trips through concat-copy. 9/9 tests pass locally. §8 format support matrix in DISTRIBUTED-RENDERING-PLAN.md is intentionally left unchanged at this PR — it flips to ✓ in PR 8.4 once the end-to-end fixture (PR 8.3) is green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(producer): include webm in plan-time needsAlpha + strengthen alpha smoke PR review feedback from Miguel and Vai on #951 caught a real bug: `plan.ts`'s `needsAlpha` disjunction excluded `"webm"`, so the plan stage froze `forceScreenshot: false` into the `LockedRenderConfig` even though distributed webm uses `yuva420p`. Every chunk worker captured opaque RGB via BeginFrame (which doesn't preserve alpha on Linux headless-shell), and libvpx-vp9 encoded uniformly-opaque alpha that the encoder then dropped — producing un-keyable webm. Two changes: 1. **plan.ts**: include `"webm"` in `needsAlpha`. Matches the in-process renderer's logic at `renderOrchestrator.ts:1469` (`const needsAlpha = isWebm || isMov || isPngSequence`); the two sites must stay in sync since the distributed pipeline's PSNR regression compares against the in-process baseline. 2. **Smoke test (yuva420p describe)**: source frames now use a real alpha gradient (`geq=a='X*255/W'` on top of `testsrc2`) instead of `testsrc2 + format=rgba` which was uniformly opaque. The decode- pix_fmt assertion is dropped (ffprobe reports `yuv420p` for VP9-with-alpha because the alpha lives in a Matroska `BlockAdditional` sidecar) and replaced with two stronger checks: - `TAG:ALPHA_MODE=1` is present on the stream — proves the encoder was actually configured for alpha - alpha plane variance after `-c:v libvpx-vp9 -i ... -pix_fmt rgba -vf extractplanes=a,signalstats` — proves the alpha sub-stream round-trips through concat-copy with spatially-varying content, not uniform/dropped alpha - decode-test gate is now exit-code-only (was `exitCode || stderr` which would flake on chatty ffmpeg `-v error` builds emitting non-fatal DTS/container notes) These checks would have caught the `needsAlpha` bug before review. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(aws-lambda): widen narrow format types to include webm CI on PR #951 was failing at typecheck/build because the producer's `DistributedRenderConfig.format` widened to include webm in this PR but the aws-lambda package's narrow `"mp4" | "mov" | "png-sequence"` type literals in `events.ts`, `handler.ts`, and `validateConfig.ts` hadn't kept up. `renderToLambda.ts:87` passed `config.format` (now including webm) into a parameter typed against the narrow union, producing TS2345. This widening originally landed in PR #952 (test fixture PR) but needs to be atomic with the producer's widening here to keep each PR independently typecheck-clean. Also refactor `formatExtension` from a switch dispatch to a `Record<DistributedFormat, string>` lookup. Adding the webm case tipped the switch's CRAP to the 30.0 fallow threshold; the lookup table drops cyclomatic from 5 to 1 with the same compile-time exhaustiveness guarantee (TS errors on missing entries when `DistributedFormat` adds a new format). The runtime `_exhaustive: never` throw was only protecting against a string slipping past TS; `validateConfig.ts`'s `ALLOWED_FORMATS` already gates untrusted input at the SDK boundary. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
James Russo
|
34d1f0e1d0 |
feat(lambda): add TypeScript SDK and CDK construct (#909)
* feat(lambda): add TypeScript SDK and CDK construct
Adds the client-side surface on top of the Phase 6a Lambda handler so
adopters can drive a deployed stack from Node without writing AWS-SDK
boilerplate:
- renderToLambda(opts) starts a Step Functions execution and returns a
handle. Does NOT poll.
- getRenderProgress({ executionArn }) returns a snapshot of progress,
frames rendered, cost (Lambda GB-seconds + SFN transitions), errors,
and the final output object once Assemble completes.
- deploySite({ projectDir, bucketName }) content-addresses the project
tree, tar.gzs it, and uploads to S3 with a HeadObject short-circuit so
re-renders of the same tree skip the tar+PUT.
- validateDistributedRenderConfig throws a typed InvalidConfigError
before StartExecution, so shape errors surface synchronously.
- computeRenderCost is exposed for callers who want to format cost out
of band.
Also ships HyperframesRenderStack, an aws-cdk-lib L2 construct that
emits the same topology as examples/aws-lambda/template.yaml. Lives on
the ./cdk subpath export so SDK-only consumers don't pull aws-cdk-lib
into their runtime graph (declared as an optional peer dependency).
Tests: 24 new unit tests across the SDK plus 9 CDK synth / contract /
snapshot tests. All 83 tests in packages/aws-lambda/src pass.
* refactor(lambda): /simplify pass on the SDK + CDK PR
Pulls shared logic out so the SDK doesn't re-invent things the handler
and the producer already have:
- `formatExtension` extracted to packages/aws-lambda/src/formatExtension.ts.
handler.ts and renderToLambda.ts both used identical 12-line copies of
this switch.
- `PLAN_PROJECT_DIR_SKIP_SEGMENTS` is now exported from
@hyperframes/producer/distributed. deploySite consumes it instead of
its own duplicate SKIP_TOP_LEVEL set; the two lists were trivially
identical and would have drifted silently.
- `FakeS3` + `drainBody` factored out of the two SDK test files into
src/sdk/__fixtures__/fakeS3.ts. Drops ~110 lines of test-file
duplication and gives future SDK tests a one-line FakeS3 import.
- S3 URI building in deploySite and renderToLambda routes through the
existing `formatS3Uri` helper instead of inline `s3://...`
concatenation; matches the convention already in handler.ts.
Net -133 lines across the touched files. All 83 aws-lambda tests still
pass; all 60 producer distributed tests still pass.
* fix(lambda): bump CDK test timeouts for CI cold-start synth
The bun:test default 5s timeout tripped the first CDK snapshot test
in CI when the cold-start `Template.fromStack(stack)` synth took ~5-8s
on the slowest GitHub Actions runner. Locally on a warm shell the
synth measures <1s, so the failure didn't reproduce until PR #909 hit
CI.
Two changes:
- Both CDK test files cache one synth in `beforeAll(..., 30000)` and
reuse the result across every test that uses the default props.
Each individual test now runs in microseconds (pure assertions
against the already-synthed template), so the 5s timeout no longer
applies on the hot path.
- The two contract tests that exercise non-default props
(reservedConcurrency, projectName) still synth fresh per-test; they
get a per-test `it(..., 30000)` timeout.
No behavior changes.
* fix(lambda): address PR review on SDK + CDK construct
Three correctness + ergonomics fixes raised in Vai's review:
- getRenderProgress over-counted SFN transitions by 3-5×. Step
Functions Standard Workflows bill per state-entry, not per
history event. Each Task produces ~5-7 history events
(Scheduled / Started / Succeeded / TaskStateExited / …);
counting `events.length` reported the runaway. Switch to
counting `*StateEntered` events explicitly.
- assembleComplete + outputFile detection was coupled to the
Lambda payload's `Action` field. Move both signals onto the
enclosing state name (`StateExited.name === "Assemble"`), which
is the state-machine identity rather than the Lambda event
contract. framesRendered increment moves to the same boundary
(RenderChunk state).
- SiteHandle now carries `bucketName` directly so README + CLI
callers don't have to re-parse `projectS3Uri.split("/")[2]`.
Test updates: getRenderProgress tests wrap renderChunk/assemble
events in matching StateEntered + StateExited pairs so the new
state-name-driven dispatch is exercised end-to-end. SiteHandle
fixture in renderToLambda.test.ts gets the new bucketName field.
All 83 aws-lambda tests still pass.
|
||
|
James Russo
|
c50f59a53b |
feat(lambda): add Lambda handler, ZIP bundling, and BeginFrame probe (#878)
* feat(lambda): add Lambda handler, ZIP bundling, and BeginFrame probe
Phase 6 of the distributed rendering plan: AWS Lambda turnkey adoption
(see DISTRIBUTED-RENDERING-PLAN.md §11 Phase 6 + §15).
This PR adds the new packages/aws-lambda/ workspace package that wraps
the OSS plan/renderChunk/assemble primitives in an AWS Lambda handler,
plus a build pipeline that bundles the handler + Chromium runtime +
ffmpeg into a deployable ZIP.
Architecture: ZIP deploy (not Docker image), Chrome via @sparticuz/chromium
with chrome-headless-shell fallback, dispatch on event.Action ∈ {plan,
renderChunk, assemble}.
The load-bearing concern — does @sparticuz/chromium's chrome-headless-shell
build honour CDP HeadlessExperimental.beginFrame? — is pinned by the new
scripts/probe-beginframe.ts regression guard. Probe boots the runtime
inside public.ecr.aws/lambda/nodejs:22, navigates to a static page, and
asserts beginFrame returns a PNG buffer. Verified locally + inside the
Docker container; both pass with hasDamage=true.
Sizes (sparticuz source): unzipped 157 MiB, zipped 99 MiB. Well under
the 240 MiB / 150 MiB in-house gates and the Lambda 250 MiB hard ceiling.
This is part of a stack of 8 PRs (3 in Phase 6a, 5 in Phase 6b); this is
PR 6.1.
* fix(lambda): address PR 878 review feedback
- Verify event.PlanHash against the untarred plan.json at the handler
boundary before invoking the producer primitive. Throws typed
PLAN_HASH_MISMATCH on divergence so Step Functions routes it as
non-retryable; previously the field was schema bloat the handler
ignored, leaving enforcement entirely inside the producer.
- Standardize on MiB throughout build-zip.ts, verify-zip-size.ts, and
the README. Lambda's hard ceiling is 250 MiB (AWS docs label "250 MB"
but use binary mebibytes); previously mixed units made the 248 MiB
budget look like a ~5 MB margin instead of the 2 MiB it actually is.
- stageChromeHeadlessShell now picks Chrome versions via numeric semver
comparison instead of lexicographic sort+reverse — the latter would
silently pick "99.x" over "131.x" once Chrome cached three-digit
majors that aren't width-aligned.
- Drop _setSparticuzChromiumForTests from the public index barrel.
Test-only DI seam imported directly from ./chromium.js in tests.
- Replace require("node:fs") inside walkSize() with the top-level fs
imports — file is ESM and the same module is already imported.
* docs(lambda): drop internal plan-doc refs from package README
* ci(windows): fix bun filter UNION bug excluding producer from Windows tests
`bun run --filter "!a" --filter "!b" test` composes as a UNION (any
package matching either negation runs), not an intersection. Effect:
@hyperframes/producer was still being tested on Windows even though
it's explicitly excluded — its regression harness (Docker + LFS golden
mp4 baselines) is Linux-only and was driving the 32min timeout.
Enumerate the packages we DO want to test instead.
|