mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-05 10:14:30 +00:00
41e23697aca1997a9408c03eb3997262d645e9de
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 James Russo
|
17a2a00ed5 |
feat(cloud): default distributed plans to v2 (#3311)
* feat(cloud): default distributed plans to v2 * fix(cloud): address plan v2 review feedback * fix(examples): document explicit v2 samples |
||
|
Miguel Ángel
|
dc4383113c |
fix(producer): mix audio into a container that can record encoder delay (#3200)
* fix(producer): mix audio into a container that can record encoder delay Every rendered composition's audio landed 1024 samples (21.33 ms at 48 kHz) after its authored `data-start`, against a frame-accurate video track. The mix is AAC-encoded, and AAC encoders emit ~1024 priming samples. The mix was written to a raw ADTS `.aac` file, which has nowhere to record that delay, so it decoded as real leading silence and every stage downstream preserved it faithfully. Measuring each intermediate localises it precisely: the source WAV is exact, the mixer's own output is already 21.33 ms late, and the pad/trim and mux stages inherit it unchanged. The filter graph itself is correct - run by hand to PCM it lands on the authored start. Switch the artifact to an MP4-family container, which stores the delay as an edit list that decoders strip. Same codec, same bitrate, so no size or quality change. The filename is a contract shared by three consumers - the mux input, the distributed plan artifact, and the PNG-sequence sidecar handed to users for NLE ingest - and its extension is what selects the muxer. Give it one owner in the engine rather than five literals, so those consumers cannot drift onto different containers. Note for reviewers: this renames the distributed plan's audio artifact, which is an on-disk contract between the plan writer and the assembler. Both move together here, but a plan written by an older build would not be found by a newer assembler. Flagging in case that mixed-version window matters for how these are deployed. * fix(cloud): read the plan audio artifact name from the producer contract The aws-lambda and gcp-cloud-run adapters each restated the plan's audio filename in five places, so renaming it in the producer left them looking for a file that is no longer written. CI caught it: the gcp dispatch test asserting a plan has no audio artifact started seeing one. Export the name from `@hyperframes/producer/distributed` and consume it in both adapters. This is the same failure the constant exists to prevent, one package boundary further out: a literal that drifts from the writer's is a silently missing audio track rather than a loud error, because both call sites only ever ask whether the file exists. * fix(cloud): accept a legacy plan's audio artifact name for one release Review raised a rolling-deploy window I had flagged but left undecided: `plan` and `assemble` are separate invocations bridged by object storage, so a pre-rollout planner can be paired with a post-rollout assembler. Both readers locate the artifact by existence alone, which makes that pairing a silently muted video rather than an error. That is reachable enough to be worth two lines, so reads now accept the old name while writes only ever emit the new one. Give the fallback one owner (`resolvePlanAudioPath` / `isPlanAudioArtifactPath`) rather than four call sites, marked for deletion one release out. Also fixes a hole in the first pass of this: the plan-v2 materializer matched either name but then joined the CURRENT one, so a legacy plan resolved to a path that was never written. It now joins the artifact's own name. Review nits in the same pass: correct the pad-branch docstring, which still described a concat-copy shape the pad branch stopped using when it moved to apad + re-encode, and fix the Windows fixture's stale `.aac` output extension so it cannot model a shape that reintroduces the priming delay. * test(producer): rebake the missing-host-comp-id golden without the audio delay The pinned reference was rendered before this branch, so it carries the 1024 sample encoder-priming delay in its audio. With the delay gone the correct audio now sits ahead of the reference and the harness's envelope correlation drops below its floor. Cross-correlating the old and new references at native 48 kHz gives a lag of exactly 1024 samples (21.33 ms) at a correlation of 0.99985: same audio, moved by exactly the amount this branch removes. Regenerated inside the CI container (Dockerfile.test, ffmpeg 5.1.9) rather than natively, so the reference matches the encoder CI will compare against - the container reproduced CI's failure to the digit (correlation 0.3938764027803616, lagWindows -12) before the rebake and passes at correlation 1.0 after it. Note for archaeology: the new reference is also 3 dB louder than the old one. That gap is not from this branch - `main` and this branch render the fixture at the same level - it is pre-existing drift the reference had accumulated, which a scale-invariant correlator could never see. The rebake absorbs it. Only output.mp4 is updated. `--update` also rewrites compiled.html, but that diff is embedded-font churn with no bearing on the comparison, which reports "Failed at compilation: 0" either way. * test(producer): rebake the variables-prod golden without the audio delay Same cause as the missing-host-comp-id rebake, caught by shard-8 once the earlier shard stopped failing and the rest of the matrix could run: this reference also carries the encoder-priming delay this branch removes. Reproduced in the CI container to the digit (correlation 0.42704173048439215, lagWindows -12), rebaked there, and it now passes at correlation 1.0. Worth recording: the shift here is 2048 samples (42.67 ms) at correlation 0.99983, exactly twice the 1024 of the other fixture. The delay compounds once per un-compensated AAC generation, and this fixture's audio needs its duration normalized, so it takes the pad/trim branch's re-encode and picks up a second frame of priming on top of the mixer's. So the pre-fix error was not a fixed 21 ms - it grew with the number of times the audio was re-encoded. All nine shards ran in that CI round with only this one failing, so the matrix has now covered every fixture against this change. |
||
|
James Russo
|
1d636f603c |
refactor(producer): share plan execution builder (#2906)
## What Refactor distributed planning around one shared local execution-plan builder: - `buildLocalExecutionPlan()` now owns compile/probe/extract/audio/freeze. - Legacy `plan()` remains a deprecated v1 transport wrapper. - Plan v2 calls the shared builder directly and publishes through the existing manifest/CAS contract. - Add neutral `createPlanV2FromExecutionPlan()`, `publishPlanV2FromExecutionPlan()`, `getPlanV2ExecutionPlanHash()`, and `PLAN_PROTOCOL_V1` names. - Retain deprecated v1-named exports and wire aliases. - Recommend explicit Plan v2 opt-in for new producer, Lambda, and Cloud Run integrations. ## Why Plan v2 previously looked like it invoked a v1 planner even though v1 and v2 share the same frozen local execution representation. This removes that migration-era coupling while preserving the public minor-version compatibility contract. ## How The shared builder returns neutral internal execution-plan fields. The v1 wrapper maps those fields back to the existing `PlanResult`; the v2 publisher consumes them directly. Compatibility is intentional and covered by exact shape tests: - omitted `planProtocol` still serializes/selects `"v1"`; - v1 layouts, descriptor-less decoding, event unions, workflow branches, and exports remain; - the v1 descriptor JSON is byte-identical and `CURRENT_PLAN_PROTOCOL` is an identity-preserving alias; - v2 manifest bytes, key order, hash framing, and `sourcePlanV1Hash` wire key remain unchanged; - no enumerable neutral hash field was added to manifests or returned result objects; - v1/v2 result objects, cloud event payloads, and SDK handle key sets remain unchanged. ## Test plan - Focused Plan v1/v2/protocol/export/size compatibility: 141 passed - `@hyperframes/core`: 1,419 passed - `@hyperframes/producer` unit lane: 990 passed - `@hyperframes/aws-lambda`: 140 passed - `@hyperframes/gcp-cloud-run`: 101 passed - Producer, Lambda, and Cloud Run typechecks - Repository-wide lint, format check, workspace/package-subpath checks - Full workspace build - `git diff --check` - [x] Unit tests added/updated - [ ] Manual testing performed - [x] Documentation updated (if applicable) |
||
|
James
|
2a284a8e3a | fix(gcp): enforce effective BeginFrame capture | ||
|
James Russo
|
5bf61d6df0 |
feat(aws-lambda): support plan protocol v2 (#2789)
* feat(aws-lambda): support plan protocol v2 * fix(aws-lambda): align SAM v2 terminal errors |
||
|
James RussoandClaude Opus 4.8
|
4da567df22 |
feat(gcp-cloud-run): Google Cloud Run + Workflows distributed render adapter (#1253)
* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda (issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble) are unchanged; this package is the storage/compute/orchestration glue. Package: Cloud Run handler (one image, three actions), runs under bun; GCS transport; in-image chrome-headless-shell resolver; client SDK (renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile; Cloud Workflows definition; Terraform module; CLI cloudrun deploy|sites|render|render-batch|progress|destroy with --output-resolution and --strict-variables; 62 unit tests + docs + live smoke script. Shared extraction (removes ~640 lines of adapter duplication): move the cloud-agnostic config validator + content-hash into producer/distributed; both adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`, failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run to the root `build` filter so its dist exists for publish + runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install The regression test image runs `bun install --frozen-lockfile` after copying each workspace package.json individually. The CLI now depends on @hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to resolve it unless its manifest is present. Add the COPY line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): add machine-sizing flags to `cloudrun deploy` Closes the parity gap with `lambda deploy` (which exposes --memory etc.). `cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout into the Terraform apply; omitted flags keep the module defaults (4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address PR review (security, waste, limits, alerts) - server.ts: bucket-allowlist guard no longer fails open silently. Unset env logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces. - server.ts: stop double-shipping audio.aac. It already rides in the plan tarball every consumer downloads, so drop the redundant standalone upload (plan) + re-download/overwrite (assemble); assemble reads it from the untar, falling back to a supplied AudioGcsUri for compat. - server.ts: chunk extension via path.extname() instead of slice(lastIndexOf). - workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20) — Cloud Workflows hard-caps concurrent iterations at 20. - Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break the image rebuild. - terraform: add min_instances var (default 0); add a workflow-failure alert (finished_execution_count status=FAILED) alongside the request-count one. - costAccounting: document that displayCost excludes GCS storage/egress. Verified against the actual APIs: @google-cloud/workflows@4.4.0 ICreateExecutionRequest has no executionId (so the idempotency-token suggestion isn't available in this client); Workflows concurrency cap is 20; failure metric is workflows.googleapis.com/finished_execution_count (status label). 174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding - workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE → PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the opposite cause), misleading anyone triaging the alert. - workflow.yaml: forward Config.cfr to the assemble step (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler but never sent, so exact-CFR was silently off for every Cloud Run render. Uses the same `in`-operator guard already proven in the retryable predicate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(release): include gcp-cloud-run in set-version PACKAGES list set-version.ts (driven by release:prepare) bumps an explicit package list to the shared version on each release. gcp-cloud-run was wired into the build + publish.yml but missing here, so a release would leave it at a stale version and publish.yml would push the wrong version. Add it so the new package version-bumps + publishes in lockstep with the others. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
James
|
71d1889da6 |
feat(distributed): add optional cfr flag for exact constant frame rate
Distributed-render output today uses -c:v copy through concat → mux → faststart, which means PTS timestamps from each chunk pass through unchanged. Container r_frame_rate is exact (#1040 + this PR's parent), but stream-level avg_frame_rate stays PTS-derived and can land on fractional rationals like 27648000/921677 over a 60s render. Same for sub-ms duration drift. This is the achievable bar within -c copy stream-copy concat. For most consumers (browser playback, YouTube, etc.) the difference is invisible. For downstream tools that strict-check avg_frame_rate or ms-precision duration (broadcast workflows, frame-accurate compositors, some third-party transcoders), it matters. Adds an opt-in cfr config flag (default false). When true, the assemble step's final pass re-encodes with -fps_mode cfr -r <fps> instead of -c copy, producing exact CFR output. Trade-off: ~2-5x the stitch time for a 60s 1080p clip; second-generation H.264 quality loss is negligible at -crf 18 but is non-zero. |
||
|
James RussoandClaude Opus 4.7
|
5d264e146c |
docs(lambda): document webm support + simplify-review fixes (#953)
* docs(lambda): document webm support in distributed mode PR 8.4 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). User-facing docs catch up with the shipped capability. Updates docs/deploy/migrating-to-hyperframes-lambda.mdx: - "Output format" row in the migration table now lists `webm` alongside mp4 / mov / png-sequence with a note that webm uses libvpx-vp9 + closed-GOP concat-copy. HDR mp4 remains the only refused format. - "No webm distributed" caveat replaced with "webm uses closed-GOP VP9" explainer covering the encoder args (`-g <chunkSize>`, `-keyint_min <chunkSize>`, `-auto-alt-ref 0`, `-cpu-used 2`), why alt-ref disable is load-bearing, and that the output preserves alpha via yuva420p with Opus audio. - Migration checklist no longer asks adopters to filter out webm compositions; only HDR-dependent renders need to stay on the previous framework. aws-lambda.mdx doesn't currently call out webm as unsupported (only HDR in the v1 surface list), so it gets no copy edits beyond the migration guide. The internal planning doc (DISTRIBUTED-RENDERING-PLAN.md §7.2, §8, §12 — kept outside the repo) gets matching updates: format support matrix flipped ✓, v1.5 backlog #1 marked shipped, HDR promoted to the new top item, and the rev-12 → rev-13 status line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: address simplify-review findings on webm stack Folds in cleanups identified by a multi-agent code-review pass over the 4-PR webm-distributed stack: - plan.ts: `resolveEncoderTriple()` webm case now calls `getEncoderPreset(quality, "webm")` for its preset string instead of hardcoding "good". The hardcode was wrong for `quality: "draft"` (`getEncoderPreset` returns "realtime" for that tier) — would have silently overridden the draft → realtime mapping for distributed webm renders. - chunkEncoder.ts: trim the new VP9 closed-GOP comment block from ~18 lines of WHY narration down to the 6 lines that actually explain why (alt-ref + cpu-used drift). Match the alpha branch's idempotent-push comment to the same standard. - chunkEncoder.test.ts: drop the duplicate WHY comment that restated the implementation comment in plain words. - webm-concat-copy.test.ts: rewrite the file-header docstring to describe the contract being tested instead of the PR-8.1-gating history; strip "PR 8.2 / Path A / Path B" references from error messages (they belong in PR bodies, not in test output). Consolidate the yuva420p alpha smoke into a single `it()` block (was a full 4-test describe with duplicated setup) — the yuv420p block already covers the probe/decode/frame-count contract; the alpha smoke only needs to prove the alpha args don't break concat-copy. - plan.test.ts: drop the "PR 8.1 proved the contract" comment. - webm-vp9 fixture: drop the aspirational "Other webm-with-audio fixtures cover the mux path separately when added" sentence (no other fixtures exist). Regenerated the baseline via `docker:test:update webm-vp9` to reflect the updated comment. - migrating-to-hyperframes-lambda.mdx: add a paragraph about distributed webm's perf cost — ~10-25% larger files at constant CRF due to forced keyframes, and slower per-chunk encode due to `-cpu-used 2` being more conservative than the libvpx default. All unit tests + the webm-vp9 distributed-simulated regression still pass after these changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): accept --format=webm in `hyperframes lambda render` The CLI's `lambda render` subcommand's FORMATS allowlist and the `RenderArgs.format` type still narrowed to `mp4 | mov | png-sequence`, so even though the producer + aws-lambda packages now support webm end-to-end, the CLI surface rejected it with `--format must be mp4|mov| png-sequence`. Add webm to both spots and update the --help description. Surfaced during real-AWS deploy prep — the local lambda-local / distributed-simulated tests didn't go through the CLI so the gap went unnoticed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(producer): font cache writes to /tmp on Lambda (read-only \$HOME) The deterministic Google Fonts cache was rooted at `\$HOME/.cache/hyperframes/fonts`, which fails on AWS Lambda — the runtime's `\$HOME` resolves to a `/home/sbx_*` directory tree that's read-only. `mkdirSync(..., { recursive: true })` can't create that path and the plan stage trips with `ENOENT: no such file or directory, mkdir '/home/sbx_user1051/.cache/hyperframes/fonts/space-mono'` on every Lambda render that pulls a Google Font (i.e. every distributed fixture using `@import url("https://fonts.googleapis.com/...")`). Detect Lambda via `\$AWS_LAMBDA_FUNCTION_NAME` and route the cache to `tmpdir()/hyperframes/fonts` in that case. Lambda's `/tmp` survives across invocations on a warm container, so cache hit rate is the same as non-Lambda runs. Also honor an explicit `\$HYPERFRAMES_FONT_CACHE_DIR` override for adopters who want a different location regardless of the runtime. Surfaced while verifying webm distributed end-to-end on real AWS — the same bug affects mp4 fixtures using Google Fonts; webm just happened to be the one I tried first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: extract DistributedFormat type + trim font-cache resolver Second simplify-review pass on the webm stack flagged two cleanups: 1. **`DistributedFormat` type duplicated 10 times.** Every file in the distributed pipeline carried its own copy of `"mp4" | "mov" | "png-sequence" | "webm"` — adding a new format meant a 10-place edit with no compile-time guarantee they stayed in sync. Extract a single source of truth in `packages/producer/src/services/distributed/shared.ts`, re-export from `@hyperframes/producer/distributed` and `@hyperframes/aws-lambda/sdk`, and have all callers pull from there. The aws-lambda `ALLOWED_FORMATS` runtime tuple and the CLI's `FORMATS` tuple now both use `satisfies readonly DistributedFormat[]` so the compiler enforces the runtime allowlist stays in sync with the type. 2. **`deterministicFonts.ts` font-cache resolver was over-commented.** Trim the 7-line block to 4 lines (drop the aspirational "and other read-only-FS execution environments" — only Lambda is detected — and the warm-container `/tmp` persistence narration — anyone reading already knows Lambda /tmp semantics). Collapse the two-step `if (explicit && explicit.length > 0)` into a single nullish-coalesce expression now that the empty-string defensive check is gone (`process.env.X` is `string | undefined`, no third shape to guard against). Out-of-scope skips (called out by the agents, deferred): - In-process `RenderConfig.format` and the in-process CLI's `render.ts` format union still carry their own inline copies. The union happens to coincide today but they're separate concerns — leaving them alone limits this PR's blast radius. - `fontCacheDir(slug)` / `resolveFontCacheRoot()` naming asymmetry flagged as taste; skipping. - Pre-existing redundant `existsSync` before `mkdirSync({ recursive: true })` in `fontCacheDir` — out of scope. All tests + typecheck still pass. Lambda render still works end-to-end (no functional changes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(lambda): drop plan-doc reference from migration checklist PR review feedback: source/docs should not mention the distributed-rendering planning doc. Tighten the migration checklist sentence to describe the webm path directly rather than referencing the doc's version label. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(producer): split resolveEncoderTriple into mp4 + non-mp4 helpers CI Fallow audit on PR #953 flagged `resolveEncoderTriple` at CRAP 31.6 — the function interleaved (a) mp4 codec validation + dispatch, (b) the non-mp4 codec-rejection throw, and (c) per-format dispatch. Splitting into `resolveMp4EncoderTriple` + `resolveNonMp4EncoderTriple` drops the top-level function's cyclomatic complexity below the threshold while preserving every error message and code path. Behavior unchanged. Also extracts an `EncoderTriple` type alias so the three functions share the return shape declaratively rather than repeating it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
James RussoandClaude Opus 4.7
|
21f5066832 |
feat(producer): enable webm in distributed mode via concat-copy (#951)
* feat(producer): enable webm in distributed mode via concat-copy PR 8.2 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). Wires libvpx-vp9 webm through the distributed pipeline now that PR 8.1 proved concat-copy works. Architectural decision: Path A (concat-copy) — based on PR 8.1's smoke test result (9/9 tests pass for both yuv420p and yuva420p VP9 streams). The simpler architecture wins; no re-encode in assemble, no encode- parallelism loss. Changes: - plan.ts: - DistributedRenderConfig.format and PlanResult.format now include "webm" — type-level acceptance matches the runtime gate. - rejectUnsupportedDistributedFormat() no longer trips on webm. HDR mp4 remains the only refused configuration. - resolveEncoderTriple() returns libvpx-vp9-software + yuva420p + preset="good" for format="webm". yuva420p preserves alpha — the format's main reason for existing for web delivery. - codec= remains rejected for non-mp4 formats (mov is always ProRes 4444; webm is always libvpx-vp9). The error message lists all four distributed-supported formats. - FormatNotSupportedInDistributedError docstring updated to reflect the new reality (only HDR is unsupported). - freezePlan.ts: LockedRenderConfig.encoder gains "libvpx-vp9-software". Mirrors libx265-software / prores-software / png-sequence in shape; the chunk worker reads this discriminant to decide encode args. - renderChunk.ts: drops the now-incorrect cast that excluded webm from buildSyntheticRenderJob's format input; tightens the preset-format cast to include webm. - assemble.ts: docstring + comment updates. The mp4/mov concat-copy path is format-agnostic — webm uses the exact same code (applyFaststart is a no-op for webm via the existing chunkEncoder.ts gate; muxVideoWithAudio already routes webm to libopus audio). - planFormatBanlist.test.ts: webm-rejection tests removed; replaced with "accepts webm" tests + a HDR+webm combo test that verifies HDR is the trip regardless of format. - plan.test.ts: new describe block pins the webm wiring contract: format="webm" produces an encoder=libvpx-vp9-software / pixelFormat=yuva420p planDir with closedGop=true and gopSize=chunkSize. - webm-concat-copy.test.ts (smoke): extended with a yuva420p variant that proves the alpha pixel format the distributed pipeline actually emits also round-trips through concat-copy. 9/9 tests pass locally. §8 format support matrix in DISTRIBUTED-RENDERING-PLAN.md is intentionally left unchanged at this PR — it flips to ✓ in PR 8.4 once the end-to-end fixture (PR 8.3) is green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(producer): include webm in plan-time needsAlpha + strengthen alpha smoke PR review feedback from Miguel and Vai on #951 caught a real bug: `plan.ts`'s `needsAlpha` disjunction excluded `"webm"`, so the plan stage froze `forceScreenshot: false` into the `LockedRenderConfig` even though distributed webm uses `yuva420p`. Every chunk worker captured opaque RGB via BeginFrame (which doesn't preserve alpha on Linux headless-shell), and libvpx-vp9 encoded uniformly-opaque alpha that the encoder then dropped — producing un-keyable webm. Two changes: 1. **plan.ts**: include `"webm"` in `needsAlpha`. Matches the in-process renderer's logic at `renderOrchestrator.ts:1469` (`const needsAlpha = isWebm || isMov || isPngSequence`); the two sites must stay in sync since the distributed pipeline's PSNR regression compares against the in-process baseline. 2. **Smoke test (yuva420p describe)**: source frames now use a real alpha gradient (`geq=a='X*255/W'` on top of `testsrc2`) instead of `testsrc2 + format=rgba` which was uniformly opaque. The decode- pix_fmt assertion is dropped (ffprobe reports `yuv420p` for VP9-with-alpha because the alpha lives in a Matroska `BlockAdditional` sidecar) and replaced with two stronger checks: - `TAG:ALPHA_MODE=1` is present on the stream — proves the encoder was actually configured for alpha - alpha plane variance after `-c:v libvpx-vp9 -i ... -pix_fmt rgba -vf extractplanes=a,signalstats` — proves the alpha sub-stream round-trips through concat-copy with spatially-varying content, not uniform/dropped alpha - decode-test gate is now exit-code-only (was `exitCode || stderr` which would flake on chatty ffmpeg `-v error` builds emitting non-fatal DTS/container notes) These checks would have caught the `needsAlpha` bug before review. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(aws-lambda): widen narrow format types to include webm CI on PR #951 was failing at typecheck/build because the producer's `DistributedRenderConfig.format` widened to include webm in this PR but the aws-lambda package's narrow `"mp4" | "mov" | "png-sequence"` type literals in `events.ts`, `handler.ts`, and `validateConfig.ts` hadn't kept up. `renderToLambda.ts:87` passed `config.format` (now including webm) into a parameter typed against the narrow union, producing TS2345. This widening originally landed in PR #952 (test fixture PR) but needs to be atomic with the producer's widening here to keep each PR independently typecheck-clean. Also refactor `formatExtension` from a switch dispatch to a `Record<DistributedFormat, string>` lookup. Adding the webm case tipped the switch's CRAP to the 30.0 fallow threshold; the lookup table drops cyclomatic from 5 to 1 with the same compile-time exhaustiveness guarantee (TS errors on missing entries when `DistributedFormat` adds a new format). The runtime `_exhaustive: never` throw was only protecting against a string slipping past TS; `validateConfig.ts`'s `ALLOWED_FORMATS` already gates untrusted input at the SDK boundary. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
James Russo
|
c50f59a53b |
feat(lambda): add Lambda handler, ZIP bundling, and BeginFrame probe (#878)
* feat(lambda): add Lambda handler, ZIP bundling, and BeginFrame probe
Phase 6 of the distributed rendering plan: AWS Lambda turnkey adoption
(see DISTRIBUTED-RENDERING-PLAN.md §11 Phase 6 + §15).
This PR adds the new packages/aws-lambda/ workspace package that wraps
the OSS plan/renderChunk/assemble primitives in an AWS Lambda handler,
plus a build pipeline that bundles the handler + Chromium runtime +
ffmpeg into a deployable ZIP.
Architecture: ZIP deploy (not Docker image), Chrome via @sparticuz/chromium
with chrome-headless-shell fallback, dispatch on event.Action ∈ {plan,
renderChunk, assemble}.
The load-bearing concern — does @sparticuz/chromium's chrome-headless-shell
build honour CDP HeadlessExperimental.beginFrame? — is pinned by the new
scripts/probe-beginframe.ts regression guard. Probe boots the runtime
inside public.ecr.aws/lambda/nodejs:22, navigates to a static page, and
asserts beginFrame returns a PNG buffer. Verified locally + inside the
Docker container; both pass with hasDamage=true.
Sizes (sparticuz source): unzipped 157 MiB, zipped 99 MiB. Well under
the 240 MiB / 150 MiB in-house gates and the Lambda 250 MiB hard ceiling.
This is part of a stack of 8 PRs (3 in Phase 6a, 5 in Phase 6b); this is
PR 6.1.
* fix(lambda): address PR 878 review feedback
- Verify event.PlanHash against the untarred plan.json at the handler
boundary before invoking the producer primitive. Throws typed
PLAN_HASH_MISMATCH on divergence so Step Functions routes it as
non-retryable; previously the field was schema bloat the handler
ignored, leaving enforcement entirely inside the producer.
- Standardize on MiB throughout build-zip.ts, verify-zip-size.ts, and
the README. Lambda's hard ceiling is 250 MiB (AWS docs label "250 MB"
but use binary mebibytes); previously mixed units made the 248 MiB
budget look like a ~5 MB margin instead of the 2 MiB it actually is.
- stageChromeHeadlessShell now picks Chrome versions via numeric semver
comparison instead of lexicographic sort+reverse — the latter would
silently pick "99.x" over "131.x" once Chrome cached three-digit
majors that aren't width-aligned.
- Drop _setSparticuzChromiumForTests from the public index barrel.
Test-only DI seam imported directly from ./chromium.js in tests.
- Replace require("node:fs") inside walkSize() with the top-level fs
imports — file is ESM and the same module is already imported.
* docs(lambda): drop internal plan-doc refs from package README
* ci(windows): fix bun filter UNION bug excluding producer from Windows tests
`bun run --filter "!a" --filter "!b" test` composes as a UNION (any
package matching either negation runs), not an intersection. Effect:
@hyperframes/producer was still being tested on Windows even though
it's explicitly excluded — its regression harness (Docker + LFS golden
mp4 baselines) is Linux-only and was driving the 32min timeout.
Enumerate the packages we DO want to test instead.
|