mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-09 03:16:38 +00:00
3f6ea0a1ab3be45a64ae307c8a5af803ac9d0c85
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Miguel Ángel
|
dca5fa90cb | fix(producer): reject impossible distributed durations (#1732) | ||
|
James RussoandClaude Opus 4.8
|
4da567df22 |
feat(gcp-cloud-run): Google Cloud Run + Workflows distributed render adapter (#1253)
* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda (issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble) are unchanged; this package is the storage/compute/orchestration glue. Package: Cloud Run handler (one image, three actions), runs under bun; GCS transport; in-image chrome-headless-shell resolver; client SDK (renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile; Cloud Workflows definition; Terraform module; CLI cloudrun deploy|sites|render|render-batch|progress|destroy with --output-resolution and --strict-variables; 62 unit tests + docs + live smoke script. Shared extraction (removes ~640 lines of adapter duplication): move the cloud-agnostic config validator + content-hash into producer/distributed; both adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`, failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run to the root `build` filter so its dist exists for publish + runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install The regression test image runs `bun install --frozen-lockfile` after copying each workspace package.json individually. The CLI now depends on @hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to resolve it unless its manifest is present. Add the COPY line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): add machine-sizing flags to `cloudrun deploy` Closes the parity gap with `lambda deploy` (which exposes --memory etc.). `cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout into the Terraform apply; omitted flags keep the module defaults (4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address PR review (security, waste, limits, alerts) - server.ts: bucket-allowlist guard no longer fails open silently. Unset env logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces. - server.ts: stop double-shipping audio.aac. It already rides in the plan tarball every consumer downloads, so drop the redundant standalone upload (plan) + re-download/overwrite (assemble); assemble reads it from the untar, falling back to a supplied AudioGcsUri for compat. - server.ts: chunk extension via path.extname() instead of slice(lastIndexOf). - workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20) — Cloud Workflows hard-caps concurrent iterations at 20. - Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break the image rebuild. - terraform: add min_instances var (default 0); add a workflow-failure alert (finished_execution_count status=FAILED) alongside the request-count one. - costAccounting: document that displayCost excludes GCS storage/egress. Verified against the actual APIs: @google-cloud/workflows@4.4.0 ICreateExecutionRequest has no executionId (so the idempotency-token suggestion isn't available in this client); Workflows concurrency cap is 20; failure metric is workflows.googleapis.com/finished_execution_count (status label). 174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding - workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE → PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the opposite cause), misleading anyone triaging the alert. - workflow.yaml: forward Config.cfr to the assemble step (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler but never sent, so exact-CFR was silently off for every Cloud Run render. Uses the same `in`-operator guard already proven in the retryable predicate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(release): include gcp-cloud-run in set-version PACKAGES list set-version.ts (driven by release:prepare) bumps an explicit package list to the shared version on each release. gcp-cloud-run was wired into the build + publish.yml but missing here, so a release would leave it at a stale version and publish.yml would push the wrong version. Add it so the new package version-bumps + publishes in lockstep with the others. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
James RussoandClaude Opus 4.7
|
5d264e146c |
docs(lambda): document webm support + simplify-review fixes (#953)
* docs(lambda): document webm support in distributed mode PR 8.4 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). User-facing docs catch up with the shipped capability. Updates docs/deploy/migrating-to-hyperframes-lambda.mdx: - "Output format" row in the migration table now lists `webm` alongside mp4 / mov / png-sequence with a note that webm uses libvpx-vp9 + closed-GOP concat-copy. HDR mp4 remains the only refused format. - "No webm distributed" caveat replaced with "webm uses closed-GOP VP9" explainer covering the encoder args (`-g <chunkSize>`, `-keyint_min <chunkSize>`, `-auto-alt-ref 0`, `-cpu-used 2`), why alt-ref disable is load-bearing, and that the output preserves alpha via yuva420p with Opus audio. - Migration checklist no longer asks adopters to filter out webm compositions; only HDR-dependent renders need to stay on the previous framework. aws-lambda.mdx doesn't currently call out webm as unsupported (only HDR in the v1 surface list), so it gets no copy edits beyond the migration guide. The internal planning doc (DISTRIBUTED-RENDERING-PLAN.md §7.2, §8, §12 — kept outside the repo) gets matching updates: format support matrix flipped ✓, v1.5 backlog #1 marked shipped, HDR promoted to the new top item, and the rev-12 → rev-13 status line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: address simplify-review findings on webm stack Folds in cleanups identified by a multi-agent code-review pass over the 4-PR webm-distributed stack: - plan.ts: `resolveEncoderTriple()` webm case now calls `getEncoderPreset(quality, "webm")` for its preset string instead of hardcoding "good". The hardcode was wrong for `quality: "draft"` (`getEncoderPreset` returns "realtime" for that tier) — would have silently overridden the draft → realtime mapping for distributed webm renders. - chunkEncoder.ts: trim the new VP9 closed-GOP comment block from ~18 lines of WHY narration down to the 6 lines that actually explain why (alt-ref + cpu-used drift). Match the alpha branch's idempotent-push comment to the same standard. - chunkEncoder.test.ts: drop the duplicate WHY comment that restated the implementation comment in plain words. - webm-concat-copy.test.ts: rewrite the file-header docstring to describe the contract being tested instead of the PR-8.1-gating history; strip "PR 8.2 / Path A / Path B" references from error messages (they belong in PR bodies, not in test output). Consolidate the yuva420p alpha smoke into a single `it()` block (was a full 4-test describe with duplicated setup) — the yuv420p block already covers the probe/decode/frame-count contract; the alpha smoke only needs to prove the alpha args don't break concat-copy. - plan.test.ts: drop the "PR 8.1 proved the contract" comment. - webm-vp9 fixture: drop the aspirational "Other webm-with-audio fixtures cover the mux path separately when added" sentence (no other fixtures exist). Regenerated the baseline via `docker:test:update webm-vp9` to reflect the updated comment. - migrating-to-hyperframes-lambda.mdx: add a paragraph about distributed webm's perf cost — ~10-25% larger files at constant CRF due to forced keyframes, and slower per-chunk encode due to `-cpu-used 2` being more conservative than the libvpx default. All unit tests + the webm-vp9 distributed-simulated regression still pass after these changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): accept --format=webm in `hyperframes lambda render` The CLI's `lambda render` subcommand's FORMATS allowlist and the `RenderArgs.format` type still narrowed to `mp4 | mov | png-sequence`, so even though the producer + aws-lambda packages now support webm end-to-end, the CLI surface rejected it with `--format must be mp4|mov| png-sequence`. Add webm to both spots and update the --help description. Surfaced during real-AWS deploy prep — the local lambda-local / distributed-simulated tests didn't go through the CLI so the gap went unnoticed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(producer): font cache writes to /tmp on Lambda (read-only \$HOME) The deterministic Google Fonts cache was rooted at `\$HOME/.cache/hyperframes/fonts`, which fails on AWS Lambda — the runtime's `\$HOME` resolves to a `/home/sbx_*` directory tree that's read-only. `mkdirSync(..., { recursive: true })` can't create that path and the plan stage trips with `ENOENT: no such file or directory, mkdir '/home/sbx_user1051/.cache/hyperframes/fonts/space-mono'` on every Lambda render that pulls a Google Font (i.e. every distributed fixture using `@import url("https://fonts.googleapis.com/...")`). Detect Lambda via `\$AWS_LAMBDA_FUNCTION_NAME` and route the cache to `tmpdir()/hyperframes/fonts` in that case. Lambda's `/tmp` survives across invocations on a warm container, so cache hit rate is the same as non-Lambda runs. Also honor an explicit `\$HYPERFRAMES_FONT_CACHE_DIR` override for adopters who want a different location regardless of the runtime. Surfaced while verifying webm distributed end-to-end on real AWS — the same bug affects mp4 fixtures using Google Fonts; webm just happened to be the one I tried first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: extract DistributedFormat type + trim font-cache resolver Second simplify-review pass on the webm stack flagged two cleanups: 1. **`DistributedFormat` type duplicated 10 times.** Every file in the distributed pipeline carried its own copy of `"mp4" | "mov" | "png-sequence" | "webm"` — adding a new format meant a 10-place edit with no compile-time guarantee they stayed in sync. Extract a single source of truth in `packages/producer/src/services/distributed/shared.ts`, re-export from `@hyperframes/producer/distributed` and `@hyperframes/aws-lambda/sdk`, and have all callers pull from there. The aws-lambda `ALLOWED_FORMATS` runtime tuple and the CLI's `FORMATS` tuple now both use `satisfies readonly DistributedFormat[]` so the compiler enforces the runtime allowlist stays in sync with the type. 2. **`deterministicFonts.ts` font-cache resolver was over-commented.** Trim the 7-line block to 4 lines (drop the aspirational "and other read-only-FS execution environments" — only Lambda is detected — and the warm-container `/tmp` persistence narration — anyone reading already knows Lambda /tmp semantics). Collapse the two-step `if (explicit && explicit.length > 0)` into a single nullish-coalesce expression now that the empty-string defensive check is gone (`process.env.X` is `string | undefined`, no third shape to guard against). Out-of-scope skips (called out by the agents, deferred): - In-process `RenderConfig.format` and the in-process CLI's `render.ts` format union still carry their own inline copies. The union happens to coincide today but they're separate concerns — leaving them alone limits this PR's blast radius. - `fontCacheDir(slug)` / `resolveFontCacheRoot()` naming asymmetry flagged as taste; skipping. - Pre-existing redundant `existsSync` before `mkdirSync({ recursive: true })` in `fontCacheDir` — out of scope. All tests + typecheck still pass. Lambda render still works end-to-end (no functional changes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(lambda): drop plan-doc reference from migration checklist PR review feedback: source/docs should not mention the distributed-rendering planning doc. Tighten the migration checklist sentence to describe the webm path directly rather than referencing the doc's version label. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(producer): split resolveEncoderTriple into mp4 + non-mp4 helpers CI Fallow audit on PR #953 flagged `resolveEncoderTriple` at CRAP 31.6 — the function interleaved (a) mp4 codec validation + dispatch, (b) the non-mp4 codec-rejection throw, and (c) per-format dispatch. Splitting into `resolveMp4EncoderTriple` + `resolveNonMp4EncoderTriple` drops the top-level function's cyclomatic complexity below the threshold while preserving every error message and code path. Behavior unchanged. Also extracts an `EncoderTriple` type alias so the three functions share the return shape declaratively rather than repeating it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
JamesandClaude Opus 4.7
|
e79a8faa42 |
feat(producer): auto-size chunkSize from maxParallelChunks when undefined
Previously, plan() defaulted chunkSize to 240 on a `?? DEFAULT_CHUNK_SIZE` line, so a 660-frame composition with maxParallelChunks=16 ended up at 3 chunks (ceil(660/240)) regardless of the caller's fan-out intent. When config.chunkSize is undefined, auto-size from maxParallelChunks: effectiveChunkSize = max(MIN_CHUNK_SIZE, ceil(totalFrames / maxParallelChunks)) MIN_CHUNK_SIZE=10 keeps per-chunk fixed overhead from swamping the parallelism gain on tiny renders. Explicit numbers, including 240, take precedence over the auto-sizer — no behavior change for callers that set chunkSize explicitly. Surfaced by the lever-1 chunk-scaling benchmark on 2026-05-17. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
James Russo
|
34d1f0e1d0 |
feat(lambda): add TypeScript SDK and CDK construct (#909)
* feat(lambda): add TypeScript SDK and CDK construct
Adds the client-side surface on top of the Phase 6a Lambda handler so
adopters can drive a deployed stack from Node without writing AWS-SDK
boilerplate:
- renderToLambda(opts) starts a Step Functions execution and returns a
handle. Does NOT poll.
- getRenderProgress({ executionArn }) returns a snapshot of progress,
frames rendered, cost (Lambda GB-seconds + SFN transitions), errors,
and the final output object once Assemble completes.
- deploySite({ projectDir, bucketName }) content-addresses the project
tree, tar.gzs it, and uploads to S3 with a HeadObject short-circuit so
re-renders of the same tree skip the tar+PUT.
- validateDistributedRenderConfig throws a typed InvalidConfigError
before StartExecution, so shape errors surface synchronously.
- computeRenderCost is exposed for callers who want to format cost out
of band.
Also ships HyperframesRenderStack, an aws-cdk-lib L2 construct that
emits the same topology as examples/aws-lambda/template.yaml. Lives on
the ./cdk subpath export so SDK-only consumers don't pull aws-cdk-lib
into their runtime graph (declared as an optional peer dependency).
Tests: 24 new unit tests across the SDK plus 9 CDK synth / contract /
snapshot tests. All 83 tests in packages/aws-lambda/src pass.
* refactor(lambda): /simplify pass on the SDK + CDK PR
Pulls shared logic out so the SDK doesn't re-invent things the handler
and the producer already have:
- `formatExtension` extracted to packages/aws-lambda/src/formatExtension.ts.
handler.ts and renderToLambda.ts both used identical 12-line copies of
this switch.
- `PLAN_PROJECT_DIR_SKIP_SEGMENTS` is now exported from
@hyperframes/producer/distributed. deploySite consumes it instead of
its own duplicate SKIP_TOP_LEVEL set; the two lists were trivially
identical and would have drifted silently.
- `FakeS3` + `drainBody` factored out of the two SDK test files into
src/sdk/__fixtures__/fakeS3.ts. Drops ~110 lines of test-file
duplication and gives future SDK tests a one-line FakeS3 import.
- S3 URI building in deploySite and renderToLambda routes through the
existing `formatS3Uri` helper instead of inline `s3://...`
concatenation; matches the convention already in handler.ts.
Net -133 lines across the touched files. All 83 aws-lambda tests still
pass; all 60 producer distributed tests still pass.
* fix(lambda): bump CDK test timeouts for CI cold-start synth
The bun:test default 5s timeout tripped the first CDK snapshot test
in CI when the cold-start `Template.fromStack(stack)` synth took ~5-8s
on the slowest GitHub Actions runner. Locally on a warm shell the
synth measures <1s, so the failure didn't reproduce until PR #909 hit
CI.
Two changes:
- Both CDK test files cache one synth in `beforeAll(..., 30000)` and
reuse the result across every test that uses the default props.
Each individual test now runs in microseconds (pure assertions
against the already-synthed template), so the 5s timeout no longer
applies on the hot path.
- The two contract tests that exercise non-default props
(reservedConcurrency, projectName) still synth fresh per-test; they
get a per-test `it(..., 30000)` timeout.
No behavior changes.
* fix(lambda): address PR review on SDK + CDK construct
Three correctness + ergonomics fixes raised in Vai's review:
- getRenderProgress over-counted SFN transitions by 3-5×. Step
Functions Standard Workflows bill per state-entry, not per
history event. Each Task produces ~5-7 history events
(Scheduled / Started / Succeeded / TaskStateExited / …);
counting `events.length` reported the runaway. Switch to
counting `*StateEntered` events explicitly.
- assembleComplete + outputFile detection was coupled to the
Lambda payload's `Action` field. Move both signals onto the
enclosing state name (`StateExited.name === "Assemble"`), which
is the state-machine identity rather than the Lambda event
contract. framesRendered increment moves to the same boundary
(RenderChunk state).
- SiteHandle now carries `bucketName` directly so README + CLI
callers don't have to re-parse `projectS3Uri.split("/")[2]`.
Test updates: getRenderProgress tests wrap renderChunk/assemble
events in matching StateEntered + StateExited pairs so the new
state-name-driven dispatch is exercised end-to-end. SiteHandle
fixture in renderToLambda.test.ts gets the new bucketName field.
All 83 aws-lambda tests still pass.
|
||
|
James Russo
|
cad9160cd8 |
chore(producer): drop internal plan-doc refs from source + docs (#903)
The producer source + docs referenced an internal coordination doc (DISTRIBUTED-RENDERING-PLAN.md) that doesn't ship in the OSS repo, leaving broken cross-links for adopters. Drops the references and the bare section-number shorthand that depended on them; behavioural content (hash contract, retry semantics, threshold rationale) is preserved inline where it was previously offloaded to a section number. |
||
|
James
|
91a91f66ba | feat(producer): export distributed render primitives |