mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
8db190dd43fdccdded89bee8ab1cea77054a3882
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 James RussoandClaude Opus 4.8
|
4da567df22 |
feat(gcp-cloud-run): Google Cloud Run + Workflows distributed render adapter (#1253)
* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda (issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble) are unchanged; this package is the storage/compute/orchestration glue. Package: Cloud Run handler (one image, three actions), runs under bun; GCS transport; in-image chrome-headless-shell resolver; client SDK (renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile; Cloud Workflows definition; Terraform module; CLI cloudrun deploy|sites|render|render-batch|progress|destroy with --output-resolution and --strict-variables; 62 unit tests + docs + live smoke script. Shared extraction (removes ~640 lines of adapter duplication): move the cloud-agnostic config validator + content-hash into producer/distributed; both adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`, failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run to the root `build` filter so its dist exists for publish + runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install The regression test image runs `bun install --frozen-lockfile` after copying each workspace package.json individually. The CLI now depends on @hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to resolve it unless its manifest is present. Add the COPY line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): add machine-sizing flags to `cloudrun deploy` Closes the parity gap with `lambda deploy` (which exposes --memory etc.). `cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout into the Terraform apply; omitted flags keep the module defaults (4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address PR review (security, waste, limits, alerts) - server.ts: bucket-allowlist guard no longer fails open silently. Unset env logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces. - server.ts: stop double-shipping audio.aac. It already rides in the plan tarball every consumer downloads, so drop the redundant standalone upload (plan) + re-download/overwrite (assemble); assemble reads it from the untar, falling back to a supplied AudioGcsUri for compat. - server.ts: chunk extension via path.extname() instead of slice(lastIndexOf). - workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20) — Cloud Workflows hard-caps concurrent iterations at 20. - Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break the image rebuild. - terraform: add min_instances var (default 0); add a workflow-failure alert (finished_execution_count status=FAILED) alongside the request-count one. - costAccounting: document that displayCost excludes GCS storage/egress. Verified against the actual APIs: @google-cloud/workflows@4.4.0 ICreateExecutionRequest has no executionId (so the idempotency-token suggestion isn't available in this client); Workflows concurrency cap is 20; failure metric is workflows.googleapis.com/finished_execution_count (status label). 174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding - workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE → PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the opposite cause), misleading anyone triaging the alert. - workflow.yaml: forward Config.cfr to the assemble step (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler but never sent, so exact-CFR was silently off for every Cloud Run render. Uses the same `in`-operator guard already proven in the retryable predicate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(release): include gcp-cloud-run in set-version PACKAGES list set-version.ts (driven by release:prepare) bumps an explicit package list to the shared version on each release. gcp-cloud-run was wired into the build + publish.yml but missing here, so a release would leave it at a stale version and publish.yml would push the wrong version. Add it so the new package version-bumps + publishes in lockstep with the others. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
James Russo
|
b6b6b8ed3b |
docs(lambda): add migration guide + non-Lambda Dockerfile example (#915)
Two adopter-facing artifacts that close out Phase 6b's user-facing
surface:
- docs/deploy/migrating-to-hyperframes-lambda.mdx — side-by-side
concept mapping for users coming from another one-command-deploy
video renderer. Covers the verb mapping (deploy/render/progress/
destroy/sites/policies), composition format (plain HTML vs JSX),
render config, and a handful of intentional differences (no HDR
in distributed mode, no webm, gpu-mode=software requirement,
fail-closed font fetch, local stack-state files, narrow-after-
first-deploy IAM pattern). Closes with a migration checklist.
Per repo convention, no competitor framework is named anywhere
in the source — adopters self-identify.
- examples/k8s-jobs/Dockerfile.example + README.md — reference
Dockerfile for adopters who want to run distributed renders
outside AWS Lambda. Bakes Node 22 + chrome-headless-shell +
ffmpeg + the producer source. Deliberately not published to
a registry; adopters build it themselves so Chrome / ffmpeg /
producer versions stay pinned to the checkout they audited.
The README documents the typical K8s Jobs orchestration shape
that points adopters at packages/aws-lambda/src/handler.ts as
the reference adapter.
Migration guide registered under the existing Deploy group in
docs.json. .gitignore extended to negate the new examples/k8s-jobs/
path the same way examples/aws-lambda/ is negated.
No source code changes.
|
||
|
James Russo
|
aa58ebc048 |
feat(lambda): add real-AWS smoke + benchmark script (#908)
* feat(lambda): add real-AWS smoke + benchmark script
Phase 6.3 of the distributed rendering plan
(DISTRIBUTED-RENDERING-PLAN.md §11 Phase 6a). Local bash script that
deploys the PR 6.2 SAM template to your own AWS account, renders a
fixture composition through the Step Functions state machine at several
chunk counts, PSNR-compares each output against the in-process
baseline, and tears the stack down.
This is the gate that proves the architecture works on real Lambda
infrastructure. After it runs green and the numbers are good, Phase 6b
(CLI, CDK, docs) can proceed with confidence.
Script lives at examples/aws-lambda/scripts/smoke.sh. Defaults:
- fixture: mp4-h264-sdr
- chunk_counts: 2,4,8
- psnr-threshold: 50 dB
- region: us-east-1
- stack-name: hyperframes-lambda-smoke-<timestamp>
AWS credentials come from the standard resolution chain (env vars,
~/.aws/credentials, SSO, IMDS). Pin a specific profile via --profile or
AWS_PROFILE; the script doesn't ship a default.
Workflow:
1. Pre-flight: verify aws/sam/bun/ffmpeg/jq/zip on PATH; check
credentials via sts:GetCallerIdentity.
2. Build the PR 6.1 ZIP and run verify:zip-size.
3. sam validate --lint + sam deploy under a per-run stack name.
4. Zip the fixture's src/ and upload to the render bucket.
5. For each chunk count, start a Step Functions execution, poll for
completion (25-min cap), download the output mp4 and the execution
history JSON, ffmpeg-psnr against the LFS-tracked in-process
baseline, and append to results.json.
6. Gate on the PSNR threshold.
7. Empty the bucket + sam delete (unless --keep-stack).
Outputs land under ./lambda-smoke-artifacts/:
- results.json (chunkCount x wallClockMs x psnrAvgDb)
- renders/N<N>-output.mp4
- renders/N<N>-history.json (full Step Functions execution history)
Per-run stack name with concurrency-safe AWS resource isolation. Run
multiple smokes in parallel without races; teardown guards against
stale stacks via cleanup_and_exit on every failure path.
Distinction from CI: this is a maintainer-run gate, not part of regular
CI. The architecture's per-PR safety net is the local Docker-based
BeginFrame probe (PR 6.1) and the upcoming Lambda RIE smoke mode (PR
6.6). No GitHub Actions / OIDC / cross-account secrets required.
This is part of the 8-PR Phase 6 stack; PR 6.3 of 8 — the last PR of
Phase 6a (validation). Phase 6b (CLI + CDK + docs) starts once 6.3's
benchmark numbers come back.
* fix(lambda): address PR 880 review feedback
- Document wall-clock methodology bias inline (eval.sh header + README):
local timing includes bun + tsx + harness scaffolding while Lambda
timing measures pure SFN execution, so "speedup" is end-to-end CLI
experience, not renderer-vs-renderer.
- Add --iterations N (default 1) with median wall-clock reporting via
awk-side median. Cold-start variance is ±5-10s per chunk; single-
sample readings made the PR-body speedup table not ground truth.
- Add --reserved-concurrency flag to both scripts; default still 16 but
no longer hardcoded. Pass-through to ReservedConcurrency CFN param.
- README: cost-per-pass estimate for both scripts.
- Replace `sed -n '2,30p' "$0"` help with usage() heredoc in both
scripts — fragile to header reflows and didn't survive the comment
expansion this commit adds anyway.
- eval.sh RMS-level parser: add a third fallback (`RMS level:` with no
`dB` suffix) for older ffmpeg builds where astats predates the unit
tag. Word-boundary guards keep `RMS peak level` from being eaten.
* docs(lambda): drop internal plan-doc + Rio refs from smoke/eval scripts
|
||
|
James Russo
|
6664e54053 |
feat(lambda): add SAM template and sample events for AWS deployment (#907)
* feat(lambda): add SAM template and sample events for AWS deployment Phase 6.2 of the distributed rendering plan (DISTRIBUTED-RENDERING-PLAN.md §15). Reference SAM template for deploying HyperFrames distributed rendering on AWS — one Lambda function in three roles, choreographed by a Step Functions standard workflow with a Map state for parallel chunk rendering. Resources created by the template: - Lambda function pointing at the Phase 6.1 ZIP - Step Functions state machine: Plan -> Map(N) RenderChunk -> Assemble - S3 bucket for plan tarballs, chunk outputs, final mp4 - IAM role for the state machine - CloudWatch alarm guarding against runaway chunk invocations Retry policy: 4 attempts, 2s initial, 2x backoff, max 60s, with the typed non-retryable error codes from plan §9.3 explicitly opted out. CodeUri points at packages/aws-lambda/dist/handler.zip; sam deploy resolves the local path and uploads to a SAM-managed bucket on first deploy. Validated: sam validate --lint passes against the template. This is part of the 8-PR Phase 6 stack; PR 6.2 of 8. * fix(lambda): address PR 879 review feedback - Add CloudWatch alarms for Lambda Errors metric (5min window, threshold 1) and Step Functions ExecutionsFailed metric. The existing runaway- invocations alarm catches too-many-calls but missed silent per-chunk failures and retry-exhaustion. - Document VersioningConfiguration: Suspended tradeoff inline. Adopters treating the final mp4 as user-keepable should bump to Enabled. - Cost-allocation Tags on RenderBucket + Lambda Globals. - Lambda Tracing: Active so X-Ray spans don't terminate at the SF→Lambda boundary (the state machine already had tracing). - State-machine top-level TimeoutSeconds: 3600 as defensive ceiling on the whole choreography — catches Plan-retry storms before they hit individual task budgets. - AssertChunkCount Choice state: if Plan ever returns ChunkCount=0 the Map would silently iterate zero times and Assemble would receive an empty ChunkS3Uris[] producing an empty output. Fail-fast with typed PLAN_TOO_LARGE error instead. - Architecture comment: explicit x86_64-only constraint from @sparticuz/chromium so adopters trying Graviton don't get bitten. * docs(lambda): drop internal plan-doc refs from SAM example + template |