mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 12:54:29 +00:00
* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda (issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble) are unchanged; this package is the storage/compute/orchestration glue. Package: Cloud Run handler (one image, three actions), runs under bun; GCS transport; in-image chrome-headless-shell resolver; client SDK (renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile; Cloud Workflows definition; Terraform module; CLI cloudrun deploy|sites|render|render-batch|progress|destroy with --output-resolution and --strict-variables; 62 unit tests + docs + live smoke script. Shared extraction (removes ~640 lines of adapter duplication): move the cloud-agnostic config validator + content-hash into producer/distributed; both adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`, failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run to the root `build` filter so its dist exists for publish + runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install The regression test image runs `bun install --frozen-lockfile` after copying each workspace package.json individually. The CLI now depends on @hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to resolve it unless its manifest is present. Add the COPY line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): add machine-sizing flags to `cloudrun deploy` Closes the parity gap with `lambda deploy` (which exposes --memory etc.). `cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout into the Terraform apply; omitted flags keep the module defaults (4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address PR review (security, waste, limits, alerts) - server.ts: bucket-allowlist guard no longer fails open silently. Unset env logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces. - server.ts: stop double-shipping audio.aac. It already rides in the plan tarball every consumer downloads, so drop the redundant standalone upload (plan) + re-download/overwrite (assemble); assemble reads it from the untar, falling back to a supplied AudioGcsUri for compat. - server.ts: chunk extension via path.extname() instead of slice(lastIndexOf). - workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20) — Cloud Workflows hard-caps concurrent iterations at 20. - Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break the image rebuild. - terraform: add min_instances var (default 0); add a workflow-failure alert (finished_execution_count status=FAILED) alongside the request-count one. - costAccounting: document that displayCost excludes GCS storage/egress. Verified against the actual APIs: @google-cloud/workflows@4.4.0 ICreateExecutionRequest has no executionId (so the idempotency-token suggestion isn't available in this client); Workflows concurrency cap is 20; failure metric is workflows.googleapis.com/finished_execution_count (status label). 174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding - workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE → PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the opposite cause), misleading anyone triaging the alert. - workflow.yaml: forward Config.cfr to the assemble step (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler but never sent, so exact-CFR was silently off for every Cloud Run render. Uses the same `in`-operator guard already proven in the retryable predicate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(release): include gcp-cloud-run in set-version PACKAGES list set-version.ts (driven by release:prepare) bumps an explicit package list to the shared version on each release. gcp-cloud-run was wired into the build + publish.yml but missing here, so a release would leave it at a stale version and publish.yml would push the wrong version. Add it so the new package version-bumps + publishes in lockstep with the others. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
100 lines
5.5 KiB
Plaintext
100 lines
5.5 KiB
Plaintext
---
|
|
title: Google Cloud Run
|
|
description: "Deploy distributed HyperFrames rendering to Google Cloud Run + Cloud Workflows, and drive renders from a laptop or CI."
|
|
---
|
|
|
|
HyperFrames ships a Google Cloud deployment that mirrors the [AWS Lambda](/deploy/aws-lambda) one: a single Cloud Run service fronts a Cloud Workflows definition that fans renders out across many parallel chunk workers, with intermediate artifacts in Google Cloud Storage. The render primitives are identical — only the storage, compute, and orchestration adapters differ.
|
|
|
|
It's the right choice for teams already running their backend and storage on Google Cloud who want distributed HyperFrames rendering without adding AWS infrastructure.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ Cloud Workflows definition │
|
|
│ Plan → parallel(for chunk) RenderChunk → Assemble │
|
|
└──────────────────────────────────────────────────────────────────┘
|
|
│ OIDC-authenticated http.post per step
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ One Cloud Run service (packages/gcp-cloud-run/Dockerfile) │
|
|
│ dist/server.js │
|
|
│ ├─ Action="plan" → @hyperframes/producer/distributed │
|
|
│ ├─ Action="renderChunk" → @hyperframes/producer/distributed │
|
|
│ └─ Action="assemble" → @hyperframes/producer/distributed │
|
|
└──────────────────────────────────────────────────────────────────┘
|
|
│ GCS download / upload
|
|
▼
|
|
Google Cloud Storage bucket
|
|
```
|
|
|
|
Each workflow step `POST`s to the same Cloud Run URL with a different `Action`. The handler downloads its inputs from GCS into the container's filesystem, runs the matching OSS primitive, uploads the output back to GCS, and returns a small JSON result. The workflow accumulates every step's result and returns `{ Plan, Chunks, Assemble }`.
|
|
|
|
## Why Cloud Run is simpler than Lambda here
|
|
|
|
Cloud Run runs a container image, so the Chrome story collapses to a `Dockerfile` line. There's no 250 MB ZIP ceiling, no `@sparticuz/chromium` runtime decompression, and no packaging probe — the image installs the same pinned `chrome-headless-shell` build the production renderer uses. Cloud Run gen2 also gives more headroom than Lambda: up to a 60-minute request timeout and 32 GiB of memory.
|
|
|
|
## Deploy
|
|
|
|
The Terraform module at `packages/gcp-cloud-run/terraform` provisions the GCS bucket, the Cloud Run service, the Cloud Workflows definition, two least-privilege service accounts, and a runaway-request alert.
|
|
|
|
```bash
|
|
# 1. Build + push the render image.
|
|
gcloud builds submit . \
|
|
--tag us-central1-docker.pkg.dev/PROJECT/hyperframes/hyperframes-render:v1
|
|
|
|
# 2. Apply the module.
|
|
cd node_modules/@hyperframes/gcp-cloud-run/terraform
|
|
terraform init
|
|
terraform apply \
|
|
-var project_id=PROJECT \
|
|
-var region=us-central1 \
|
|
-var image=us-central1-docker.pkg.dev/PROJECT/hyperframes/hyperframes-render:v1
|
|
```
|
|
|
|
Terraform outputs `render_bucket_name`, `service_url`, `workflow_name`, and `region`. Pass those into the SDK.
|
|
|
|
<Note>
|
|
The target GCP project must have **billing enabled** — Cloud Run, Cloud Workflows, Artifact Registry, and Cloud Build are all billed services.
|
|
</Note>
|
|
|
|
## Render
|
|
|
|
```ts
|
|
import {
|
|
renderToCloudRun,
|
|
getRenderProgress,
|
|
} from "@hyperframes/gcp-cloud-run/sdk";
|
|
|
|
const handle = await renderToCloudRun({
|
|
projectDir: "./my-composition",
|
|
config: { fps: 30, width: 1920, height: 1080, format: "mp4" },
|
|
bucketName: "hyperframes-render-my-project",
|
|
projectId: "my-project",
|
|
location: "us-central1",
|
|
workflowId: "hyperframes-render",
|
|
serviceUrl: "https://hyperframes-render-abc.us-central1.run.app",
|
|
});
|
|
|
|
let progress = await getRenderProgress({ executionName: handle.executionName });
|
|
while (progress.status === "running") {
|
|
await new Promise((r) => setTimeout(r, 5000));
|
|
progress = await getRenderProgress({ executionName: handle.executionName });
|
|
}
|
|
console.log(progress.status, progress.outputFile, progress.costs.displayCost);
|
|
```
|
|
|
|
Templates with [variables](/concepts/variables) work the same way — declare `data-composition-variables` on the composition and pass `config.variables`. The Cloud Workflows execution argument is capped at 512 KiB, so pass media as URL references the composition resolves at render time rather than inlining base64.
|
|
|
|
## End-to-end smoke
|
|
|
|
`examples/gcp-cloud-run/scripts/smoke.sh` builds the image, applies the Terraform module, renders a fixture composition through the workflow at one or more chunk sizes, PSNR-compares each output against the in-process baseline, and tears the stack down.
|
|
|
|
```bash
|
|
examples/gcp-cloud-run/scripts/smoke.sh --project my-project --region us-central1
|
|
```
|
|
|
|
## Supported formats
|
|
|
|
Same as the distributed pipeline everywhere: `mp4` (H.264 / H.265), `mov` (ProRes), `webm` (VP9), and `png-sequence`. HDR mp4 is not supported in distributed mode.
|