* perf(producer): distributed chunks download only the videos they show A project video under compiled/ that chunks extract their own frames from now goes only to the chunks that show it, like video-sources/ files. A chunk skips the readiness wait for a deferred video it does not show, so it no longer logs media_load_failed for a file it was never given. * fix(producer): cloud site uploads pack the files their id hashes, symlinks included * perf(aws-lambda): chunks read only the byte ranges of a video they need A chunk that extracts its own frames from a deferred project video now reads it from S3 by byte range through a short-lived pre-signed URL, instead of downloading and hashing the whole object first. S3 attests the object's stored SHA-256 before the URL is signed; every other artifact is still downloaded and verified as before. * perf(aws-lambda): upload project files once to a shared content-addressed area * chore(producer): keep planV2 comment share at the base level * fix(engine): verify the TLS certificate of https video inputs and never surface their signed query * fix(aws-lambda): renders re-upload a site's expired files instead of failing * perf(aws-lambda): the plan step probes project videos in place instead of downloading them * fix(engine): https video reads reconnect and time out, and redaction covers wrapped causes Signed-URL reads now pass -rw_timeout, -reconnect and -reconnect_delay_max (both Lambda builds accept them) and ffmpeg also -reconnect_on_network_error, so a transient S3 error or a stalled read fails fast or recovers instead of waiting out the process timeout. Query redaction now descends into error.cause. * fix(aws-lambda): share entry bundles and ratchet package budget * fix(aws-lambda): site ids uploaded before files.json still render A site id from an earlier sites create has only sites/<id>/project.tar.gz. renderToLambda now falls back to that tarball, which the plan step still reads, and when neither key exists it fails before starting with both keys named and the fix (run sites create again). * fix(aws-lambda): renders fail before upload on a stack too old to read sites A stack deployed before files.json sites untars the site file in its plan step and fails after four retries. renderToLambda now reads the state machine definition first and stops with STACK_TOO_OLD and the redeploy command when the plan task does not list SITE_MANIFEST_INVALID, which new SAM and CDK stacks now treat as terminal. The README's upgrade section drops the planProtocol v1 fallback, which hits the same failure. * docs(aws-lambda): document shared content storage and the upgrade path The deploy guide still described the tarball HeadObject short-circuit. It now covers the cas/sha256/ store and its 30-day lifecycle rule, the STACK_TOO_OLD, SITE_FILES_MISSING and SITE_NOT_FOUND errors, that old site ids keep working, and that every project uploads once more after the upgrade. A test keeps a troubleshooting entry for each pre-submit error. * fix(producer): chunks log each video they read by URL, without its query A chunk reading a video straight from the bucket left no trace in its log. It now logs one line per remote source with the path and the URL minus its query string, so signatures never reach the log. * fix(engine): ffmpeg errors never keep part of a signed URL's query A process's stderr tail was trimmed before its URLs were redacted, so a cut inside a signed URL left a query fragment the URL pattern no longer matched, and the fragment reached the error. The stderr tail is now redacted before every trim, and the redactor also strips a signing parameter cut off from its URL. * fix(aws-lambda): comments name the site manifest instead of a file path * fix(engine): stderr redaction survives chunk splits, and every ffmpeg spawn uses it The stderr tail was redacted on each chunk and then cut, so once a cut severed a long URL's scheme, the rest of its signature in the next chunk matched neither pattern. The tail is now kept raw with 2x slack and redacted only when the text is produced, then cut to size. The unused producer audioExtractor (its own spawn, unredacted stderr) is deleted and the parity harness's PCM decode goes through ManagedChildProcess; a test fails on any ffmpeg/ffprobe spawn in engine, producer or aws-lambda that does not. * chore(producer): record the audioExtractor deletion as intended * fix(engine): drop a signature fragment the stderr tail cut from its URL The raw tail is kept at twice the cap, but that cut can itself land inside a signature, and when redaction shrinks the rest below the cap the fragment survived. After a cut, the severed token is dropped before redaction. * refactor(engine): name the severed-token drop instead of commenting it * test(engine): a cut-off URL leaves nothing behind; aws-lambda size budget to measured * fix(producer): chunk URL-read tests use main's frame list; producer size budget to measured * chore(producer): size budget at the measured size, no headroom * chore(producer): size budget back to measured x1.05, the check's own rule
@hyperframes/gcp-cloud-run
Google Cloud Run + Cloud Workflows adapter for HyperFrames distributed
rendering. The OSS render primitives (plan → renderChunk × N →
assemble) are pure functions over local file paths; this package is the
deployment, orchestration, and storage glue that runs them on Google Cloud —
the GCP counterpart to @hyperframes/aws-lambda.
Two surfaces, one package:
- Server-side handler (
./server) — a Cloud Run HTTP service that dispatchesplan/renderChunk/assembleon the request body'sActionfield, bridging GCS ↔ the container's filesystem around each OSS primitive. This is what the bundledDockerfileruns. - Client-side SDK (
./sdk) —renderToCloudRun,getRenderProgress,deploySite,validateDistributedRenderConfig, andcomputeRenderCost. Call these from a Node process (CI, CLI, app backend) to drive a deployed stack without writing GCS / Workflows boilerplate.
The package is not a dependency of @hyperframes/producer; install it
separately.
Architecture
GCS bucket ←→ Cloud Run service (plan / renderChunk / assemble)
▲
│ OIDC-authenticated http.post, one per step
│
Cloud Workflows (Plan → parallel RenderChunk → Assemble)
- Plan downloads the project tarball and publishes either a legacy v1 planDir tarball or a v2 manifest plus content-addressed artifacts.
- RenderChunk runs in a parallel
forloop in the workflow, fanned out up to the plan's chunk count. Each invocation renders one chunk and uploads it. - Assemble downloads every chunk + audio, stitches the final deliverable, and uploads it.
Every step is a POST to the same Cloud Run URL with a different Action.
The workflow accumulates each step's small result body and returns
{ Plan, Chunks, Assemble } so getRenderProgress can read frame totals and
per-step durations on success.
Plan transport selection
Plan v2 is the default for new renders. When planProtocol is omitted,
renderToCloudRun sends an explicit PlanProtocol: "v2" so the SDK and
the deployed workflow agree:
await renderToCloudRun({
// ...project, bucket, workflow, service, and config...
});
V2 uses separate manifest and content-addressed artifact locators throughout the workflow. Unknown protocols and integrity failures fail closed; a render never mixes v1 and v2 artifacts.
The monolithic v1 transport remains available as deprecated compatibility by
passing planProtocol: "v1" explicitly.
Upgrade order
Redeploy the Cloud Run image and Cloud Workflows definition from the same new
package version before upgrading an application that calls
renderToCloudRun. Pause new renders and drain active workflow executions
during the infrastructure update. Older workflows can default omission to v1
or lack the v2 branch, while the new SDK sends explicit v2. If infrastructure
cannot be redeployed first, keep the previous SDK version or pass
planProtocol: "v1" explicitly until the Terraform/workflow redeploy is
complete.
Chrome runtime
Unlike the Lambda adapter — which fights a 250 MB ZIP ceiling and
decompresses @sparticuz/chromium into /tmp at runtime — Cloud Run runs a
container image. The Dockerfile installs the same pinned
chrome-headless-shell build and font set the production renderer uses, at a
fixed path, and exports HYPERFRAMES_CHROME_PATH. CDP-level BeginFrame
support is a binary/runtime capability, so the image build launches that
exact executable and requires an enable + warm-up + PNG-returning
HeadlessExperimental.beginFrame probe to pass. The end-to-end smoke also
requires every chunk to report effective CaptureMode: "beginframe", which
catches runtime fallback separately from build-time packaging. There is no
runtime decompression step and no packaging ceiling.
Deploying
The terraform/ module provisions everything: the GCS render bucket, the
Cloud Run service, the Cloud Workflows definition, two least-privilege
service accounts (the service reads/writes the bucket; the workflow invokes
the service), and a runaway-request alert.
# 1. Build + push the image (Cloud Build or local docker).
gcloud builds submit . \
--tag REGION-docker.pkg.dev/PROJECT/REPO/hyperframes-render:TAG
# 2. Apply the module.
terraform -chdir=node_modules/@hyperframes/gcp-cloud-run/terraform init
terraform -chdir=node_modules/@hyperframes/gcp-cloud-run/terraform apply \
-var project_id=PROJECT \
-var region=us-central1 \
-var image=REGION-docker.pkg.dev/PROJECT/REPO/hyperframes-render:TAG
Terraform outputs render_bucket_name, service_url, workflow_name, and
region — pass them straight into the SDK.
Using the SDK
import { renderToCloudRun, getRenderProgress } from "@hyperframes/gcp-cloud-run/sdk";
const handle = await renderToCloudRun({
projectDir: "./my-composition",
config: { fps: 30, width: 1920, height: 1080, format: "mp4" },
bucketName: "hyperframes-render-my-project", // from terraform output
projectId: "my-project",
location: "us-central1",
workflowId: "hyperframes-render",
serviceUrl: "https://hyperframes-render-abc.us-central1.run.app",
});
// Poll until done.
let progress = await getRenderProgress({ executionName: handle.executionName });
while (progress.status === "running") {
await new Promise((r) => setTimeout(r, 5000));
progress = await getRenderProgress({ executionName: handle.executionName });
}
console.log(progress.status, progress.outputFile, progress.costs.displayCost);
deploySite is called implicitly when you pass projectDir; call it
yourself to pre-upload once and reuse the siteHandle across many renders
(e.g. personalised template batches).
Running tests
bun test # unit tests over an in-memory GCS double — no network
bun run typecheck
The live end-to-end smoke (build image → terraform apply → render a fixture
through the workflow → PSNR-compare → destroy) lives at
examples/gcp-cloud-run/scripts/smoke.sh and needs a GCP project with
billing enabled.
What's still ahead
- Mid-flight per-chunk progress.
getRenderProgressreports coarserunningprogress and exact numbers on success. Reading the Cloud Workflows step-entries API would give per-chunk progress while the render is in flight; tracked as a follow-up. - Cloud Run Jobs / Firebase Functions variants. This first version targets Cloud Run services + Workflows (the closest analog to Lambda + Step Functions). The same handler runs unchanged under Cloud Run Jobs; only the orchestration trigger differs.