feat(gcp-cloud-run): Google Cloud Run + Workflows distributed render adapter (#1253)

* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda (issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble) are unchanged; this package is the storage/compute/orchestration glue. Package: Cloud Run handler (one image, three actions), runs under bun; GCS transport; in-image chrome-headless-shell resolver; client SDK (renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile; Cloud Workflows definition; Terraform module; CLI cloudrun deploy|sites|render|render-batch|progress|destroy with --output-resolution and --strict-variables; 62 unit tests + docs + live smoke script. Shared extraction (removes ~640 lines of adapter duplication): move the cloud-agnostic config validator + content-hash into producer/distributed; both adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`, failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run to the root `build` filter so its dist exists for publish + runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install The regression test image runs `bun install --frozen-lockfile` after copying each workspace package.json individually. The CLI now depends on @hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to resolve it unless its manifest is present. Add the COPY line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): add machine-sizing flags to `cloudrun deploy` Closes the parity gap with `lambda deploy` (which exposes --memory etc.). `cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout into the Terraform apply; omitted flags keep the module defaults (4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address PR review (security, waste, limits, alerts) - server.ts: bucket-allowlist guard no longer fails open silently. Unset env logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces. - server.ts: stop double-shipping audio.aac. It already rides in the plan tarball every consumer downloads, so drop the redundant standalone upload (plan) + re-download/overwrite (assemble); assemble reads it from the untar, falling back to a supplied AudioGcsUri for compat. - server.ts: chunk extension via path.extname() instead of slice(lastIndexOf). - workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20) — Cloud Workflows hard-caps concurrent iterations at 20. - Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break the image rebuild. - terraform: add min_instances var (default 0); add a workflow-failure alert (finished_execution_count status=FAILED) alongside the request-count one. - costAccounting: document that displayCost excludes GCS storage/egress. Verified against the actual APIs: @google-cloud/workflows@4.4.0 ICreateExecutionRequest has no executionId (so the idempotency-token suggestion isn't available in this client); Workflows concurrency cap is 20; failure metric is workflows.googleapis.com/finished_execution_count (status label). 174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding - workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE → PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the opposite cause), misleading anyone triaging the alert. - workflow.yaml: forward Config.cfr to the assemble step (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler but never sent, so exact-CFR was silently off for every Cloud Run render. Uses the same `in`-operator guard already proven in the retryable predicate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(release): include gcp-cloud-run in set-version PACKAGES list set-version.ts (driven by release:prepare) bumps an explicit package list to the shared version on each release. gcp-cloud-run was wired into the build + publish.yml but missing here, so a release would leave it at a stale version and publish.yml would push the wrong version. Add it so the new package version-bumps + publishes in lockstep with the others. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-09-11 06:30:03 +00:00 · 2026-06-07 14:43:38 -07:00
parent 806b226b34
commit 4da567df22
60 changed files with 5782 additions and 362 deletions
@@ -0,0 +1,116 @@
+/**
+ * In-memory `@google-cloud/storage` stand-in for the adapter's unit tests.
+ *
+ * Mimics just the surface the transport + deploySite use:
+ *   storage.bucket(name).file(key) → { createReadStream, exists, getMetadata }
+ *   storage.bucket(name).upload(localPath, { destination, contentType })
+ *
+ * Objects live in a Map keyed `bucket/key`. `upload` reads the real local
+ * file from disk (the handler writes real tarballs), so round-trips through
+ * tar/untar exercise the actual code path. Every op is recorded for
+ * sequence assertions.
+ */
+
+import { readFileSync, writeFileSync } from "node:fs";
+import { Readable } from "node:stream";
+import type { Storage } from "@google-cloud/storage";
+
+export interface FakeGcsOp {
+  kind: "download" | "upload" | "exists" | "getMetadata";
+  uri: string;
+  bytes?: number;
+}
+
+export class FakeGcs {
+  ops: FakeGcsOp[] = [];
+  objects = new Map<string, Buffer>();
+
+  // Accessed only through the `Storage` cast in tests, so fallow's static
+  // analysis can't see the reference.
+  // fallow-ignore-next-line unused-class-member
+  bucket(bucketName: string): FakeBucket {
+    return new FakeBucket(this, bucketName);
+  }
+
+  /** Seed an object directly (e.g. a pre-built project tarball). */
+  seed(uri: string, bytes: Buffer): void {
+    this.objects.set(uri, bytes);
+  }
+
+  /** Seed from a local file on disk. */
+  seedFromFile(uri: string, localPath: string): void {
+    this.objects.set(uri, readFileSync(localPath));
+  }
+}
+
+class FakeBucket {
+  constructor(
+    private readonly gcs: FakeGcs,
+    private readonly bucketName: string,
+  ) {}
+
+  get name(): string {
+    return this.bucketName;
+  }
+
+  file(key: string): FakeFile {
+    return new FakeFile(this.gcs, this.bucketName, key);
+  }
+
+  async upload(
+    localPath: string,
+    opts: { destination: string; contentType?: string },
+  ): Promise<unknown> {
+    const uri = `gs://${this.bucketName}/${opts.destination}`;
+    const bytes = readFileSync(localPath);
+    this.gcs.objects.set(uri, bytes);
+    this.gcs.ops.push({ kind: "upload", uri, bytes: bytes.length });
+    return [{}];
+  }
+}
+
+class FakeFile {
+  private readonly uri: string;
+  constructor(
+    private readonly gcs: FakeGcs,
+    bucketName: string,
+    key: string,
+  ) {
+    this.uri = `gs://${bucketName}/${key}`;
+  }
+
+  createReadStream(): Readable {
+    const bytes = this.gcs.objects.get(this.uri);
+    if (!bytes) {
+      const r = new Readable({ read() {} });
+      r.destroy(new Error(`FakeGcs: object not found: ${this.uri}`));
+      return r;
+    }
+    this.gcs.ops.push({ kind: "download", uri: this.uri, bytes: bytes.length });
+    return Readable.from([bytes]);
+  }
+
+  async exists(): Promise<[boolean]> {
+    const has = this.gcs.objects.has(this.uri);
+    this.gcs.ops.push({ kind: "exists", uri: this.uri });
+    return [has];
+  }
+
+  async getMetadata(): Promise<[{ size?: string | number; updated?: string }]> {
+    const bytes = this.gcs.objects.get(this.uri);
+    this.gcs.ops.push({ kind: "getMetadata", uri: this.uri });
+    return [{ size: bytes?.length ?? 0, updated: "2026-06-06T00:00:00.000Z" }];
+  }
+
+  /** Helper for tests that want to materialize an object to disk. */
+  writeToDisk(destPath: string): void {
+    const bytes = this.gcs.objects.get(this.uri);
+    if (!bytes) throw new Error(`FakeGcs: object not found: ${this.uri}`);
+    writeFileSync(destPath, bytes);
+  }
+}
+
+/** Cast so `deploySite({ storage: asStorage(fake) })` reads cleanly. */
+export function asStorage(fake: FakeGcs): Storage {
+  return fake as unknown as Storage;
+}