mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-05 17:30:50 +00:00
James RussoandClaude Opus 4.8
4da567df22
* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda (issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble) are unchanged; this package is the storage/compute/orchestration glue. Package: Cloud Run handler (one image, three actions), runs under bun; GCS transport; in-image chrome-headless-shell resolver; client SDK (renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile; Cloud Workflows definition; Terraform module; CLI cloudrun deploy|sites|render|render-batch|progress|destroy with --output-resolution and --strict-variables; 62 unit tests + docs + live smoke script. Shared extraction (removes ~640 lines of adapter duplication): move the cloud-agnostic config validator + content-hash into producer/distributed; both adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`, failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run to the root `build` filter so its dist exists for publish + runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install The regression test image runs `bun install --frozen-lockfile` after copying each workspace package.json individually. The CLI now depends on @hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to resolve it unless its manifest is present. Add the COPY line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): add machine-sizing flags to `cloudrun deploy` Closes the parity gap with `lambda deploy` (which exposes --memory etc.). `cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout into the Terraform apply; omitted flags keep the module defaults (4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address PR review (security, waste, limits, alerts) - server.ts: bucket-allowlist guard no longer fails open silently. Unset env logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces. - server.ts: stop double-shipping audio.aac. It already rides in the plan tarball every consumer downloads, so drop the redundant standalone upload (plan) + re-download/overwrite (assemble); assemble reads it from the untar, falling back to a supplied AudioGcsUri for compat. - server.ts: chunk extension via path.extname() instead of slice(lastIndexOf). - workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20) — Cloud Workflows hard-caps concurrent iterations at 20. - Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break the image rebuild. - terraform: add min_instances var (default 0); add a workflow-failure alert (finished_execution_count status=FAILED) alongside the request-count one. - costAccounting: document that displayCost excludes GCS storage/egress. Verified against the actual APIs: @google-cloud/workflows@4.4.0 ICreateExecutionRequest has no executionId (so the idempotency-token suggestion isn't available in this client); Workflows concurrency cap is 20; failure metric is workflows.googleapis.com/finished_execution_count (status label). 174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding - workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE → PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the opposite cause), misleading anyone triaging the alert. - workflow.yaml: forward Config.cfr to the assemble step (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler but never sent, so exact-CFR was silently off for every Cloud Run render. Uses the same `in`-operator guard already proven in the retryable predicate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(release): include gcp-cloud-run in set-version PACKAGES list set-version.ts (driven by release:prepare) bumps an explicit package list to the shared version on each release. gcp-cloud-run was wired into the build + publish.yml but missing here, so a release would leave it at a stale version and publish.yml would push the wrong version. Add it so the new package version-bumps + publishes in lockstep with the others. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
333 lines
12 KiB
TypeScript
333 lines
12 KiB
TypeScript
/**
|
|
* Handler dispatch + HTTP-shell unit tests.
|
|
*
|
|
* Asserts that:
|
|
* - `dispatch` routes Action="plan"/"renderChunk"/"assemble" to the
|
|
* matching OSS primitive and plumbs GCS download/upload around it.
|
|
* - It unwraps `{ Payload }` / `{ Input }` envelopes and rejects unknown
|
|
* actions.
|
|
* - The handler-boundary guards fire: plan-hash mismatch + bucket
|
|
* allowlist throw the typed, non-retryable errors.
|
|
* - `createApp` maps non-retryable errors → 400 and retryable → 500.
|
|
*
|
|
* The real OSS primitives are NOT exercised — they have their own coverage
|
|
* in `packages/producer`. This file pins the adapter glue's contract.
|
|
*/
|
|
|
|
import { afterEach, describe, expect, it } from "bun:test";
|
|
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import type { AssembleResult, ChunkResult, PlanResult } from "@hyperframes/producer/distributed";
|
|
import { asStorage, FakeGcs } from "./__fixtures__/fakeGcs.js";
|
|
import type { AssembleEvent, CloudRunEvent, PlanEvent, RenderChunkEvent } from "./events.js";
|
|
import { createApp, dispatch, type HandlerDeps, unwrapEvent } from "./server.js";
|
|
import { tarDirectory } from "./gcsTransport.js";
|
|
|
|
const tmpDirs: string[] = [];
|
|
function mkTmp(prefix: string): string {
|
|
const dir = mkdtempSync(join(tmpdir(), prefix));
|
|
tmpDirs.push(dir);
|
|
return dir;
|
|
}
|
|
afterEach(() => {
|
|
for (const d of tmpDirs.splice(0)) rmSync(d, { recursive: true, force: true });
|
|
});
|
|
|
|
const PLAN_HASH = "abc123planhash";
|
|
|
|
/** Build a real project tarball, seed it into the fake, return its URI. */
|
|
async function seedProjectTar(gcs: FakeGcs, uri: string): Promise<void> {
|
|
const src = mkTmp("hf-proj-");
|
|
writeFileSync(join(src, "index.html"), "<html></html>");
|
|
const tarPath = join(mkTmp("hf-proj-tar-"), "project.tar.gz");
|
|
await tarDirectory(src, tarPath);
|
|
gcs.seedFromFile(uri, tarPath);
|
|
}
|
|
|
|
/** Build a real plan tarball containing plan.json, seed it, return its URI. */
|
|
async function seedPlanTar(gcs: FakeGcs, uri: string, planHash: string): Promise<void> {
|
|
const planDir = mkTmp("hf-plan-");
|
|
writeFileSync(join(planDir, "plan.json"), JSON.stringify({ planHash }));
|
|
const tarPath = join(mkTmp("hf-plan-tar-"), "plan.tar.gz");
|
|
await tarDirectory(planDir, tarPath);
|
|
gcs.seedFromFile(uri, tarPath);
|
|
}
|
|
|
|
const planResult: PlanResult = {
|
|
planDir: "(set at call time)",
|
|
planHash: PLAN_HASH,
|
|
chunkCount: 3,
|
|
totalFrames: 90,
|
|
fps: 30,
|
|
width: 1920,
|
|
height: 1080,
|
|
format: "mp4",
|
|
ffmpegVersion: "ffmpeg version 6.1.1",
|
|
producerVersion: "0.6.79",
|
|
};
|
|
|
|
function depsWith(
|
|
gcs: FakeGcs,
|
|
overrides: Partial<NonNullable<HandlerDeps["primitives"]>> = {},
|
|
): HandlerDeps {
|
|
const plan = async (_projectDir: string, _config: unknown, planDir: string) => {
|
|
mkdirSync(planDir, { recursive: true });
|
|
writeFileSync(join(planDir, "plan.json"), JSON.stringify({ planHash: PLAN_HASH }));
|
|
return planResult;
|
|
};
|
|
const renderChunk = async (_planDir: string, chunkIndex: number, outputBase: string) => {
|
|
writeFileSync(outputBase, Buffer.from(`chunk-${chunkIndex}`));
|
|
return {
|
|
outputPath: outputBase,
|
|
outputKind: "file",
|
|
framesEncoded: 30,
|
|
sha256: `sha-${chunkIndex}`,
|
|
} satisfies ChunkResult;
|
|
};
|
|
const assemble = async (
|
|
_planDir: string,
|
|
_chunkPaths: string[],
|
|
_audio: string | null,
|
|
finalOutput: string,
|
|
) => {
|
|
writeFileSync(finalOutput, Buffer.from("final-output"));
|
|
return { framesEncoded: 90, fileSize: 12 } satisfies AssembleResult;
|
|
};
|
|
return {
|
|
storage: asStorage(gcs),
|
|
skipChromeResolution: true,
|
|
primitives: { plan, renderChunk, assemble, ...overrides } as NonNullable<
|
|
HandlerDeps["primitives"]
|
|
>,
|
|
};
|
|
}
|
|
|
|
describe("unwrapEvent", () => {
|
|
const plan: PlanEvent = {
|
|
Action: "plan",
|
|
ProjectGcsUri: "gs://b/p.tar.gz",
|
|
PlanOutputGcsPrefix: "gs://b/out/",
|
|
Config: { fps: 30, width: 1920, height: 1080, format: "mp4" } as PlanEvent["Config"],
|
|
};
|
|
|
|
it("returns a bare event unchanged", () => {
|
|
expect(unwrapEvent(plan).Action).toBe("plan");
|
|
});
|
|
|
|
it("unwraps { Payload }", () => {
|
|
expect(unwrapEvent({ Payload: plan } as CloudRunEvent).Action).toBe("plan");
|
|
});
|
|
|
|
it("unwraps nested { Input: { Payload } }", () => {
|
|
expect(unwrapEvent({ Input: { Payload: plan } } as CloudRunEvent).Action).toBe("plan");
|
|
});
|
|
|
|
it("throws when no Action is found", () => {
|
|
expect(() => unwrapEvent({ foo: "bar" } as unknown as CloudRunEvent)).toThrow(
|
|
/no recognised Action/,
|
|
);
|
|
});
|
|
});
|
|
|
|
describe("dispatch", () => {
|
|
it("routes plan, uploads the plan tarball", async () => {
|
|
const gcs = new FakeGcs();
|
|
await seedProjectTar(gcs, "gs://b/sites/x/project.tar.gz");
|
|
const event: PlanEvent = {
|
|
Action: "plan",
|
|
ProjectGcsUri: "gs://b/sites/x/project.tar.gz",
|
|
PlanOutputGcsPrefix: "gs://b/renders/r1/",
|
|
Config: { fps: 30, width: 1920, height: 1080, format: "mp4" } as PlanEvent["Config"],
|
|
};
|
|
const res = await dispatch(event, depsWith(gcs));
|
|
expect(res.Action).toBe("plan");
|
|
if (res.Action !== "plan") throw new Error("unreachable");
|
|
expect(res.PlanHash).toBe(PLAN_HASH);
|
|
expect(res.ChunkCount).toBe(3);
|
|
expect(res.PlanGcsUri).toBe("gs://b/renders/r1/plan.tar.gz");
|
|
expect(gcs.objects.has("gs://b/renders/r1/plan.tar.gz")).toBe(true);
|
|
});
|
|
|
|
it("routes renderChunk and uploads the chunk", async () => {
|
|
const gcs = new FakeGcs();
|
|
await seedPlanTar(gcs, "gs://b/renders/r1/plan.tar.gz", PLAN_HASH);
|
|
const event: RenderChunkEvent = {
|
|
Action: "renderChunk",
|
|
PlanGcsUri: "gs://b/renders/r1/plan.tar.gz",
|
|
PlanHash: PLAN_HASH,
|
|
ChunkIndex: 2,
|
|
ChunkOutputGcsPrefix: "gs://b/renders/r1/",
|
|
Format: "mp4",
|
|
};
|
|
const res = await dispatch(event, depsWith(gcs));
|
|
if (res.Action !== "renderChunk") throw new Error("unreachable");
|
|
expect(res.ChunkIndex).toBe(2);
|
|
expect(res.ChunkGcsUri).toBe("gs://b/renders/r1/chunks/0002.mp4");
|
|
expect(gcs.objects.has("gs://b/renders/r1/chunks/0002.mp4")).toBe(true);
|
|
});
|
|
|
|
it("throws PLAN_HASH_MISMATCH when the event hash disagrees", async () => {
|
|
const gcs = new FakeGcs();
|
|
await seedPlanTar(gcs, "gs://b/renders/r1/plan.tar.gz", PLAN_HASH);
|
|
const event: RenderChunkEvent = {
|
|
Action: "renderChunk",
|
|
PlanGcsUri: "gs://b/renders/r1/plan.tar.gz",
|
|
PlanHash: "WRONG_HASH",
|
|
ChunkIndex: 0,
|
|
ChunkOutputGcsPrefix: "gs://b/renders/r1/",
|
|
Format: "mp4",
|
|
};
|
|
await expect(dispatch(event, depsWith(gcs))).rejects.toThrow(/PLAN_HASH_MISMATCH/);
|
|
});
|
|
|
|
it("routes assemble and uploads the final output", async () => {
|
|
const gcs = new FakeGcs();
|
|
await seedPlanTar(gcs, "gs://b/renders/r1/plan.tar.gz", PLAN_HASH);
|
|
gcs.seed("gs://b/renders/r1/chunks/0000.mp4", Buffer.from("c0"));
|
|
gcs.seed("gs://b/renders/r1/chunks/0001.mp4", Buffer.from("c1"));
|
|
const event: AssembleEvent = {
|
|
Action: "assemble",
|
|
PlanGcsUri: "gs://b/renders/r1/plan.tar.gz",
|
|
ChunkGcsUris: ["gs://b/renders/r1/chunks/0000.mp4", "gs://b/renders/r1/chunks/0001.mp4"],
|
|
AudioGcsUri: null,
|
|
OutputGcsUri: "gs://b/renders/r1/output.mp4",
|
|
Format: "mp4",
|
|
};
|
|
const res = await dispatch(event, depsWith(gcs));
|
|
if (res.Action !== "assemble") throw new Error("unreachable");
|
|
expect(res.FramesEncoded).toBe(90);
|
|
expect(gcs.objects.has("gs://b/renders/r1/output.mp4")).toBe(true);
|
|
});
|
|
|
|
it("rejects an unknown action", async () => {
|
|
const gcs = new FakeGcs();
|
|
await expect(
|
|
dispatch({ Action: "nope" } as unknown as CloudRunEvent, depsWith(gcs)),
|
|
).rejects.toThrow(/no recognised Action/);
|
|
});
|
|
});
|
|
|
|
describe("bucket allowlist guard", () => {
|
|
it("throws GCS_URI_NOT_ALLOWED for an off-bucket URI", async () => {
|
|
const gcs = new FakeGcs();
|
|
const prev = process.env.HYPERFRAMES_RENDER_BUCKET;
|
|
process.env.HYPERFRAMES_RENDER_BUCKET = "allowed-bucket";
|
|
try {
|
|
const event: RenderChunkEvent = {
|
|
Action: "renderChunk",
|
|
PlanGcsUri: "gs://evil-bucket/plan.tar.gz",
|
|
PlanHash: PLAN_HASH,
|
|
ChunkIndex: 0,
|
|
ChunkOutputGcsPrefix: "gs://allowed-bucket/renders/r1/",
|
|
Format: "mp4",
|
|
};
|
|
await expect(dispatch(event, depsWith(gcs))).rejects.toThrow(/GCS_URI_NOT_ALLOWED/);
|
|
} finally {
|
|
if (prev === undefined) delete process.env.HYPERFRAMES_RENDER_BUCKET;
|
|
else process.env.HYPERFRAMES_RENDER_BUCKET = prev;
|
|
}
|
|
});
|
|
|
|
it('treats HYPERFRAMES_RENDER_BUCKET="*" as an explicit opt-out (off-bucket allowed)', async () => {
|
|
const gcs = new FakeGcs();
|
|
await seedPlanTar(gcs, "gs://any-bucket/renders/r1/plan.tar.gz", PLAN_HASH);
|
|
const prev = process.env.HYPERFRAMES_RENDER_BUCKET;
|
|
process.env.HYPERFRAMES_RENDER_BUCKET = "*";
|
|
try {
|
|
const event: RenderChunkEvent = {
|
|
Action: "renderChunk",
|
|
PlanGcsUri: "gs://any-bucket/renders/r1/plan.tar.gz",
|
|
PlanHash: PLAN_HASH,
|
|
ChunkIndex: 0,
|
|
ChunkOutputGcsPrefix: "gs://any-bucket/renders/r1/",
|
|
Format: "mp4",
|
|
};
|
|
const res = await dispatch(event, depsWith(gcs));
|
|
expect(res.Action).toBe("renderChunk");
|
|
} finally {
|
|
if (prev === undefined) delete process.env.HYPERFRAMES_RENDER_BUCKET;
|
|
else process.env.HYPERFRAMES_RENDER_BUCKET = prev;
|
|
}
|
|
});
|
|
});
|
|
|
|
describe("createApp HTTP mapping", () => {
|
|
it("returns 200 with the result body on success", async () => {
|
|
const gcs = new FakeGcs();
|
|
await seedPlanTar(gcs, "gs://b/renders/r1/plan.tar.gz", PLAN_HASH);
|
|
const app = createApp(depsWith(gcs));
|
|
const res = await app.request("/", {
|
|
method: "POST",
|
|
headers: { "content-type": "application/json" },
|
|
body: JSON.stringify({
|
|
Action: "renderChunk",
|
|
PlanGcsUri: "gs://b/renders/r1/plan.tar.gz",
|
|
PlanHash: PLAN_HASH,
|
|
ChunkIndex: 0,
|
|
ChunkOutputGcsPrefix: "gs://b/renders/r1/",
|
|
Format: "mp4",
|
|
}),
|
|
});
|
|
expect(res.status).toBe(200);
|
|
const body = (await res.json()) as { Action: string };
|
|
expect(body.Action).toBe("renderChunk");
|
|
});
|
|
|
|
it("returns 400 for a non-retryable error (plan-hash mismatch)", async () => {
|
|
const gcs = new FakeGcs();
|
|
await seedPlanTar(gcs, "gs://b/renders/r1/plan.tar.gz", PLAN_HASH);
|
|
const app = createApp(depsWith(gcs));
|
|
const res = await app.request("/", {
|
|
method: "POST",
|
|
headers: { "content-type": "application/json" },
|
|
body: JSON.stringify({
|
|
Action: "renderChunk",
|
|
PlanGcsUri: "gs://b/renders/r1/plan.tar.gz",
|
|
PlanHash: "WRONG",
|
|
ChunkIndex: 0,
|
|
ChunkOutputGcsPrefix: "gs://b/renders/r1/",
|
|
Format: "mp4",
|
|
}),
|
|
});
|
|
expect(res.status).toBe(400);
|
|
const body = (await res.json()) as { error: string };
|
|
expect(body.error).toBe("PLAN_HASH_MISMATCH");
|
|
});
|
|
|
|
it("returns 500 for a retryable/unknown error", async () => {
|
|
const gcs = new FakeGcs(); // plan tar NOT seeded → download fails (retryable)
|
|
const app = createApp(depsWith(gcs));
|
|
const res = await app.request("/", {
|
|
method: "POST",
|
|
headers: { "content-type": "application/json" },
|
|
body: JSON.stringify({
|
|
Action: "renderChunk",
|
|
PlanGcsUri: "gs://b/renders/r1/missing.tar.gz",
|
|
PlanHash: PLAN_HASH,
|
|
ChunkIndex: 0,
|
|
ChunkOutputGcsPrefix: "gs://b/renders/r1/",
|
|
Format: "mp4",
|
|
}),
|
|
});
|
|
expect(res.status).toBe(500);
|
|
});
|
|
|
|
it("returns 400 when the body is not JSON", async () => {
|
|
const gcs = new FakeGcs();
|
|
const app = createApp(depsWith(gcs));
|
|
const res = await app.request("/", {
|
|
method: "POST",
|
|
headers: { "content-type": "application/json" },
|
|
body: "not json{",
|
|
});
|
|
expect(res.status).toBe(400);
|
|
});
|
|
|
|
it("healthz returns ok", async () => {
|
|
const app = createApp(depsWith(new FakeGcs()));
|
|
const res = await app.request("/healthz");
|
|
expect(res.status).toBe(200);
|
|
});
|
|
});
|