ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-08 02:36:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(gcp-cloud-run): Google Cloud Run + Workflows distributed render adapter (#1253)

Browse Source 
* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter

Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda
(issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble)
are unchanged; this package is the storage/compute/orchestration glue.

Package: Cloud Run handler (one image, three actions), runs under bun; GCS
transport; in-image chrome-headless-shell resolver; client SDK
(renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile;
Cloud Workflows definition; Terraform module; CLI cloudrun
deploy|sites|render|render-batch|progress|destroy with --output-resolution and
--strict-variables; 62 unit tests + docs + live smoke script.

Shared extraction (removes ~640 lines of adapter duplication): move the
cloud-agnostic config validator + content-hash into producer/distributed; both
adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build

The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`,
failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that
build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk
subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run
to the root `build` filter so its dist exists for publish + runtime.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install

The regression test image runs `bun install --frozen-lockfile` after copying
each workspace package.json individually. The CLI now depends on
@hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to
resolve it unless its manifest is present. Add the COPY line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(cli): add machine-sizing flags to `cloudrun deploy`

Closes the parity gap with `lambda deploy` (which exposes --memory etc.).
`cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout
into the Terraform apply; omitted flags keep the module defaults
(4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module
directly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(gcp-cloud-run): address PR review (security, waste, limits, alerts)

- server.ts: bucket-allowlist guard no longer fails open silently. Unset env
  logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces.
- server.ts: stop double-shipping audio.aac. It already rides in the plan
  tarball every consumer downloads, so drop the redundant standalone upload
  (plan) + re-download/overwrite (assemble); assemble reads it from the untar,
  falling back to a supplied AudioGcsUri for compat.
- server.ts: chunk extension via path.extname() instead of slice(lastIndexOf).
- workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20)
  — Cloud Workflows hard-caps concurrent iterations at 20.
- Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break
  the image rebuild.
- terraform: add min_instances var (default 0); add a workflow-failure alert
  (finished_execution_count status=FAILED) alongside the request-count one.
- costAccounting: document that displayCost excludes GCS storage/egress.

Verified against the actual APIs: @google-cloud/workflows@4.4.0
ICreateExecutionRequest has no executionId (so the idempotency-token suggestion
isn't available in this client); Workflows concurrency cap is 20; failure
metric is workflows.googleapis.com/finished_execution_count (status label).
174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding

- workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE →
  PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the
  opposite cause), misleading anyone triaging the alert.
- workflow.yaml: forward Config.cfr to the assemble step
  (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler
  but never sent, so exact-CFR was silently off for every Cloud Run render.
  Uses the same `in`-operator guard already proven in the retryable predicate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(release): include gcp-cloud-run in set-version PACKAGES list

set-version.ts (driven by release:prepare) bumps an explicit package list to
the shared version on each release. gcp-cloud-run was wired into the build +
publish.yml but missing here, so a release would leave it at a stale version
and publish.yml would push the wrong version. Add it so the new package
version-bumps + publishes in lockstep with the others.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
James Russo
2026-06-07 14:43:38 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 806b226b34
commit 4da567df22
60 changed files with 5782 additions and 362 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/aws-lambda/src/sdk/deploySite.ts
+3 -39
View File
@@ -12,12 +12,11 @@
* produce the same `siteId` and `HeadObject`-short-circuit the upload.
*/
import { mkdtempSync, readdirSync, readFileSync, rmSync, statSync } from "node:fs";
import { createHash } from "node:crypto";
import { mkdtempSync, rmSync, statSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, relative } from "node:path";
import { join } from "node:path";
import { HeadObjectCommand, S3Client } from "@aws-sdk/client-s3";
import { PLAN_PROJECT_DIR_SKIP_SEGMENTS } from "@hyperframes/producer/distributed";
import { hashProjectDir } from "@hyperframes/producer/distributed";
import { formatS3Uri, tarDirectory, uploadFileToS3 } from "../s3Transport.js";
/** Options for {@link deploySite}. */
@@ -110,41 +109,6 @@ export async function deploySite(opts: DeploySiteOptions): Promise<SiteHandle> {
}
}
/**
* SHA-256 over every regular file under `projectDir` (sorted by relative
* path) → 16-character hex prefix. The prefix is the `siteId`.
*
* The hash includes the relative path plus every byte of each file, so a
* same-bytes rename still yields a fresh id. We trim to 16 chars because
* the full 64 isn't useful in an S3 key for legibility.
*
* Reads are synchronous: project trees are typically tens of MB at most
* (HTML/CSS/JS plus a few composition assets), so the simpler shape wins
* over a streaming pipeline.
*/
function hashProjectDir(projectDir: string): string {
const hash = createHash("sha256");
const files: string[] = [];
function walk(dir: string, isRoot: boolean): void {
for (const entry of readdirSync(dir, { withFileTypes: true }).sort((a, b) =>
a.name < b.name ? -1 : a.name > b.name ? 1 : 0,
)) {
if (isRoot && PLAN_PROJECT_DIR_SKIP_SEGMENTS.has(entry.name)) continue;
const full = join(dir, entry.name);
if (entry.isDirectory()) walk(full, false);
else if (entry.isFile()) files.push(full);
}
}
walk(projectDir, true);
for (const file of files) {
const rel = relative(projectDir, file).replaceAll("\\", "/");
hash.update(rel);
hash.update("\0");
hash.update(readFileSync(file));
}
return hash.digest("hex").slice(0, 16);
}
async function headObject(
s3: S3Client,
bucket: string,
packages/aws-lambda/src/sdk/validateConfig.ts
+24 -300
View File
@@ -1,194 +1,32 @@
/**
* Client-side validation of `SerializableDistributedRenderConfig` so the
* SDK fails on shape errors with a typed `InvalidConfigError` *before* a
* Step Functions execution starts.
* Client-side validation for the AWS Lambda adapter.
*
* The producer's `plan` stage validates the same fields server-side, but a
* caller staring at "ExecutionFailed: BROWSER_GPU_NOT_SOFTWARE" five
* minutes after StartExecution has to dig through Step Functions history
* to learn that the renderToLambda call passed an unsupported format.
* Catching the obvious mistakes locally turns that wait into a synchronous
* throw.
*
* The check is deliberately narrow — it covers the *shape* errors any
* caller could have surfaced with `tsc` if they passed a literal, plus
* the `force-hdr` rejection (HDR mp4 isn't supported in distributed
* mode). webm was previously rejected here too; v0.7+ supports it via
* closed-GOP concat-copy. Anything deeper (font availability, plan
* size cap, GPU mode at runtime) needs the actual planner.
* The cloud-agnostic config-shape validation (`validateDistributedRenderConfig`,
* `validateVariablesPayload`, `InvalidConfigError`) lives in
* `@hyperframes/producer/distributed` and is shared with the other adapters.
* This module re-exports those and adds the one piece specific to Step
* Functions: the 256 KiB Standard-workflow execution-input size cap.
*/
import type { DistributedFormat } from "../formatExtension.js";
import type { SerializableDistributedRenderConfig } from "../events.js";
import { InvalidConfigError } from "@hyperframes/producer/distributed";
/** Thrown for any client-side `SerializableDistributedRenderConfig` violation. */
export class InvalidConfigError extends Error {
// Read via Error.prototype.toString; fallow can't see it.
// fallow-ignore-next-line unused-class-member
override readonly name = "InvalidConfigError";
/** Dotted JSON-pointer-ish path to the offending field, e.g. `config.fps`. */
readonly field: string;
constructor(field: string, message: string) {
super(`[validateConfig] ${field}: ${message}`);
this.field = field;
}
}
const ALLOWED_FPS = [24, 30, 60] as const;
const ALLOWED_FORMATS = [
"mp4",
"mov",
"png-sequence",
"webm",
] as const satisfies readonly DistributedFormat[];
const ALLOWED_CODECS = ["h264", "h265"] as const;
const ALLOWED_QUALITIES = ["draft", "standard", "high"] as const;
const ALLOWED_RUNTIME_CAPS = ["lambda", "temporal", "cloud-run-job", "k8s-job", "none"] as const;
const ALLOWED_HDR_MODES = ["auto", "force-sdr"] as const;
const MAX_DIMENSION = 7680;
const MIN_DIMENSION = 16;
const MAX_CHUNK_SIZE = 3600;
const MAX_PARALLEL_CHUNKS_CEILING = 256;
export {
InvalidConfigError,
validateDistributedRenderConfig,
validateVariablesPayload,
} from "@hyperframes/producer/distributed";
/**
* Throw an `InvalidConfigError` if `config` is not a valid
* `SerializableDistributedRenderConfig`. Returns the same reference on
* success so the call site reads:
*
* const validated = validateDistributedRenderConfig(input);
*/
export function validateDistributedRenderConfig(
config: SerializableDistributedRenderConfig,
): SerializableDistributedRenderConfig {
if (config === null || typeof config !== "object") {
throw new InvalidConfigError("config", "must be an object");
}
if (!ALLOWED_FPS.includes(config.fps as 24 | 30 | 60)) {
throw new InvalidConfigError(
"config.fps",
`must be one of ${ALLOWED_FPS.join(", ")}; got ${String(config.fps)}`,
);
}
validateIntDimension("config.width", config.width);
validateIntDimension("config.height", config.height);
if (!ALLOWED_FORMATS.includes(config.format)) {
throw new InvalidConfigError(
"config.format",
`must be one of ${ALLOWED_FORMATS.join(", ")}; got ${String(config.format)}`,
);
}
if (config.codec !== undefined) {
if (config.format !== "mp4") {
throw new InvalidConfigError(
"config.codec",
`is only valid with format="mp4"; got format=${String(config.format)}`,
);
}
if (!ALLOWED_CODECS.includes(config.codec)) {
throw new InvalidConfigError(
"config.codec",
`must be one of ${ALLOWED_CODECS.join(", ")}; got ${String(config.codec)}`,
);
}
}
if (config.quality !== undefined && !ALLOWED_QUALITIES.includes(config.quality)) {
throw new InvalidConfigError(
"config.quality",
`must be one of ${ALLOWED_QUALITIES.join(", ")}; got ${String(config.quality)}`,
);
}
if (config.crf !== undefined && config.bitrate !== undefined) {
throw new InvalidConfigError("config.crf", "is mutually exclusive with config.bitrate");
}
if (
config.crf !== undefined &&
(!Number.isInteger(config.crf) || config.crf < 0 || config.crf > 51)
) {
throw new InvalidConfigError("config.crf", `must be an integer in [0, 51]; got ${config.crf}`);
}
if (config.bitrate !== undefined && !/^\d+(\.\d+)?[kKmM]?$/.test(config.bitrate)) {
throw new InvalidConfigError(
"config.bitrate",
`must look like "10M" or "5000k"; got ${JSON.stringify(config.bitrate)}`,
);
}
if (config.chunkSize !== undefined) {
if (!Number.isInteger(config.chunkSize) || config.chunkSize < 1) {
throw new InvalidConfigError(
"config.chunkSize",
`must be a positive integer; got ${config.chunkSize}`,
);
}
if (config.chunkSize > MAX_CHUNK_SIZE) {
throw new InvalidConfigError(
"config.chunkSize",
// Lambda 15-min cap leaves no useful headroom past ~3600 frames
// at 4 fps capture-equivalent throughput; rejecting up front
// avoids a 14-minute Plan-state retry storm.
`must be ≤ ${MAX_CHUNK_SIZE} (Lambda 15-min cap); got ${config.chunkSize}`,
);
}
}
if (config.maxParallelChunks !== undefined) {
if (!Number.isInteger(config.maxParallelChunks) || config.maxParallelChunks < 1) {
throw new InvalidConfigError(
"config.maxParallelChunks",
`must be a positive integer; got ${config.maxParallelChunks}`,
);
}
if (config.maxParallelChunks > MAX_PARALLEL_CHUNKS_CEILING) {
throw new InvalidConfigError(
"config.maxParallelChunks",
`must be ≤ ${MAX_PARALLEL_CHUNKS_CEILING}; got ${config.maxParallelChunks}`,
);
}
}
if (config.runtimeCap !== undefined && !ALLOWED_RUNTIME_CAPS.includes(config.runtimeCap)) {
throw new InvalidConfigError(
"config.runtimeCap",
`must be one of ${ALLOWED_RUNTIME_CAPS.join(", ")}; got ${String(config.runtimeCap)}`,
);
}
if (config.hdrMode !== undefined && !ALLOWED_HDR_MODES.includes(config.hdrMode)) {
// `force-hdr` is rejected here on top of the producer's plan-stage
// rejection — it makes the typical typo (`"force-hdr"` from a copy-
// paste of in-process config) surface synchronously instead of as a
// typed Step Functions failure two minutes in.
throw new InvalidConfigError(
"config.hdrMode",
`distributed mode supports only ${ALLOWED_HDR_MODES.join(", ")}; got ${String(config.hdrMode)}`,
);
}
if (config.variables !== undefined) {
validateVariablesPayload(config.variables);
}
return config;
}
/**
* Hard cap on Step Functions Standard workflow execution input — 256 KiB
* per the AWS limits page. Express workflows cap at 32 KiB; the render
* stack runs Standard for execution-history visibility, so the larger
* limit applies. The cap is on the entire serialized input, not just the
* variables, because users hit it at the wire boundary regardless of
* which field caused the bloat.
* Hard cap on Step Functions Standard workflow execution input — 256 KiB per
* the AWS limits page. Express workflows cap at 32 KiB; the render stack runs
* Standard for execution-history visibility, so the larger limit applies. The
* cap is on the entire serialized input, not just the variables, because
* users hit it at the wire boundary regardless of which field caused the
* bloat.
*
* Specific to Step Functions Standard. Other workflow runtimes (Temporal,
* Express SFN, raw Lambda invoke) have different caps; this constant
* shouldn't be reused for those without confirming the limit.
* Express SFN, Cloud Workflows, raw Lambda invoke) have different caps; don't
* reuse this constant for those without confirming the limit.
*/
export const MAX_STEP_FUNCTIONS_INPUT_BYTES = 256 * 1024;
@@ -197,10 +35,10 @@ const LARGE_VARIABLES_DOCS_URL =
"https://hyperframes.heygen.com/deploy/templates-on-lambda#working-with-large-variables";
/**
* Validate that the serialized Step Functions execution input fits inside
* the 256 KiB Standard-workflow cap. Measured in UTF-8 bytes (the format
* Step Functions uses on the wire) — JS strings count UTF-16 code units,
* which under-reports for any multi-byte character.
* Validate that the serialized Step Functions execution input fits inside the
* 256 KiB Standard-workflow cap. Measured in UTF-8 bytes (the format Step
* Functions uses on the wire) — JS strings count UTF-16 code units, which
* under-reports for any multi-byte character.
*
* Throws {@link InvalidConfigError} with a clear message naming the actual
* byte count, the cap, and a pointer to the "working with large variables"
@@ -214,17 +52,12 @@ export function validateStepFunctionsInputSize(input: unknown): void {
try {
serialized = JSON.stringify(input);
} catch (err) {
// JSON.stringify throws on circular refs and BigInt. The variables
// walker catches both inside `config.variables`, but a non-variables
// field could hit the same case in a future field addition.
throw new InvalidConfigError(
"config",
`Step Functions execution input is not JSON-serializable: ${err instanceof Error ? err.message : String(err)}`,
);
}
if (serialized === undefined) {
// JSON.stringify returns undefined for non-serializable roots
// (functions, Symbols at the top level).
throw new InvalidConfigError(
"config",
"Step Functions execution input is not JSON-serializable (JSON.stringify returned undefined). " +
@@ -244,112 +77,3 @@ export function validateStepFunctionsInputSize(input: unknown): void {
);
}
}
/**
* Validate that `variables` is a plain JSON-safe object — no functions,
* Symbols, `undefined` leaves, BigInts, non-finite numbers, or non-plain
* objects (Dates, Maps, Sets, class instances). Rejected values would
* either round-trip incorrectly through Step Functions (`undefined` is
* silently dropped by `JSON.stringify`) or throw at the wire boundary
* (`bigint`), so we surface the offending path synchronously.
*
* The check is purely structural — semantic constraints (e.g. "is this
* variable declared in `data-composition-variables`?") belong to the CLI
* layer where the project's HTML is on disk.
*/
export function validateVariablesPayload(value: unknown): void {
if (value === null || typeof value !== "object" || Array.isArray(value)) {
throw new InvalidConfigError(
"config.variables",
`must be a plain JSON object (got ${describeValue(value)})`,
);
}
walkVariables(value, "config.variables", new WeakSet());
}
/** Per-typeof rejection messages for JSON-unsafe leaves. */
const LEAF_REJECTIONS: Partial<Record<string, string>> = {
// `JSON.stringify` silently drops `undefined` leaves — caller would never
// notice their value isn't actually being sent.
undefined:
"undefined leaves are silently dropped by JSON.stringify — use null if you mean an absent value",
function: "functions are not JSON-serializable",
symbol: "Symbols are not JSON-serializable",
bigint: "BigInt values throw at JSON.stringify — encode as a string if you need 64-bit integers",
};
// fallow-ignore-next-line complexity
function walkVariables(value: unknown, path: string, seen: WeakSet<object>): void {
const t = typeof value;
if (value === null || t === "string" || t === "boolean") return;
if (t === "number") {
if (!Number.isFinite(value as number)) {
throw new InvalidConfigError(
path,
`non-finite numbers (NaN / Infinity) are not JSON-serializable; got ${String(value)}`,
);
}
return;
}
const leafReject = LEAF_REJECTIONS[t];
if (leafReject !== undefined) {
throw new InvalidConfigError(path, leafReject);
}
// t === "object" from here on. Reject circular refs up front — recursing
// through a back-edge would stack-overflow with no actionable error.
if (seen.has(value as object)) {
throw new InvalidConfigError(
path,
"circular reference detected — JSON.stringify cannot serialize cycles",
);
}
seen.add(value as object);
if (Array.isArray(value)) {
for (let i = 0; i < value.length; i++) {
walkVariables(value[i], `${path}[${i}]`, seen);
}
return;
}
// Reject non-plain objects (Date, Map, Set, class instances) up front.
// Date's `toJSON` does round-trip as a string, but the composition gets a
// string, not a Date — explicit reject is clearer than silent type-loss.
const proto = Object.getPrototypeOf(value);
if (proto !== Object.prototype && proto !== null) {
throw new InvalidConfigError(
path,
`non-plain objects are not supported (got ${describeValue(value)}); use a plain {…} object`,
);
}
for (const key of Object.keys(value as Record<string, unknown>)) {
walkVariables((value as Record<string, unknown>)[key], `${path}.${key}`, seen);
}
}
// fallow-ignore-next-line complexity
function describeValue(value: unknown): string {
if (value === null) return "null";
if (Array.isArray(value)) return "array";
if (typeof value !== "object") return typeof value;
// Class instances expose their constructor name; plain objects fall through
// to the generic "object" label. `Object.create(null)` has no constructor —
// treat its absent name the same as "Object" for reporting.
const ctorName = (value as { constructor?: { name?: string } }).constructor?.name ?? "Object";
return ctorName === "Object" ? "object" : ctorName;
}
function validateIntDimension(field: string, value: unknown): void {
if (typeof value !== "number" || !Number.isInteger(value)) {
throw new InvalidConfigError(field, `must be an integer; got ${String(value)}`);
}
if (value < MIN_DIMENSION || value > MAX_DIMENSION) {
throw new InvalidConfigError(
field,
`must be in [${MIN_DIMENSION}, ${MAX_DIMENSION}]; got ${value}`,
);
}
if (value % 2 !== 0) {
// libx264 / libx265 yuv420p require even dimensions; rejecting now
// beats a Plan-stage ffmpeg crash on dimension parity.
throw new InvalidConfigError(field, `must be even (yuv420p constraint); got ${value}`);
}
}