mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-11 06:30:03 +00:00
feat(producer): add --mode=lambda-local to the regression harness (#913)
* feat(producer): add --mode=lambda-local to the regression harness
Third harness mode that drives the OSS @hyperframes/aws-lambda handler
through the exact event sequence Step Functions produces in
production:
handler({Action: "plan"}) → planDir tarball on fake S3
handler({Action: "renderChunk"}) × N → chunk artifacts on fake S3
handler({Action: "assemble"}) → final mp4/mov/png-sequence
The S3 client is a filesystem-backed fake (every s3://<bucket>/<key>
URI maps to <tempRoot>/s3/<key>), so the harness exercises the
handler's event-parsing + tar/S3 conventions + dispatch logic on top
of the underlying producer primitives. Regressions in event JSON
shape, S3 key layout, or plan-hash boundary checks now surface in
the same CI run as the in-process and distributed-simulated modes
without paying for a real AWS round-trip.
Deliberately NOT a Docker/RIE invocation — that would gate the
producer test suite on Docker-in-Docker support which most CI
runners lack. Real-ZIP-via-RIE tests live in
packages/aws-lambda/scripts/ (probe:beginframe) and the
maintainer-run smoke.sh.
Wired up via:
- HarnessMode union extended to include "lambda-local"
- parseHarnessModeFlag accepts --mode=lambda-local
- regression-harness.ts dispatches to runLambdaLocalRender for
the new mode, sharing the distributed-support gate +
pathology-floor threshold with distributed-simulated mode
- package.json scripts: test:lambda-local + docker:test:lambda-local
- producer.devDependencies += @hyperframes/aws-lambda (workspace)
- producer/tsconfig.json gains path mappings to self so the type
cycle through aws-lambda's source resolves at typecheck time
without needing producer to be pre-built
Tests: 3 new unit tests on parseHarnessModeFlag + resolveMinPsnrForMode
cover the new mode. End-to-end PSNR contract still runs through
Dockerfile.test (manual + CI).
* refactor(producer): /simplify pass on lambda-local harness imports
Three small cleanups on top of the lambda-local harness:
- Drop the unused createReadStream import + its `void` workaround
comment. The aws-lambda handler's tar / S3 transport pulls
createReadStream from its own imports; this file never references
it directly.
- Hoist the dynamic `await import("node:fs")` calls for
writeFileSync out of FilesystemBackedFakeS3.send into the static
import block. Repeated PutObject calls don't need to repay the
dynamic-import cost.
- Hoist the dynamic `await import("@hyperframes/aws-lambda")` call
for untarDirectory similarly. Drops the now-redundant duplicate
aws-lambda import statement.
The PutObject body branch also collapses: `body instanceof Buffer`
and `typeof body === "string"` both call writeFileSync identically,
so they share one branch.
No behavior changes.
* fix(producer): lazy-import lambda-local harness module
The static import of regression-harness-lambda-local.ts pulled
@hyperframes/aws-lambda (and its @aws-sdk/* + @sparticuz/chromium
transitive deps) at module-load time. Dockerfile.test only copies
the producer's own files into the container, so aws-lambda's src
isn't present at runtime — and even `--mode=in-process` failed:
Error [ERR_MODULE_NOT_FOUND]: Cannot find module
'/app/packages/producer/node_modules/@hyperframes/aws-lambda/src/index.ts'
imported from /app/packages/producer/src/regression-harness-lambda-local.ts
Load the module on demand instead. `--mode=lambda-local` callers
pay the import cost; the existing in-process and distributed-
simulated modes don't.
* fix(producer): address PR review on lambda-local harness
Three review items from Vai:
- `Config.width`/`Config.height` are now plumbed through
RunLambdaLocalInput rather than hardcoded inside
runLambdaLocalRender. Lambda-local's whole point is to catch
event-shape drift; if the handler ever starts honouring
Config.width/height (e.g. for canvas sizing), having those
values flow from the caller means the harness sees what the
fixture authored. The interface change makes the eventual
upgrade-to-real-fixture-resolution a one-line dispatch swap.
- Drop the dead `export type { Fps }` and its unused import
from @hyperframes/core. The module never re-exports it.
- The dispatch site in regression-harness.ts now passes 1920×1080
explicitly with a comment marking it as a placeholder until
the harness compiles the composition HTML up-front to surface
the authored data-width/data-height. distributed-simulated
mode uses the same placeholder internally, kept for parity.
No behavior change in the existing modes; lambda-local now has a
clear extension point for honouring fixture dimensions.
This commit is contained in:
@@ -0,0 +1,227 @@
|
||||
/**
|
||||
* Lambda-local render path for the regression harness.
|
||||
*
|
||||
* Drives the OSS `@hyperframes/aws-lambda` handler through the exact
|
||||
* sequence Step Functions invokes in production:
|
||||
*
|
||||
* handler({ Action: "plan" }) → planDir tarball on S3
|
||||
* handler({ Action: "renderChunk" }) × N → chunk artifacts on S3
|
||||
* handler({ Action: "assemble" }) → final mp4 / mov / png-seq
|
||||
*
|
||||
* The S3 client is a filesystem-backed fake: every `s3://test-bucket/<key>`
|
||||
* URI maps to `<tempRoot>/s3/<key>`. This means the harness exercises
|
||||
* the handler's event-parsing + tar / S3 layout + dispatch logic in
|
||||
* addition to the underlying producer primitives, catching regressions
|
||||
* (event JSON drift, S3 key conventions, plan-hash boundary checks)
|
||||
* that `distributed-simulated` mode wouldn't.
|
||||
*
|
||||
* `lambda-local` is **deliberately** not a Docker / RIE invocation —
|
||||
* that would gate the producer test suite on Docker-in-Docker support
|
||||
* which most CI runners lack. Real-ZIP-via-RIE tests live in
|
||||
* `packages/aws-lambda/scripts/` (`probe:beginframe`) and the
|
||||
* maintainer-run `smoke.sh`.
|
||||
*/
|
||||
|
||||
import {
|
||||
createWriteStream,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readFileSync,
|
||||
statSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
import { Readable } from "node:stream";
|
||||
import { downloadS3ObjectToFile, tarDirectory, untarDirectory } from "@hyperframes/aws-lambda";
|
||||
import { handler } from "@hyperframes/aws-lambda/handler";
|
||||
import type {
|
||||
AssembleEvent,
|
||||
AssembleLambdaResult,
|
||||
HandlerDeps,
|
||||
PlanEvent,
|
||||
PlanLambdaResult,
|
||||
RenderChunkEvent,
|
||||
RenderChunkLambdaResult,
|
||||
SerializableDistributedRenderConfig,
|
||||
} from "@hyperframes/aws-lambda";
|
||||
|
||||
/** Inputs for {@link runLambdaLocalRender}. Same contract as `runDistributedSimulatedRender`. */
|
||||
export interface RunLambdaLocalInput {
|
||||
projectDir: string;
|
||||
tempRoot: string;
|
||||
renderedOutputPath: string;
|
||||
fps: 24 | 30 | 60;
|
||||
/**
|
||||
* Width/height from the fixture's renderConfig. Forwarded directly to
|
||||
* the Lambda event so this mode catches drift if the handler ever
|
||||
* starts honouring `Config.width/height` for canvas sizing rather
|
||||
* than reading the composition's `data-width`/`data-height`. The
|
||||
* `distributed-simulated` mode hardcodes 1920×1080 because it
|
||||
* bypasses the event-serialization boundary; lambda-local goes
|
||||
* through it, which is the whole point.
|
||||
*/
|
||||
width: number;
|
||||
height: number;
|
||||
format: "mp4" | "mov" | "png-sequence";
|
||||
codec?: "h264" | "h265";
|
||||
chunkSize?: number;
|
||||
maxParallelChunks?: number;
|
||||
variables?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
const FAKE_BUCKET = "harness-lambda-local";
|
||||
|
||||
/** S3 URI helpers — keep the URI shape identical to what SFN uses in production. */
|
||||
function uri(key: string): string {
|
||||
return `s3://${FAKE_BUCKET}/${key}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run plan → renderChunk × N → assemble through the OSS handler with a
|
||||
* filesystem-backed fake S3. Output lands at `input.renderedOutputPath`.
|
||||
*/
|
||||
export async function runLambdaLocalRender(input: RunLambdaLocalInput): Promise<void> {
|
||||
const s3Root = join(input.tempRoot, "s3");
|
||||
mkdirSync(s3Root, { recursive: true });
|
||||
|
||||
// STEP 0: stage the project as a tar.gz at the fake-S3 path the Plan
|
||||
// event will reference, mirroring what `deploySite` does in prod.
|
||||
const projectKey = `sites/harness/${Date.now()}/project.tar.gz`;
|
||||
const projectS3Path = join(s3Root, projectKey);
|
||||
mkdirSync(dirname(projectS3Path), { recursive: true });
|
||||
await tarDirectory(input.projectDir, projectS3Path);
|
||||
|
||||
const fakeS3 = new FilesystemBackedFakeS3(s3Root);
|
||||
const deps: HandlerDeps = {
|
||||
s3: fakeS3 as unknown as HandlerDeps["s3"],
|
||||
// The handler resolves a Chrome path via `@sparticuz/chromium` by
|
||||
// default; that's the Lambda-specific binary. In Dockerfile.test
|
||||
// we want the producer's already-configured Chrome instead. The
|
||||
// skip flag tells the handler not to override PRODUCER_HEADLESS_SHELL_PATH.
|
||||
skipChromeResolution: true,
|
||||
tmpRoot: join(input.tempRoot, "lambda-tmp"),
|
||||
};
|
||||
mkdirSync(deps.tmpRoot as string, { recursive: true });
|
||||
|
||||
const config: SerializableDistributedRenderConfig = {
|
||||
fps: input.fps,
|
||||
width: input.width,
|
||||
height: input.height,
|
||||
format: input.format,
|
||||
...(input.format === "mp4" && input.codec !== undefined ? { codec: input.codec } : {}),
|
||||
chunkSize: input.chunkSize,
|
||||
maxParallelChunks: input.maxParallelChunks,
|
||||
hdrMode: "force-sdr",
|
||||
};
|
||||
|
||||
// STEP A: plan
|
||||
const planPrefix = `renders/harness/${Date.now()}/`;
|
||||
const planEvent: PlanEvent = {
|
||||
Action: "plan",
|
||||
ProjectS3Uri: uri(projectKey),
|
||||
PlanOutputS3Prefix: uri(planPrefix),
|
||||
Config: config,
|
||||
};
|
||||
const planResult = (await handler(planEvent, deps)) as PlanLambdaResult;
|
||||
|
||||
// STEP B: render every chunk through the handler.
|
||||
const chunkUris: string[] = [];
|
||||
for (let i = 0; i < planResult.ChunkCount; i++) {
|
||||
const chunkEvent: RenderChunkEvent = {
|
||||
Action: "renderChunk",
|
||||
PlanS3Uri: planResult.PlanS3Uri,
|
||||
PlanHash: planResult.PlanHash,
|
||||
ChunkIndex: i,
|
||||
ChunkOutputS3Prefix: uri(planPrefix),
|
||||
Format: input.format,
|
||||
};
|
||||
const chunkResult = (await handler(chunkEvent, deps)) as RenderChunkLambdaResult;
|
||||
chunkUris.push(chunkResult.ChunkS3Uri);
|
||||
}
|
||||
|
||||
// STEP C: assemble
|
||||
const finalUri = uri(
|
||||
`${planPrefix}output${input.format === "png-sequence" ? ".tar.gz" : `.${input.format}`}`,
|
||||
);
|
||||
const assembleEvent: AssembleEvent = {
|
||||
Action: "assemble",
|
||||
PlanS3Uri: planResult.PlanS3Uri,
|
||||
ChunkS3Uris: chunkUris,
|
||||
AudioS3Uri: planResult.AudioS3Uri,
|
||||
OutputS3Uri: finalUri,
|
||||
Format: input.format,
|
||||
};
|
||||
(await handler(assembleEvent, deps)) as AssembleLambdaResult;
|
||||
|
||||
// Copy the final output from fake-S3 land back out to the path the
|
||||
// harness expects. For png-sequence, untar into the dir.
|
||||
const finalKey = finalUri.slice(`s3://${FAKE_BUCKET}/`.length);
|
||||
if (input.format === "png-sequence") {
|
||||
const tarPath = join(s3Root, finalKey);
|
||||
mkdirSync(input.renderedOutputPath, { recursive: true });
|
||||
await untarDirectory(tarPath, input.renderedOutputPath);
|
||||
} else {
|
||||
await downloadS3ObjectToFile(
|
||||
fakeS3 as unknown as Parameters<typeof downloadS3ObjectToFile>[0],
|
||||
finalUri,
|
||||
input.renderedOutputPath,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Minimum AWS-SDK-shaped fake S3 the handler's `send(GetObject)` and
|
||||
* `send(PutObject)` calls land in. Stores blobs on the local filesystem
|
||||
* under `root/<key>` so the harness can pre-stage inputs (tarball'd
|
||||
* project) and post-inspect outputs (per-chunk artifacts, final video)
|
||||
* without going through a real S3 endpoint.
|
||||
*/
|
||||
class FilesystemBackedFakeS3 {
|
||||
constructor(private readonly root: string) {}
|
||||
|
||||
async send(command: unknown): Promise<unknown> {
|
||||
const cmdName = (command as { constructor: { name: string } }).constructor.name;
|
||||
const input = (command as { input: { Bucket: string; Key: string; Body?: unknown } }).input;
|
||||
const fsPath = join(this.root, input.Key);
|
||||
|
||||
if (cmdName === "GetObjectCommand") {
|
||||
if (!existsSync(fsPath)) {
|
||||
const err = new Error(
|
||||
`FakeS3: GetObject for missing key ${input.Bucket}/${input.Key}`,
|
||||
) as Error & {
|
||||
$metadata: { httpStatusCode: number };
|
||||
};
|
||||
err.$metadata = { httpStatusCode: 404 };
|
||||
throw err;
|
||||
}
|
||||
const bytes = readFileSync(fsPath);
|
||||
return { Body: Readable.from([bytes]) };
|
||||
}
|
||||
if (cmdName === "PutObjectCommand") {
|
||||
mkdirSync(dirname(fsPath), { recursive: true });
|
||||
const body = input.Body;
|
||||
if (body instanceof Buffer || typeof body === "string") {
|
||||
writeFileSync(fsPath, body);
|
||||
} else if (body && typeof (body as NodeJS.ReadableStream).pipe === "function") {
|
||||
await pipeline(body as NodeJS.ReadableStream, createWriteStream(fsPath));
|
||||
} else {
|
||||
throw new Error(`FakeS3: PutObject body shape not supported (${typeof body})`);
|
||||
}
|
||||
return { ETag: `"fake-${statSync(fsPath).size}"` };
|
||||
}
|
||||
if (cmdName === "HeadObjectCommand") {
|
||||
if (!existsSync(fsPath)) {
|
||||
const err = new Error(
|
||||
`FakeS3: HeadObject for missing key ${input.Bucket}/${input.Key}`,
|
||||
) as Error & {
|
||||
$metadata: { httpStatusCode: number };
|
||||
};
|
||||
err.$metadata = { httpStatusCode: 404 };
|
||||
throw err;
|
||||
}
|
||||
return { ContentLength: statSync(fsPath).size, LastModified: new Date() };
|
||||
}
|
||||
throw new Error(`FakeS3: unexpected command ${cmdName}`);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user