feat(producer): add --mode=lambda-local to the regression harness (#913)

* feat(producer): add --mode=lambda-local to the regression harness Third harness mode that drives the OSS @hyperframes/aws-lambda handler through the exact event sequence Step Functions produces in production: handler({Action: "plan"}) → planDir tarball on fake S3 handler({Action: "renderChunk"}) × N → chunk artifacts on fake S3 handler({Action: "assemble"}) → final mp4/mov/png-sequence The S3 client is a filesystem-backed fake (every s3://<bucket>/<key> URI maps to <tempRoot>/s3/<key>), so the harness exercises the handler's event-parsing + tar/S3 conventions + dispatch logic on top of the underlying producer primitives. Regressions in event JSON shape, S3 key layout, or plan-hash boundary checks now surface in the same CI run as the in-process and distributed-simulated modes without paying for a real AWS round-trip. Deliberately NOT a Docker/RIE invocation — that would gate the producer test suite on Docker-in-Docker support which most CI runners lack. Real-ZIP-via-RIE tests live in packages/aws-lambda/scripts/ (probe:beginframe) and the maintainer-run smoke.sh. Wired up via: - HarnessMode union extended to include "lambda-local" - parseHarnessModeFlag accepts --mode=lambda-local - regression-harness.ts dispatches to runLambdaLocalRender for the new mode, sharing the distributed-support gate + pathology-floor threshold with distributed-simulated mode - package.json scripts: test:lambda-local + docker:test:lambda-local - producer.devDependencies += @hyperframes/aws-lambda (workspace) - producer/tsconfig.json gains path mappings to self so the type cycle through aws-lambda's source resolves at typecheck time without needing producer to be pre-built Tests: 3 new unit tests on parseHarnessModeFlag + resolveMinPsnrForMode cover the new mode. End-to-end PSNR contract still runs through Dockerfile.test (manual + CI). * refactor(producer): /simplify pass on lambda-local harness imports Three small cleanups on top of the lambda-local harness: - Drop the unused createReadStream import + its `void` workaround comment. The aws-lambda handler's tar / S3 transport pulls createReadStream from its own imports; this file never references it directly. - Hoist the dynamic `await import("node:fs")` calls for writeFileSync out of FilesystemBackedFakeS3.send into the static import block. Repeated PutObject calls don't need to repay the dynamic-import cost. - Hoist the dynamic `await import("@hyperframes/aws-lambda")` call for untarDirectory similarly. Drops the now-redundant duplicate aws-lambda import statement. The PutObject body branch also collapses: `body instanceof Buffer` and `typeof body === "string"` both call writeFileSync identically, so they share one branch. No behavior changes. * fix(producer): lazy-import lambda-local harness module The static import of regression-harness-lambda-local.ts pulled @hyperframes/aws-lambda (and its @aws-sdk/* + @sparticuz/chromium transitive deps) at module-load time. Dockerfile.test only copies the producer's own files into the container, so aws-lambda's src isn't present at runtime — and even `--mode=in-process` failed: Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/app/packages/producer/node_modules/@hyperframes/aws-lambda/src/index.ts' imported from /app/packages/producer/src/regression-harness-lambda-local.ts Load the module on demand instead. `--mode=lambda-local` callers pay the import cost; the existing in-process and distributed- simulated modes don't. * fix(producer): address PR review on lambda-local harness Three review items from Vai: - `Config.width`/`Config.height` are now plumbed through RunLambdaLocalInput rather than hardcoded inside runLambdaLocalRender. Lambda-local's whole point is to catch event-shape drift; if the handler ever starts honouring Config.width/height (e.g. for canvas sizing), having those values flow from the caller means the harness sees what the fixture authored. The interface change makes the eventual upgrade-to-real-fixture-resolution a one-line dispatch swap. - Drop the dead `export type { Fps }` and its unused import from @hyperframes/core. The module never re-exports it. - The dispatch site in regression-harness.ts now passes 1920×1080 explicitly with a comment marking it as a placeholder until the harness compiles the composition HTML up-front to surface the authored data-width/data-height. distributed-simulated mode uses the same placeholder internally, kept for parity. No behavior change in the existing modes; lambda-local now has a clear extension point for honouring fixture dimensions.
2026-09-04 07:19:52 +00:00 · 2026-05-17 14:04:49 -04:00
parent 62ddd29b74
commit b05a22e69e
7 changed files with 312 additions and 20 deletions
@@ -50,6 +50,8 @@
    "docker:test": "docker run --rm --security-opt seccomp=unconfined --shm-size=2g -v ./tests:/app/packages/producer/tests hyperframes-producer:test",
    "docker:test:update": "docker run --rm --security-opt seccomp=unconfined --shm-size=2g -v ./tests:/app/packages/producer/tests hyperframes-producer:test --update",
    "docker:test:distributed": "docker run --rm --security-opt seccomp=unconfined --shm-size=2g -v ./tests:/app/packages/producer/tests hyperframes-producer:test --mode=distributed-simulated",
+    "test:lambda-local": "tsx src/regression-harness.ts --exclude-tags transparency --mode=lambda-local",
+    "docker:test:lambda-local": "docker run --rm --security-opt seccomp=unconfined --shm-size=2g -v ./tests:/app/packages/producer/tests hyperframes-producer:test --mode=lambda-local",
    "prepublishOnly": "echo skip"
  },
  "dependencies": {
@@ -81,6 +83,7 @@
    "@fontsource/poppins": "^5.2.7",
    "@fontsource/roboto": "^5.2.10",
    "@fontsource/source-code-pro": "^5.2.7",
+    "@hyperframes/aws-lambda": "workspace:*",
    "@types/node": "^25.0.10",
    "@webgpu/types": "^0.1.69",
    "esbuild": "^0.25.12",
@@ -18,6 +18,10 @@ describe("parseHarnessModeFlag()", () => {
    expect(parseHarnessModeFlag("--mode=distributed-simulated")).toBe("distributed-simulated");
  });

+  it("parses --mode=lambda-local", () => {
+    expect(parseHarnessModeFlag("--mode=lambda-local")).toBe("lambda-local");
+  });
+
  it("returns null for tokens that aren't --mode", () => {
    expect(parseHarnessModeFlag("--update")).toBeNull();
    expect(parseHarnessModeFlag("font-variant-numeric")).toBeNull();
@@ -28,6 +32,11 @@ describe("parseHarnessModeFlag()", () => {
    expect(() => parseHarnessModeFlag("--mode=foo")).toThrow(/--mode must be/);
    expect(() => parseHarnessModeFlag("--mode=")).toThrow(/--mode must be/);
  });
+
+  it("error message lists all three accepted modes", () => {
+    expect(() => parseHarnessModeFlag("--mode=foo")).toThrow(/lambda-local/);
+    expect(() => parseHarnessModeFlag("--mode=foo")).toThrow(/distributed-simulated/);
+  });
 });

 describe("checkDistributedSupport()", () => {
@@ -113,6 +122,13 @@ describe("resolveMinPsnrForMode()", () => {
    );
  });

+  it("lambda-local mirrors distributed-simulated's pathology floor", () => {
+    // Both non-in-process modes run through the same producer primitives,
+    // so they share the same pathology threshold.
+    expect(resolveMinPsnrForMode("lambda-local", 30)).toBe(30);
+    expect(resolveMinPsnrForMode("lambda-local", 0)).toBe(DISTRIBUTED_SIMULATED_MIN_PSNR_DB);
+  });
+
  it("every committed fixture authors a minPsnr above the absolute floor", async () => {
    // The pathology floor only fires for a fixture whose authored minPsnr
    // is below it — by design that should be no committed fixture. If
@@ -36,8 +36,20 @@ import { join } from "node:path";
 import type { Fps } from "@hyperframes/core";
 import { assemble, plan, renderChunk } from "./distributed.js";

-/** Two-mode contract that backs `--mode=<value>` on the regression harness CLI. */
-export type HarnessMode = "in-process" | "distributed-simulated";
+/**
+ * Three-mode contract that backs `--mode=<value>` on the regression
+ * harness CLI:
+ *
+ *   - `in-process` — `executeRenderJob`, the same path the CLI takes.
+ *   - `distributed-simulated` — `plan` → `renderChunk` × N → `assemble`
+ *     in-process. No adapter (no Temporal, no Lambda).
+ *   - `lambda-local` — drives the OSS `@hyperframes/aws-lambda` handler
+ *     dispatch through a filesystem-backed fake S3, so every event
+ *     shape SFN sends in production also lands here. Catches regressions
+ *     in event JSON / S3 path conventions without paying for a real AWS
+ *     round-trip.
+ */
+export type HarnessMode = "in-process" | "distributed-simulated" | "lambda-local";

 /**
 * Absolute pathology floor for `--mode=distributed-simulated` — catches
@@ -207,6 +219,8 @@ export async function runDistributedSimulatedRender(
 */
 export function resolveMinPsnrForMode(mode: HarnessMode, fixtureMinPsnr: number): number {
  if (mode === "in-process") return fixtureMinPsnr;
+  // `lambda-local` shares the distributed-simulated pathology floor —
+  // both modes go through the same plan/renderChunk/assemble primitives.
  return Math.max(fixtureMinPsnr, DISTRIBUTED_SIMULATED_MIN_PSNR_DB);
 }

@@ -220,10 +234,11 @@ export function resolveMinPsnrForMode(mode: HarnessMode, fixtureMinPsnr: number)
 export function parseHarnessModeFlag(token: string): HarnessMode | null {
  if (token === "--mode=in-process") return "in-process";
  if (token === "--mode=distributed-simulated") return "distributed-simulated";
+  if (token === "--mode=lambda-local") return "lambda-local";
  if (token.startsWith("--mode=")) {
    const value = token.slice("--mode=".length);
    throw new Error(
-      `regression-harness: --mode must be 'in-process' or 'distributed-simulated' (got ${JSON.stringify(value)})`,
+      `regression-harness: --mode must be 'in-process', 'distributed-simulated', or 'lambda-local' (got ${JSON.stringify(value)})`,
    );
  }
  return null;
@@ -0,0 +1,227 @@
+/**
+ * Lambda-local render path for the regression harness.
+ *
+ * Drives the OSS `@hyperframes/aws-lambda` handler through the exact
+ * sequence Step Functions invokes in production:
+ *
+ *     handler({ Action: "plan" })              → planDir tarball on S3
+ *     handler({ Action: "renderChunk" }) × N   → chunk artifacts on S3
+ *     handler({ Action: "assemble" })          → final mp4 / mov / png-seq
+ *
+ * The S3 client is a filesystem-backed fake: every `s3://test-bucket/<key>`
+ * URI maps to `<tempRoot>/s3/<key>`. This means the harness exercises
+ * the handler's event-parsing + tar / S3 layout + dispatch logic in
+ * addition to the underlying producer primitives, catching regressions
+ * (event JSON drift, S3 key conventions, plan-hash boundary checks)
+ * that `distributed-simulated` mode wouldn't.
+ *
+ * `lambda-local` is **deliberately** not a Docker / RIE invocation —
+ * that would gate the producer test suite on Docker-in-Docker support
+ * which most CI runners lack. Real-ZIP-via-RIE tests live in
+ * `packages/aws-lambda/scripts/` (`probe:beginframe`) and the
+ * maintainer-run `smoke.sh`.
+ */
+
+import {
+  createWriteStream,
+  existsSync,
+  mkdirSync,
+  readFileSync,
+  statSync,
+  writeFileSync,
+} from "node:fs";
+import { dirname, join } from "node:path";
+import { pipeline } from "node:stream/promises";
+import { Readable } from "node:stream";
+import { downloadS3ObjectToFile, tarDirectory, untarDirectory } from "@hyperframes/aws-lambda";
+import { handler } from "@hyperframes/aws-lambda/handler";
+import type {
+  AssembleEvent,
+  AssembleLambdaResult,
+  HandlerDeps,
+  PlanEvent,
+  PlanLambdaResult,
+  RenderChunkEvent,
+  RenderChunkLambdaResult,
+  SerializableDistributedRenderConfig,
+} from "@hyperframes/aws-lambda";
+
+/** Inputs for {@link runLambdaLocalRender}. Same contract as `runDistributedSimulatedRender`. */
+export interface RunLambdaLocalInput {
+  projectDir: string;
+  tempRoot: string;
+  renderedOutputPath: string;
+  fps: 24 | 30 | 60;
+  /**
+   * Width/height from the fixture's renderConfig. Forwarded directly to
+   * the Lambda event so this mode catches drift if the handler ever
+   * starts honouring `Config.width/height` for canvas sizing rather
+   * than reading the composition's `data-width`/`data-height`. The
+   * `distributed-simulated` mode hardcodes 1920×1080 because it
+   * bypasses the event-serialization boundary; lambda-local goes
+   * through it, which is the whole point.
+   */
+  width: number;
+  height: number;
+  format: "mp4" | "mov" | "png-sequence";
+  codec?: "h264" | "h265";
+  chunkSize?: number;
+  maxParallelChunks?: number;
+  variables?: Record<string, unknown>;
+}
+
+const FAKE_BUCKET = "harness-lambda-local";
+
+/** S3 URI helpers — keep the URI shape identical to what SFN uses in production. */
+function uri(key: string): string {
+  return `s3://${FAKE_BUCKET}/${key}`;
+}
+
+/**
+ * Run plan → renderChunk × N → assemble through the OSS handler with a
+ * filesystem-backed fake S3. Output lands at `input.renderedOutputPath`.
+ */
+export async function runLambdaLocalRender(input: RunLambdaLocalInput): Promise<void> {
+  const s3Root = join(input.tempRoot, "s3");
+  mkdirSync(s3Root, { recursive: true });
+
+  // STEP 0: stage the project as a tar.gz at the fake-S3 path the Plan
+  // event will reference, mirroring what `deploySite` does in prod.
+  const projectKey = `sites/harness/${Date.now()}/project.tar.gz`;
+  const projectS3Path = join(s3Root, projectKey);
+  mkdirSync(dirname(projectS3Path), { recursive: true });
+  await tarDirectory(input.projectDir, projectS3Path);
+
+  const fakeS3 = new FilesystemBackedFakeS3(s3Root);
+  const deps: HandlerDeps = {
+    s3: fakeS3 as unknown as HandlerDeps["s3"],
+    // The handler resolves a Chrome path via `@sparticuz/chromium` by
+    // default; that's the Lambda-specific binary. In Dockerfile.test
+    // we want the producer's already-configured Chrome instead. The
+    // skip flag tells the handler not to override PRODUCER_HEADLESS_SHELL_PATH.
+    skipChromeResolution: true,
+    tmpRoot: join(input.tempRoot, "lambda-tmp"),
+  };
+  mkdirSync(deps.tmpRoot as string, { recursive: true });
+
+  const config: SerializableDistributedRenderConfig = {
+    fps: input.fps,
+    width: input.width,
+    height: input.height,
+    format: input.format,
+    ...(input.format === "mp4" && input.codec !== undefined ? { codec: input.codec } : {}),
+    chunkSize: input.chunkSize,
+    maxParallelChunks: input.maxParallelChunks,
+    hdrMode: "force-sdr",
+  };
+
+  // STEP A: plan
+  const planPrefix = `renders/harness/${Date.now()}/`;
+  const planEvent: PlanEvent = {
+    Action: "plan",
+    ProjectS3Uri: uri(projectKey),
+    PlanOutputS3Prefix: uri(planPrefix),
+    Config: config,
+  };
+  const planResult = (await handler(planEvent, deps)) as PlanLambdaResult;
+
+  // STEP B: render every chunk through the handler.
+  const chunkUris: string[] = [];
+  for (let i = 0; i < planResult.ChunkCount; i++) {
+    const chunkEvent: RenderChunkEvent = {
+      Action: "renderChunk",
+      PlanS3Uri: planResult.PlanS3Uri,
+      PlanHash: planResult.PlanHash,
+      ChunkIndex: i,
+      ChunkOutputS3Prefix: uri(planPrefix),
+      Format: input.format,
+    };
+    const chunkResult = (await handler(chunkEvent, deps)) as RenderChunkLambdaResult;
+    chunkUris.push(chunkResult.ChunkS3Uri);
+  }
+
+  // STEP C: assemble
+  const finalUri = uri(
+    `${planPrefix}output${input.format === "png-sequence" ? ".tar.gz" : `.${input.format}`}`,
+  );
+  const assembleEvent: AssembleEvent = {
+    Action: "assemble",
+    PlanS3Uri: planResult.PlanS3Uri,
+    ChunkS3Uris: chunkUris,
+    AudioS3Uri: planResult.AudioS3Uri,
+    OutputS3Uri: finalUri,
+    Format: input.format,
+  };
+  (await handler(assembleEvent, deps)) as AssembleLambdaResult;
+
+  // Copy the final output from fake-S3 land back out to the path the
+  // harness expects. For png-sequence, untar into the dir.
+  const finalKey = finalUri.slice(`s3://${FAKE_BUCKET}/`.length);
+  if (input.format === "png-sequence") {
+    const tarPath = join(s3Root, finalKey);
+    mkdirSync(input.renderedOutputPath, { recursive: true });
+    await untarDirectory(tarPath, input.renderedOutputPath);
+  } else {
+    await downloadS3ObjectToFile(
+      fakeS3 as unknown as Parameters<typeof downloadS3ObjectToFile>[0],
+      finalUri,
+      input.renderedOutputPath,
+    );
+  }
+}
+
+/**
+ * Minimum AWS-SDK-shaped fake S3 the handler's `send(GetObject)` and
+ * `send(PutObject)` calls land in. Stores blobs on the local filesystem
+ * under `root/<key>` so the harness can pre-stage inputs (tarball'd
+ * project) and post-inspect outputs (per-chunk artifacts, final video)
+ * without going through a real S3 endpoint.
+ */
+class FilesystemBackedFakeS3 {
+  constructor(private readonly root: string) {}
+
+  async send(command: unknown): Promise<unknown> {
+    const cmdName = (command as { constructor: { name: string } }).constructor.name;
+    const input = (command as { input: { Bucket: string; Key: string; Body?: unknown } }).input;
+    const fsPath = join(this.root, input.Key);
+
+    if (cmdName === "GetObjectCommand") {
+      if (!existsSync(fsPath)) {
+        const err = new Error(
+          `FakeS3: GetObject for missing key ${input.Bucket}/${input.Key}`,
+        ) as Error & {
+          $metadata: { httpStatusCode: number };
+        };
+        err.$metadata = { httpStatusCode: 404 };
+        throw err;
+      }
+      const bytes = readFileSync(fsPath);
+      return { Body: Readable.from([bytes]) };
+    }
+    if (cmdName === "PutObjectCommand") {
+      mkdirSync(dirname(fsPath), { recursive: true });
+      const body = input.Body;
+      if (body instanceof Buffer || typeof body === "string") {
+        writeFileSync(fsPath, body);
+      } else if (body && typeof (body as NodeJS.ReadableStream).pipe === "function") {
+        await pipeline(body as NodeJS.ReadableStream, createWriteStream(fsPath));
+      } else {
+        throw new Error(`FakeS3: PutObject body shape not supported (${typeof body})`);
+      }
+      return { ETag: `"fake-${statSync(fsPath).size}"` };
+    }
+    if (cmdName === "HeadObjectCommand") {
+      if (!existsSync(fsPath)) {
+        const err = new Error(
+          `FakeS3: HeadObject for missing key ${input.Bucket}/${input.Key}`,
+        ) as Error & {
+          $metadata: { httpStatusCode: number };
+        };
+        err.$metadata = { httpStatusCode: 404 };
+        throw err;
+      }
+      return { ContentLength: statSync(fsPath).size, LastModified: new Date() };
+    }
+    throw new Error(`FakeS3: unexpected command ${cmdName}`);
+  }
+}
@@ -32,6 +32,18 @@ import {
  runDistributedSimulatedRender,
 } from "./regression-harness-distributed.js";

+// `regression-harness-lambda-local` statically imports
+// `@hyperframes/aws-lambda`, which depends on @aws-sdk + @sparticuz/chromium.
+// In Dockerfile.test the workspace copy of aws-lambda's src isn't present,
+// so a static import here would fail at module-load time even when
+// running `--mode=in-process`. Load it on demand instead.
+async function loadLambdaLocalRender(): Promise<
+  typeof import("./regression-harness-lambda-local.js").runLambdaLocalRender
+> {
+  const mod = await import("./regression-harness-lambda-local.js");
+  return mod.runLambdaLocalRender;
+}
+
 // ── Types ────────────────────────────────────────────────────────────────────

 type TestMetadata = {
@@ -228,13 +240,13 @@ function parseArgs(argv: string[]): CliOptions {
    }
  }

-  if (update && mode === "distributed-simulated") {
+  if (update && (mode === "distributed-simulated" || mode === "lambda-local")) {
    // The in-process renderer is the source of truth for golden baselines —
-    // distributed-simulated's job is to verify the contract against the
-    // same baseline, not to author its own. Surfacing this at parse time
-    // saves a multi-minute render before the user notices.
+    // the other two modes verify the contract against the same baseline,
+    // not author their own. Surfacing this at parse time saves a multi-
+    // minute render before the user notices.
    throw new Error(
-      "regression-harness: --update is incompatible with --mode=distributed-simulated. " +
+      `regression-harness: --update is incompatible with --mode=${mode}. ` +
        "Generate baselines with the in-process renderer (the default mode), then re-run " +
        "without --update to verify both modes match.",
    );
@@ -889,13 +901,14 @@ async function runTestSuite(
    copyFixtureSupportFiles(suite, tempRoot);
    cpSync(suite.srcDir, tempSrcDir, { recursive: true });

-    if (options.mode === "distributed-simulated") {
+    if (options.mode === "distributed-simulated" || options.mode === "lambda-local") {
      const support = checkDistributedSupport(suite.meta.renderConfig);
      if (!support.supported) {
-        // Skipping is a clean outcome — the distributed pipeline can't
-        // run this fixture, but in-process mode already covers it. Mark
-        // passed so the suite summary doesn't trip CI; the `skipped`
-        // field is what distinguishes a real pass from a skip.
+        // Skipping is a clean outcome — the distributed pipeline (which
+        // both modes go through) can't run this fixture, but in-process
+        // mode already covers it. Mark passed so the suite summary
+        // doesn't trip CI; the `skipped` field is what distinguishes a
+        // real pass from a skip.
        console.log(
          JSON.stringify({
            event: "test_skipped",
@@ -912,21 +925,33 @@ async function runTestSuite(
      // `checkDistributedSupport` already narrowed fps to {24,30,60} and
      // rejected webm; the cast surfaces that guarantee to TS.
      const fpsNum = suite.meta.renderConfig.fps.num as 24 | 30 | 60;
-      // `runDistributedSimulatedRender`'s `format` parameter accepts the
-      // distributed-supported set; the harness type allows `"webm"` too
-      // but `checkDistributedSupport` rejected that above. Narrow the cast
-      // accordingly.
-      await runDistributedSimulatedRender({
+      const distributedInput = {
        projectDir: tempSrcDir,
        tempRoot,
        renderedOutputPath,
        fps: fpsNum,
+        // `runDistributedSimulatedRender` / `runLambdaLocalRender`'s
+        // `format` parameter accepts the distributed-supported set;
+        // the harness type allows `"webm"` too but
+        // `checkDistributedSupport` rejected that above. Narrow.
        format: outputFormat as "mp4" | "mov" | "png-sequence",
        codec: suite.meta.renderConfig.codec,
        chunkSize: suite.meta.renderConfig.chunkSize,
        maxParallelChunks: suite.meta.renderConfig.maxParallelChunks,
        variables: suite.meta.renderConfig.variables,
-      });
+      };
+      if (options.mode === "lambda-local") {
+        const runLambdaLocalRender = await loadLambdaLocalRender();
+        // The fixture's authored dimensions live in the composition's
+        // `data-width`/`data-height` attributes, not in `meta.json`'s
+        // renderConfig. Until the harness compiles the HTML up-front
+        // to surface them here, pass 1920×1080 — the same placeholder
+        // `runDistributedSimulatedRender` uses internally. The
+        // composition attrs override at plan time.
+        await runLambdaLocalRender({ ...distributedInput, width: 1920, height: 1080 });
+      } else {
+        await runDistributedSimulatedRender(distributedInput);
+      }
    } else {
      const job = createRenderJob({
        fps: suite.meta.renderConfig.fps,
@@ -12,7 +12,12 @@
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true,
-    "types": ["@webgpu/types"]
+    "types": ["@webgpu/types"],
+    "baseUrl": ".",
+    "paths": {
+      "@hyperframes/producer": ["./src/index.ts"],
+      "@hyperframes/producer/distributed": ["./src/distributed.ts"]
+    }
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules", "dist", "src/**/*.test.ts", "src/**/__test_utils__/**"]