mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-16 14:27:26 +00:00
* fix(producer): mix audio into a container that can record encoder delay Every rendered composition's audio landed 1024 samples (21.33 ms at 48 kHz) after its authored `data-start`, against a frame-accurate video track. The mix is AAC-encoded, and AAC encoders emit ~1024 priming samples. The mix was written to a raw ADTS `.aac` file, which has nowhere to record that delay, so it decoded as real leading silence and every stage downstream preserved it faithfully. Measuring each intermediate localises it precisely: the source WAV is exact, the mixer's own output is already 21.33 ms late, and the pad/trim and mux stages inherit it unchanged. The filter graph itself is correct - run by hand to PCM it lands on the authored start. Switch the artifact to an MP4-family container, which stores the delay as an edit list that decoders strip. Same codec, same bitrate, so no size or quality change. The filename is a contract shared by three consumers - the mux input, the distributed plan artifact, and the PNG-sequence sidecar handed to users for NLE ingest - and its extension is what selects the muxer. Give it one owner in the engine rather than five literals, so those consumers cannot drift onto different containers. Note for reviewers: this renames the distributed plan's audio artifact, which is an on-disk contract between the plan writer and the assembler. Both move together here, but a plan written by an older build would not be found by a newer assembler. Flagging in case that mixed-version window matters for how these are deployed. * fix(cloud): read the plan audio artifact name from the producer contract The aws-lambda and gcp-cloud-run adapters each restated the plan's audio filename in five places, so renaming it in the producer left them looking for a file that is no longer written. CI caught it: the gcp dispatch test asserting a plan has no audio artifact started seeing one. Export the name from `@hyperframes/producer/distributed` and consume it in both adapters. This is the same failure the constant exists to prevent, one package boundary further out: a literal that drifts from the writer's is a silently missing audio track rather than a loud error, because both call sites only ever ask whether the file exists. * fix(cloud): accept a legacy plan's audio artifact name for one release Review raised a rolling-deploy window I had flagged but left undecided: `plan` and `assemble` are separate invocations bridged by object storage, so a pre-rollout planner can be paired with a post-rollout assembler. Both readers locate the artifact by existence alone, which makes that pairing a silently muted video rather than an error. That is reachable enough to be worth two lines, so reads now accept the old name while writes only ever emit the new one. Give the fallback one owner (`resolvePlanAudioPath` / `isPlanAudioArtifactPath`) rather than four call sites, marked for deletion one release out. Also fixes a hole in the first pass of this: the plan-v2 materializer matched either name but then joined the CURRENT one, so a legacy plan resolved to a path that was never written. It now joins the artifact's own name. Review nits in the same pass: correct the pad-branch docstring, which still described a concat-copy shape the pad branch stopped using when it moved to apad + re-encode, and fix the Windows fixture's stale `.aac` output extension so it cannot model a shape that reintroduces the priming delay. * test(producer): rebake the missing-host-comp-id golden without the audio delay The pinned reference was rendered before this branch, so it carries the 1024 sample encoder-priming delay in its audio. With the delay gone the correct audio now sits ahead of the reference and the harness's envelope correlation drops below its floor. Cross-correlating the old and new references at native 48 kHz gives a lag of exactly 1024 samples (21.33 ms) at a correlation of 0.99985: same audio, moved by exactly the amount this branch removes. Regenerated inside the CI container (Dockerfile.test, ffmpeg 5.1.9) rather than natively, so the reference matches the encoder CI will compare against - the container reproduced CI's failure to the digit (correlation 0.3938764027803616, lagWindows -12) before the rebake and passes at correlation 1.0 after it. Note for archaeology: the new reference is also 3 dB louder than the old one. That gap is not from this branch - `main` and this branch render the fixture at the same level - it is pre-existing drift the reference had accumulated, which a scale-invariant correlator could never see. The rebake absorbs it. Only output.mp4 is updated. `--update` also rewrites compiled.html, but that diff is embedded-font churn with no bearing on the comparison, which reports "Failed at compilation: 0" either way. * test(producer): rebake the variables-prod golden without the audio delay Same cause as the missing-host-comp-id rebake, caught by shard-8 once the earlier shard stopped failing and the rest of the matrix could run: this reference also carries the encoder-priming delay this branch removes. Reproduced in the CI container to the digit (correlation 0.42704173048439215, lagWindows -12), rebaked there, and it now passes at correlation 1.0. Worth recording: the shift here is 2048 samples (42.67 ms) at correlation 0.99983, exactly twice the 1024 of the other fixture. The delay compounds once per un-compensated AAC generation, and this fixture's audio needs its duration normalized, so it takes the pad/trim branch's re-encode and picks up a second frame of priming on top of the mixer's. So the pre-fix error was not a fixed 21 ms - it grew with the number of times the audio was re-encoded. All nine shards ran in that CI round with only this one failing, so the matrix has now covered every fixture against this change.
866 lines
32 KiB
TypeScript
866 lines
32 KiB
TypeScript
/**
|
||
* AWS Lambda handler for HyperFrames distributed rendering.
|
||
*
|
||
* One Lambda function, three roles. Step Functions dispatches by setting
|
||
* `event.Action`; the handler unwraps Map-state envelopes, primes the
|
||
* Lambda environment (Chrome path, ffmpeg path, tmpdir), and forwards to
|
||
* the matching OSS primitive from `@hyperframes/producer/distributed`.
|
||
*
|
||
* Everything heavy — capture, encode, audio mix — happens inside the OSS
|
||
* primitives. The handler is thin glue: parse event → S3 download → call
|
||
* primitive → S3 upload → return small JSON result.
|
||
*/
|
||
|
||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync } from "node:fs";
|
||
import { tmpdir } from "node:os";
|
||
import { basename, join } from "node:path";
|
||
import { S3Client } from "@aws-sdk/client-s3";
|
||
import {
|
||
assemble,
|
||
type AssembleResult,
|
||
type ChunkRenderer,
|
||
type ChunkResult,
|
||
type DistributedRenderConfig,
|
||
listPlanV2ArtifactsForTarget,
|
||
materializePlanV2Target,
|
||
plan,
|
||
isPlanAudioArtifactPath,
|
||
PLAN_AUDIO_RELATIVE_PATH,
|
||
resolvePlanAudioPath,
|
||
planV2WithPublisher,
|
||
type PlanResult,
|
||
type PlanV2Artifact,
|
||
type PlanV2Manifest,
|
||
type PlanV2MaterializationTarget,
|
||
readPlanV2Manifest,
|
||
renderChunk,
|
||
} from "@hyperframes/producer/distributed";
|
||
import { resolveChromeExecutablePath } from "./chromium.js";
|
||
import { type DistributedFormat, formatExtension } from "./formatExtension.js";
|
||
import type {
|
||
AssembleEvent,
|
||
AssembleLambdaResult,
|
||
LambdaAction,
|
||
LambdaEvent,
|
||
LambdaResult,
|
||
PlanEvent,
|
||
PlanLambdaResult,
|
||
RenderChunkEvent,
|
||
RenderChunkLambdaResult,
|
||
} from "./events.js";
|
||
import {
|
||
downloadS3ObjectToFile,
|
||
downloadS3ObjectToFileVerified,
|
||
parseS3Uri,
|
||
tarDirectory,
|
||
untarDirectory,
|
||
uploadFileToS3,
|
||
} from "./s3Transport.js";
|
||
import { S3PlanV2ArtifactPublisher } from "./s3PlanV2Publisher.js";
|
||
|
||
/**
|
||
* Lazily-constructed S3 client. Cached at module scope so warm Lambda
|
||
* containers reuse the underlying HTTP keep-alive pool across invocations.
|
||
*/
|
||
let cachedS3Client: S3Client | null = null;
|
||
function getS3Client(): S3Client {
|
||
if (cachedS3Client) return cachedS3Client;
|
||
cachedS3Client = new S3Client({});
|
||
return cachedS3Client;
|
||
}
|
||
|
||
/**
|
||
* Optional injection points used by the handler's unit tests. Production
|
||
* callers leave these unset; the real OSS primitives are used. Tests
|
||
* inject `s3` and `primitives` directly rather than mutating module
|
||
* state — the dependency-injection seam is sufficient and avoids a
|
||
* second leak point for cross-test contamination.
|
||
*/
|
||
export interface HandlerDeps {
|
||
s3?: S3Client;
|
||
primitives?: {
|
||
plan: typeof plan;
|
||
planV2WithPublisher?: typeof planV2WithPublisher;
|
||
renderChunk: ChunkRenderer;
|
||
assemble: typeof assemble;
|
||
};
|
||
/** Override the per-invocation `/tmp` workdir root (defaults to Lambda's `/tmp`). */
|
||
tmpRoot?: string;
|
||
/** Skip Chrome resolution (used by handler dispatch tests that mock renderChunk). */
|
||
skipChromeResolution?: boolean;
|
||
}
|
||
|
||
/**
|
||
* Lambda entry. Step Functions sometimes wraps the event in
|
||
* `{ Payload: ... }` or `{ Input: ... }` depending on the state machine
|
||
* shape; unwrap until we hit a discriminated event.
|
||
*/
|
||
export async function handler(event: LambdaEvent, deps?: HandlerDeps): Promise<LambdaResult> {
|
||
const unwrapped = unwrapEvent(event);
|
||
validateEventS3Uris(unwrapped);
|
||
primeRuntimeEnv();
|
||
// Single structured boot log line — CloudWatch Logs Insights queries
|
||
// key off `event=handler_start` to grep for a specific Action / S3 URI
|
||
// when triaging without attaching a debugger.
|
||
logEvent({ event: "handler_start", action: unwrapped.Action, input: summarizeEvent(unwrapped) });
|
||
try {
|
||
switch (unwrapped.Action) {
|
||
case "plan":
|
||
return await handlePlan(unwrapped, deps);
|
||
case "renderChunk":
|
||
return await handleRenderChunk(unwrapped, deps);
|
||
case "assemble":
|
||
return await handleAssemble(unwrapped, deps);
|
||
default: {
|
||
// Compile-time exhaustiveness: a new LambdaAction member trips
|
||
// the `never` assignment before the runtime error is reachable.
|
||
const _exhaustive: never = unwrapped;
|
||
throw new Error(
|
||
`[handler] unknown Action: ${JSON.stringify(
|
||
(_exhaustive as { Action?: string }).Action,
|
||
)}. Expected one of "plan", "renderChunk", "assemble".`,
|
||
);
|
||
}
|
||
}
|
||
} catch (err) {
|
||
normalizeTerminalErrorName(err);
|
||
// Log before re-throwing so CloudWatch captures the structured
|
||
// error context alongside Lambda's default stack trace. Otherwise
|
||
// ops only sees the trace and has to correlate with execution
|
||
// history to recover the action + input.
|
||
logEvent({
|
||
event: "handler_error",
|
||
action: unwrapped.Action,
|
||
input: summarizeEvent(unwrapped),
|
||
message: err instanceof Error ? err.message : String(err),
|
||
name: err instanceof Error ? err.name : undefined,
|
||
});
|
||
throw err;
|
||
}
|
||
}
|
||
|
||
/**
|
||
* AWS Lambda reports `Error.name` to Step Functions, while producer errors
|
||
* expose stable machine codes separately. Normalize workflow-facing codes
|
||
* whose historical class names differ from their orchestration contracts.
|
||
*/
|
||
// The explicit error-name mapping is the public Step Functions failure contract.
|
||
// fallow-ignore-next-line complexity
|
||
function normalizeTerminalErrorName(error: unknown): void {
|
||
if (!error || typeof error !== "object") return;
|
||
const candidate = error as { code?: unknown; name?: string };
|
||
if (
|
||
candidate.code === "PLAN_PROTOCOL_UNSUPPORTED" ||
|
||
candidate.code === "PLAN_TOO_LARGE" ||
|
||
candidate.code === "PLAN_V2_INTEGRITY_UNRECOVERABLE" ||
|
||
candidate.code === "FONT_FETCH_FAILED" ||
|
||
candidate.code === "FONT_FETCH_UNAVAILABLE" ||
|
||
candidate.code === "NOT_MEDIA_PAYLOAD" ||
|
||
candidate.code === "VIDEO_SOURCE_UNRENDERABLE" ||
|
||
candidate.code === "VIDEO_EXTRACTION_FAILED" ||
|
||
candidate.code === "INVALID_VIDEO_METADATA"
|
||
) {
|
||
candidate.name = candidate.code;
|
||
}
|
||
}
|
||
|
||
/**
|
||
* Walk through Step Functions' Map-state and Task-state envelopes until
|
||
* the discriminated event is found.
|
||
*/
|
||
// Step Functions wraps at most `{Payload: {Input: ...}}` in our state
|
||
// machine; 4 levels is 2× headroom for unusual Map / Wait state
|
||
// configurations and prevents infinite loops on malformed input.
|
||
const MAX_ENVELOPE_DEPTH = 4;
|
||
|
||
export function unwrapEvent(event: LambdaEvent): PlanEvent | RenderChunkEvent | AssembleEvent {
|
||
let cursor: LambdaEvent = event;
|
||
for (let i = 0; i < MAX_ENVELOPE_DEPTH; i++) {
|
||
if (cursor && typeof cursor === "object") {
|
||
const obj = cursor as Record<string, unknown>;
|
||
if (typeof obj.Action === "string" && isLambdaAction(obj.Action)) {
|
||
return cursor as PlanEvent | RenderChunkEvent | AssembleEvent;
|
||
}
|
||
if ("Payload" in obj) {
|
||
cursor = obj.Payload as LambdaEvent;
|
||
continue;
|
||
}
|
||
if ("Input" in obj) {
|
||
cursor = obj.Input as LambdaEvent;
|
||
continue;
|
||
}
|
||
}
|
||
break;
|
||
}
|
||
throw new Error(
|
||
`[handler] event has no recognised Action; unwrapped ${MAX_ENVELOPE_DEPTH} levels of Payload/Input without finding one.`,
|
||
);
|
||
}
|
||
|
||
function isLambdaAction(value: string): value is LambdaAction {
|
||
return value === "plan" || value === "renderChunk" || value === "assemble";
|
||
}
|
||
|
||
/**
|
||
* Emit a single JSON line to stdout. CloudWatch ingests each line as a
|
||
* structured event; Logs Insights queries can `filter event="..."` and
|
||
* project specific fields. We write to stdout (not stderr) because
|
||
* Lambda's default destination for both is the same log group, and
|
||
* Logs Insights' INFO/ERROR level parser keys off the JSON `level`
|
||
* field, not the stream.
|
||
*/
|
||
function logEvent(payload: Record<string, unknown>): void {
|
||
console.log(JSON.stringify(payload));
|
||
}
|
||
|
||
/**
|
||
* Compact, non-PII summary of a Lambda event for logging. The full
|
||
* event payload can include the entire project config; we only emit
|
||
* the routable fields (S3 URIs, chunk index, format) needed to triage
|
||
* a failure from CloudWatch.
|
||
*/
|
||
// Keep event variants together so logs share one redaction and summarization boundary.
|
||
// fallow-ignore-next-line complexity
|
||
function summarizeEvent(
|
||
event: PlanEvent | RenderChunkEvent | AssembleEvent,
|
||
): Record<string, unknown> {
|
||
switch (event.Action) {
|
||
case "plan":
|
||
return {
|
||
projectS3Uri: event.ProjectS3Uri,
|
||
planOutputS3Prefix: event.PlanOutputS3Prefix,
|
||
planProtocol: event.PlanProtocol ?? "v1",
|
||
format: event.Config.format,
|
||
fps: event.Config.fps,
|
||
};
|
||
case "renderChunk":
|
||
return {
|
||
planProtocol: event.PlanProtocol ?? "v1",
|
||
...(event.PlanProtocol === "v2"
|
||
? { planV2ManifestS3Uri: event.PlanV2ManifestS3Uri }
|
||
: { planS3Uri: event.PlanS3Uri }),
|
||
chunkIndex: event.ChunkIndex,
|
||
format: event.Format,
|
||
};
|
||
case "assemble":
|
||
return {
|
||
planProtocol: event.PlanProtocol ?? "v1",
|
||
...(event.PlanProtocol === "v2"
|
||
? { planV2ManifestS3Uri: event.PlanV2ManifestS3Uri }
|
||
: { planS3Uri: event.PlanS3Uri }),
|
||
chunkCount: event.ChunkS3Uris.length,
|
||
hasAudio: event.AudioS3Uri !== null,
|
||
outputS3Uri: event.OutputS3Uri,
|
||
format: event.Format,
|
||
};
|
||
}
|
||
}
|
||
|
||
/**
|
||
* Lambda sets `TMPDIR` to `/tmp` already, but the bundled binaries (Chrome
|
||
* + ffmpeg) live alongside the handler at `/var/task/bin/`. Add that to
|
||
* PATH the first time the handler runs so spawn("ffmpeg", …) inside the
|
||
* OSS primitives resolves to the bundled binary.
|
||
*/
|
||
let runtimeEnvPrimed = false;
|
||
function primeRuntimeEnv(): void {
|
||
if (runtimeEnvPrimed) return;
|
||
runtimeEnvPrimed = true;
|
||
const taskRoot = process.env.LAMBDA_TASK_ROOT ?? "/var/task";
|
||
const bin = join(taskRoot, "bin");
|
||
if (existsSync(bin)) {
|
||
process.env.PATH = `${bin}:${process.env.PATH ?? ""}`;
|
||
}
|
||
}
|
||
|
||
// ── Plan ────────────────────────────────────────────────────────────────────
|
||
|
||
// The v1 handler owns one transactional download, plan, archive, upload, and cleanup lifecycle.
|
||
// fallow-ignore-next-line complexity
|
||
async function handlePlan(event: PlanEvent, deps?: HandlerDeps): Promise<PlanLambdaResult> {
|
||
if (event.PlanProtocol === "v2") {
|
||
return handlePlanV2(event, deps);
|
||
}
|
||
const started = Date.now();
|
||
const s3 = deps?.s3 ?? getS3Client();
|
||
const primitive = deps?.primitives?.plan ?? plan;
|
||
|
||
// The producer's probe stage launches Chromium whenever the composition
|
||
// needs a runtime duration probe or has unresolved sub-compositions, so
|
||
// plan has to resolve Chrome the same way renderChunk does. Without this
|
||
// the probe throws "An `executablePath` or `channel` must be specified
|
||
// for `puppeteer-core`" the moment runProbeStage calls puppeteer.launch.
|
||
if (!deps?.skipChromeResolution && !process.env.PRODUCER_HEADLESS_SHELL_PATH) {
|
||
const chromePath = await resolveChromeExecutablePath();
|
||
process.env.PRODUCER_HEADLESS_SHELL_PATH = chromePath;
|
||
}
|
||
|
||
const work = mkdtempSync(join(deps?.tmpRoot ?? tmpdir(), "hf-lambda-plan-"));
|
||
// We use `.tar.gz` (not `.zip`) as the project archive's on-the-wire
|
||
// format because Lambda's Amazon Linux base image ships GNU `tar` but
|
||
// not `unzip` in `/usr/bin`. The smoke script + future CLI both
|
||
// produce tar.gz uploads.
|
||
const projectArchive = join(work, "project.tar.gz");
|
||
const projectDir = join(work, "project");
|
||
const planDir = join(work, "plan");
|
||
|
||
try {
|
||
await downloadS3ObjectToFile(s3, event.ProjectS3Uri, projectArchive);
|
||
await untarDirectory(projectArchive, projectDir);
|
||
|
||
const config: DistributedRenderConfig = {
|
||
...event.Config,
|
||
};
|
||
const result: PlanResult = await primitive(projectDir, config, planDir);
|
||
|
||
// Upload the planDir as a single tarball. Step Functions cannot pass
|
||
// a directory-shaped artifact between states; we serialize and rely on
|
||
// the consumer (renderChunk / assemble) to untar. Audio is co-located
|
||
// alongside the plan so RenderChunk doesn't have to pull the whole
|
||
// plan tarball when audio isn't relevant to the chunk.
|
||
const planTar = join(work, "plan.tar.gz");
|
||
await tarDirectory(planDir, planTar);
|
||
const planTarUri = `${trimTrailingSlash(event.PlanOutputS3Prefix)}/plan.tar.gz`;
|
||
const audioPath = join(planDir, PLAN_AUDIO_RELATIVE_PATH);
|
||
const hasAudio = existsSync(audioPath) && statSync(audioPath).size > 0;
|
||
const audioUri = hasAudio
|
||
? `${trimTrailingSlash(event.PlanOutputS3Prefix)}/${PLAN_AUDIO_RELATIVE_PATH}`
|
||
: null;
|
||
// Plan and audio are independent S3 PUTs; run them in parallel so
|
||
// the response returns as soon as the slower of the two completes.
|
||
await Promise.all([
|
||
uploadFileToS3(s3, planTar, planTarUri, "application/gzip"),
|
||
hasAudio && audioUri ? uploadFileToS3(s3, audioPath, audioUri, "audio/aac") : null,
|
||
]);
|
||
|
||
return {
|
||
Action: "plan",
|
||
PlanS3Uri: planTarUri,
|
||
PlanHash: result.planHash,
|
||
ChunkCount: result.chunkCount,
|
||
TotalFrames: result.totalFrames,
|
||
Fps: result.fps,
|
||
Width: result.width,
|
||
Height: result.height,
|
||
Format: result.format,
|
||
HasAudio: audioUri !== null,
|
||
AudioS3Uri: audioUri,
|
||
FfmpegVersion: result.ffmpegVersion,
|
||
ProducerVersion: result.producerVersion,
|
||
DurationMs: Date.now() - started,
|
||
};
|
||
} finally {
|
||
cleanupDir(work);
|
||
}
|
||
}
|
||
|
||
// Plan v2 orchestration is kept as one transactional boundary: stage, CAS upload,
|
||
// and manifest-last publication must stay ordered and fail together.
|
||
// fallow-ignore-next-line complexity
|
||
async function handlePlanV2(
|
||
event: Extract<PlanEvent, { PlanProtocol: "v2" }>,
|
||
deps?: HandlerDeps,
|
||
): Promise<Extract<PlanLambdaResult, { PlanProtocol: "v2" }>> {
|
||
const started = Date.now();
|
||
const s3 = deps?.s3 ?? getS3Client();
|
||
const primitive = deps?.primitives?.planV2WithPublisher ?? planV2WithPublisher;
|
||
if (!deps?.skipChromeResolution && !process.env.PRODUCER_HEADLESS_SHELL_PATH) {
|
||
process.env.PRODUCER_HEADLESS_SHELL_PATH = await resolveChromeExecutablePath();
|
||
}
|
||
|
||
const work = mkdtempSync(join(deps?.tmpRoot ?? tmpdir(), "hf-lambda-plan-v2-"));
|
||
const projectArchive = join(work, "project.tar.gz");
|
||
const projectDir = join(work, "project");
|
||
try {
|
||
await downloadS3ObjectToFile(s3, event.ProjectS3Uri, projectArchive);
|
||
await untarDirectory(projectArchive, projectDir);
|
||
const publisher = new S3PlanV2ArtifactPublisher({
|
||
s3,
|
||
planOutputS3Prefix: event.PlanOutputS3Prefix,
|
||
temporaryRoot: work,
|
||
});
|
||
const manifest: PlanV2Manifest = await primitive(projectDir, { ...event.Config }, publisher, {
|
||
stagingParentDir: work,
|
||
});
|
||
|
||
return {
|
||
Action: "plan",
|
||
PlanProtocol: "v2",
|
||
PlanV2ManifestS3Uri: publisher.manifestUri,
|
||
PlanV2ArtifactS3Prefix: publisher.artifactPrefix,
|
||
PlanHash: manifest.planHash,
|
||
ChunkCount: manifest.chunkCount,
|
||
TotalFrames: manifest.totalFrames,
|
||
Fps: manifest.fps,
|
||
Width: manifest.width,
|
||
Height: manifest.height,
|
||
Format: manifest.format,
|
||
HasAudio: manifest.artifacts.some((artifact) => isPlanAudioArtifactPath(artifact.path)),
|
||
AudioS3Uri: null,
|
||
FfmpegVersion: manifest.ffmpegVersion,
|
||
ProducerVersion: manifest.producerVersion,
|
||
DurationMs: Date.now() - started,
|
||
};
|
||
} finally {
|
||
cleanupDir(work);
|
||
}
|
||
}
|
||
|
||
// ── RenderChunk ─────────────────────────────────────────────────────────────
|
||
|
||
async function handleRenderChunk(
|
||
event: RenderChunkEvent,
|
||
deps?: HandlerDeps,
|
||
): Promise<RenderChunkLambdaResult> {
|
||
if (event.PlanProtocol === "v2") {
|
||
return handleRenderChunkV2(event, deps);
|
||
}
|
||
const started = Date.now();
|
||
const s3 = deps?.s3 ?? getS3Client();
|
||
const primitive = deps?.primitives?.renderChunk ?? renderChunk;
|
||
|
||
// Sparticuz decompresses Chromium into /tmp on first call; warm starts
|
||
// skip the work (path already cached). Guard the env-var mutation too so
|
||
// a caller-supplied PRODUCER_HEADLESS_SHELL_PATH (e.g. the SAM-local
|
||
// RIE smoke) wins over the auto-resolution.
|
||
if (!deps?.skipChromeResolution && !process.env.PRODUCER_HEADLESS_SHELL_PATH) {
|
||
const chromePath = await resolveChromeExecutablePath();
|
||
// The OSS engine resolves Chrome via `PRODUCER_HEADLESS_SHELL_PATH`
|
||
// first (see `browserManager.resolveHeadlessShellPath`); set it before
|
||
// invoking the primitive so launch picks up the bundled binary.
|
||
process.env.PRODUCER_HEADLESS_SHELL_PATH = chromePath;
|
||
}
|
||
|
||
const work = mkdtempSync(join(deps?.tmpRoot ?? tmpdir(), "hf-lambda-chunk-"));
|
||
const planTar = join(work, "plan.tar.gz");
|
||
const planDir = join(work, "plan");
|
||
|
||
try {
|
||
await downloadS3ObjectToFile(s3, event.PlanS3Uri, planTar);
|
||
await untarDirectory(planTar, planDir);
|
||
|
||
// Verify the plan's hash matches what Step Functions told us to render.
|
||
// The producer's renderChunk re-checks internally (defense-in-depth),
|
||
// but doing it here at the handler boundary lets us fail before paying
|
||
// the Chrome-launch + render cost on a misrouted chunk. Throws a
|
||
// typed PLAN_HASH_MISMATCH that Step Functions can route as
|
||
// non-retryable.
|
||
verifyPlanHash(planDir, event.PlanHash);
|
||
|
||
const chunkOutputBase = join(
|
||
work,
|
||
event.Format === "png-sequence"
|
||
? `chunk-${pad(event.ChunkIndex)}`
|
||
: `chunk-${pad(event.ChunkIndex)}${formatExtension(event.Format)}`,
|
||
);
|
||
|
||
const result: ChunkResult = await primitive(planDir, event.ChunkIndex, chunkOutputBase);
|
||
|
||
const chunkUri = await uploadChunkOutput(
|
||
s3,
|
||
result,
|
||
event.ChunkOutputS3Prefix,
|
||
event.ChunkIndex,
|
||
);
|
||
|
||
return {
|
||
Action: "renderChunk",
|
||
ChunkS3Uri: chunkUri,
|
||
ChunkIndex: event.ChunkIndex,
|
||
Sha256: result.sha256,
|
||
FramesEncoded: result.framesEncoded,
|
||
CaptureMode: result.captureMode,
|
||
DurationMs: Date.now() - started,
|
||
};
|
||
} finally {
|
||
cleanupDir(work);
|
||
}
|
||
}
|
||
|
||
// The v2 chunk handler deliberately keeps download, verified materialization,
|
||
// render, and upload in one lifecycle so cleanup and errors remain atomic.
|
||
// fallow-ignore-next-line complexity
|
||
async function handleRenderChunkV2(
|
||
event: Extract<RenderChunkEvent, { PlanProtocol: "v2" }>,
|
||
deps?: HandlerDeps,
|
||
): Promise<RenderChunkLambdaResult> {
|
||
const started = Date.now();
|
||
const s3 = deps?.s3 ?? getS3Client();
|
||
const primitive = deps?.primitives?.renderChunk ?? renderChunk;
|
||
if (!deps?.skipChromeResolution && !process.env.PRODUCER_HEADLESS_SHELL_PATH) {
|
||
process.env.PRODUCER_HEADLESS_SHELL_PATH = await resolveChromeExecutablePath();
|
||
}
|
||
const work = mkdtempSync(join(deps?.tmpRoot ?? tmpdir(), "hf-lambda-chunk-v2-"));
|
||
try {
|
||
const planDir = await downloadAndMaterializePlanV2(
|
||
s3,
|
||
event,
|
||
{ role: "chunk", chunkIndex: event.ChunkIndex },
|
||
work,
|
||
);
|
||
const chunkOutputBase = join(
|
||
work,
|
||
event.Format === "png-sequence"
|
||
? `chunk-${pad(event.ChunkIndex)}`
|
||
: `chunk-${pad(event.ChunkIndex)}${formatExtension(event.Format)}`,
|
||
);
|
||
const result = await primitive(planDir, event.ChunkIndex, chunkOutputBase);
|
||
const chunkUri = await uploadChunkOutput(
|
||
s3,
|
||
result,
|
||
event.ChunkOutputS3Prefix,
|
||
event.ChunkIndex,
|
||
);
|
||
return {
|
||
Action: "renderChunk",
|
||
ChunkS3Uri: chunkUri,
|
||
ChunkIndex: event.ChunkIndex,
|
||
Sha256: result.sha256,
|
||
FramesEncoded: result.framesEncoded,
|
||
CaptureMode: result.captureMode,
|
||
DurationMs: Date.now() - started,
|
||
};
|
||
} finally {
|
||
cleanupDir(work);
|
||
}
|
||
}
|
||
|
||
async function uploadChunkOutput(
|
||
s3: S3Client,
|
||
result: ChunkResult,
|
||
prefix: string,
|
||
chunkIndex: number,
|
||
): Promise<string> {
|
||
const trimmed = trimTrailingSlash(prefix);
|
||
if (result.outputKind === "file") {
|
||
const ext = result.outputPath.slice(result.outputPath.lastIndexOf("."));
|
||
const uri = `${trimmed}/chunks/${pad(chunkIndex)}${ext}`;
|
||
await uploadFileToS3(s3, result.outputPath, uri);
|
||
return uri;
|
||
}
|
||
// frame-dir: upload as a tarball so a single S3 object represents the chunk.
|
||
// Assemble's png-sequence path expects a directory per chunk; it untars on
|
||
// its end.
|
||
const tarball = `${result.outputPath}.tar.gz`;
|
||
await tarDirectory(result.outputPath, tarball);
|
||
const uri = `${trimmed}/chunks/${pad(chunkIndex)}.tar.gz`;
|
||
await uploadFileToS3(s3, tarball, uri, "application/gzip");
|
||
return uri;
|
||
}
|
||
|
||
// ── Assemble ────────────────────────────────────────────────────────────────
|
||
|
||
async function handleAssemble(
|
||
event: AssembleEvent,
|
||
deps?: HandlerDeps,
|
||
): Promise<AssembleLambdaResult> {
|
||
if (event.PlanProtocol === "v2") {
|
||
return handleAssembleV2(event, deps);
|
||
}
|
||
const started = Date.now();
|
||
const s3 = deps?.s3 ?? getS3Client();
|
||
const primitive = deps?.primitives?.assemble ?? assemble;
|
||
|
||
const work = mkdtempSync(join(deps?.tmpRoot ?? tmpdir(), "hf-lambda-assemble-"));
|
||
const planTar = join(work, "plan.tar.gz");
|
||
const planDir = join(work, "plan");
|
||
|
||
try {
|
||
await downloadS3ObjectToFile(s3, event.PlanS3Uri, planTar);
|
||
await untarDirectory(planTar, planDir);
|
||
|
||
const chunkPaths = await downloadChunkObjects(s3, event.ChunkS3Uris, work, event.Format);
|
||
|
||
let audioPath: string | null = null;
|
||
if (event.AudioS3Uri) {
|
||
audioPath = resolvePlanAudioPath(planDir) ?? join(planDir, PLAN_AUDIO_RELATIVE_PATH);
|
||
await downloadS3ObjectToFile(s3, event.AudioS3Uri, audioPath);
|
||
}
|
||
|
||
const finalOutput =
|
||
event.Format === "png-sequence"
|
||
? join(work, "output-frames")
|
||
: join(work, `output${formatExtension(event.Format)}`);
|
||
|
||
const result: AssembleResult = await primitive(planDir, chunkPaths, audioPath, finalOutput, {
|
||
cfr: event.Cfr === true,
|
||
});
|
||
|
||
if (event.Format === "png-sequence") {
|
||
const tarball = `${finalOutput}.tar.gz`;
|
||
await tarDirectory(finalOutput, tarball);
|
||
await uploadFileToS3(s3, tarball, event.OutputS3Uri, "application/gzip");
|
||
} else {
|
||
await uploadFileToS3(s3, finalOutput, event.OutputS3Uri);
|
||
}
|
||
|
||
return {
|
||
Action: "assemble",
|
||
OutputS3Uri: event.OutputS3Uri,
|
||
FramesEncoded: result.framesEncoded,
|
||
FileSize: result.fileSize,
|
||
DurationMs: Date.now() - started,
|
||
};
|
||
} finally {
|
||
cleanupDir(work);
|
||
}
|
||
}
|
||
|
||
// Assembly mirrors the chunk lifecycle while adding assembler-only artifacts;
|
||
// keeping the steps local makes its temporary-storage ownership explicit.
|
||
// fallow-ignore-next-line complexity
|
||
async function handleAssembleV2(
|
||
event: Extract<AssembleEvent, { PlanProtocol: "v2" }>,
|
||
deps?: HandlerDeps,
|
||
): Promise<AssembleLambdaResult> {
|
||
const started = Date.now();
|
||
const s3 = deps?.s3 ?? getS3Client();
|
||
const primitive = deps?.primitives?.assemble ?? assemble;
|
||
const work = mkdtempSync(join(deps?.tmpRoot ?? tmpdir(), "hf-lambda-assemble-v2-"));
|
||
try {
|
||
const planDir = await downloadAndMaterializePlanV2(s3, event, { role: "assembler" }, work);
|
||
// `downloadAndMaterializePlanV2` materializes atomically. Audio is
|
||
// assembler-only and lives at the familiar v1-compatible location.
|
||
const audioPath = resolvePlanAudioPath(planDir);
|
||
const chunkPaths = await downloadChunkObjects(s3, event.ChunkS3Uris, work, event.Format);
|
||
const finalOutput =
|
||
event.Format === "png-sequence"
|
||
? join(work, "output-frames")
|
||
: join(work, `output${formatExtension(event.Format)}`);
|
||
const result = await primitive(planDir, chunkPaths, audioPath, finalOutput, {
|
||
cfr: event.Cfr === true,
|
||
});
|
||
if (event.Format === "png-sequence") {
|
||
const tarball = `${finalOutput}.tar.gz`;
|
||
await tarDirectory(finalOutput, tarball);
|
||
await uploadFileToS3(s3, tarball, event.OutputS3Uri, "application/gzip");
|
||
} else {
|
||
await uploadFileToS3(s3, finalOutput, event.OutputS3Uri);
|
||
}
|
||
return {
|
||
Action: "assemble",
|
||
OutputS3Uri: event.OutputS3Uri,
|
||
FramesEncoded: result.framesEncoded,
|
||
FileSize: result.fileSize,
|
||
DurationMs: Date.now() - started,
|
||
};
|
||
} finally {
|
||
cleanupDir(work);
|
||
}
|
||
}
|
||
|
||
async function downloadAndMaterializePlanV2(
|
||
s3: S3Client,
|
||
event: {
|
||
PlanV2ManifestS3Uri: string;
|
||
PlanV2ArtifactS3Prefix: string;
|
||
PlanHash: string;
|
||
},
|
||
target: PlanV2MaterializationTarget,
|
||
work: string,
|
||
): Promise<string> {
|
||
const transportDir = join(work, "plan-v2");
|
||
mkdirSync(transportDir, { recursive: true });
|
||
await downloadS3ObjectToFile(s3, event.PlanV2ManifestS3Uri, join(transportDir, "plan.json"));
|
||
const manifest = readPlanV2Manifest(transportDir);
|
||
if (manifest.planHash !== event.PlanHash) {
|
||
throwPlanHashMismatch(event.PlanHash, manifest.planHash);
|
||
}
|
||
const artifacts = listPlanV2ArtifactsForTarget(manifest, target);
|
||
const uniqueArtifacts = [
|
||
...new Map(artifacts.map((artifact) => [artifact.sha256, artifact])).values(),
|
||
];
|
||
await mapConcurrent(uniqueArtifacts, 16, async (artifact) => {
|
||
await downloadPlanV2Artifact(s3, event.PlanV2ArtifactS3Prefix, transportDir, artifact);
|
||
});
|
||
const planDir = join(work, "plan");
|
||
materializePlanV2Target(transportDir, target, planDir);
|
||
return planDir;
|
||
}
|
||
|
||
async function downloadPlanV2Artifact(
|
||
s3: S3Client,
|
||
artifactPrefix: string,
|
||
planV2Dir: string,
|
||
artifact: Readonly<PlanV2Artifact>,
|
||
): Promise<void> {
|
||
await downloadS3ObjectToFileVerified(
|
||
s3,
|
||
planV2BlobUri(artifactPrefix, artifact.sha256),
|
||
planV2BlobPath(planV2Dir, artifact.sha256),
|
||
artifact.sha256,
|
||
);
|
||
}
|
||
|
||
function planV2BlobPath(planV2Dir: string, digest: string): string {
|
||
return join(planV2Dir, "artifacts", "sha256", digest.slice(0, 2), digest);
|
||
}
|
||
|
||
function planV2BlobUri(prefix: string, digest: string): string {
|
||
return `${trimTrailingSlash(prefix)}/${digest.slice(0, 2)}/${digest}`;
|
||
}
|
||
|
||
function throwPlanHashMismatch(expected: string, actual: string): never {
|
||
const error = new Error(
|
||
`PLAN_HASH_MISMATCH: event PlanHash=${expected} did not match v2 manifest planHash=${actual}`,
|
||
);
|
||
error.name = "PLAN_HASH_MISMATCH";
|
||
throw error;
|
||
}
|
||
|
||
async function mapConcurrent<T>(
|
||
values: readonly T[],
|
||
concurrency: number,
|
||
fn: (value: T) => Promise<void>,
|
||
): Promise<void> {
|
||
let cursor = 0;
|
||
async function worker(): Promise<void> {
|
||
while (cursor < values.length) {
|
||
const index = cursor++;
|
||
await fn(values[index]!);
|
||
}
|
||
}
|
||
const results = await Promise.allSettled(
|
||
Array.from({ length: Math.min(concurrency, values.length) }, () => worker()),
|
||
);
|
||
const failure = results.find(
|
||
(result): result is PromiseRejectedResult => result.status === "rejected",
|
||
);
|
||
// Do not reject while sibling workers may still be writing into invocation
|
||
// scratch. The caller removes that directory in `finally`; draining the pool
|
||
// first prevents late S3 streams from racing cleanup after another GET fails.
|
||
if (failure) throw failure.reason;
|
||
}
|
||
|
||
async function downloadChunkObjects(
|
||
s3: S3Client,
|
||
uris: string[],
|
||
workDir: string,
|
||
format: DistributedFormat,
|
||
): Promise<string[]> {
|
||
const chunksDir = join(workDir, "chunks");
|
||
mkdirSync(chunksDir, { recursive: true });
|
||
// Each chunk is an independent S3 GET (+ untar for png-sequence). Run
|
||
// them in parallel — assemble's wall-clock is otherwise dominated by
|
||
// `Σ chunk-download-ms` instead of `max(chunk-download-ms)`. Preserve
|
||
// the input order by writing into a pre-sized array rather than
|
||
// pushing as each task settles.
|
||
const local: string[] = new Array<string>(uris.length);
|
||
await Promise.all(
|
||
uris.map(async (uri, i) => {
|
||
if (!uri) {
|
||
throw new Error(`[handler] chunk URI at index ${i} is empty`);
|
||
}
|
||
const { key } = parseS3Uri(uri);
|
||
const localPath = join(chunksDir, basename(key));
|
||
await downloadS3ObjectToFile(s3, uri, localPath);
|
||
if (format === "png-sequence") {
|
||
const dirPath = join(chunksDir, `frames-${pad(i)}`);
|
||
await untarDirectory(localPath, dirPath);
|
||
local[i] = dirPath;
|
||
} else {
|
||
local[i] = localPath;
|
||
}
|
||
}),
|
||
);
|
||
return local;
|
||
}
|
||
|
||
// ── Helpers ─────────────────────────────────────────────────────────────────
|
||
|
||
/** Collect every S3 URI that the handler will touch for a given event. */
|
||
// This is an exhaustive event-union projection used only for safe log summaries.
|
||
// fallow-ignore-next-line complexity
|
||
function getEventS3Uris(event: PlanEvent | RenderChunkEvent | AssembleEvent): string[] {
|
||
switch (event.Action) {
|
||
case "plan":
|
||
return [event.ProjectS3Uri, event.PlanOutputS3Prefix];
|
||
case "renderChunk":
|
||
return event.PlanProtocol === "v2"
|
||
? [event.PlanV2ManifestS3Uri, event.PlanV2ArtifactS3Prefix, event.ChunkOutputS3Prefix]
|
||
: [event.PlanS3Uri, event.ChunkOutputS3Prefix];
|
||
case "assemble":
|
||
return [
|
||
...(event.PlanProtocol === "v2"
|
||
? [event.PlanV2ManifestS3Uri, event.PlanV2ArtifactS3Prefix]
|
||
: [event.PlanS3Uri]),
|
||
...event.ChunkS3Uris,
|
||
event.OutputS3Uri,
|
||
event.AudioS3Uri,
|
||
].filter((u): u is string => u != null);
|
||
}
|
||
}
|
||
|
||
/**
|
||
* Verify every S3 URI in the event resolves to the configured render bucket.
|
||
* Throws `S3_URI_NOT_ALLOWED` (non-retryable) when a URI targets a different
|
||
* bucket, preventing event injection from reading or writing arbitrary S3 data.
|
||
*
|
||
* Skipped when `HYPERFRAMES_RENDER_BUCKET` is unset so existing deployments
|
||
* without the env var continue to work.
|
||
*/
|
||
function validateEventS3Uris(event: PlanEvent | RenderChunkEvent | AssembleEvent): void {
|
||
const allowedBucket = process.env.HYPERFRAMES_RENDER_BUCKET?.trim();
|
||
if (!allowedBucket) return;
|
||
|
||
for (const uri of getEventS3Uris(event)) {
|
||
const { bucket } = parseS3Uri(uri);
|
||
if (bucket !== allowedBucket) {
|
||
const err = new Error(
|
||
`[handler] S3_URI_NOT_ALLOWED: URI ${JSON.stringify(uri)} targets bucket "${bucket}" but only "${allowedBucket}" is permitted`,
|
||
);
|
||
err.name = "S3_URI_NOT_ALLOWED";
|
||
throw err;
|
||
}
|
||
}
|
||
}
|
||
|
||
function pad(n: number): string {
|
||
return n.toString().padStart(4, "0");
|
||
}
|
||
|
||
function trimTrailingSlash(prefix: string): string {
|
||
return prefix.endsWith("/") ? prefix.slice(0, -1) : prefix;
|
||
}
|
||
|
||
function cleanupDir(dir: string): void {
|
||
try {
|
||
// Lambda warm starts can reuse `/tmp` across invocations; clean up
|
||
// aggressively so we don't leak a chunk-sized footprint between renders.
|
||
rmSync(dir, { recursive: true, force: true });
|
||
} catch {
|
||
// Best-effort — leak is preferable to crashing on success path.
|
||
}
|
||
}
|
||
|
||
/**
|
||
* Read the untarred planDir's `plan.json` and assert its `planHash`
|
||
* matches what the Step Functions event claims. Throws on mismatch with
|
||
* a typed `PLAN_HASH_MISMATCH` error name so the state machine's typed
|
||
* non-retryable list routes it correctly.
|
||
*
|
||
* This is defense-in-depth — the producer's `renderChunk` does the same
|
||
* check internally — but performing it here lets us fail before paying
|
||
* the Chrome-launch + per-frame capture cost on a misrouted chunk.
|
||
*/
|
||
function verifyPlanHash(planDir: string, expected: string): void {
|
||
const planJsonPath = join(planDir, "plan.json");
|
||
let parsed: { planHash?: unknown };
|
||
try {
|
||
parsed = JSON.parse(readFileSync(planJsonPath, "utf-8")) as { planHash?: unknown };
|
||
} catch (err) {
|
||
const msg = err instanceof Error ? err.message : String(err);
|
||
const error = new Error(`PLAN_HASH_MISMATCH: failed to read ${planJsonPath}: ${msg}`);
|
||
error.name = "PLAN_HASH_MISMATCH";
|
||
throw error;
|
||
}
|
||
const actual = parsed.planHash;
|
||
if (typeof actual !== "string" || actual !== expected) {
|
||
const error = new Error(
|
||
`PLAN_HASH_MISMATCH: event PlanHash=${expected} did not match plan.json planHash=${String(actual)}`,
|
||
);
|
||
error.name = "PLAN_HASH_MISMATCH";
|
||
throw error;
|
||
}
|
||
}
|