Files
hyperframes/packages/cli/src/commands/lambda/render.ts
T
Miguel Ángel 992df8b203 perf(cloud): move project bytes once through shared content storage (#5772)
* perf(producer): distributed chunks download only the videos they show

A project video under compiled/ that chunks extract their own frames from
now goes only to the chunks that show it, like video-sources/ files. A
chunk skips the readiness wait for a deferred video it does not show, so
it no longer logs media_load_failed for a file it was never given.

* fix(producer): cloud site uploads pack the files their id hashes, symlinks included

* perf(aws-lambda): chunks read only the byte ranges of a video they need

A chunk that extracts its own frames from a deferred project video now
reads it from S3 by byte range through a short-lived pre-signed URL,
instead of downloading and hashing the whole object first. S3 attests the
object's stored SHA-256 before the URL is signed; every other artifact is
still downloaded and verified as before.

* perf(aws-lambda): upload project files once to a shared content-addressed area

* chore(producer): keep planV2 comment share at the base level

* fix(engine): verify the TLS certificate of https video inputs and never surface their signed query

* fix(aws-lambda): renders re-upload a site's expired files instead of failing

* perf(aws-lambda): the plan step probes project videos in place instead of downloading them

* fix(engine): https video reads reconnect and time out, and redaction covers wrapped causes

Signed-URL reads now pass -rw_timeout, -reconnect and -reconnect_delay_max (both Lambda builds accept them) and ffmpeg also -reconnect_on_network_error, so a transient S3 error or a stalled read fails fast or recovers instead of waiting out the process timeout. Query redaction now descends into error.cause.

* fix(aws-lambda): share entry bundles and ratchet package budget

* fix(aws-lambda): site ids uploaded before files.json still render

A site id from an earlier sites create has only sites/<id>/project.tar.gz.
renderToLambda now falls back to that tarball, which the plan step still
reads, and when neither key exists it fails before starting with both keys
named and the fix (run sites create again).

* fix(aws-lambda): renders fail before upload on a stack too old to read sites

A stack deployed before files.json sites untars the site file in its plan
step and fails after four retries. renderToLambda now reads the state
machine definition first and stops with STACK_TOO_OLD and the redeploy
command when the plan task does not list SITE_MANIFEST_INVALID, which new
SAM and CDK stacks now treat as terminal. The README's upgrade section
drops the planProtocol v1 fallback, which hits the same failure.

* docs(aws-lambda): document shared content storage and the upgrade path

The deploy guide still described the tarball HeadObject short-circuit. It
now covers the cas/sha256/ store and its 30-day lifecycle rule, the
STACK_TOO_OLD, SITE_FILES_MISSING and SITE_NOT_FOUND errors, that old site
ids keep working, and that every project uploads once more after the
upgrade. A test keeps a troubleshooting entry for each pre-submit error.

* fix(producer): chunks log each video they read by URL, without its query

A chunk reading a video straight from the bucket left no trace in its log.
It now logs one line per remote source with the path and the URL minus its
query string, so signatures never reach the log.

* fix(engine): ffmpeg errors never keep part of a signed URL's query

A process's stderr tail was trimmed before its URLs were redacted, so a
cut inside a signed URL left a query fragment the URL pattern no longer
matched, and the fragment reached the error. The stderr tail is now
redacted before every trim, and the redactor also strips a signing
parameter cut off from its URL.

* fix(aws-lambda): comments name the site manifest instead of a file path

* fix(engine): stderr redaction survives chunk splits, and every ffmpeg spawn uses it

The stderr tail was redacted on each chunk and then cut, so once a cut
severed a long URL's scheme, the rest of its signature in the next chunk
matched neither pattern. The tail is now kept raw with 2x slack and
redacted only when the text is produced, then cut to size. The unused
producer audioExtractor (its own spawn, unredacted stderr) is deleted and
the parity harness's PCM decode goes through ManagedChildProcess; a test
fails on any ffmpeg/ffprobe spawn in engine, producer or aws-lambda that
does not.

* chore(producer): record the audioExtractor deletion as intended

* fix(engine): drop a signature fragment the stderr tail cut from its URL

The raw tail is kept at twice the cap, but that cut can itself land inside a
signature, and when redaction shrinks the rest below the cap the fragment
survived. After a cut, the severed token is dropped before redaction.

* refactor(engine): name the severed-token drop instead of commenting it

* test(engine): a cut-off URL leaves nothing behind; aws-lambda size budget to measured

* fix(producer): chunk URL-read tests use main's frame list; producer size budget to measured

* chore(producer): size budget at the measured size, no headroom

* chore(producer): size budget back to measured x1.05, the check's own rule
2026-10-19 19:23:51 +00:00

265 lines
10 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { setCommandExitCode } from "../../utils/commandResult.js";
/**
* `hyperframes lambda render <projectDir>` — start a distributed render
* against the deployed stack. Wraps {@link renderToLambda}. Does NOT
* poll — use `hyperframes lambda progress` for that.
*/
import { existsSync } from "node:fs";
import { join, resolve as resolvePath } from "node:path";
import type {
DistributedFormat,
SerializableDistributedRenderConfig,
} from "@hyperframes/aws-lambda/sdk";
import type { CanvasResolution, DistributedFps } from "@hyperframes/core";
import { c } from "../../ui/colors.js";
import {
reportVariableIssues,
resolveVariablesArg,
validateVariablesAgainstProject,
} from "../../utils/variables.js";
import { warnOnDimensionMismatch } from "./_dimensions.js";
import { requireStack, stateFilePath } from "./state.js";
// Dynamic-import the SDK so tsup keeps it out of the static-import head of
// the CLI bundle. See sites.ts loadSDK() for the full rationale.
async function loadSDK(): Promise<typeof import("@hyperframes/aws-lambda/sdk")> {
return import("@hyperframes/aws-lambda/sdk");
}
export interface RenderArgs {
projectDir: string;
stackName: string;
siteId?: string;
/** Composition config — fps/width/height/format required, rest optional. */
fps: DistributedFps;
width: number;
height: number;
/**
* Optional output resolution preset that engages Chrome `deviceScaleFactor`
* supersampling. When set, the composition is laid out at the canvas size
* declared by `data-width`/`data-height` (or `--width`/`--height`) and
* supersampled to the preset's dimensions — `landscape-4k` (3840×2160)
* from a 1920×1080 layout uses DPR 2, etc.
*/
outputResolution?: CanvasResolution;
/**
* True when {@link outputResolution} was normalized from an aspect-agnostic
* alias (`1080p` / `hd` / `4k` / `uhd`). Threaded through
* `SerializableDistributedRenderConfig` so the Lambda worker's compile stage
* remaps `landscape` → `portrait` / `square` when the composition's
* orientation demands it. Dropping this field is what shipped the
* portrait-1080p regression at the sibling entrypoints (see PR #2529).
*/
outputResolutionAspectAgnostic?: boolean;
format: DistributedFormat;
codec?: "h264" | "h265";
quality?: "draft" | "standard" | "high";
chunkSize?: number;
maxParallelChunks?: number;
targetChunkFrames?: number;
executionName?: string;
outputKey?: string;
/** Inline JSON for `--variables '{...}'`. Mutually exclusive with `variablesFile`. */
variables?: string;
/** Path to a JSON file for `--variables-file ./vars.json`. */
variablesFile?: string;
/**
* Fail the command if any `--variables` key is undeclared or has a wrong
* type vs the composition's `data-composition-variables`. Without this
* flag, mismatches are warnings (matches the local `hyperframes render`
* behavior).
*/
strictVariables?: boolean;
/** Print machine-readable JSON instead of the human-friendly summary. */
json: boolean;
/** Block until the render finishes. Polls `progress` until SUCCEEDED/FAILED. */
wait: boolean;
/** Poll cadence in ms when `--wait` is set. */
waitIntervalMs: number;
}
// fallow-ignore-next-line complexity
export async function runRender(args: RenderArgs): Promise<void> {
const stack = requireStack(args.stackName);
const projectDir = resolvePath(args.projectDir);
warnOnDimensionMismatch({
projectDir,
cliWidth: args.width,
cliHeight: args.height,
outputResolution: args.outputResolution,
quiet: args.json,
});
// Resolve --variables / --variables-file using the same parser the local
// `hyperframes render` uses. `resolveVariablesArg` exits(1) with a friendly
// errorBox on parse errors so callers don't have to.
const variables = resolveVariablesArg(args.variables, args.variablesFile);
// Validate against the composition's `data-composition-variables`
// declaration when present. The local CLI silently treats unreadable
// index.html as "no declarations" — mirror that. Skip validation
// entirely when the project dir is missing on disk (e.g. `--site-id`
// pointing at a pre-uploaded site that was packaged on another machine).
if (variables && Object.keys(variables).length > 0) {
const indexPath = join(projectDir, "index.html");
if (existsSync(indexPath)) {
const issues = validateVariablesAgainstProject(indexPath, variables);
// Suppress the warning block when --json is set; stdout is reserved
// for the manifest. The strict-mode errorBox still prints to stderr
// and exits, so machine consumers still get a non-zero exit.
reportVariableIssues(issues, { strict: args.strictVariables ?? false, quiet: args.json });
} else if (args.strictVariables && !args.json) {
// --strict-variables asks for typed checking but there's no
// index.html to check against (typical with --site-id pointing at a
// pre-uploaded site). Make that silent skip visible so the flag
// doesn't quietly become a no-op.
console.warn(
c.warn(
`--strict-variables: no ${indexPath} on disk — schema validation skipped. ` +
"Variables flow through unchecked. To enable strict checking, run from a project dir that contains the composition.",
),
);
}
}
const config: SerializableDistributedRenderConfig = buildLambdaRenderConfig(
args,
variables,
stack.reservedConcurrency,
);
// --site-id renders a site an earlier `lambda sites create` uploaded.
const { renderToLambda, siteHandleFromId } = await loadSDK();
const siteHandle = args.siteId ? siteHandleFromId(stack.bucketName, args.siteId) : undefined;
const handle = await renderToLambda({
projectDir,
siteHandle,
bucketName: stack.bucketName,
stateMachineArn: stack.stateMachineArn,
region: stack.region,
config,
executionName: args.executionName,
outputKey: args.outputKey,
});
if (args.json) {
// --wait + --json should emit a single parseable JSON document: the
// final progress snapshot. Without --wait, emit the handle (the
// caller will poll progress separately). Previously this printed
// both, producing two concatenated JSON blobs that `jq -r` would
// misparse.
if (args.wait) {
await waitForCompletion(handle.executionArn, stack, args.waitIntervalMs, args.json);
} else {
console.log(JSON.stringify(handle, null, 2));
}
return;
}
console.log(c.success("Render started."));
console.log(` ${c.dim("Render ID:")} ${handle.renderId}`);
console.log(` ${c.dim("Execution ARN:")} ${handle.executionArn}`);
console.log(` ${c.dim("Output S3 URI:")} ${handle.outputS3Uri}`);
console.log(` ${c.dim("Project S3:")} ${handle.projectS3Uri}`);
console.log(` ${c.dim("Stack state:")} ${stateFilePath(args.stackName)}`);
console.log();
if (args.wait) {
await waitForCompletion(handle.executionArn, stack, args.waitIntervalMs, args.json);
return;
}
console.log(c.dim(`Poll with: hyperframes lambda progress ${handle.renderId}`));
}
async function waitForCompletion(
executionArn: string,
stack: { region: string; functionName: string; lambdaMemoryMb: number },
intervalMs: number,
json: boolean,
): Promise<void> {
// Lazy import to avoid pulling SFN client when only `render --no-wait` is used.
const { getRenderProgress } = await loadSDK();
let lastRendered = -1;
const warned = new Set<string>();
while (true) {
const progress = await getRenderProgress({
executionArn,
region: stack.region,
defaultMemorySizeMb: stack.lambdaMemoryMb,
});
for (const warning of progress.warnings) {
if (!warned.has(warning)) console.warn(c.warn(warning));
warned.add(warning);
}
if (!json && progress.framesRendered !== lastRendered) {
lastRendered = progress.framesRendered;
const total = progress.totalFrames ?? "?";
const pct = Math.round(progress.overallProgress * 100);
console.log(
` ${c.dim(`[${progress.status}]`)} ${pct}% • ${progress.framesRendered}/${total} frames • ${progress.costs.displayCost}`,
);
}
if (progress.status !== "RUNNING") {
if (json) {
console.log(JSON.stringify(progress, null, 2));
} else if (progress.status === "SUCCEEDED" && progress.outputFile) {
console.log();
console.log(c.success("Render complete."));
console.log(` ${c.dim("Output:")} ${progress.outputFile.s3Uri}`);
console.log(` ${c.dim("Size:")} ${progress.outputFile.bytes ?? "?"} bytes`);
console.log(` ${c.dim("Total cost:")} ${progress.costs.displayCost}`);
} else {
console.log();
console.log(c.error(`Render ended with status ${progress.status}.`));
for (const err of progress.errors) {
console.log(` ${c.dim(err.state)}: ${err.error} — ${err.cause}`);
}
setCommandExitCode(1);
}
return;
}
await sleep(intervalMs);
}
}
function sleep(ms: number): Promise<void> {
return new Promise((res) => setTimeout(res, ms));
}
/**
* Build the wire {@link SerializableDistributedRenderConfig} for
* `hyperframes lambda render`. Extracted for boundary-test coverage of the
* aspect-agnostic flag threading — the field this helper preserves is what
* lets the Lambda worker's compile stage remap `landscape` → `portrait` on
* an aspect-agnostic alias (`1080p` / `hd` / `4k` / `uhd`).
*/
export function buildLambdaRenderConfig(
args: RenderArgs,
variables: Record<string, unknown> | undefined,
workerConcurrency?: number,
): SerializableDistributedRenderConfig {
const config: SerializableDistributedRenderConfig = {
fps: args.fps,
width: args.width,
height: args.height,
outputResolution: args.outputResolution,
format: args.format,
codec: args.codec,
quality: args.quality,
chunkSize: args.chunkSize,
maxParallelChunks: args.maxParallelChunks,
targetChunkFrames: args.targetChunkFrames,
workerConcurrency,
runtimeCap: "lambda",
variables,
};
// Keep the wire shape sparse when the alias flag wasn't set — matches how
// `SerializableDistributedRenderConfig` is emitted at every other adapter.
if (args.outputResolutionAspectAgnostic) {
config.outputResolutionAspectAgnostic = true;
}
return config;
}