perf(aws-lambda): size distributed renders from account concurrency limits (#5813)

* perf(aws-lambda): run chunks through a Distributed Map to lift the 40-chunk cap

The inline Map runs at most 40 chunk renders at once. Both state machine
definitions (CDK and SAM) now use a Distributed Map with STANDARD child
executions, capped at the function's reserved concurrency so chunks past it
queue instead of being throttled. Progress reads each chunk's child
execution history so frames, cost and chunk errors still show up.

* fix(aws-lambda): retry Lambda throttles for minutes and stop on oversized chunks

A throttled task used to burn its four catch-all retries in about 30 s.
It now gets its own jittered backoff of several minutes, because another
render on the same function can still hold the reserved slots. Plan
treats CHUNK_EXCEEDS_WORKER_TIMEOUT as terminal in both definitions.

* chore(aws-lambda): trim stack comments to keep the comment share flat

* fix(aws-lambda): keep render --wait working at hundreds of chunks

Progress polls read the chunk Map Run's counts (one DescribeMapRun) instead of
every child execution's history, take exact chunk frames and cost from the Map
output once it exits, and read failed children only when the counts show a
failure. The SDK client retries throttled calls up to 8 times with backoff.

* feat(aws-lambda): size each render's concurrency to the account instead of a fixed 8

* fix(lambda): keep the concurrency SDK external in published modules

* fix(cli): keep an existing stack's reserved concurrency on redeploy

Redeploying without --concurrency sent ReservedConcurrency=-1, which
removed the cap of 8 that earlier CLI versions reserved by default and
stopped recording it in the state file. The override is now left out
when the flag is omitted, so SAM keeps the stack's current value (a new
stack still gets the template's unreserved default). The state file
records the deployed value, and deploy prints the kept reservation.

* fix(aws-lambda): keep progress working for callers without DescribeMapRun

Callers whose IAM policy came from an earlier `lambda policies user`
lack states:DescribeMapRun, so reading a running Distributed Map threw
AccessDeniedException out of `lambda progress` and `render --wait`
while the render itself went on to succeed. getRenderProgress now
catches only that error around the map-run read, returns the parent
snapshot (status, plan and assemble progress) and adds a warning to
re-apply the policy, which both commands print. Other errors still
throw.

* fix(cli): let the documented deploy policy drop a reserved concurrency

When a redeploy removes a function's reserved concurrency, CloudFormation
calls lambda:DeleteFunctionConcurrency. The policy printed by
`lambda policies role|user` granted only PutFunctionConcurrency, so the
update failed and rolled back. Grant it, and pin it with
states:DescribeMapRun in the required-actions test.

* fix(aws-lambda): read smoke chunk hashes from the Distributed Map output

With chunks in a Distributed Map, their task results live in child
executions, so the smoke gate's chunk-hash check found 0 chunks in the
parent history and failed every v1/v2 comparison. Read Chunks[] from the
RenderChunks / RenderChunksV2 MapStateExited output instead, as
getRenderProgress does. The stack this script deploys only has the
Distributed Map, so the inline-Map path is gone.

* fix(producer): cap a pinned chunk size at the worker concurrency, not 16

With --chunk-size or --target-chunk-frames and no --max-parallel-chunks,
the plan still capped the render at 16 chunks however many workers the
account allowed, so --chunk-size 30 on a 60 s render gave 16 chunks of
113 frames. The cap is now one wave across the worker concurrency, as
the --max-parallel-chunks docs say; 16 remains when the concurrency is
unknown.

* feat(cli): remove a stack's reserved concurrency with --concurrency=none

Since an omitted --concurrency now keeps the stack's reservation, there
was no CLI way back to the unreserved default. --concurrency=none sends
ReservedConcurrency=-1, the state file records the stack as unreserved
and deploy says the reservation was removed. Positive N and an omitted
flag behave as before; any other value is rejected instead of being
silently ignored.

* chore(aws-lambda): raise the package size budget for account-sized concurrency

Measured with pnpm@10.17.1 pack after rebasing on main with the S3 assembly change: 67,763 packed / 244,685 unpacked; the budget is the check's own 5% headroom over that.
This commit is contained in:
Miguel Ángel
2026-10-19 00:57:30 +00:00
committed by GitHub
parent 5658f70065
commit 2a14cbc924
28 changed files with 982 additions and 127 deletions
+21 -7
View File
@@ -19,7 +19,8 @@ export interface DeployArgs {
stackName: string;
region: string;
awsProfile?: string;
reservedConcurrency: number;
/** -1 removes the reservation. Omitted: a redeploy keeps it; a new stack reserves none. */
reservedConcurrency?: number;
chromeSource: "sparticuz" | "chrome-headless-shell";
lambdaMemoryMb: number;
/** Override the handler-ZIP rebuild step. Useful for CI runs that already built it. */
@@ -28,17 +29,13 @@ export interface DeployArgs {
const DEFAULT_REGION = "us-east-1";
const DEFAULT_MEMORY_MB = 10240;
// Low default reserved-concurrency so a first-time user doesn't get
// surprise-billed by a runaway Map state. Adopters with their own quota
// raise this explicitly via `--concurrency`.
const DEFAULT_CONCURRENCY = 8;
export async function runDeploy(args: Partial<DeployArgs> = {}): Promise<void> {
const resolved: DeployArgs = {
stackName: args.stackName ?? DEFAULT_STACK_NAME,
region: args.region ?? process.env.AWS_REGION ?? DEFAULT_REGION,
awsProfile: args.awsProfile ?? process.env.AWS_PROFILE,
reservedConcurrency: args.reservedConcurrency ?? DEFAULT_CONCURRENCY,
reservedConcurrency: args.reservedConcurrency,
chromeSource: args.chromeSource ?? "sparticuz",
lambdaMemoryMb: args.lambdaMemoryMb ?? DEFAULT_MEMORY_MB,
skipBuild: args.skipBuild ?? false,
@@ -86,10 +83,27 @@ export async function runDeploy(args: Partial<DeployArgs> = {}): Promise<void> {
stateMachineArn: outputs.stateMachineArn,
functionName: outputs.functionName,
lambdaMemoryMb: resolved.lambdaMemoryMb,
reservedConcurrency: resolved.reservedConcurrency,
reservedConcurrency: outputs.reservedConcurrency,
deployedAt: new Date().toISOString(),
});
if (resolved.reservedConcurrency === -1) {
console.log(
c.warn(
"Removed the stack's reserved concurrency; each render now sizes itself to the account.",
),
);
} else if (
resolved.reservedConcurrency === undefined &&
outputs.reservedConcurrency !== undefined
) {
console.log(
c.warn(
`Kept the stack's reserved concurrency of ${outputs.reservedConcurrency}. Pass --concurrency=N to change it.`,
),
);
}
console.log();
console.log(c.success("Stack deployed."));
console.log(` ${c.dim("Bucket:")} ${outputs.bucketName}`);
@@ -39,6 +39,9 @@ describe("policies — required actions", () => {
"iam:CreateRole",
"logs:CreateLogGroup",
"cloudwatch:PutMetricAlarm",
// A redeploy that drops a reserved concurrency rolls back without it.
"lambda:DeleteFunctionConcurrency",
"states:DescribeMapRun",
]) {
expect(actions).toContain(must);
}
@@ -98,6 +98,7 @@ const REQUIRED_ACTIONS = {
"lambda:AddPermission",
"lambda:CreateFunction",
"lambda:DeleteFunction",
"lambda:DeleteFunctionConcurrency",
"lambda:GetFunction",
"lambda:GetFunctionConfiguration",
"lambda:InvokeFunction",
@@ -144,6 +145,7 @@ const REQUIRED_ACTIONS = {
"states:CreateStateMachine",
"states:DeleteStateMachine",
"states:DescribeExecution",
"states:DescribeMapRun",
"states:DescribeStateMachine",
"states:GetExecutionHistory",
"states:ListExecutions",
@@ -39,6 +39,7 @@ export async function runProgress(args: ProgressArgs): Promise<void> {
console.log(JSON.stringify(progress, null, 2));
return;
}
for (const warning of progress.warnings) console.warn(c.warn(warning));
const pct = Math.round(progress.overallProgress * 100);
console.log(`${c.dim("Status:")} ${statusColor(progress.status)}`);
@@ -194,12 +194,17 @@ async function waitForCompletion(
// Lazy import to avoid pulling SFN client when only `render --no-wait` is used.
const { getRenderProgress } = await loadSDK();
let lastRendered = -1;
const warned = new Set<string>();
while (true) {
const progress = await getRenderProgress({
executionArn,
region: stack.region,
defaultMemorySizeMb: stack.lambdaMemoryMb,
});
for (const warning of progress.warnings) {
if (!warned.has(warning)) console.warn(c.warn(warning));
warned.add(warning);
}
if (!json && progress.framesRendered !== lastRendered) {
lastRendered = progress.framesRendered;
const total = progress.totalFrames ?? "?";
+60 -2
View File
@@ -1,8 +1,19 @@
import { afterEach, describe, expect, it } from "vitest";
import { afterEach, describe, expect, it, vi } from "vitest";
import { execFileSync, spawnSync } from "node:child_process";
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { locateSamTemplate } from "./sam.js";
import {
fetchStackOutputs,
locateSamTemplate,
parseReservedConcurrency,
samDeploy,
} from "./sam.js";
vi.mock("node:child_process", () => ({
execFileSync: vi.fn(),
spawnSync: vi.fn(() => ({ status: 0 })),
}));
let dir: string | undefined;
@@ -30,3 +41,50 @@ describe("locateSamTemplate", () => {
expect(() => locateSamTemplate(dir!)).not.toThrow(/--sam-template/);
});
});
describe("samDeploy", () => {
const overrides = () => vi.mocked(spawnSync).mock.lastCall?.[1]?.join(" ") ?? "";
// An omitted override makes SAM send UsePreviousValue, so a redeploy keeps an existing cap.
it("sends no ReservedConcurrency when --concurrency is omitted", () => {
samDeploy({ repoRoot: "/repo", stackName: "s", region: "us-east-2" });
expect(overrides()).not.toContain("ReservedConcurrency");
});
it("forwards an explicit --concurrency", () => {
samDeploy({ repoRoot: "/repo", stackName: "s", region: "us-east-2", reservedConcurrency: 8 });
expect(overrides()).toContain("ReservedConcurrency=8");
});
});
describe("fetchStackOutputs", () => {
const stack = (reserved: string) =>
JSON.stringify({
Outputs: [
{ OutputKey: "RenderBucketName", OutputValue: "bucket" },
{ OutputKey: "RenderFunctionArn", OutputValue: "arn:aws:lambda:r:1:function:fn" },
{ OutputKey: "RenderStateMachineArn", OutputValue: "arn:sm" },
],
Parameters: [{ ParameterKey: "ReservedConcurrency", ParameterValue: reserved }],
});
it("reports the deployed reservation, and none for -1", () => {
vi.mocked(execFileSync).mockReturnValue(stack("8"));
expect(fetchStackOutputs({ stackName: "s", region: "r" }).reservedConcurrency).toBe(8);
vi.mocked(execFileSync).mockReturnValue(stack("-1"));
expect(fetchStackOutputs({ stackName: "s", region: "r" }).reservedConcurrency).toBeUndefined();
});
});
describe("parseReservedConcurrency", () => {
it("maps none to -1, keeps N, leaves an omitted flag undefined and rejects anything else", () => {
expect(parseReservedConcurrency("none")).toBe(-1);
expect(parseReservedConcurrency("8")).toBe(8);
expect(parseReservedConcurrency(undefined)).toBeUndefined();
for (const bad of ["abc", "0", "-1", "2.5"]) {
expect(() => parseReservedConcurrency(bad)).toThrow('positive integer or "none"');
}
samDeploy({ repoRoot: "/repo", stackName: "s", region: "r", reservedConcurrency: -1 });
expect(vi.mocked(spawnSync).mock.lastCall?.[1]).toContain("ReservedConcurrency=-1");
});
});
+29 -7
View File
@@ -75,13 +75,27 @@ export function locateSamTemplate(repoRoot: string): string {
return candidate;
}
/** `--concurrency`: a positive integer, or `none` (-1, the template's "unreserved"). */
export function parseReservedConcurrency(raw: unknown): number | undefined {
if (raw === undefined || raw === null || raw === "") return undefined;
if (raw === "none") return -1;
const n = Number(raw);
if (!Number.isInteger(n) || n < 1) {
throw new Error(
`[lambda] --concurrency must be a positive integer or "none"; got ${String(raw)}`,
);
}
return n;
}
/** Run `sam deploy` non-interactively. Returns when SAM exits 0; throws on non-zero. */
export function samDeploy(opts: DeployOptions): void {
assertSamAvailable();
const paramOverrides = [
`ChromeSource=${opts.chromeSource ?? "sparticuz"}`,
`ReservedConcurrency=${opts.reservedConcurrency ?? -1}`,
];
const paramOverrides = [`ChromeSource=${opts.chromeSource ?? "sparticuz"}`];
// Omitted, SAM sends UsePreviousValue: a redeploy keeps the stack's reservation, a new stack gets -1.
if (opts.reservedConcurrency !== undefined) {
paramOverrides.push(`ReservedConcurrency=${opts.reservedConcurrency}`);
}
if (opts.lambdaMemoryMb !== undefined) {
paramOverrides.push(`LambdaMemoryMb=${opts.lambdaMemoryMb}`);
}
@@ -140,6 +154,7 @@ export interface StackOutputBag {
bucketName: string;
functionName: string;
stateMachineArn: string;
reservedConcurrency?: number;
}
/**
@@ -160,7 +175,7 @@ export function fetchStackOutputs(opts: {
"--region",
opts.region,
"--query",
"Stacks[0].Outputs",
"Stacks[0]",
"--output",
"json",
];
@@ -168,8 +183,14 @@ export function fetchStackOutputs(opts: {
args.unshift("--profile", opts.awsProfile);
}
const out = execFileSync("aws", args, { encoding: "utf-8" });
const parsed = JSON.parse(out) as { OutputKey: string; OutputValue: string }[];
const byKey = new Map(parsed.map((o) => [o.OutputKey, o.OutputValue]));
const stack = JSON.parse(out) as {
Outputs?: { OutputKey: string; OutputValue: string }[];
Parameters?: { ParameterKey: string; ParameterValue: string }[];
};
const byKey = new Map((stack.Outputs ?? []).map((o) => [o.OutputKey, o.OutputValue]));
const reserved = Number(
stack.Parameters?.find((p) => p.ParameterKey === "ReservedConcurrency")?.ParameterValue,
);
const bucketName = byKey.get("RenderBucketName");
const functionName = byKey.get("RenderFunctionArn");
const stateMachineArn = byKey.get("RenderStateMachineArn");
@@ -185,5 +206,6 @@ export function fetchStackOutputs(opts: {
// for cost math + CloudWatch lookups.
functionName: functionName.split(":").pop() ?? functionName,
stateMachineArn,
reservedConcurrency: reserved > 0 ? reserved : undefined,
};
}