fix(aws-lambda): chrome-headless-shell 158 in the arm64 video image, sparticuz 153 in the x86 ZIP (#8155)

* fix(aws-lambda): run chrome-headless-shell 158 in the arm64 video image

The arm64 video image now installs Chrome for Testing's linux-arm64 chrome-headless-shell
158.0.8127.36, pinned by sha256 and fetched with curl -f, as the whole shell folder. The Lambda
Node 22 base has none of the shared libraries the shell links and no fonts, so the image installs
them from the base's repository at the same pinned release. The image sets
HYPERFRAMES_LAMBDA_CHROME_SOURCE and HYPERFRAMES_LAMBDA_CHROME_PATH itself, and the nested video
function no longer passes ChromeSource, which would override them. stage-arm64-image.ts stops
fetching the sparticuz arm64 pack and drops the ZIP's x86 sparticuz Chrome instead.

The x86 handler ZIP stays on @sparticuz/chromium, bumped from 148.0.0 to 153.0.0. It builds at
258,502,222 bytes unzipped, under Lambda's 262,144,000-byte limit and the 248 MiB in-house gate.

Two fixes for --source=chrome-headless-shell:
- build-zip.ts staged only the executable, which exits at launch without its ICU data and .pak
  files. It now stages the whole shell folder at chrome-headless-shell-linux64/.
- Nothing set HYPERFRAMES_LAMBDA_CHROME_PATH, which src/chromium.ts requires for that source.
  The SAM template and the CDK stack now set it to the bundled shell when ChromeSource asks for it.

Drops the unused @types/aws-lambda devDependency, which the dead-code audit reports whenever this
package's manifest changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(publish): build the Lambda arm64 video image on an ARM runner

The video stage now installs Chrome's system libraries with microdnf, so it runs code for arm64.
The pack job built that image on ubuntu-latest with no emulation, which it could do only while the
stage copied files. A new lambda-video-image job builds it natively on ubuntu-24.04-arm, from the
handler zip whose sha256 the zip job reported, and reports the OCI tarball's sha256. Pack downloads
that tarball, checks the sha256 against the job's, and pins it as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(aws-lambda): fit the comment ratchet and the package size budget

chromium.ts loses the two doc lines on BUNDLED_SHELL_DIR and BUNDLED_SHELL_PATH; the names and the
SAM template's comment already say what they are.

@hyperframes/aws-lambda's unpacked budget goes from 290,508 to 298,000 bytes. The package unpacks to
295,324 bytes, up from the shared chrome-headless-shell staging module the ZIP and image scripts now
import and from the README and docs that describe the two Chrome sources.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
James Russo
2026-11-18 09:07:46 +00:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 3b79b1858e
commit 81d73d2278
27 changed files with 394 additions and 202 deletions
+12 -4
View File
@@ -93,7 +93,7 @@ The package supports two Chromium sources:
| Source | Default | Size | When to pick it |
| ------------------------------- | ------- | ------------------ | --------------------------------------------------------------------------------------------------------------------- |
| `@sparticuz/chromium` | yes | ~70 MiB compressed | Lambda. Decompresses into `/tmp` at runtime; the rest of the ecosystem already uses it for headless-Chrome-in-Lambda. |
| Bundled `chrome-headless-shell` | no | ~140 MiB | Fallback. Used if `@sparticuz/chromium` ever drops `HeadlessExperimental.beginFrame` support. |
| Bundled `chrome-headless-shell` | no | ~296 MB | Fallback. Used if `@sparticuz/chromium` ever drops `HeadlessExperimental.beginFrame` support. |
Pick the source at build time:
@@ -102,9 +102,17 @@ bun run --cwd packages/aws-lambda build:zip
bun run --cwd packages/aws-lambda build:zip -- --source=chrome-headless-shell
```
The handler reads `HYPERFRAMES_LAMBDA_CHROME_SOURCE` at boot. The build
script sets that env var via Lambda function configuration in
`examples/aws-lambda/template.yaml`.
The handler reads `HYPERFRAMES_LAMBDA_CHROME_SOURCE` at boot, and for
`chrome-headless-shell` the executable from `HYPERFRAMES_LAMBDA_CHROME_PATH`.
`examples/aws-lambda/template.yaml` and the CDK construct set both from the
Chrome source. The build stages the whole shell folder (the executable needs
its ICU data and `.pak` files beside it), which is over Lambda's 250 MiB
unzipped limit for a ZIP function, so the x86 ZIP ships `@sparticuz/chromium`.
The arm64 video image (`image/Dockerfile`, `video` target) does not use
`@sparticuz/chromium`: it installs Chrome for Testing's linux-arm64
`chrome-headless-shell` folder, pinned by version and sha256, with the shared
libraries and fonts it needs, and sets both env vars itself.
## BeginFrame regression guard
+20 -1
View File
@@ -37,7 +37,26 @@ COPY --from=check /check.txt /hf-media-check.txt
# The video function. /var/task comes from scripts/stage-arm64-image.ts as the `task` build context:
# docker buildx build --platform linux/arm64 --target video --build-context task=<out>/task packages/aws-lambda/image
# Copy only, so any host builds it without emulation.
# Its Chrome is Chrome for Testing's linux-arm64 chrome-headless-shell, the whole folder: the executable
# alone finds no ICU data or .pak resources. The base has none of the shared libraries the shell links
# and no fonts, so they come from the base's own repository; this RUN needs an arm64 host or emulation.
# Bump the version and the archive's sha256 together, with the rest of the repo's pins.
FROM ${BASE} AS video
ARG CHROME_HEADLESS_SHELL_VERSION=158.0.8127.36
ARG CHROME_HEADLESS_SHELL_SHA256=176ee5c55486c22de43128871189061365ea28e881fc96016fd18e7cb59437c8
COPY fetch.sh /fetch.sh
RUN . /fetch.sh && retry microdnf install -y --nodocs --setopt=install_weak_deps=0 --releasever=2023.12.20260914 \
alsa-lib at-spi2-atk at-spi2-core atk dbus-libs expat fontconfig libX11 libXcomposite libXdamage \
libXext libXfixes libXrandr libxcb libxkbcommon liberation-fonts mesa-libgbm nspr nss nss-util \
systemd-libs unzip && \
curl -fsSL --retry 5 --retry-all-errors --retry-delay 3 --connect-timeout 20 --max-time 600 -o /tmp/chs.zip \
"https://storage.googleapis.com/chrome-for-testing-public/${CHROME_HEADLESS_SHELL_VERSION}/linux-arm64/chrome-headless-shell-linux-arm64.zip" && \
echo "${CHROME_HEADLESS_SHELL_SHA256} /tmp/chs.zip" | sha256sum -c - && \
unzip -q /tmp/chs.zip -d /opt && rm /tmp/chs.zip /fetch.sh && \
microdnf remove -y unzip && microdnf clean all && \
/opt/chrome-headless-shell-linux-arm64/chrome-headless-shell --version | grep -F "${CHROME_HEADLESS_SHELL_VERSION}"
# src/chromium.ts launches this executable.
ENV HYPERFRAMES_LAMBDA_CHROME_SOURCE=chrome-headless-shell \
HYPERFRAMES_LAMBDA_CHROME_PATH=/opt/chrome-headless-shell-linux-arm64/chrome-headless-shell
COPY --from=task / /var/task/
CMD ["handler.handler"]
+4 -5
View File
@@ -1,8 +1,7 @@
// A handler mounted into the video image and invoked through its own Lambda bootstrap, which sets
// AWS_EXECUTION_ENV as Lambda does: the bundled ffmpeg answers and arm64 Chrome renders, launched as
// src/chromium.ts launches it.
// AWS_EXECUTION_ENV as Lambda does: the bundled ffmpeg answers and the image's chrome-headless-shell
// renders, found where src/chromium.ts finds it and launched with the engine's Lambda flags.
import { execFileSync } from "node:child_process";
import chromium from "@sparticuz/chromium";
import puppeteer from "puppeteer-core";
export const handler = async () => {
@@ -10,8 +9,8 @@ export const handler = async () => {
encoding: "utf-8",
}).split("\n")[0];
const browser = await puppeteer.launch({
executablePath: await chromium.executablePath(),
args: chromium.args,
executablePath: process.env.HYPERFRAMES_LAMBDA_CHROME_PATH,
args: ["--no-sandbox", "--disable-setuid-sandbox", "--no-zygote", "--disable-dev-shm-usage"],
});
try {
const page = await browser.newPage();
+1 -2
View File
@@ -80,14 +80,13 @@
"@aws-sdk/client-sfn": "^3.1146.0",
"@aws-sdk/s3-request-presigner": "3.1146.0",
"@hyperframes/producer": "workspace:^",
"@sparticuz/chromium": "148.0.0",
"@sparticuz/chromium": "153.0.0",
"ffmpeg-static": "^5.2.0",
"ffprobe-static": "^3.1.0",
"puppeteer-core": "^25.8.0",
"tar": "^7.5.22"
},
"devDependencies": {
"@types/aws-lambda": "^8.10.146",
"@types/node": "^25.0.10",
"@types/tar": "^6.1.13",
"aws-cdk-lib": "^2.130.0",
@@ -0,0 +1,62 @@
import { cpSync, existsSync, readdirSync } from "node:fs";
import { join } from "node:path";
import { BUNDLED_SHELL_DIR } from "../src/chromium.js";
/**
* Stage the newest linux64 chrome-headless-shell from the Puppeteer cache under `home` into
* `<stagingDir>/<BUNDLED_SHELL_DIR>/`. The whole folder: the executable alone exits at launch
* ("Invalid file descriptor to ICU data received") without icudtl.dat and its .pak files.
* Returns the cached version it staged.
*/
export function stageChromeHeadlessShell(
stagingDir: string,
home = process.env.HOME ?? "/root",
): string {
// The fallback path bundles the same chrome-headless-shell build the K8s deploy uses. It is
// fetched via `@puppeteer/browsers` into the host's `~/.cache/puppeteer/`; the build re-uses that
// cache rather than redownloading.
const baseDir = join(home, ".cache", "puppeteer", "chrome-headless-shell");
if (!existsSync(baseDir)) {
throw new Error(
`[build-zip] chrome-headless-shell cache missing at ${baseDir}. Run\n` +
` npx --yes @puppeteer/browsers install chrome-headless-shell@158.0.8127.36 --path ${home}/.cache/puppeteer\n` +
`before --source=chrome-headless-shell.`,
);
}
// Sort by numeric semver descending. `sort().reverse()` is lexicographic,
// which silently picks "99.0.0" over "131.0.0" once Chrome ships
// three-digit majors that aren't strictly width-aligned. `compareSemver`
// returns negative/zero/positive on (a, b), so descending = `b - a`.
const versions = readdirSync(baseDir).sort((a, b) => compareSemver(b, a));
for (const v of versions) {
const folder = join(baseDir, v, "chrome-headless-shell-linux64");
if (existsSync(join(folder, "chrome-headless-shell"))) {
// cpSync keeps each file's mode, so the executable stays executable.
cpSync(folder, join(stagingDir, BUNDLED_SHELL_DIR), { recursive: true });
return v;
}
}
throw new Error(`[build-zip] no linux64 chrome-headless-shell binary found under ${baseDir}.`);
}
/**
* Compare two semver-shaped strings like "131.0.6778.108". Treats any
* non-numeric directory name as `-Infinity` so it sorts to the bottom
* (Puppeteer's cache layout sometimes includes `latest` or branch tags).
* Used by `stageChromeHeadlessShell` to pick the newest cached Chrome
* without tripping on the lexicographic "99 > 131" trap.
*/
function compareSemver(a: string, b: string): number {
const partsA = a.split(".").map((s) => Number.parseInt(s, 10));
const partsB = b.split(".").map((s) => Number.parseInt(s, 10));
const len = Math.max(partsA.length, partsB.length);
for (let i = 0; i < len; i++) {
const ai = partsA[i] ?? 0;
const bi = partsB[i] ?? 0;
if (Number.isNaN(ai) && Number.isNaN(bi)) continue;
if (Number.isNaN(ai)) return -1;
if (Number.isNaN(bi)) return 1;
if (ai !== bi) return ai - bi;
}
return 0;
}
+59 -2
View File
@@ -1,10 +1,12 @@
import { describe, expect, it } from "bun:test";
import { spawnSync } from "node:child_process";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { dirname, join } from "node:path";
import { pathToFileURL } from "node:url";
import * as esbuild from "esbuild";
import { BUNDLED_SHELL_DIR, BUNDLED_SHELL_PATH } from "../src/chromium.js";
import { stageChromeHeadlessShell } from "./_chromeHeadlessShell.js";
import { HANDLER_BANNER } from "./_handlerBanner.js";
// The handler ships as ESM (.mjs) but inlines CJS deps that assume Node's CJS
@@ -75,3 +77,58 @@ describe("build-zip handler banner", () => {
}
});
});
describe("build-zip chrome-headless-shell staging", () => {
function puppeteerCache(versions: Record<string, Record<string, string>>): string {
const home = mkdtempSync(join(tmpdir(), "hf-shell-cache-"));
for (const [version, files] of Object.entries(versions)) {
const folder = join(home, ".cache/puppeteer/chrome-headless-shell", version);
for (const [path, body] of Object.entries(files)) {
const file = join(folder, "chrome-headless-shell-linux64", path);
mkdirSync(dirname(file), { recursive: true });
writeFileSync(file, body, { mode: path === "chrome-headless-shell" ? 0o755 : 0o644 });
}
}
return home;
}
it("stages the whole newest shell folder where the template and the CDK stack point", () => {
const home = puppeteerCache({
"99.0.0.1": { "chrome-headless-shell": "old" },
"158.0.8127.36": {
"chrome-headless-shell": "exe",
"icudtl.dat": "icu",
"headless_lib_data.pak": "pak",
"locales/en-US.pak": "locale",
},
});
const staging = mkdtempSync(join(tmpdir(), "hf-shell-staging-"));
try {
expect(stageChromeHeadlessShell(staging, home)).toBe("158.0.8127.36");
const shell = join(staging, BUNDLED_SHELL_DIR);
// Without icudtl.dat beside it the shell exits at launch, so the folder ships whole.
expect(readFileSync(join(shell, "icudtl.dat"), "utf-8")).toBe("icu");
expect(readFileSync(join(shell, "headless_lib_data.pak"), "utf-8")).toBe("pak");
expect(readFileSync(join(shell, "locales/en-US.pak"), "utf-8")).toBe("locale");
expect(readFileSync(join(shell, "chrome-headless-shell"), "utf-8")).toBe("exe");
expect(statSync(join(shell, "chrome-headless-shell")).mode & 0o111).not.toBe(0);
expect(join("/var/task", BUNDLED_SHELL_DIR, "chrome-headless-shell")).toBe(
BUNDLED_SHELL_PATH,
);
} finally {
rmSync(home, { recursive: true, force: true });
rmSync(staging, { recursive: true, force: true });
}
});
it("names the install command when the cache is missing", () => {
const home = mkdtempSync(join(tmpdir(), "hf-shell-nocache-"));
try {
expect(() => stageChromeHeadlessShell(home, home)).toThrow(
/@puppeteer\/browsers install chrome-headless-shell@/,
);
} finally {
rmSync(home, { recursive: true, force: true });
}
});
});
+7 -57
View File
@@ -8,7 +8,9 @@
* handler.mjs — bundled entry, set as Lambda's Handler
* handler.mjs.map — sourcemap (debugging aid; small)
* bin/ffmpeg — ffmpeg-static binary
* bin/chrome-headless-shell — fallback Chrome (only when CHROME_SOURCE=shell)
* chrome-headless-shell-linux64/
* — fallback Chrome, the whole shell folder (only
* with --source=chrome-headless-shell)
* node_modules/@sparticuz/chromium/
* — primary Chrome (lives under node_modules so
* runtime `import("@sparticuz/chromium")`
@@ -47,6 +49,8 @@ import {
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import * as esbuild from "esbuild";
import { BUNDLED_SHELL_DIR } from "../src/chromium.js";
import { stageChromeHeadlessShell } from "./_chromeHeadlessShell.js";
import { formatBytes } from "./_formatBytes.js";
import { HANDLER_BANNER } from "./_handlerBanner.js";
import { RUNTIME_LOCK_DIR, assertRuntimeLockCurrent } from "./runtimeLock.js";
@@ -139,7 +143,8 @@ async function main(): Promise<void> {
// 4. If we're on the chrome-headless-shell fallback, stage that binary.
if (opts.source === "chrome-headless-shell") {
stageChromeHeadlessShell(stagingDir);
const version = stageChromeHeadlessShell(stagingDir);
console.log(`[build-zip] staged chrome-headless-shell (${version}) → ${BUNDLED_SHELL_DIR}/`);
}
// 5. Compute the unzipped size BEFORE zipping so we fail loud when over budget.
@@ -394,61 +399,6 @@ function stageFfmpeg(stagingDir: string): void {
console.log(`[build-zip] staged ffmpeg + ffprobe → bin/`);
}
function stageChromeHeadlessShell(stagingDir: string): void {
// The fallback path bundles the same chrome-headless-shell binary the
// K8s deploy uses. The binary is fetched via `@puppeteer/browsers` on
// first build into the host's `~/.cache/puppeteer/`; the build script
// re-uses that cache rather than redownloading.
const home = process.env.HOME ?? "/root";
const baseDir = join(home, ".cache", "puppeteer", "chrome-headless-shell");
if (!existsSync(baseDir)) {
throw new Error(
`[build-zip] chrome-headless-shell cache missing at ${baseDir}. Run\n` +
` npx --yes @puppeteer/browsers install chrome-headless-shell@158.0.8127.36 --path ${home}/.cache/puppeteer\n` +
`before --source=chrome-headless-shell.`,
);
}
// Sort by numeric semver descending. `sort().reverse()` is lexicographic,
// which silently picks "99.0.0" over "131.0.0" once Chrome ships
// three-digit majors that aren't strictly width-aligned. `compareSemver`
// returns negative/zero/positive on (a, b), so descending = `b - a`.
const versions = readdirSync(baseDir).sort((a, b) => compareSemver(b, a));
for (const v of versions) {
const candidate = join(baseDir, v, "chrome-headless-shell-linux64", "chrome-headless-shell");
if (existsSync(candidate)) {
const dest = join(stagingDir, "bin", "chrome-headless-shell");
mkdirSync(dirname(dest), { recursive: true });
cpSync(candidate, dest);
chmodSync(dest, 0o755);
console.log(`[build-zip] staged chrome-headless-shell (${v}) → bin/chrome-headless-shell`);
return;
}
}
throw new Error(`[build-zip] no linux64 chrome-headless-shell binary found under ${baseDir}.`);
}
/**
* Compare two semver-shaped strings like "131.0.6778.108". Treats any
* non-numeric directory name as `-Infinity` so it sorts to the bottom
* (Puppeteer's cache layout sometimes includes `latest` or branch tags).
* Used by `stageChromeHeadlessShell` to pick the newest cached Chrome
* without tripping on the lexicographic "99 > 131" trap.
*/
function compareSemver(a: string, b: string): number {
const partsA = a.split(".").map((s) => Number.parseInt(s, 10));
const partsB = b.split(".").map((s) => Number.parseInt(s, 10));
const len = Math.max(partsA.length, partsB.length);
for (let i = 0; i < len; i++) {
const ai = partsA[i] ?? 0;
const bi = partsB[i] ?? 0;
if (Number.isNaN(ai) && Number.isNaN(bi)) continue;
if (Number.isNaN(ai)) return -1;
if (Number.isNaN(bi)) return 1;
if (ai !== bi) return ai - bi;
}
return 0;
}
function zipDirectory(sourceDir: string, zipPath: string): void {
const result = spawnSync("zip", ["-rq", zipPath, "."], { cwd: sourceDir, stdio: "inherit" });
if (result.status !== 0) {
@@ -47,7 +47,7 @@ COPY packages/aws-lambda/scripts/ scripts/
RUN printf '{"name":"hf-lambda-probe","version":"1.0.0","type":"module"}\n' > package.json \
&& npm install --no-audit --no-fund --omit=optional \
@sparticuz/chromium@148.0.0 \
@sparticuz/chromium@153.0.0 \
puppeteer-core@^24.39.1 \
tsx@^4.21.0
@@ -8,7 +8,7 @@
"name": "hyperframes-aws-lambda-bundled",
"version": "0.0.0",
"dependencies": {
"@sparticuz/chromium": "148.0.0",
"@sparticuz/chromium": "153.0.0",
"puppeteer-core": "25.13.0"
}
},
@@ -41,15 +41,15 @@
}
},
"node_modules/@sparticuz/chromium": {
"version": "148.0.0",
"resolved": "https://registry.npmjs.org/@sparticuz/chromium/-/chromium-148.0.0.tgz",
"integrity": "sha512-na5beDSZkrlcEWEMt+eHu4Xe+MLUgCtHBjHaXGsNaQu5tJWwXE+McxAcMtyumEM/JzXrxGpkO5vAPD9TWhil3g==",
"version": "153.0.0",
"resolved": "https://registry.npmjs.org/@sparticuz/chromium/-/chromium-153.0.0.tgz",
"integrity": "sha512-jiGWqSvPMdu4+P9TilYqAR7yEGPEHINx+lCvE/NqNO/EHYzF/x1hUiI5T6ioVIt7dEy7n88byyT0vk+dnsgQwA==",
"license": "MIT",
"dependencies": {
"tar-fs": "^3.1.2"
},
"engines": {
"node": ">=22.17.0"
"node": "^22.17.0 || >=24.0.0"
}
},
"node_modules/ansi-regex": {
@@ -3,7 +3,7 @@
"version": "0.0.0",
"private": true,
"dependencies": {
"@sparticuz/chromium": "148.0.0",
"@sparticuz/chromium": "153.0.0",
"puppeteer-core": "25.13.0"
}
}
@@ -1,10 +1,10 @@
import { afterEach, describe, expect, it } from "bun:test";
import { spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { CHROMIUM_ARM64_PACK_SHA256, stageArm64Image } from "./stage-arm64-image.js";
import { BUNDLED_SHELL_DIR } from "../src/chromium.js";
import { stageArm64Image } from "./stage-arm64-image.js";
const X86_64 = 0x3e;
const AARCH64 = 0xb7;
@@ -53,27 +53,21 @@ function fixture(opts: { ffmpegMachine?: number; zipExtra?: Record<string, strin
"hf-media/lib/libgnutls.so.30": "gnutls",
});
pack("tar", ["-czf", "hf-media.tar.gz", "hf-media"], mediaDir);
const packDir = tree({ "chromium.br": "arm64 chrome", "al2023.tar.br": "arm64 libs" });
pack("tar", ["-cf", "pack.tar", "chromium.br", "al2023.tar.br"], packDir);
const packBytes = readFileSync(join(packDir, "pack.tar"));
return {
handlerZip: join(zipDir, "handler.zip"),
ffmpegTarball: join(mediaDir, "hf-media.tar.gz"),
outDir: join(tree({}), "out"),
fetchPack: async () => packBytes,
packPins: { "148.0.0": createHash("sha256").update(packBytes).digest("hex") },
};
}
describe("stageArm64Image", () => {
it("swaps the handler ZIP's x86 ffmpeg and Chrome for the arm64 ones", async () => {
it("swaps the handler ZIP's x86 ffmpeg for the arm64 one and drops its x86 Chrome", async () => {
const task = await stageArm64Image(fixture());
expect(readFileSync(join(task, "bin/ffmpeg")).readUInt16LE(18)).toBe(AARCH64);
expect(readFileSync(join(task, "bin/ffprobe")).readUInt16LE(18)).toBe(AARCH64);
expect(readFileSync(join(task, "lib/libgnutls.so.30"), "utf-8")).toBe("gnutls");
const chromeBin = join(task, "node_modules/@sparticuz/chromium/bin");
expect(readFileSync(join(chromeBin, "chromium.br"), "utf-8")).toBe("arm64 chrome");
expect(readFileSync(join(chromeBin, "al2023.tar.br"), "utf-8")).toBe("arm64 libs");
// The image installs its own chrome-headless-shell; the ZIP's x86 Chrome cannot run on arm64.
expect(existsSync(join(task, "node_modules/@sparticuz/chromium/bin"))).toBe(false);
expect(readFileSync(join(task, "handler.mjs"), "utf-8")).toContain("handler");
});
@@ -83,19 +77,8 @@ describe("stageArm64Image", () => {
);
});
it("refuses a Chrome pack whose sha256 is not the pinned one", async () => {
const opts = { ...fixture(), packPins: CHROMIUM_ARM64_PACK_SHA256 };
await expect(stageArm64Image(opts)).rejects.toThrow("pinned c18a5241");
});
it("refuses a @sparticuz/chromium version with no pinned arm64 pack", async () => {
await expect(stageArm64Image({ ...fixture(), packPins: {} })).rejects.toThrow(
"no arm64 Chrome pack pinned for @sparticuz/chromium 148.0.0",
);
});
it("refuses a handler ZIP built with chrome-headless-shell", async () => {
const opts = fixture({ zipExtra: { "bin/chrome-headless-shell": "x86 chrome" } });
await expect(stageArm64Image(opts)).rejects.toThrow("sparticuz Chrome only");
const opts = fixture({ zipExtra: { [`${BUNDLED_SHELL_DIR}/chrome-headless-shell`]: "x86" } });
await expect(stageArm64Image(opts)).rejects.toThrow("built from the sparticuz handler ZIP");
});
});
@@ -1,45 +1,22 @@
#!/usr/bin/env bun
// Stages /var/task for the arm64 video function image: the handler ZIP with its x86 ffmpeg and Chrome
// swapped for the native ffmpeg tree (image/Dockerfile's artifact) and Sparticuz's arm64 Chrome pack.
// Stages /var/task for the arm64 video function image: the handler ZIP with its x86 ffmpeg swapped for the
// native ffmpeg tree (image/Dockerfile's artifact) and its x86 sparticuz Chrome dropped. The image installs
// its own chrome-headless-shell (image/Dockerfile's video target).
// bun packages/aws-lambda/scripts/stage-arm64-image.ts --ffmpeg=<hf-media tarball> [--zip=..] [--out=..]
import { spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
import {
closeSync,
existsSync,
mkdirSync,
openSync,
readFileSync,
readSync,
rmSync,
} from "node:fs";
import { closeSync, existsSync, mkdirSync, openSync, readSync, rmSync } from "node:fs";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
// The npm package ships x64 Chrome only. Bumping @sparticuz/chromium fails here until its arm64 pack is pinned.
export const CHROMIUM_ARM64_PACK_SHA256: Record<string, string> = {
"148.0.0": "c18a5241b0a460cba233fff3abccd6678ebdcf855b8548bb7c58240a477f5f0e",
};
import { BUNDLED_SHELL_DIR } from "../src/chromium.js";
export interface StageOptions {
handlerZip: string;
ffmpegTarball: string;
outDir: string;
fetchPack?: (url: string) => Promise<Uint8Array>;
packPins?: Record<string, string>;
}
async function fetchBytes(url: string): Promise<Uint8Array> {
const res = await fetch(url);
if (!res.ok) throw new Error(`[stage-arm64-image] GET ${url} answered ${res.status}`);
return new Uint8Array(await res.arrayBuffer());
}
function run(cmd: string, args: string[], input?: Uint8Array): void {
const result = spawnSync(cmd, args, {
stdio: [input ? "pipe" : "inherit", "inherit", "inherit"],
input,
});
function run(cmd: string, args: string[]): void {
const result = spawnSync(cmd, args, { stdio: "inherit" });
if (result.status !== 0) {
throw new Error(`[stage-arm64-image] ${cmd} ${args.join(" ")} exited ${result.status}`);
}
@@ -59,42 +36,22 @@ export function assertArm64Elf(path: string): void {
}
}
async function stageChromiumArm64(chromiumDir: string, opts: StageOptions) {
const { version } = JSON.parse(readFileSync(join(chromiumDir, "package.json"), "utf-8")) as {
version: string;
};
// The URL is built from the pin table's own key, never from the version read off disk.
const [pinnedVersion, pinned] =
Object.entries(opts.packPins ?? CHROMIUM_ARM64_PACK_SHA256).find(([v]) => v === version) ?? [];
if (!pinnedVersion || !pinned) {
throw new Error(
`[stage-arm64-image] no arm64 Chrome pack pinned for @sparticuz/chromium ${version}; ` +
"add its sha256 to CHROMIUM_ARM64_PACK_SHA256",
);
}
const url = `https://github.com/Sparticuz/chromium/releases/download/v${pinnedVersion}/chromium-v${pinnedVersion}-pack.arm64.tar`;
const pack = await (opts.fetchPack ?? fetchBytes)(url);
const actual = createHash("sha256").update(pack).digest("hex");
if (actual !== pinned) {
throw new Error(`[stage-arm64-image] ${url} has sha256 ${actual}, pinned ${pinned}`);
}
// Only bytes that matched the pin reach tar, over stdin; the pack is never written to a file.
run("tar", ["-xf", "-", "-C", join(chromiumDir, "bin")], pack);
}
export async function stageArm64Image(opts: StageOptions): Promise<string> {
const task = join(opts.outDir, "task");
rmSync(opts.outDir, { recursive: true, force: true });
mkdirSync(task, { recursive: true });
run("unzip", ["-q", opts.handlerZip, "-d", task]);
if (existsSync(join(task, "bin", "chrome-headless-shell"))) {
throw new Error("[stage-arm64-image] the arm64 image bundles sparticuz Chrome only");
if (existsSync(join(task, BUNDLED_SHELL_DIR))) {
throw new Error(
"[stage-arm64-image] the arm64 image is built from the sparticuz handler ZIP; it installs its own chrome-headless-shell",
);
}
rmSync(join(task, "bin", "ffmpeg"));
rmSync(join(task, "bin", "ffprobe"));
run("tar", ["-xzf", opts.ffmpegTarball, "-C", task, "--strip-components=1"]);
for (const bin of ["ffmpeg", "ffprobe"]) assertArm64Elf(join(task, "bin", bin));
await stageChromiumArm64(join(task, "node_modules", "@sparticuz", "chromium"), opts);
// x86 Chrome cannot run here; the image's HYPERFRAMES_LAMBDA_CHROME_SOURCE never loads it.
rmSync(join(task, "node_modules", "@sparticuz", "chromium", "bin"), { recursive: true });
return task;
}
@@ -31,8 +31,12 @@ import {
CONTENT_STORE_REFRESH_AFTER_DAYS,
CONTENT_STORE_RETENTION_DAYS,
} from "../contentStore.js";
import { BUNDLED_SHELL_PATH } from "../chromium.js";
import { SITE_MANIFEST_INVALID } from "../siteManifest.js";
import { HyperframesRenderStack } from "./HyperframesRenderStack.js";
import {
HyperframesRenderStack,
type HyperframesRenderStackProps,
} from "./HyperframesRenderStack.js";
// CDK synth + Template.fromStack is slow on cold start in CI (~5-8s on
// the first call). The default bun:test 5s timeout trips it on the
@@ -86,7 +90,10 @@ const EXPECTED_NON_RETRYABLE_ERRORS = new Set([
"SITE_MANIFEST_INVALID",
]);
function doSynth(reservedConcurrency?: number): {
function doSynth(
reservedConcurrency?: number,
chromeSource?: HyperframesRenderStackProps["chromeSource"],
): {
template: Template;
definition: { States: Record<string, unknown>; StartAt: string };
} {
@@ -97,6 +104,7 @@ function doSynth(reservedConcurrency?: number): {
new HyperframesRenderStack(stack, "Render", {
handlerZipPath: join(zipDir, "handler.zip"),
reservedConcurrency,
chromeSource,
});
const template = Template.fromStack(stack);
const stateMachine = Object.values(
@@ -389,18 +397,60 @@ describe("HyperframesRenderStack — snapshot", () => {
);
});
it("gives the nested arm64 video function every environment variable the render function gets", () => {
it("gives the nested arm64 video function every environment variable the render function gets but its Chrome's", () => {
const sam = readExampleTemplate("template.yaml");
// The image sets these for the chrome-headless-shell it installs; a function value would override them.
const imageOwned = ["HYPERFRAMES_LAMBDA_CHROME_SOURCE", "HYPERFRAMES_LAMBDA_CHROME_PATH"];
const render = [
...envNames(sam, "Globals", "Function", "Environment", "Variables"),
...envNames(sam, "Resources", "RenderFunction", "Properties", "Environment", "Variables"),
];
].filter((name) => !imageOwned.includes(name));
const video = readExampleTemplate("video-function.yaml");
expect(
envNames(video, "Resources", "Function", "Properties", "Environment", "Variables").sort(),
).toEqual([...new Set(render)].sort());
});
it("points a chrome-headless-shell ZIP at its bundled shell, in CDK and SAM", () => {
const chromeEnv = (synthed: ReturnType<typeof doSynth>) => {
const fn = Object.values(synthed.template.findResources("AWS::Lambda::Function"))[0] as {
Properties: { Environment: { Variables: Record<string, unknown> } };
};
const vars = fn.Properties.Environment.Variables;
return [vars.HYPERFRAMES_LAMBDA_CHROME_SOURCE, vars.HYPERFRAMES_LAMBDA_CHROME_PATH];
};
expect(chromeEnv(SYNTHED)).toEqual(["sparticuz", undefined]);
expect(chromeEnv(doSynth(undefined, "chrome-headless-shell"))).toEqual([
"chrome-headless-shell",
BUNDLED_SHELL_PATH,
]);
const globals = requireRecordProperty(
requireRecordProperty(
requireRecordProperty(readExampleTemplate("template.yaml"), "Globals", "SAM globals"),
"Function",
"SAM function globals",
),
"Environment",
"SAM global environment",
) as { Variables: Record<string, unknown> };
// The YAML parser drops the !If and !Ref tags and keeps [condition, then, else].
expect(globals.Variables.HYPERFRAMES_LAMBDA_CHROME_PATH).toEqual([
"BundlesChromeHeadlessShell",
BUNDLED_SHELL_PATH,
"AWS::NoValue",
]);
const conditions = requireRecordProperty(
readExampleTemplate("template.yaml"),
"Conditions",
"SAM conditions",
);
expect(conditions.BundlesChromeHeadlessShell).toEqual([
"ChromeSource",
"chrome-headless-shell",
]);
}, 30000);
it("lets the render function abort the multipart uploads its S3 stitch starts", () => {
const cdkPolicies = JSON.stringify(SYNTHED.template.findResources("AWS::IAM::Policy"));
expect(cdkPolicies).toContain("s3:Abort*");
@@ -30,6 +30,7 @@ import * as s3 from "aws-cdk-lib/aws-s3";
import * as sfn from "aws-cdk-lib/aws-stepfunctions";
import * as tasks from "aws-cdk-lib/aws-stepfunctions-tasks";
import { Construct } from "constructs";
import { BUNDLED_SHELL_PATH } from "../chromium.js";
import { CONTENT_STORE_PREFIX, CONTENT_STORE_RETENTION_DAYS } from "../contentStore.js";
/** Construction-time props for {@link HyperframesRenderStack}. */
@@ -62,6 +63,17 @@ const DEFAULT_TIMEOUT_SEC = 900;
const DEFAULT_CHROME_SOURCE = "sparticuz";
const DEFAULT_ALARM_THRESHOLD = 1000;
/** The handler's Chrome source, and for the bundled shell, where build-zip.ts put it. */
function chromeEnvironment(
source: NonNullable<HyperframesRenderStackProps["chromeSource"]>,
): Record<string, string> {
if (source === "sparticuz") return { HYPERFRAMES_LAMBDA_CHROME_SOURCE: source };
return {
HYPERFRAMES_LAMBDA_CHROME_SOURCE: source,
HYPERFRAMES_LAMBDA_CHROME_PATH: BUNDLED_SHELL_PATH,
};
}
export class HyperframesRenderStack extends Construct {
/** S3 bucket for plan tarballs, chunk outputs, and final renders. */
readonly bucket: s3.Bucket;
@@ -117,7 +129,7 @@ export class HyperframesRenderStack extends Construct {
environment: {
NODE_OPTIONS: "--enable-source-maps",
TMPDIR: "/tmp",
HYPERFRAMES_LAMBDA_CHROME_SOURCE: chromeSource,
...chromeEnvironment(chromeSource),
HYPERFRAMES_RENDER_BUCKET: this.bucket.bucketName,
HYPERFRAMES_LAMBDA_RESERVED_CONCURRENCY: String(props.reservedConcurrency ?? -1),
HYPERFRAMES_STITCH_FUNCTION: `${projectName}-render`,
+8 -5
View File
@@ -18,11 +18,12 @@
* 2. A bundled `chrome-headless-shell` binary (fallback). If
* `@sparticuz/chromium`'s build ever drops `HeadlessExperimental`
* support, we fall back to the same `chrome-headless-shell` build
* the K8s deploy uses. The fallback raises the ZIP from ~70 MB
* Chrome to ~140 MB Chrome — still well under 250 MB.
* the K8s deploy uses. The whole shell folder (~296 MB unzipped)
* does not fit a ZIP function's 250 MiB limit; the arm64 container
* image installs it instead.
*
* The runtime path is selected by the `HYPERFRAMES_LAMBDA_CHROME_SOURCE`
* env var (set by `build-zip.ts`):
* env var (set by the SAM template, the CDK stack, or the arm64 image):
*
* "sparticuz" → use `@sparticuz/chromium.executablePath()`
* "chrome-headless-shell" → use `process.env.HYPERFRAMES_LAMBDA_CHROME_PATH`
@@ -37,6 +38,9 @@ import { existsSync } from "node:fs";
/** Discriminator for the two supported Chrome sources. */
export type ChromeSource = "sparticuz" | "chrome-headless-shell";
export const BUNDLED_SHELL_DIR = "chrome-headless-shell-linux64";
export const BUNDLED_SHELL_PATH = `/var/task/${BUNDLED_SHELL_DIR}/chrome-headless-shell`;
/**
* Thrown when the Chrome binary resolver can't produce a usable path.
* The class name is the SFN `Retry: { ErrorEquals: [...] }` discriminator —
@@ -45,8 +49,7 @@ export type ChromeSource = "sparticuz" | "chrome-headless-shell";
export class ChromeBinaryUnavailableError extends Error {
// Lambda's runtime serializes the error envelope's `errorType` from
// `err.name`; this class-field override sets it across the structured
// clone. Read indirectly; fallow can't follow.
// fallow-ignore-next-line unused-class-member
// clone.
override readonly name = "ChromeBinaryUnavailableError";
readonly source: ChromeSource;
readonly resolvedPath: string | null;