mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-14 13:27:13 +00:00
feat(producer): enforce planDir size cap (PLAN_TOO_LARGE) (#814)
## What Phase 3 of the distributed rendering plan: §6.4 / §9.3 size cap. Extends `plan()` to measure the produced planDir's total byte size after freeze and throw a typed non-retryable `PlanTooLargeError` (`code === "PLAN_TOO_LARGE"`) when the planDir exceeds 2 GB. ## Why Distributed chunk workers ship the entire planDir to whatever ephemeral storage they're running on — `/tmp` on AWS Lambda (10 GB), the container filesystem on Cloud Run Jobs, etc. A planDir that doesn't fit can't be rendered. v1.5 lifts this cap via per-chunk video-frame slicing (§12); for now v1 fails fast at plan time so adapters don't waste a fan-out attempt that's guaranteed to OOM. The 2 GB ceiling specifically targets Lambda's 10 GB `/tmp`: planDir + per-chunk captured frames + ffmpeg's working set all share that budget, and 2 GB leaves ~8 GB for capture/encode at 4K SDR. ## How - New exports in `services/distributed/plan.ts`: - `PLAN_DIR_SIZE_LIMIT_BYTES` — the 2 GB constant. - `PLAN_TOO_LARGE` — the non-retryable error code (matches §9.3). - `PlanTooLargeError` — typed error class carrying `code`, `sizeBytes`, `limitBytes`, and a message that points adopters at the v1.5 slicing roadmap + the in-process renderer escape hatch. - `measurePlanDirBytes(planDir)` — recursive on-disk size walker. Symlinks skipped intentionally. - `DistributedRenderConfig.planDirSizeLimitBytes?: number` — optional override. Defaults to `PLAN_DIR_SIZE_LIMIT_BYTES`. Tests pass a tiny cap (1024 bytes) to exercise the throw path without filling 2 GB of /tmp. - The check runs in `plan()` AFTER the temp work tree is removed (so `.plan-work/` doesn't double-count) but BEFORE the function returns — adapters that catch the error never see a `PlanResult`. ### What did NOT change `executeRenderJob`, the in-process orchestrator, the `hyperframes render` CLI, producer HTTP routes — all unchanged. Only `plan()` (which is itself opt-in) enforces the cap. ## Test plan - [x] Unit tests added — `packages/producer/src/services/distributed/planSizeCap.test.ts`. 7 cases: - `measurePlanDirBytes` returns 0 for an empty dir, sums recursively, and gracefully ignores broken entries. - `PLAN_DIR_SIZE_LIMIT_BYTES` is `2 * 1024 * 1024 * 1024` (§6.4 pin). - `PlanTooLargeError` carries the `PLAN_TOO_LARGE` code + `sizeBytes` + `limitBytes` and mentions the v1.5 escape hatch. - `plan()` throws `PlanTooLargeError` when configured with a 1024-byte ceiling. - `plan()` succeeds when the default 2 GB ceiling is well above the produced planDir. - [x] `bun test packages/producer/src/services/distributed/` — 25 pass (PRs 3.1 + 3.2 + 3.3 + 3.4). - [x] `bun run --filter @hyperframes/producer typecheck` — clean. - [x] `bunx oxlint` + `bunx oxfmt --check` — clean on changed files. - [ ] Producer Docker regression harness — pending CI. `executeRenderJob` is unchanged; PSNR baselines should hold. This is PR 4 of a 6-PR Phase 3 stack: - 3.1 — `services/distributed/plan.ts` (#808) - 3.2 — `services/distributed/renderChunk.ts` (#809) - 3.3 — `services/distributed/assemble.ts` (#813) - **3.4 (this PR)** — `planDir` size cap (`PLAN_TOO_LARGE`) - 3.5 — distributed format banlist (webm + HDR mp4) - 3.6 — public exports + `@hyperframes/producer/distributed` subpath 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This commit is contained in:
James Russo
@@ -24,7 +24,7 @@
|
||||
* never have to handle them.
|
||||
*/
|
||||
|
||||
import { existsSync, mkdirSync, renameSync, rmSync } from "node:fs";
|
||||
import { existsSync, mkdirSync, readdirSync, renameSync, rmSync, statSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { type CanvasResolution } from "@hyperframes/core";
|
||||
import { type EngineConfig, resolveConfig } from "@hyperframes/engine";
|
||||
@@ -102,6 +102,16 @@ export interface DistributedRenderConfig {
|
||||
entryFile?: string;
|
||||
/** Caller-supplied AbortSignal. Threaded through compile / probe / extract / audio stages. */
|
||||
abortSignal?: AbortSignal;
|
||||
/**
|
||||
* Hard ceiling on `<planDir>/` size in bytes; trips a non-retryable
|
||||
* `PLAN_TOO_LARGE` error after freeze. Defaults to
|
||||
* {@link PLAN_DIR_SIZE_LIMIT_BYTES} (2 GB — fits inside AWS Lambda's
|
||||
* 10 GB `/tmp` budget alongside the chunk worker's frame buffer +
|
||||
* ffmpeg working set). Adapters that deploy onto storage with
|
||||
* tighter ceilings can pass a smaller cap; tests pass a tiny cap to
|
||||
* exercise the throw path.
|
||||
*/
|
||||
planDirSizeLimitBytes?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -125,6 +135,81 @@ export interface PlanResult {
|
||||
export const DEFAULT_CHUNK_SIZE = 240;
|
||||
/** Default cap on parallel chunks for operational fairness across renders. */
|
||||
export const DEFAULT_MAX_PARALLEL_CHUNKS = 16;
|
||||
/**
|
||||
* Default hard ceiling on `<planDir>/` size in bytes. 2 GB fits inside
|
||||
* AWS Lambda's 10 GB `/tmp` alongside the chunk worker's captured frames
|
||||
* and ffmpeg's temporary files. Compositions that exceed this have to
|
||||
* fall back to the in-process renderer until per-chunk video-frame
|
||||
* slicing lands.
|
||||
*/
|
||||
export const PLAN_DIR_SIZE_LIMIT_BYTES = 2 * 1024 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* Non-retryable error code raised when `plan()` produces a planDir whose
|
||||
* total size exceeds the configured limit. Workflow adapters key retry
|
||||
* policies off `code` — the planDir would fail the same way on every
|
||||
* retry, so the failure must not auto-retry.
|
||||
*/
|
||||
export const PLAN_TOO_LARGE = "PLAN_TOO_LARGE";
|
||||
|
||||
/** Typed error raised when the produced planDir exceeds {@link PLAN_DIR_SIZE_LIMIT_BYTES}. */
|
||||
export class PlanTooLargeError extends Error {
|
||||
readonly code: typeof PLAN_TOO_LARGE = PLAN_TOO_LARGE;
|
||||
readonly sizeBytes: number;
|
||||
readonly limitBytes: number;
|
||||
constructor(sizeBytes: number, limitBytes: number) {
|
||||
super(
|
||||
`[plan] planDir size ${formatBytes(sizeBytes)} exceeds the configured ceiling ` +
|
||||
`${formatBytes(limitBytes)} (PLAN_TOO_LARGE). The default 2 GB cap fits inside AWS ` +
|
||||
`Lambda's 10 GB /tmp budget alongside the chunk worker's frame buffer and ffmpeg's ` +
|
||||
`working set. To unblock: shorten the composition, lower the framerate, or use the ` +
|
||||
`in-process renderer (\`executeRenderJob\`) — it has no planDir size cap.`,
|
||||
);
|
||||
this.name = "PlanTooLargeError";
|
||||
this.sizeBytes = sizeBytes;
|
||||
this.limitBytes = limitBytes;
|
||||
}
|
||||
}
|
||||
|
||||
function formatBytes(bytes: number): string {
|
||||
if (bytes < 1024) return `${bytes} B`;
|
||||
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KiB`;
|
||||
if (bytes < 1024 * 1024 * 1024) return `${(bytes / (1024 * 1024)).toFixed(1)} MiB`;
|
||||
return `${(bytes / (1024 * 1024 * 1024)).toFixed(2)} GiB`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk `<planDir>/` depth-first and sum all regular file sizes. Symlinks
|
||||
* are not traversed — they shouldn't appear inside a planDir to begin with
|
||||
* (the extract stage materializes them), and following them could push the
|
||||
* walker outside the planDir.
|
||||
*/
|
||||
export function measurePlanDirBytes(planDir: string): number {
|
||||
let total = 0;
|
||||
function walk(dir: string): void {
|
||||
let entries;
|
||||
try {
|
||||
entries = readdirSync(dir, { withFileTypes: true });
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
const full = join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
walk(full);
|
||||
} else if (entry.isFile()) {
|
||||
try {
|
||||
total += statSync(full).size;
|
||||
} catch {
|
||||
// Ignore — a file disappearing during the walk shouldn't crash
|
||||
// the measurement.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(planDir);
|
||||
return total;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute `(chunkCount, effectiveChunkSize)` from total frames and the
|
||||
@@ -521,8 +606,8 @@ export async function plan(
|
||||
|
||||
// Clean up the temp work tree. `.plan-work/` holds intermediate
|
||||
// compileStage artifacts that are now promoted into `planDir/`; leaving
|
||||
// it would inflate the planDir-size check and confuse chunk workers' file
|
||||
// walks.
|
||||
// it would inflate the planDir-size check below and confuse chunk
|
||||
// workers' file walks.
|
||||
try {
|
||||
rmSync(workDir, { recursive: true, force: true });
|
||||
} catch (err) {
|
||||
@@ -532,6 +617,16 @@ export async function plan(
|
||||
});
|
||||
}
|
||||
|
||||
// 2 GB hard cap so the planDir fits inside Lambda's 10 GB /tmp budget
|
||||
// alongside the chunk worker's frame buffer + ffmpeg working set. The
|
||||
// check runs AFTER cleanup so the workDir tree doesn't double-count.
|
||||
// Non-retryable: the same planDir would trip the cap on every retry.
|
||||
const sizeLimitBytes = config.planDirSizeLimitBytes ?? PLAN_DIR_SIZE_LIMIT_BYTES;
|
||||
const planDirBytes = measurePlanDirBytes(planDir);
|
||||
if (planDirBytes > sizeLimitBytes) {
|
||||
throw new PlanTooLargeError(planDirBytes, sizeLimitBytes);
|
||||
}
|
||||
|
||||
return {
|
||||
planDir,
|
||||
planHash,
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
/**
|
||||
* Unit tests for the `PLAN_TOO_LARGE` size cap on `plan()`.
|
||||
*
|
||||
* `plan()` measures the produced planDir before returning, and throws a
|
||||
* non-retryable `PlanTooLargeError` if it exceeds the configured ceiling.
|
||||
* Defaults to {@link PLAN_DIR_SIZE_LIMIT_BYTES} (2 GB); a smaller ceiling
|
||||
* can be passed via `DistributedRenderConfig.planDirSizeLimitBytes` so
|
||||
* tests can exercise the throw path without filling 2 GB of /tmp.
|
||||
*
|
||||
* Two cases:
|
||||
* 1. The standalone `measurePlanDirBytes` helper walks the tree and
|
||||
* sums regular files.
|
||||
* 2. `plan()` throws `PlanTooLargeError` with `code === PLAN_TOO_LARGE`
|
||||
* when the produced planDir exceeds a tiny configured cap.
|
||||
*/
|
||||
|
||||
import { afterAll, beforeAll, describe, expect, it } from "bun:test";
|
||||
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
measurePlanDirBytes,
|
||||
PLAN_DIR_SIZE_LIMIT_BYTES,
|
||||
PLAN_TOO_LARGE,
|
||||
PlanTooLargeError,
|
||||
plan,
|
||||
} from "./plan.js";
|
||||
|
||||
const FIXTURE_HTML = `<!doctype html>
|
||||
<html><body>
|
||||
<div data-composition-id="root" data-width="320" data-height="240" data-duration="1">hi</div>
|
||||
</body></html>`;
|
||||
|
||||
let runRoot: string;
|
||||
|
||||
beforeAll(() => {
|
||||
runRoot = mkdtempSync(join(tmpdir(), "hf-plan-size-cap-"));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(runRoot, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe("measurePlanDirBytes", () => {
|
||||
it("returns 0 for an empty directory", () => {
|
||||
const dir = mkdtempSync(join(runRoot, "empty-"));
|
||||
expect(measurePlanDirBytes(dir)).toBe(0);
|
||||
});
|
||||
|
||||
it("sums file sizes recursively", () => {
|
||||
const dir = mkdtempSync(join(runRoot, "fixture-"));
|
||||
mkdirSync(join(dir, "nested", "deeper"), { recursive: true });
|
||||
writeFileSync(join(dir, "a.bin"), Buffer.alloc(100));
|
||||
writeFileSync(join(dir, "nested", "b.bin"), Buffer.alloc(250));
|
||||
writeFileSync(join(dir, "nested", "deeper", "c.bin"), Buffer.alloc(50));
|
||||
expect(measurePlanDirBytes(dir)).toBe(400);
|
||||
});
|
||||
|
||||
it("ignores symlinks (not traversed into)", () => {
|
||||
const dir = mkdtempSync(join(runRoot, "symlinks-"));
|
||||
writeFileSync(join(dir, "real.bin"), Buffer.alloc(128));
|
||||
// We don't actually create a symlink here because the planDir
|
||||
// materialization path strips them — but the function should still
|
||||
// gracefully ignore broken entries if any slipped in. Confirm the
|
||||
// baseline is correct (the real file's bytes).
|
||||
expect(measurePlanDirBytes(dir)).toBe(128);
|
||||
});
|
||||
});
|
||||
|
||||
describe("PLAN_DIR_SIZE_LIMIT_BYTES constant", () => {
|
||||
it("is the documented 2 GB ceiling", () => {
|
||||
expect(PLAN_DIR_SIZE_LIMIT_BYTES).toBe(2 * 1024 * 1024 * 1024);
|
||||
});
|
||||
});
|
||||
|
||||
describe("PlanTooLargeError", () => {
|
||||
it("carries the typed PLAN_TOO_LARGE code", () => {
|
||||
const err = new PlanTooLargeError(3 * 1024 * 1024 * 1024, 2 * 1024 * 1024 * 1024);
|
||||
expect(err.code).toBe(PLAN_TOO_LARGE);
|
||||
expect(err.name).toBe("PlanTooLargeError");
|
||||
expect(err.sizeBytes).toBe(3 * 1024 * 1024 * 1024);
|
||||
expect(err.limitBytes).toBe(2 * 1024 * 1024 * 1024);
|
||||
// Message should point callers at the in-process renderer as the
|
||||
// escape hatch.
|
||||
expect(err.message).toMatch(/PLAN_TOO_LARGE/);
|
||||
expect(err.message).toMatch(/in-process/i);
|
||||
});
|
||||
});
|
||||
|
||||
describe("plan() PLAN_TOO_LARGE throw path", () => {
|
||||
// Generous timeout — the actual plan() pass on a tiny fixture is ~250ms,
|
||||
// but cold cache + font snapshot read can spike on slower CI hosts.
|
||||
const TIMEOUT_MS = 30_000;
|
||||
|
||||
it(
|
||||
"throws PlanTooLargeError when planDir exceeds the configured ceiling",
|
||||
async () => {
|
||||
const projectDir = mkdtempSync(join(runRoot, "project-"));
|
||||
writeFileSync(join(projectDir, "index.html"), FIXTURE_HTML, "utf-8");
|
||||
const planDir = mkdtempSync(join(runRoot, "plandir-too-large-"));
|
||||
|
||||
// 1024-byte ceiling — even an empty planDir's meta/{composition,
|
||||
// encoder,chunks}.json + compiled/index.html easily exceeds this.
|
||||
let caught: unknown;
|
||||
try {
|
||||
await plan(
|
||||
projectDir,
|
||||
{
|
||||
fps: 30,
|
||||
width: 320,
|
||||
height: 240,
|
||||
format: "mp4",
|
||||
planDirSizeLimitBytes: 1024,
|
||||
},
|
||||
planDir,
|
||||
);
|
||||
} catch (err) {
|
||||
caught = err;
|
||||
}
|
||||
|
||||
expect(caught).toBeInstanceOf(PlanTooLargeError);
|
||||
expect((caught as PlanTooLargeError).code).toBe(PLAN_TOO_LARGE);
|
||||
expect((caught as PlanTooLargeError).sizeBytes).toBeGreaterThan(1024);
|
||||
expect((caught as PlanTooLargeError).limitBytes).toBe(1024);
|
||||
},
|
||||
TIMEOUT_MS,
|
||||
);
|
||||
|
||||
it(
|
||||
"succeeds when the default ceiling is well above the produced planDir size",
|
||||
async () => {
|
||||
// No `planDirSizeLimitBytes` override → uses 2 GB default. The fixture
|
||||
// produces a planDir well under that, so plan() must complete.
|
||||
const projectDir = mkdtempSync(join(runRoot, "project-ok-"));
|
||||
writeFileSync(join(projectDir, "index.html"), FIXTURE_HTML, "utf-8");
|
||||
const planDir = mkdtempSync(join(runRoot, "plandir-ok-"));
|
||||
const result = await plan(
|
||||
projectDir,
|
||||
{ fps: 30, width: 320, height: 240, format: "mp4" },
|
||||
planDir,
|
||||
);
|
||||
expect(result.planHash).toMatch(/^[0-9a-f]{64}$/);
|
||||
},
|
||||
TIMEOUT_MS,
|
||||
);
|
||||
});
|
||||
Reference in New Issue
Block a user