fix(producer): address PR review feedback on harness mode + plan() copy filter

Miguel (approved) and Vai (commented) both flagged the same
PSNR-threshold doc/code mismatch; Vai additionally flagged a
path-anchoring bug in the projectDir-copy filter and a dishonest type
cast. Addressed all five findings:

PSNR threshold doc/code mismatch (important):
- Module docstring, `resolveMinPsnrForMode` JSDoc, and tests/README.md all
  claimed distributed-simulated tightens to ≥50 dB. The actual code uses
  `max(fixture.minPsnr, 10)` — 10 dB is a pathology floor, the per-test
  gate is the fixture's authored `minPsnr`. Updated all three doc sites
  to describe what the code does. The 50 dB target in §5.1 is a per-
  render distributed-vs-in-process contract; against the frozen baseline
  it's unreachable for either mode (shared encoder/JPEG jitter), so it
  can't be a per-fixture gate.

`PLAN_PROJECT_DIR_COPY_SKIP` regex matched absolute paths (important):
- `cpSync` calls the filter with the absolute source path, so a
  `projectDir` whose absolute path happens to contain a blocklisted
  segment (`/home/user/work/output/comp/`, `~/projects/dist/foo/`, etc.)
  caused the filter to return false for every descendant — empty
  compiled directory, broken render. Now matches relative-to-projectDir
  segments via `path.relative()` + `split(sep)`. Switched from a regex
  to a Set for clarity. Harness fixtures don't hit this because they
  live under `tests/<name>/src/`, but adapters call `plan()` with
  caller-supplied paths.

Dishonest type cast in regression-harness.ts (important):
- `as "mp4" | "mov" | "png-sequence"` claimed reachability for formats
  that `validateMetadata` doesn't accept (the schema is `"mp4" | "webm"`,
  and webm is rejected by `checkDistributedSupport`). Narrowed to
  hardcoded `format: "mp4"` with a comment naming the metadata-schema
  invariant that lets us do that.

Renamed `chunkVideoInjectorFactory` (nit):
- The variable was invoked once and never used again — "factory" implied
  repeated calls. Inlined as a plain `videoInjector: BeforeCaptureHook | null`
  ternary.

Replaced tautology test (nit):
- `expect(DISTRIBUTED_SIMULATED_MIN_PSNR_DB).toBe(10)` was a value-pin
  over an exported constant. The invariant the JSDoc actually asserts is
  "10 dB is below any real fixture's authored minPsnr"; if someone lands
  a permissive fixture (minPsnr: 5), the value-pin doesn't catch it.
  Replaced with a test that walks `tests/*/meta.json` and asserts every
  authored `minPsnr` is ≥ the floor.

Validated in `docker:test --mode=distributed-simulated`:
  font-variant-numeric, many-cuts, gsap-letters-render-compat,
  style-1-prod, sub-composition-video — all PASSED.
Unit tests: 15/15 pass (new fixture-scan test included).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
James
2026-05-14 20:33:03 +00:00
co-authored by Claude Opus 4.7
parent b8e8617f80
commit e50587496f
6 changed files with 110 additions and 58 deletions
@@ -113,11 +113,32 @@ describe("resolveMinPsnrForMode()", () => {
);
});
it("DISTRIBUTED_SIMULATED_MIN_PSNR_DB is the absolute-pathology floor", () => {
// 10 dB is far below any real fixture's authored minPsnr (the lowest
// among committed fixtures is 30 dB). It exists as a non-zero guard
// for distributed-mode regressions that render fully-black frames
// against a fixture authored with `minPsnr: 0`.
expect(DISTRIBUTED_SIMULATED_MIN_PSNR_DB).toBe(10);
it("every committed fixture authors a minPsnr above the absolute floor", async () => {
// The pathology floor only fires for a fixture whose authored minPsnr
// is below it — by design that should be no committed fixture. If
// someone lands a permissive fixture (minPsnr: 5), distributed mode
// will silently use 10 dB instead, which is the right behavior but
// worth flagging so reviewers ask "is this fixture really meant to
// accept near-black output?". This test prevents accidental misuse
// by failing loudly when a fixture drops below the floor.
const { readdirSync, readFileSync, statSync } = await import("node:fs");
const { join: pathJoin } = await import("node:path");
const testsDir = pathJoin(import.meta.dir, "..", "tests");
const offenders: Array<{ fixture: string; minPsnr: number }> = [];
for (const entry of readdirSync(testsDir)) {
const metaPath = pathJoin(testsDir, entry, "meta.json");
let stat;
try {
stat = statSync(metaPath);
} catch {
continue;
}
if (!stat.isFile()) continue;
const meta = JSON.parse(readFileSync(metaPath, "utf-8")) as { minPsnr?: unknown };
if (typeof meta.minPsnr === "number" && meta.minPsnr < DISTRIBUTED_SIMULATED_MIN_PSNR_DB) {
offenders.push({ fixture: entry, minPsnr: meta.minPsnr });
}
}
expect(offenders).toEqual([]);
});
});
@@ -12,18 +12,23 @@
* or Lambda involvement: the controller and chunk worker are both this
* process, but they go through the same artifact (planDir + frozen
* `meta/encoder.json` + per-chunk concat-copy) that a real fan-out
* would. Validates the determinism contract from §5.1 — the distributed
* pipeline's output must be PSNR ≥ 50 dB against the in-process
* baseline (the contract isn't byte-identical because streaming-encode
* fusion is unavailable when chunks are encoded separately).
* would.
*
* Both modes share the per-fixture `minPsnr` threshold — distributed must
* pass the same quality bar the in-process renderer passes against the
* same frozen baseline. A separate {@link DISTRIBUTED_SIMULATED_MIN_PSNR_DB}
* pathology floor catches the case where a fixture authored a permissive
* threshold and distributed regresses to fully-black output. The §5.1
* 50 dB target was written for per-render comparison (fresh in-process vs
* fresh distributed); against the frozen baseline file it's unreachable
* for either mode due to shared encoder/JPEG-capture jitter, so the
* harness can't use it as a per-test gate.
*
* Not every fixture can run in distributed-simulated mode. Distributed mode
* refuses webm, HDR mp4, NTSC framerates, and non-{24,30,60} fps at plan
* time. Fixtures that don't meet the constraints are skipped — the harness
* logs the reason and the fixture is treated as "passed (skipped)" in
* distributed-simulated mode. The full determinism contract lands when the
* Phase 4 fixtures (`tests/distributed/<name>/`) cover every supported
* format and adapter.
* distributed-simulated mode.
*/
import { existsSync, mkdirSync } from "node:fs";
@@ -183,12 +188,13 @@ export async function runDistributedSimulatedRender(
}
/**
* Pick the PSNR threshold for a fixture given the harness mode. In-process
* uses the fixture's authored `minPsnr` (which may be as low as 30 dB to
* absorb per-fixture jitter). Distributed-simulated tightens to the
* §5.1 contract floor (50 dB) but never goes below the fixture's own
* threshold — a fixture that demands ≥55 dB in-process stays at ≥55 dB
* distributed.
* Pick the PSNR threshold for a fixture given the harness mode. Both modes
* share the fixture's authored `minPsnr` — distributed must clear the same
* quality bar in-process clears against the same frozen baseline.
* Distributed-simulated additionally lifts the threshold to
* {@link DISTRIBUTED_SIMULATED_MIN_PSNR_DB} for fixtures with a permissive
* authored threshold; that absolute floor catches fully-black-output
* regressions independent of fixture tolerance.
*/
export function resolveMinPsnrForMode(mode: HarnessMode, fixtureMinPsnr: number): number {
if (mode === "in-process") return fixtureMinPsnr;
+7 -6
View File
@@ -718,18 +718,19 @@ async function runTestSuite(
return result;
}
// `checkDistributedSupport` already narrowed fps to {24,30,60} and
// rejected webm; the `as` casts surface those guarantees to TS.
// rejected webm; the cast surfaces that guarantee to TS.
const fpsNum = suite.meta.renderConfig.fps.num as 24 | 30 | 60;
const distributedFormat = (suite.meta.renderConfig.format ?? "mp4") as
| "mp4"
| "mov"
| "png-sequence";
// `validateMetadata` only accepts `format: "mp4" | "webm"` in
// `renderConfig`, and `checkDistributedSupport` rejected webm above,
// so by here only `mp4` (or the unset default) can reach this call.
// If the metadata schema grows to accept "mov" / "png-sequence"
// someday, narrow this cast accordingly.
await runDistributedSimulatedRender({
projectDir: tempSrcDir,
tempRoot,
renderedOutputPath,
fps: fpsNum,
format: distributedFormat,
format: "mp4",
chunkSize: suite.meta.renderConfig.chunkSize,
maxParallelChunks: suite.meta.renderConfig.maxParallelChunks,
variables: suite.meta.renderConfig.variables,
@@ -34,7 +34,7 @@ import {
statSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { join, relative, sep } from "node:path";
import { type CanvasResolution } from "@hyperframes/core";
import { type EngineConfig, resolveConfig } from "@hyperframes/engine";
import { defaultLogger, type ProducerLogger } from "../../logger.js";
@@ -147,13 +147,24 @@ export interface PlanResult {
}
/**
* Skip patterns for the `projectDir → planDir/compiled/` pre-seed copy.
* Real projects often contain `node_modules/`, VCS metadata, and harness
* artifacts that have no business in a planDir they bloat the 2 GB
* planDir cap and slow the S3/Lambda round-trip for no benefit.
* Top-level directory names skipped by the `projectDir → planDir/compiled/`
* pre-seed copy. Real projects often contain `node_modules/`, VCS metadata,
* and harness artifacts that have no business in a planDir they bloat
* the 2 GB planDir cap and slow the S3/Lambda round-trip for no benefit.
* Matched against the path relative to `projectDir` so a `projectDir`
* whose absolute path happens to contain one of these names (e.g.
* `~/work/output/comp/`) doesn't false-positive-skip the entire copy.
*/
const PLAN_PROJECT_DIR_COPY_SKIP =
/(^|\/)(node_modules|\.git|\.cache|output|failures|dist|\.next|\.turbo)(\/|$)/;
const PLAN_PROJECT_DIR_SKIP_SEGMENTS = new Set([
"node_modules",
".git",
".cache",
"output",
"failures",
"dist",
".next",
".turbo",
]);
/** Default chunk size in frames (~8s @ 30fps; fits Lambda's 15-min cap). */
export const DEFAULT_CHUNK_SIZE = 240;
@@ -527,7 +538,15 @@ export async function plan(
cpSync(projectDir, compiledDir, {
recursive: true,
dereference: true,
filter: (src) => !PLAN_PROJECT_DIR_COPY_SKIP.test(src),
filter: (src) => {
// cpSync passes the absolute source path. Compare relative-to-projectDir
// so a parent directory of projectDir matching a skip name doesn't
// false-positive every descendant.
const rel = relative(projectDir, src);
if (rel === "" || rel.startsWith("..")) return true;
const firstSegment = rel.split(sep, 1)[0];
return firstSegment === undefined || !PLAN_PROJECT_DIR_SKIP_SEGMENTS.has(firstSegment);
},
});
// The compiled directory lives at `<planDir>/compiled/` in the final
@@ -439,21 +439,21 @@ export async function renderChunk(
forceScreenshot: encoder.forceScreenshot,
};
// Rebuild the BeforeCaptureHook that injects pre-extracted video
// frames into the page. Computed once per chunk and reusedthe
// `createRenderVideoFrameInjector` callback `runCaptureStage` accepts
// can fire on every frame, and re-listing `planDir/video-frames/`
// each time would be wasteful (the directory contents don't change
// for the duration of a chunk render). Compositions with no video
// elements produce `null` here, matching the in-process renderer's
// skip path.
const chunkVideoInjectorFactory = (): BeforeCaptureHook | null => {
if (!planVideos || planVideos.extracted.length === 0) return null;
const extracted = rebuildExtractedFramesFromPlanDir(planDir, planVideos.extracted);
const frameLookup = createFrameLookupTable(planVideos.videos, extracted);
return createVideoFrameInjector(frameLookup);
};
const cachedVideoInjector = chunkVideoInjectorFactory();
// Build the BeforeCaptureHook that injects pre-extracted video frames
// into the page once per chunk and reuse — `runCaptureStage` may
// invoke `createRenderVideoFrameInjector` multiple times, and
// re-listing `planDir/video-frames/` each call would be wasteful.
// Compositions with no video elements produce `null`, matching the
// in-process renderer's skip path.
const videoInjector: BeforeCaptureHook | null =
planVideos && planVideos.extracted.length > 0
? createVideoFrameInjector(
createFrameLookupTable(
planVideos.videos,
rebuildExtractedFramesFromPlanDir(planDir, planVideos.extracted),
),
)
: null;
// ── Per-chunk work + frames directories ──
// Suffix workDir with pid + random bytes so concurrent invocations on
@@ -530,7 +530,7 @@ export async function renderChunk(
needsAlpha: plan.dimensions.format !== "mp4",
captureAttempts: [],
buildCaptureOptions: () => captureOptions,
createRenderVideoFrameInjector: () => cachedVideoInjector,
createRenderVideoFrameInjector: () => videoInjector,
abortSignal: undefined,
assertNotAborted: () => {},
frameRange: { startFrame: slice.startFrame, endFrame: slice.endFrame },
+14 -9
View File
@@ -96,11 +96,16 @@ passing in the summary):
- `format === "webm"``plan()` refuses webm.
- `hdr === true` — distributed mode is SDR-only at v1.
For fixtures that *are* supported, the PSNR threshold tightens to **≥ 50
dB** (the §5.1 determinism contract) or the fixture's own `minPsnr`,
whichever is higher. A failure at this threshold means the distributed
pipeline has drifted from in-process output — file an issue rather than
adjusting the threshold.
Both modes use the fixture's authored `minPsnr` as the per-test
threshold — distributed must clear the same quality bar in-process
clears against the same frozen baseline. (`DISTRIBUTED-RENDERING-PLAN.md`
§5.1's 50 dB target is a per-render distributed-vs-in-process contract;
against the frozen baseline file, neither mode reaches it consistently
due to shared encoder/JPEG-capture jitter.) An absolute 10 dB pathology
floor catches fully-black-output regressions when a fixture authors a
permissive threshold. A distributed failure at the fixture's own
threshold means the distributed pipeline has drifted — file an issue
rather than relaxing the fixture.
`--update` is incompatible with `--mode=distributed-simulated`: the
in-process renderer is the source of truth for baselines, and the
@@ -124,10 +129,10 @@ bun run --cwd packages/producer docker:test font-variant-numeric -- --mode=distr
bun run --cwd packages/producer docker:test many-cuts -- --mode=distributed-simulated
```
Both modes must produce PSNR ≥ 50 dB against the existing baseline. If
`--mode=distributed-simulated` fails on a baseline, the distributed
primitive has a regression — stop, file an issue, do not paper over it
by adjusting the threshold.
Both modes must pass at each fixture's authored `minPsnr` against the
existing baseline. If `--mode=distributed-simulated` fails where
`--mode=in-process` passes, the distributed primitive has a regression —
file an issue rather than relaxing the fixture's threshold.
## Distributed-only fixtures