mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
fix(producer): address PR review feedback on harness mode + plan() copy filter
Miguel (approved) and Vai (commented) both flagged the same PSNR-threshold doc/code mismatch; Vai additionally flagged a path-anchoring bug in the projectDir-copy filter and a dishonest type cast. Addressed all five findings: PSNR threshold doc/code mismatch (important): - Module docstring, `resolveMinPsnrForMode` JSDoc, and tests/README.md all claimed distributed-simulated tightens to ≥50 dB. The actual code uses `max(fixture.minPsnr, 10)` — 10 dB is a pathology floor, the per-test gate is the fixture's authored `minPsnr`. Updated all three doc sites to describe what the code does. The 50 dB target in §5.1 is a per- render distributed-vs-in-process contract; against the frozen baseline it's unreachable for either mode (shared encoder/JPEG jitter), so it can't be a per-fixture gate. `PLAN_PROJECT_DIR_COPY_SKIP` regex matched absolute paths (important): - `cpSync` calls the filter with the absolute source path, so a `projectDir` whose absolute path happens to contain a blocklisted segment (`/home/user/work/output/comp/`, `~/projects/dist/foo/`, etc.) caused the filter to return false for every descendant — empty compiled directory, broken render. Now matches relative-to-projectDir segments via `path.relative()` + `split(sep)`. Switched from a regex to a Set for clarity. Harness fixtures don't hit this because they live under `tests/<name>/src/`, but adapters call `plan()` with caller-supplied paths. Dishonest type cast in regression-harness.ts (important): - `as "mp4" | "mov" | "png-sequence"` claimed reachability for formats that `validateMetadata` doesn't accept (the schema is `"mp4" | "webm"`, and webm is rejected by `checkDistributedSupport`). Narrowed to hardcoded `format: "mp4"` with a comment naming the metadata-schema invariant that lets us do that. Renamed `chunkVideoInjectorFactory` (nit): - The variable was invoked once and never used again — "factory" implied repeated calls. Inlined as a plain `videoInjector: BeforeCaptureHook | null` ternary. Replaced tautology test (nit): - `expect(DISTRIBUTED_SIMULATED_MIN_PSNR_DB).toBe(10)` was a value-pin over an exported constant. The invariant the JSDoc actually asserts is "10 dB is below any real fixture's authored minPsnr"; if someone lands a permissive fixture (minPsnr: 5), the value-pin doesn't catch it. Replaced with a test that walks `tests/*/meta.json` and asserts every authored `minPsnr` is ≥ the floor. Validated in `docker:test --mode=distributed-simulated`: font-variant-numeric, many-cuts, gsap-letters-render-compat, style-1-prod, sub-composition-video — all PASSED. Unit tests: 15/15 pass (new fixture-scan test included). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -113,11 +113,32 @@ describe("resolveMinPsnrForMode()", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("DISTRIBUTED_SIMULATED_MIN_PSNR_DB is the absolute-pathology floor", () => {
|
||||
// 10 dB is far below any real fixture's authored minPsnr (the lowest
|
||||
// among committed fixtures is 30 dB). It exists as a non-zero guard
|
||||
// for distributed-mode regressions that render fully-black frames
|
||||
// against a fixture authored with `minPsnr: 0`.
|
||||
expect(DISTRIBUTED_SIMULATED_MIN_PSNR_DB).toBe(10);
|
||||
it("every committed fixture authors a minPsnr above the absolute floor", async () => {
|
||||
// The pathology floor only fires for a fixture whose authored minPsnr
|
||||
// is below it — by design that should be no committed fixture. If
|
||||
// someone lands a permissive fixture (minPsnr: 5), distributed mode
|
||||
// will silently use 10 dB instead, which is the right behavior but
|
||||
// worth flagging so reviewers ask "is this fixture really meant to
|
||||
// accept near-black output?". This test prevents accidental misuse
|
||||
// by failing loudly when a fixture drops below the floor.
|
||||
const { readdirSync, readFileSync, statSync } = await import("node:fs");
|
||||
const { join: pathJoin } = await import("node:path");
|
||||
const testsDir = pathJoin(import.meta.dir, "..", "tests");
|
||||
const offenders: Array<{ fixture: string; minPsnr: number }> = [];
|
||||
for (const entry of readdirSync(testsDir)) {
|
||||
const metaPath = pathJoin(testsDir, entry, "meta.json");
|
||||
let stat;
|
||||
try {
|
||||
stat = statSync(metaPath);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (!stat.isFile()) continue;
|
||||
const meta = JSON.parse(readFileSync(metaPath, "utf-8")) as { minPsnr?: unknown };
|
||||
if (typeof meta.minPsnr === "number" && meta.minPsnr < DISTRIBUTED_SIMULATED_MIN_PSNR_DB) {
|
||||
offenders.push({ fixture: entry, minPsnr: meta.minPsnr });
|
||||
}
|
||||
}
|
||||
expect(offenders).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -12,18 +12,23 @@
|
||||
* or Lambda involvement: the controller and chunk worker are both this
|
||||
* process, but they go through the same artifact (planDir + frozen
|
||||
* `meta/encoder.json` + per-chunk concat-copy) that a real fan-out
|
||||
* would. Validates the determinism contract from §5.1 — the distributed
|
||||
* pipeline's output must be PSNR ≥ 50 dB against the in-process
|
||||
* baseline (the contract isn't byte-identical because streaming-encode
|
||||
* fusion is unavailable when chunks are encoded separately).
|
||||
* would.
|
||||
*
|
||||
* Both modes share the per-fixture `minPsnr` threshold — distributed must
|
||||
* pass the same quality bar the in-process renderer passes against the
|
||||
* same frozen baseline. A separate {@link DISTRIBUTED_SIMULATED_MIN_PSNR_DB}
|
||||
* pathology floor catches the case where a fixture authored a permissive
|
||||
* threshold and distributed regresses to fully-black output. The §5.1
|
||||
* 50 dB target was written for per-render comparison (fresh in-process vs
|
||||
* fresh distributed); against the frozen baseline file it's unreachable
|
||||
* for either mode due to shared encoder/JPEG-capture jitter, so the
|
||||
* harness can't use it as a per-test gate.
|
||||
*
|
||||
* Not every fixture can run in distributed-simulated mode. Distributed mode
|
||||
* refuses webm, HDR mp4, NTSC framerates, and non-{24,30,60} fps at plan
|
||||
* time. Fixtures that don't meet the constraints are skipped — the harness
|
||||
* logs the reason and the fixture is treated as "passed (skipped)" in
|
||||
* distributed-simulated mode. The full determinism contract lands when the
|
||||
* Phase 4 fixtures (`tests/distributed/<name>/`) cover every supported
|
||||
* format and adapter.
|
||||
* distributed-simulated mode.
|
||||
*/
|
||||
|
||||
import { existsSync, mkdirSync } from "node:fs";
|
||||
@@ -183,12 +188,13 @@ export async function runDistributedSimulatedRender(
|
||||
}
|
||||
|
||||
/**
|
||||
* Pick the PSNR threshold for a fixture given the harness mode. In-process
|
||||
* uses the fixture's authored `minPsnr` (which may be as low as 30 dB to
|
||||
* absorb per-fixture jitter). Distributed-simulated tightens to the
|
||||
* §5.1 contract floor (50 dB) but never goes below the fixture's own
|
||||
* threshold — a fixture that demands ≥55 dB in-process stays at ≥55 dB
|
||||
* distributed.
|
||||
* Pick the PSNR threshold for a fixture given the harness mode. Both modes
|
||||
* share the fixture's authored `minPsnr` — distributed must clear the same
|
||||
* quality bar in-process clears against the same frozen baseline.
|
||||
* Distributed-simulated additionally lifts the threshold to
|
||||
* {@link DISTRIBUTED_SIMULATED_MIN_PSNR_DB} for fixtures with a permissive
|
||||
* authored threshold; that absolute floor catches fully-black-output
|
||||
* regressions independent of fixture tolerance.
|
||||
*/
|
||||
export function resolveMinPsnrForMode(mode: HarnessMode, fixtureMinPsnr: number): number {
|
||||
if (mode === "in-process") return fixtureMinPsnr;
|
||||
|
||||
@@ -718,18 +718,19 @@ async function runTestSuite(
|
||||
return result;
|
||||
}
|
||||
// `checkDistributedSupport` already narrowed fps to {24,30,60} and
|
||||
// rejected webm; the `as` casts surface those guarantees to TS.
|
||||
// rejected webm; the cast surfaces that guarantee to TS.
|
||||
const fpsNum = suite.meta.renderConfig.fps.num as 24 | 30 | 60;
|
||||
const distributedFormat = (suite.meta.renderConfig.format ?? "mp4") as
|
||||
| "mp4"
|
||||
| "mov"
|
||||
| "png-sequence";
|
||||
// `validateMetadata` only accepts `format: "mp4" | "webm"` in
|
||||
// `renderConfig`, and `checkDistributedSupport` rejected webm above,
|
||||
// so by here only `mp4` (or the unset default) can reach this call.
|
||||
// If the metadata schema grows to accept "mov" / "png-sequence"
|
||||
// someday, narrow this cast accordingly.
|
||||
await runDistributedSimulatedRender({
|
||||
projectDir: tempSrcDir,
|
||||
tempRoot,
|
||||
renderedOutputPath,
|
||||
fps: fpsNum,
|
||||
format: distributedFormat,
|
||||
format: "mp4",
|
||||
chunkSize: suite.meta.renderConfig.chunkSize,
|
||||
maxParallelChunks: suite.meta.renderConfig.maxParallelChunks,
|
||||
variables: suite.meta.renderConfig.variables,
|
||||
|
||||
@@ -34,7 +34,7 @@ import {
|
||||
statSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { join, relative, sep } from "node:path";
|
||||
import { type CanvasResolution } from "@hyperframes/core";
|
||||
import { type EngineConfig, resolveConfig } from "@hyperframes/engine";
|
||||
import { defaultLogger, type ProducerLogger } from "../../logger.js";
|
||||
@@ -147,13 +147,24 @@ export interface PlanResult {
|
||||
}
|
||||
|
||||
/**
|
||||
* Skip patterns for the `projectDir → planDir/compiled/` pre-seed copy.
|
||||
* Real projects often contain `node_modules/`, VCS metadata, and harness
|
||||
* artifacts that have no business in a planDir — they bloat the 2 GB
|
||||
* planDir cap and slow the S3/Lambda round-trip for no benefit.
|
||||
* Top-level directory names skipped by the `projectDir → planDir/compiled/`
|
||||
* pre-seed copy. Real projects often contain `node_modules/`, VCS metadata,
|
||||
* and harness artifacts that have no business in a planDir — they bloat
|
||||
* the 2 GB planDir cap and slow the S3/Lambda round-trip for no benefit.
|
||||
* Matched against the path relative to `projectDir` so a `projectDir`
|
||||
* whose absolute path happens to contain one of these names (e.g.
|
||||
* `~/work/output/comp/`) doesn't false-positive-skip the entire copy.
|
||||
*/
|
||||
const PLAN_PROJECT_DIR_COPY_SKIP =
|
||||
/(^|\/)(node_modules|\.git|\.cache|output|failures|dist|\.next|\.turbo)(\/|$)/;
|
||||
const PLAN_PROJECT_DIR_SKIP_SEGMENTS = new Set([
|
||||
"node_modules",
|
||||
".git",
|
||||
".cache",
|
||||
"output",
|
||||
"failures",
|
||||
"dist",
|
||||
".next",
|
||||
".turbo",
|
||||
]);
|
||||
|
||||
/** Default chunk size in frames (~8s @ 30fps; fits Lambda's 15-min cap). */
|
||||
export const DEFAULT_CHUNK_SIZE = 240;
|
||||
@@ -527,7 +538,15 @@ export async function plan(
|
||||
cpSync(projectDir, compiledDir, {
|
||||
recursive: true,
|
||||
dereference: true,
|
||||
filter: (src) => !PLAN_PROJECT_DIR_COPY_SKIP.test(src),
|
||||
filter: (src) => {
|
||||
// cpSync passes the absolute source path. Compare relative-to-projectDir
|
||||
// so a parent directory of projectDir matching a skip name doesn't
|
||||
// false-positive every descendant.
|
||||
const rel = relative(projectDir, src);
|
||||
if (rel === "" || rel.startsWith("..")) return true;
|
||||
const firstSegment = rel.split(sep, 1)[0];
|
||||
return firstSegment === undefined || !PLAN_PROJECT_DIR_SKIP_SEGMENTS.has(firstSegment);
|
||||
},
|
||||
});
|
||||
|
||||
// The compiled directory lives at `<planDir>/compiled/` in the final
|
||||
|
||||
@@ -439,21 +439,21 @@ export async function renderChunk(
|
||||
forceScreenshot: encoder.forceScreenshot,
|
||||
};
|
||||
|
||||
// Rebuild the BeforeCaptureHook that injects pre-extracted video
|
||||
// frames into the page. Computed once per chunk and reused — the
|
||||
// `createRenderVideoFrameInjector` callback `runCaptureStage` accepts
|
||||
// can fire on every frame, and re-listing `planDir/video-frames/`
|
||||
// each time would be wasteful (the directory contents don't change
|
||||
// for the duration of a chunk render). Compositions with no video
|
||||
// elements produce `null` here, matching the in-process renderer's
|
||||
// skip path.
|
||||
const chunkVideoInjectorFactory = (): BeforeCaptureHook | null => {
|
||||
if (!planVideos || planVideos.extracted.length === 0) return null;
|
||||
const extracted = rebuildExtractedFramesFromPlanDir(planDir, planVideos.extracted);
|
||||
const frameLookup = createFrameLookupTable(planVideos.videos, extracted);
|
||||
return createVideoFrameInjector(frameLookup);
|
||||
};
|
||||
const cachedVideoInjector = chunkVideoInjectorFactory();
|
||||
// Build the BeforeCaptureHook that injects pre-extracted video frames
|
||||
// into the page once per chunk and reuse — `runCaptureStage` may
|
||||
// invoke `createRenderVideoFrameInjector` multiple times, and
|
||||
// re-listing `planDir/video-frames/` each call would be wasteful.
|
||||
// Compositions with no video elements produce `null`, matching the
|
||||
// in-process renderer's skip path.
|
||||
const videoInjector: BeforeCaptureHook | null =
|
||||
planVideos && planVideos.extracted.length > 0
|
||||
? createVideoFrameInjector(
|
||||
createFrameLookupTable(
|
||||
planVideos.videos,
|
||||
rebuildExtractedFramesFromPlanDir(planDir, planVideos.extracted),
|
||||
),
|
||||
)
|
||||
: null;
|
||||
|
||||
// ── Per-chunk work + frames directories ──
|
||||
// Suffix workDir with pid + random bytes so concurrent invocations on
|
||||
@@ -530,7 +530,7 @@ export async function renderChunk(
|
||||
needsAlpha: plan.dimensions.format !== "mp4",
|
||||
captureAttempts: [],
|
||||
buildCaptureOptions: () => captureOptions,
|
||||
createRenderVideoFrameInjector: () => cachedVideoInjector,
|
||||
createRenderVideoFrameInjector: () => videoInjector,
|
||||
abortSignal: undefined,
|
||||
assertNotAborted: () => {},
|
||||
frameRange: { startFrame: slice.startFrame, endFrame: slice.endFrame },
|
||||
|
||||
@@ -96,11 +96,16 @@ passing in the summary):
|
||||
- `format === "webm"` — `plan()` refuses webm.
|
||||
- `hdr === true` — distributed mode is SDR-only at v1.
|
||||
|
||||
For fixtures that *are* supported, the PSNR threshold tightens to **≥ 50
|
||||
dB** (the §5.1 determinism contract) or the fixture's own `minPsnr`,
|
||||
whichever is higher. A failure at this threshold means the distributed
|
||||
pipeline has drifted from in-process output — file an issue rather than
|
||||
adjusting the threshold.
|
||||
Both modes use the fixture's authored `minPsnr` as the per-test
|
||||
threshold — distributed must clear the same quality bar in-process
|
||||
clears against the same frozen baseline. (`DISTRIBUTED-RENDERING-PLAN.md`
|
||||
§5.1's 50 dB target is a per-render distributed-vs-in-process contract;
|
||||
against the frozen baseline file, neither mode reaches it consistently
|
||||
due to shared encoder/JPEG-capture jitter.) An absolute 10 dB pathology
|
||||
floor catches fully-black-output regressions when a fixture authors a
|
||||
permissive threshold. A distributed failure at the fixture's own
|
||||
threshold means the distributed pipeline has drifted — file an issue
|
||||
rather than relaxing the fixture.
|
||||
|
||||
`--update` is incompatible with `--mode=distributed-simulated`: the
|
||||
in-process renderer is the source of truth for baselines, and the
|
||||
@@ -124,10 +129,10 @@ bun run --cwd packages/producer docker:test font-variant-numeric -- --mode=distr
|
||||
bun run --cwd packages/producer docker:test many-cuts -- --mode=distributed-simulated
|
||||
```
|
||||
|
||||
Both modes must produce PSNR ≥ 50 dB against the existing baseline. If
|
||||
`--mode=distributed-simulated` fails on a baseline, the distributed
|
||||
primitive has a regression — stop, file an issue, do not paper over it
|
||||
by adjusting the threshold.
|
||||
Both modes must pass at each fixture's authored `minPsnr` against the
|
||||
existing baseline. If `--mode=distributed-simulated` fails where
|
||||
`--mode=in-process` passes, the distributed primitive has a regression —
|
||||
file an issue rather than relaxing the fixture's threshold.
|
||||
|
||||
## Distributed-only fixtures
|
||||
|
||||
|
||||
Reference in New Issue
Block a user