Files
hyperframes/packages/producer/scripts/test-classification.mjs
Aditya PoudyalandClaude Opus 5 462f9db73d feat(producer): add hls output format (#4014)
* feat(engine): add packageHls for HLS VOD packaging

Adds `packageHls` next to `muxVideoWithAudio`: stream-copies an
already-encoded H.264 video and optional AAC sidecar into an HLS VOD
directory (`master.m3u8` + split video/audio renditions, MPEG-TS
segments) with `-c copy`, so no re-encode happens at package time.

Depends on the locked GOP from the previous commit. `-hls_time` can only
cut at existing keyframes, so `gopSize = segmentSeconds x fps` is what
makes every segment exactly `segmentSeconds` long and IDR-aligned. A
test packages an unlocked encode to pin that: the same call yields one
4s segment instead of four 1s ones.

Deliberately omits the mp4 mux's provenance tags, which MPEG-TS drops
and whose `-movflags` is invalid for `-f hls`, and `-avoid_negative_ts`,
which would discard the AAC priming and shift audio against video (see
issue 3487). A real-ffmpeg test asserts the first presentable audio
sample lands on the first video frame.

No caller yet; `format: "hls"` in the producer follows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(engine): zero HLS mux base and reject % in packageHls outputDir

- Pass -muxdelay 0 -muxpreload 0 so the first segment starts at PTS 0
  instead of the mpegts muxer's 1.4 s base; A/V alignment through the
  AAC priming packet is unchanged
- Throw on an outputDir containing %, which ffmpeg expands as a printf
  conversion in -hls_segment_filename and fails with a misleading
  "No such file or directory"
- Assert alignment on relative spacing rather than on how one ffmpeg
  build clamps the priming packet's negative PTS
- Encode the test fixture with the full software GOP lock
  (lockedGopCodecParams), not just appendLockedGopArgs
- Trim comments to the non-obvious rationale

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(engine): satisfy the ffprobe argv contract in packageHls test

`producer/src/utils/ffprobeArgvContract.test.ts` sweeps every package at
the source level and requires `--` to be the argv entry immediately before
an ffprobe input path. The `probe()` helper here passed
`join(outputDir, HLS_VIDEO_PLAYLIST)` in that slot, which the contract's
comma-splitting parser reads as two entries, so the producer suite was red
on this branch while the engine suite stayed green.

`probe()` now takes the input path as a parameter and appends the
terminator itself. The sibling ffmpeg `run()` helper switches from a
spread-in-array-literal to `.concat`, because a verbosity-flag-then-spread
literal is indistinguishable from the ffprobe wrapper shape the contract
demands a terminator on. The matcher reads raw source including comments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(engine): drop the audio-only variant from the HLS master playlist

With `a:0,agroup:aud` in `-var_stream_map`, ffmpeg lists the audio
rendition twice in `master.m3u8`: as the `#EXT-X-MEDIA:TYPE=AUDIO` group
the video variant references, and again as a standalone `#EXT-X-STREAM-INF`
variant with no `RESOLUTION`. That is valid HLS, but a player choosing by
bandwidth can select it and play sound with no picture, and the VOD
consumer asked for a single rendition.

`stripAudioOnlyVariants` removes any variant tag without a `RESOLUTION`
attribute together with its URI line; `packageHls` applies it to the
master after a successful audio+video run. The rendition group and the
audio playlist are untouched, so `audio.m3u8` is still referenced from the
master. Pinned by a unit test on the playlist text and by the real-ffmpeg
test, which now asserts exactly one `#EXT-X-STREAM-INF`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(producer): extract hdrMode warning and decision-log helpers

`resolveEffectiveHdrMode` was already over the fallow complexity gate on
`main` (21 cyclomatic / 25 cognitive / CRAP 462) and only surfaces now
because the HLS format work needs to touch its downgrade warning. Pull the
downgrade message into `formatDowngradeWarning` and the "which mode and
why" info line into `logHdrDecision`, which puts the function under the
threshold.

No behavior change: every log string is byte-identical and `hdrMode.test.ts`
asserts them unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(producer): add hls output format

Adds `format: "hls"` to `@hyperframes/producer`. An HLS render is the same
opaque H.264 + AAC encode as `mp4`, stream-copied at assemble time (no
re-encode) into a directory: `master.m3u8`, `video.m3u8` + `video_%05d.ts`,
and `audio.m3u8` + `audio_%05d.ts` when the composition has audio.

The load-bearing part is that `-c copy -f hls` can only cut a segment at an
existing keyframe, so the encoder's GOP is locked to
`hlsSegmentSeconds x fps` frames. `resolveHlsEncoderGopLock` returns that
pair as one value spread into both encoder inputs, so the streaming and
disk/chunked paths cannot disagree — a render reaching only one of them
still succeeds, it just emits variable-length segments silently.

- `renderFormat.ts`: `hls` is opaque (`outputNeedsAlpha` is now
  `!== "mp4" && !== "hls"` — missing this flips HLS to RGBA screenshot
  capture with nothing failing), supports page-side compositing, and
  `outputUsesH264Pipeline` replaces the three `outputFormat === "mp4"`
  router gates.
- `hlsConfig.ts`: `hlsSegmentSeconds` (default 4, positive integer), the
  GOP derivation, and the fail-fast gates. `force-hdr` is rejected (v1 is
  SDR-only; HDR10 needs HEVC in fMP4), and so is `useGpu` — the GOP lock is
  software-only, so a GPU HLS render segments correctly by accident on
  videotoolbox and wrong on nvenc.
- `assembleStage.ts`: `hls` branches to `packageHls` instead of
  mux/faststart, after the same `padOrTrimAudioToVideoFrameCount` the mp4
  mux uses.
- `artifactTransaction.ts`: a directory artifact can now carry a
  `probeTarget` + `requiredFiles`, so the #3395 truncation gate still runs
  for HLS against `video.m3u8`. `expectedFrames` is inert there (ffprobe
  reports no frame count through the hls demuxer); duration carries it.
- `renderRequest.ts`: accepts `hls` + `hlsSegmentSeconds`; distributed
  conversion rejects it alongside `gif`. `DistributedFormat` is unchanged.
  `RENDER_REQUEST_VERSION` stays at 1: the new field is optional, the
  parser ignores unknown option keys, and an older worker already rejects
  `format: "hls"` with "Render request format is invalid".
- `hdrMode.ts`: the non-mp4 SDR downgrade warning no longer claims
  "HDR + alpha" for `hls`; the helper extraction landed in the preceding
  refactor commit.

Verified: `hlsRender.integration.test.ts` renders a real 10 s composition
and asserts 3 segments, the first two exactly 4.000 s, every segment
starting on an IDR frame, and PTS 0. Removing the GOP wiring collapses it
to 2 segments, so the test is load-bearing. Both producer lanes (unit and
integration) run on every PR in `ci.yml`, so that wiring check is on the
PR gate.

CLI flags and docs are deliberately left to the next PR in the series.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(engine): close the HLS master playlist rewrite race

CodeQL flagged two high-severity alerts on the `stripAudioOnlyVariants`
rewrite: a check-then-use race (`existsSync` then a separate path-based
read and write) and insecure creation of a file at a predictable path in
the OS temp dir.

Both came from resolving `master.m3u8` three times. Open it once with
`r+` and owner-only mode — which neither creates the playlist nor widens
its permissions, ffmpeg owns that — then read, truncate and write through
that one descriptor. Absence is still tolerated for the argument-level
tests that stub ffmpeg, now via ENOENT rather than a prior existence
check.

No behavior change: `hlsRender.integration.test.ts` still renders a real
composition with audio and asserts the stripped master playlist.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 20:33:55 -04:00

84 lines
3.4 KiB
JavaScript

// fallow-ignore-file complexity
import { readdirSync, readFileSync } from "node:fs";
import { dirname, relative, resolve } from "node:path";
import { fileURLToPath } from "node:url";
export const PRODUCER_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "..");
// These tests need host capabilities such as Chrome, ffmpeg, worker threads,
// or local sockets. Keep the list explicit so a filename-only rename does not
// make Git/fallow re-audit thousands of unchanged test lines as new code.
const INTEGRATION_TEST_FILES = new Set([
"src/regression-harness-psnr.test.ts",
"src/services/coreRuntimeBrowser.test.ts",
"src/services/deterministicFonts-systemCapture.test.ts",
"src/services/distributed/assemble.test.ts",
"src/services/distributed/chunkBoundary.test.ts",
"src/services/distributed/crossWorkerIdempotency.test.ts",
"src/services/distributed/plan.test.ts",
"src/services/distributed/planSizeCap.test.ts",
"src/services/distributed/renderChunk.test.ts",
"src/services/fileServer.test.ts",
"src/services/fileServer.file-race.test.ts",
"src/services/healthWorker.test.ts",
"src/services/hyperframeLint.file-race.test.ts",
"src/services/assetMediaType.test.ts",
"src/services/htmlCompiler.mediaType.test.ts",
"src/services/htmlCompiler.naturalDuration.test.ts",
"src/services/render/hlsRender.integration.test.ts",
"src/services/render/stages/compileStage.mediaType.test.ts",
"src/utils/audioRegression.test.ts",
"src/utils/streamDurationParity.test.ts",
]);
function collectTestFiles(directory, files = []) {
for (const entry of readdirSync(directory, { withFileTypes: true })) {
const entryPath = resolve(directory, entry.name);
if (entry.isDirectory()) collectTestFiles(entryPath, files);
else if (entry.isFile() && entry.name.endsWith(".test.ts")) files.push(entryPath);
}
return files;
}
export function classifyTestSource(filePath, source, integrationFiles = INTEGRATION_TEST_FILES) {
const importsBun = /\bfrom\s+["']bun:test["']/.test(source);
const importsVitest = /\bfrom\s+["']vitest["']/.test(source);
if (importsBun === importsVitest) {
const detail = importsBun ? "imports both bun:test and vitest" : "imports neither runner";
throw new Error(`${filePath}: ${detail}`);
}
return {
file: filePath,
runner: importsBun ? "bun" : "vitest",
lane: integrationFiles.has(filePath) ? "integration" : "unit",
};
}
export function discoverProducerTests(producerRoot = PRODUCER_ROOT) {
const srcDir = resolve(producerRoot, "src");
const files = collectTestFiles(srcDir).map((absolutePath) => ({
absolutePath,
filePath: relative(producerRoot, absolutePath).replaceAll("\\", "/"),
}));
const discoveredFiles = new Set(files.map((test) => test.filePath));
const staleEntries = [...INTEGRATION_TEST_FILES].filter((file) => !discoveredFiles.has(file));
if (staleEntries.length > 0) {
throw new Error(`Integration test manifest contains missing files: ${staleEntries.join(", ")}`);
}
return files
.map(({ absolutePath, filePath }) => {
return classifyTestSource(filePath, readFileSync(absolutePath, "utf8"));
})
.sort((left, right) => left.file.localeCompare(right.file));
}
export function summarizeTests(tests) {
const summary = {
total: tests.length,
unit: { bun: 0, vitest: 0 },
integration: { bun: 0, vitest: 0 },
};
for (const test of tests) summary[test.lane][test.runner] += 1;
return summary;
}