ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-10 22:20:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(engine): segment-scope SDR→HDR preflight (#445)

Browse Source 
## What

Scopes the SDR→HDR preflight re-encode to the segment the composition actually uses, mirroring the existing VFR→CFR segment-scope fix.

## Why

`convertSdrToHdr` was re-encoding entire source files, so a 30-minute SDR screen recording contributing a 2-second clip in a mixed HDR/SDR composition ate multi-second preflight time that produced frames no one would ever read. Validated on a mixed 30s-SDR + 2s-HDR fixture: `hdrPreflightMs` drops **87%** (1162→148ms), `videoExtractMs` drops **82%** (1272→231ms), `tmpPeakBytes` drops **45%** (8.2MB→4.5MB).

Depends on #444 (phase-level instrumentation) for the measurement surface.

## How

- `convertSdrToHdr` gains `startTime` and `duration` parameters ahead of the upstream `targetTransfer` arg added by #370. New signature: `convertSdrToHdr(input, output, startTime, duration, targetTransfer, signal, config)`. `-ss $start -t $duration` is added to the ffmpeg args.
- Phase 2 now captures the full `VideoMetadata` per `resolvedVideos` entry (previously just `colorSpace`) so the caller can compute `segDuration` from `video.end - video.start` with a fallback to `metadata.durationSeconds - video.mediaStart` for unbounded (Infinity) clips — without firing another ffprobe.
- After a successful convert, `entry.video.mediaStart` is zeroed out via shallow-copy (doesn't mutate the caller's `VideoElement`) so downstream extraction seeks from 0 instead of the original offset. Mirrors what the VFR→CFR path already does.

## Test plan

Validation on `/tmp/hf-fixtures/hdr-sdr-mixed-scope`:
```
hdrPreflightMs: >1000 → 150   (gate: <300)   ✓
videoExtractMs: 1272  → 237   (-82%)
tmpPeakBytes:   8.2MB → 4.5MB (-45%)
```

- [x] Unit test: new regression test synthesizes 10s SDR + 2s HDR fixture inline and asserts the converted file's duration matches the 2s used segment (pre-fix matched the 10s source)
- [x] Lint + format
- [x] Typecheck
- [x] Manual perf validation against synthesized fixture
This commit is contained in:
James Russo
2026-04-24 00:15:04 -04:00
committed by GitHub
parent 31354d52da
commit 9912d3730a
2 changed files with 157 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/engine/src/services/videoFrameExtractor.ts
+76 -7
View File
@@ -293,22 +293,38 @@ export async function extractVideoFramesRange(
* function (PQ for HDR10, HLG for broadcast HDR). The output transfer must
* match the dominant transfer of the surrounding HDR content; otherwise the
* downstream encoder will tag the final video with the wrong curve.
*
* `startTime` and `duration` bound the re-encode to the segment the composition
* actually uses. Without them a 30-minute screen recording that contributes a
* 2-second clip was transcoded in full — a >100× waste for long sources.
* Mirrors the segment-scope fix already applied to the VFR→CFR preflight.
*/
async function convertSdrToHdr(
inputPath: string,
outputPath: string,
startTime: number,
duration: number,
targetTransfer: HdrTransfer,
signal?: AbortSignal,
config?: Partial<Pick<EngineConfig, "ffmpegProcessTimeout">>,
): Promise<void> {
// Positive duration is required — FFmpeg's `-t 0` silently produces a 0-byte
// output that the downstream extractor then treats as a valid (empty) file.
if (duration <= 0) {
throw new Error(`convertSdrToHdr: duration must be positive (got ${duration})`);
}
const timeout = config?.ffmpegProcessTimeout ?? DEFAULT_CONFIG.ffmpegProcessTimeout;
// smpte2084 = PQ (HDR10), arib-std-b67 = HLG.
const colorTrc = targetTransfer === "pq" ? "smpte2084" : "arib-std-b67";
const args = [
"-ss",
String(startTime),
"-i",
inputPath,
"-t",
String(duration),
"-vf",
"colorspace=all=bt2020:iall=bt709:range=tv",
"-color_primaries",
@@ -455,16 +471,21 @@ export async function extractAllVideoFrames(
// Phase 2: Probe color spaces and normalize if mixed HDR/SDR
const phase2ProbeStart = Date.now();
const videoColorSpaces = await Promise.all(
resolvedVideos.map(async ({ videoPath }) => {
const metadata = await extractMediaMetadata(videoPath);
return metadata.colorSpace;
}),
const videoMetadata = await Promise.all(
resolvedVideos.map(({ videoPath }) => extractMediaMetadata(videoPath)),
);
const videoColorSpaces = videoMetadata.map((m) => m.colorSpace);
breakdown.hdrProbeMs = Date.now() - phase2ProbeStart;
const hdrPreflightStart = Date.now();
const hdrInfo = analyzeCompositionHdr(videoColorSpaces);
// Track entries the HDR preflight validated as non-extractable so they can
// be removed from every parallel array before Phase 2b and Phase 3 see them.
// Without this, `errors.push({...}); continue;` only short-circuits the
// normalization step — the invalid entry stays in `resolvedVideos` and
// Phase 3 still calls `extractVideoFramesRange` on the same past-EOF
// mediaStart, surfacing a second raw FFmpeg error for the same clip.
const hdrSkippedIndices = new Set<number>();
if (hdrInfo.hasHdr && hdrInfo.dominantTransfer) {
// dominantTransfer is "majority wins" — if a composition mixes PQ and HLG
// sources (rare but legal), the minority transfer's videos get converted
@@ -483,11 +504,46 @@ export async function extractAllVideoFrames(
// SDR video in a mixed timeline — convert to the dominant HDR transfer
// so the encoder tags the final video correctly (PQ vs HLG).
const entry = resolvedVideos[i];
if (!entry) continue;
const metadata = videoMetadata[i];
if (!entry || !metadata) continue;
// Guard against mediaStart past EOF — FFmpeg's `-ss` silently produces
// a 0-byte file when seeking beyond the source duration, and the
// downstream extractor then points at a broken input.
if (entry.video.mediaStart >= metadata.durationSeconds) {
errors.push({
videoId: entry.video.id,
error: `SDR→HDR conversion skipped: mediaStart (${entry.video.mediaStart}s) ≥ source duration (${metadata.durationSeconds}s)`,
});
hdrSkippedIndices.add(i);
continue;
}
// Scope the re-encode to the segment the composition actually uses.
// Long sources (e.g. 30-minute screen recordings) contributing short
// clips were transcoded in full pre-fix — a >100× waste.
let segDuration = entry.video.end - entry.video.start;
if (!Number.isFinite(segDuration) || segDuration <= 0) {
const sourceRemaining = metadata.durationSeconds - entry.video.mediaStart;
segDuration = sourceRemaining > 0 ? sourceRemaining : metadata.durationSeconds;
}
const convertedPath = join(convertDir, `${entry.video.id}_hdr.mp4`);
try {
await convertSdrToHdr(entry.videoPath, convertedPath, targetTransfer, signal, config);
await convertSdrToHdr(
entry.videoPath,
convertedPath,
entry.video.mediaStart,
segDuration,
targetTransfer,
signal,
config,
);
entry.videoPath = convertedPath;
// Segment-scoped re-encode starts the new file at t=0, so downstream
// extraction must seek from 0, not the original mediaStart. Shallow-copy
// to avoid mutating the caller's VideoElement (mirrors the VFR fix).
entry.video = { ...entry.video, mediaStart: 0 };
breakdown.hdrPreflightCount += 1;
} catch (err) {
errors.push({
@@ -500,6 +556,19 @@ export async function extractAllVideoFrames(
}
breakdown.hdrPreflightMs = Date.now() - hdrPreflightStart;
// Remove HDR-preflight-skipped entries from every parallel array so Phase 2b
// (VFR) and Phase 3 (extract) don't re-process them. Iterate backwards to
// keep indices stable while splicing.
if (hdrSkippedIndices.size > 0) {
for (let i = resolvedVideos.length - 1; i >= 0; i--) {
if (hdrSkippedIndices.has(i)) {
resolvedVideos.splice(i, 1);
videoMetadata.splice(i, 1);
videoColorSpaces.splice(i, 1);
}
}
}
// Phase 2b: Re-encode VFR inputs to CFR so the fps filter in Phase 3 produces
// the expected frame count. Only the used segment is transcoded.
const vfrPreflightStart = Date.now();