mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-12 23:29:50 +00:00
fix(engine): auto-normalize VFR video inputs to CFR before frame extraction (#360)
* fix(engine): auto-normalize VFR video inputs to CFR before frame extraction
Screen recordings (macOS ScreenCaptureKit, QuickTime, phone videos) are
commonly variable-frame-rate. When such inputs hit the extractor's
`-ss <start> -i <video> -t <dur> -vf fps=N` pipeline, the fps filter
can emit fewer frames than requested — for a 4-second 30fps segment
starting mid-file, the output was ~90 frames instead of 120.
`FrameLookupTable.getFrameAtTime` returns null for out-of-range indices,
so the compositor held the last valid frame and the user perceived the
video as freezing. This matches the bug report from an X community post
where a user said "all of them freezes" on their screen recording scenes.
The engine already detects VFR via `metadata.isVFR` in ffprobe.ts but
never acted on it — the compiler only logged a warning. This change
mirrors the existing SDR→HDR normalization pattern: when a source is
detected as VFR, re-encode only the used segment with
`-fps_mode cfr -r <fps> -preset fast -crf 18` before extraction.
Scoping the re-encode to `[mediaStart, mediaStart+duration]` means a
30-second clip cut from a 60-minute screen recording pays ~1s of
transcode cost, not 18s. Benchmarked locally:
Baseline (current): 32-39% duplicate frames, 25% frame-count
shortfall on mid-file segments.
Tier 1 (flag changes only): ~same — fps filter issue is not flag-fixable.
Tier 2 (CFR preflight): 1.7-6% duplicate frames, correct frame
count in every scenario tested.
The compiler warning that previously told users to manually re-encode
is downgraded to `console.info` since the engine now handles it.
— Rames Jusso
* refactor(engine): clean up VFR normalization loop after review
- Drop the `vfrNormDirCreated` flag; `mkdirSync({recursive:true})` is
idempotent and cheap.
- Don't re-wrap the `VFR→CFR conversion failed` prefix — `convertVfrToCfr`
already throws a message with that label; adding it again in the catch
produced "VFR→CFR conversion failed: VFR→CFR conversion failed (exit 1)".
- Shorten the Phase 2b header comment; the function docstring above
`convertVfrToCfr` already explains the failure modes and rationale.
- Note which frame windows the VFR fixture's select filter drops so the
magic numbers are scannable.
No behavior change; 311/311 engine tests still pass.
— Rames Jusso
* test(engine): add VFR regression unit tests
Adds a describe block that synthesizes a VFR fixture via ffmpeg and asserts
the extractor produces the expected frame count (no shortfall) and no long
runs of duplicate frames — the user-visible "frozen screen recording"
symptom. Covers both a mid-file segment and the full-file case.
Guarded with describe.skipIf(!HAS_FFMPEG) because the CI Test job on
ubuntu-24.04 and the Windows test-windows job don't install ffmpeg. The
producer-level regression test in packages/producer/tests/vfr-screen-recording/
runs inside Dockerfile.test (which has ffmpeg) and is the primary CI signal
for this bug; these unit tests are supplementary coverage for local and
any ffmpeg-equipped CI environment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(producer): add vfr-screen-recording regression test
End-to-end CI regression coverage for PR #360 via the existing
regression-harness: renders a 3s composition containing a real macOS
ScreenCaptureKit clip (r_frame_rate=120, avg≈36fps) seeked to
mediaStart=1, then PSNR-compares against a committed output.mp4.
Fixture src/clip.mp4 (108 KB) is a 5-second excerpt downscaled to 480×332
with -fps_mode passthrough to preserve the VFR timestamps. Content is the
public hyperframes OSS repo root page — see NOTICE.md for provenance.
With the fix applied, all 100 PSNR checkpoints pass. With the fix reverted,
66 of 100 fail (PSNR drops from ~43 dB to ~20 dB in the duplicate-frame
windows). Tagged "regression,video,vfr" so it runs in the fast shard
of .github/workflows/regression.yml automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(producer): regenerate vfr-screen-recording baseline in Docker
The committed golden output.mp4 was initially rendered on the host machine;
CI runs the renderer inside Dockerfile.test with a different Chrome +
ffmpeg build, producing pixel-level drift that failed PSNR at 54/100
checkpoints (~20 dB vs 41 dB in the VFR sparse-content windows). Both
renders are valid — the VFR source has inherent sampling ambiguity in
static segments, and different Chrome/ffmpeg builds make different valid
choices.
Regenerated the baseline via `bun run docker:test:update vfr-screen-recording`
so it matches the Docker environment CI actually uses. Matches the flow
the existing sub-composition-video, hdr-pq, etc. baselines were captured
with.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: document that producer test baselines must be captured in Docker
Hit this 2026-04-21 with the vfr-screen-recording regression test:
host-generated output.mp4 baseline tripped 54/100 PSNR checkpoints in CI
because Chrome + ffmpeg drift between the host and Dockerfile.test.
Document the `bun run --cwd packages/producer docker:test:update <name>`
flow so future contributors don't repeat the mistake.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
James Russo
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
b98093aa1c
commit
ffc06827c4
@@ -0,0 +1,22 @@
|
||||
# Source attribution
|
||||
|
||||
`src/clip.mp4` is a 5-second excerpt from a macOS ScreenCaptureKit (ReplayKit)
|
||||
recording, used here as a regression fixture for the VFR (variable-frame-rate)
|
||||
freeze bug fixed in PR #360.
|
||||
|
||||
- **Original duration**: 21s, recorded via `ReplayKitRecording` (the
|
||||
`com.apple.quicktime.author` QuickTime tag identifies this).
|
||||
- **Excerpt**: 16s–21s of the original, downscaled from 2746×1902 to 480×332,
|
||||
re-encoded with `ffmpeg -fps_mode passthrough -c:v libx264 -preset slow
|
||||
-crf 28 -an` to preserve the original VFR timestamps.
|
||||
- **Recorded content**: the public `heygen-com/hyperframes` GitHub repo root
|
||||
page. No private, proprietary, or user-identifying content.
|
||||
|
||||
## Properties preserved from the original
|
||||
|
||||
- `r_frame_rate`: 120/1
|
||||
- `avg_frame_rate`: ~36.1fps (21720/601)
|
||||
- `isVFR`: true (70% delta vs `r_frame_rate`, well over the 10% threshold in
|
||||
`ffprobe.ts`)
|
||||
- Pre-fix duplicate-frame rate: ~34% on a mid-file 3s segment extracted at
|
||||
30fps — matches the 18–44% observed across segments of the full recording.
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "vfr-screen-recording",
|
||||
"description": "Regression test for the VFR (variable-frame-rate) screen-recording freeze bug (PR #360). Renders a 3-second composition with a macOS ScreenCaptureKit clip (r_frame_rate=120, avg≈36fps) seeked to mediaStart=1. Pre-fix, the fps filter emitted long runs of duplicate frames that the compositor held as a frozen image; post-fix, VFR→CFR normalization keeps frame-accurate timing.",
|
||||
"tags": ["regression", "video", "vfr"],
|
||||
"minPsnr": 28,
|
||||
"maxFrameFailures": 2,
|
||||
"minAudioCorrelation": 0,
|
||||
"maxAudioLagWindows": 1,
|
||||
"renderConfig": {
|
||||
"fps": 30,
|
||||
"workers": 1
|
||||
}
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7aec7945d08be6b3c69464d9cf89445c44d52c9e0f89bad367592268964b2b22
|
||||
size 472149
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:602459ff96bcd7cd4bf306935b857676434bb65b2a1e395d30ce19a225bbd359
|
||||
size 108396
|
||||
@@ -0,0 +1,69 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<title>VFR Screen Recording Regression</title>
|
||||
<style>
|
||||
html,
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
background: #000;
|
||||
}
|
||||
#main {
|
||||
position: relative;
|
||||
width: 480px;
|
||||
height: 332px;
|
||||
overflow: hidden;
|
||||
}
|
||||
#clip {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: cover;
|
||||
}
|
||||
#label {
|
||||
position: absolute;
|
||||
top: 16px;
|
||||
left: 16px;
|
||||
z-index: 10;
|
||||
padding: 6px 12px;
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
font-size: 16px;
|
||||
font-weight: 700;
|
||||
color: #ffffff;
|
||||
background: rgba(0, 0, 0, 0.55);
|
||||
border-radius: 6px;
|
||||
letter-spacing: 0.04em;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div
|
||||
id="main"
|
||||
data-composition-id="vfr-screen-recording"
|
||||
data-start="0"
|
||||
data-duration="3"
|
||||
data-width="480"
|
||||
data-height="332"
|
||||
>
|
||||
<video
|
||||
id="clip"
|
||||
class="clip"
|
||||
data-start="0"
|
||||
data-duration="3"
|
||||
data-media-start="1"
|
||||
data-track-index="0"
|
||||
src="clip.mp4"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
|
||||
<div id="label">VFR</div>
|
||||
</div>
|
||||
<script>
|
||||
window.__timelines = window.__timelines || {};
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user