mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 12:54:29 +00:00
James RussoandClaude Opus 4.7
ffc06827c4
* fix(engine): auto-normalize VFR video inputs to CFR before frame extraction
Screen recordings (macOS ScreenCaptureKit, QuickTime, phone videos) are
commonly variable-frame-rate. When such inputs hit the extractor's
`-ss <start> -i <video> -t <dur> -vf fps=N` pipeline, the fps filter
can emit fewer frames than requested — for a 4-second 30fps segment
starting mid-file, the output was ~90 frames instead of 120.
`FrameLookupTable.getFrameAtTime` returns null for out-of-range indices,
so the compositor held the last valid frame and the user perceived the
video as freezing. This matches the bug report from an X community post
where a user said "all of them freezes" on their screen recording scenes.
The engine already detects VFR via `metadata.isVFR` in ffprobe.ts but
never acted on it — the compiler only logged a warning. This change
mirrors the existing SDR→HDR normalization pattern: when a source is
detected as VFR, re-encode only the used segment with
`-fps_mode cfr -r <fps> -preset fast -crf 18` before extraction.
Scoping the re-encode to `[mediaStart, mediaStart+duration]` means a
30-second clip cut from a 60-minute screen recording pays ~1s of
transcode cost, not 18s. Benchmarked locally:
Baseline (current): 32-39% duplicate frames, 25% frame-count
shortfall on mid-file segments.
Tier 1 (flag changes only): ~same — fps filter issue is not flag-fixable.
Tier 2 (CFR preflight): 1.7-6% duplicate frames, correct frame
count in every scenario tested.
The compiler warning that previously told users to manually re-encode
is downgraded to `console.info` since the engine now handles it.
— Rames Jusso
* refactor(engine): clean up VFR normalization loop after review
- Drop the `vfrNormDirCreated` flag; `mkdirSync({recursive:true})` is
idempotent and cheap.
- Don't re-wrap the `VFR→CFR conversion failed` prefix — `convertVfrToCfr`
already throws a message with that label; adding it again in the catch
produced "VFR→CFR conversion failed: VFR→CFR conversion failed (exit 1)".
- Shorten the Phase 2b header comment; the function docstring above
`convertVfrToCfr` already explains the failure modes and rationale.
- Note which frame windows the VFR fixture's select filter drops so the
magic numbers are scannable.
No behavior change; 311/311 engine tests still pass.
— Rames Jusso
* test(engine): add VFR regression unit tests
Adds a describe block that synthesizes a VFR fixture via ffmpeg and asserts
the extractor produces the expected frame count (no shortfall) and no long
runs of duplicate frames — the user-visible "frozen screen recording"
symptom. Covers both a mid-file segment and the full-file case.
Guarded with describe.skipIf(!HAS_FFMPEG) because the CI Test job on
ubuntu-24.04 and the Windows test-windows job don't install ffmpeg. The
producer-level regression test in packages/producer/tests/vfr-screen-recording/
runs inside Dockerfile.test (which has ffmpeg) and is the primary CI signal
for this bug; these unit tests are supplementary coverage for local and
any ffmpeg-equipped CI environment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(producer): add vfr-screen-recording regression test
End-to-end CI regression coverage for PR #360 via the existing
regression-harness: renders a 3s composition containing a real macOS
ScreenCaptureKit clip (r_frame_rate=120, avg≈36fps) seeked to
mediaStart=1, then PSNR-compares against a committed output.mp4.
Fixture src/clip.mp4 (108 KB) is a 5-second excerpt downscaled to 480×332
with -fps_mode passthrough to preserve the VFR timestamps. Content is the
public hyperframes OSS repo root page — see NOTICE.md for provenance.
With the fix applied, all 100 PSNR checkpoints pass. With the fix reverted,
66 of 100 fail (PSNR drops from ~43 dB to ~20 dB in the duplicate-frame
windows). Tagged "regression,video,vfr" so it runs in the fast shard
of .github/workflows/regression.yml automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(producer): regenerate vfr-screen-recording baseline in Docker
The committed golden output.mp4 was initially rendered on the host machine;
CI runs the renderer inside Dockerfile.test with a different Chrome +
ffmpeg build, producing pixel-level drift that failed PSNR at 54/100
checkpoints (~20 dB vs 41 dB in the VFR sparse-content windows). Both
renders are valid — the VFR source has inherent sampling ambiguity in
static segments, and different Chrome/ffmpeg builds make different valid
choices.
Regenerated the baseline via `bun run docker:test:update vfr-screen-recording`
so it matches the Docker environment CI actually uses. Matches the flow
the existing sub-composition-video, hdr-pq, etc. baselines were captured
with.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: document that producer test baselines must be captured in Docker
Hit this 2026-04-21 with the vfr-screen-recording regression test:
host-generated output.mp4 baseline tripped 54/100 PSNR checkpoints in CI
because Chrome + ffmpeg drift between the host and Dockerfile.test.
Document the `bun run --cwd packages/producer docker:test:update <name>`
flow so future contributors don't repeat the mistake.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
71 lines
3.3 KiB
Markdown
71 lines
3.3 KiB
Markdown
# Hyperframes
|
|
|
|
Open-source video rendering framework: write HTML, render video.
|
|
|
|
```
|
|
packages/
|
|
cli/ → hyperframes CLI (create, preview, lint, render)
|
|
core/ → Types, parsers, generators, linter, runtime, frame adapters
|
|
engine/ → Seekable page-to-video capture engine (Puppeteer + FFmpeg)
|
|
player/ → Embeddable <hyperframes-player> web component
|
|
producer/ → Full rendering pipeline (capture + encode + audio mix)
|
|
studio/ → Browser-based composition editor UI
|
|
```
|
|
|
|
## Development
|
|
|
|
```bash
|
|
bun install # Install dependencies
|
|
bun run build # Build all packages
|
|
bun run test # Run tests
|
|
```
|
|
|
|
**This repo uses bun**, not pnpm. Do NOT run `pnpm install` — it creates a `pnpm-lock.yaml` that should not exist. Workspace linking relies on bun's resolution from `"workspaces"` in root `package.json`.
|
|
|
|
### Linting & Formatting
|
|
|
|
This project uses **oxlint** and **oxfmt** (not biome, not eslint, not prettier).
|
|
|
|
```bash
|
|
bunx oxlint <files> # Lint
|
|
bunx oxfmt <files> # Format (write)
|
|
bunx oxfmt --check <files> # Format (check only, used by pre-commit hook)
|
|
```
|
|
|
|
Always run both on changed files before committing. The lefthook pre-commit hook runs `bunx oxlint` and `bunx oxfmt --check` automatically.
|
|
|
|
### Adding CLI Commands
|
|
|
|
When adding a new CLI command:
|
|
|
|
1. Define the command in `packages/cli/src/commands/<name>.ts` using `defineCommand` from citty
|
|
2. **Export `examples`** in the same file — `export const examples: Example[] = [...]` (import `Example` from `./_examples.js`). These are displayed by `--help`.
|
|
3. Register it in `packages/cli/src/cli.ts` under `subCommands` (lazy-loaded)
|
|
4. **Add to help groups** in `packages/cli/src/help.ts` — add the command name and description to the appropriate `GROUPS` entry. Without this, the command won't appear in `hyperframes --help` even though it works.
|
|
5. **Document it** in `docs/packages/cli.mdx` — add a section with usage examples and flags.
|
|
6. Validate by running `npx tsx packages/cli/src/cli.ts --help` (command appears in the list) and `npx tsx packages/cli/src/cli.ts <name> --help` (examples appear).
|
|
|
|
### Regression Test Golden Baselines (producer)
|
|
|
|
`packages/producer/tests/<name>/output/output.mp4` baselines MUST be generated
|
|
inside `Dockerfile.test`, not on your host. CI renders inside that Docker image
|
|
with a specific Chrome + ffmpeg build; pixel-level output drifts across
|
|
different host Chrome/ffmpeg versions and will fail PSNR at dozens of
|
|
checkpoints even when the code is correct.
|
|
|
|
```bash
|
|
# Build the test image once:
|
|
docker build -t hyperframes-producer:test -f Dockerfile.test .
|
|
|
|
# Generate or update a baseline (runs the harness with --update inside Docker):
|
|
bun run --cwd packages/producer docker:test:update <test-name>
|
|
```
|
|
|
|
Never run `bun run --cwd packages/producer test:update` directly from the
|
|
host to capture a baseline that will be committed — the resulting output.mp4
|
|
will not match CI. Use it only for local-only experimentation.
|
|
|
|
## Skills
|
|
|
|
Composition authoring (not repo development) is guided by skills installed via `npx skills add heygen-com/hyperframes`. See `skills/` for source. Invoke `/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, or `/gsap` when authoring compositions. When a user provides a website URL and wants a video, invoke `/website-to-hyperframes` — it runs the full 7-step capture-to-video pipeline.
|