mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
Merge pull request #627 from heygen-com/fix/sub-comp-video-path-resolution-and-render-fixes
fix: render robustness — sub-comp src paths, alpha tag case, encoder + matter improvements
This commit is contained in:
James Russo
@@ -25,7 +25,7 @@ The CLI ships a built-in `remove-background` command that runs locally — no AP
|
||||
</Step>
|
||||
<Step title="Remove the background from your video">
|
||||
```bash Terminal
|
||||
npx hyperframes remove-background avatar.mp4 -o transparent.webm
|
||||
npx hyperframes remove-background subject.mp4 -o transparent.webm
|
||||
```
|
||||
|
||||
On the first run, the CLI downloads ~168 MB of model weights to `~/.cache/hyperframes/background-removal/models/`. Subsequent runs reuse the cache.
|
||||
@@ -44,7 +44,7 @@ The CLI ships a built-in `remove-background` command that runs locally — no AP
|
||||
<!-- background layer -->
|
||||
<img src="city.jpg" class="bg" />
|
||||
|
||||
<!-- transparent avatar floats on top -->
|
||||
<!-- transparent subject floats on top -->
|
||||
<video src="transparent.webm" autoplay muted loop playsinline></video>
|
||||
</div>
|
||||
```
|
||||
@@ -75,8 +75,8 @@ The output is encoded with the exact ffmpeg flags Chrome's `<video>` element nee
|
||||
| `.png` | PNG with alpha | Single-image cutout (only when the input is also a single image) | varies |
|
||||
|
||||
```bash Terminal
|
||||
npx hyperframes remove-background avatar.mp4 -o transparent.webm # web playback
|
||||
npx hyperframes remove-background avatar.mp4 -o transparent.mov # editing
|
||||
npx hyperframes remove-background subject.mp4 -o transparent.webm # web playback
|
||||
npx hyperframes remove-background subject.mp4 -o transparent.mov # editing
|
||||
npx hyperframes remove-background portrait.jpg -o cutout.png # still image
|
||||
```
|
||||
|
||||
@@ -91,7 +91,7 @@ Real-world numbers from the [matting eval](https://www.heygenverse.com/a/0dd5a43
|
||||
| Linux x86 | CPU | ~1100 | ~16 min |
|
||||
| macOS Intel | CPU | ~900 | ~13 min |
|
||||
|
||||
Matting is offline preprocessing — you run it once per asset and reuse the output. CPU-only is slow but always works; if you reuse the same avatar repeatedly, run it once on a faster machine and check the transparent output into your project.
|
||||
Matting is offline preprocessing — you run it once per asset and reuse the output. CPU-only is slow but always works; if you reuse the same subject clip repeatedly, run it once on a faster machine and check the transparent output into your project.
|
||||
|
||||
## Picking a device explicitly
|
||||
|
||||
@@ -100,13 +100,13 @@ Matting is offline preprocessing — you run it once per asset and reuse the out
|
||||
- **Force CPU on a GPU box** when you want to keep the GPU free for other work, or are debugging an EP-specific issue:
|
||||
|
||||
```bash Terminal
|
||||
npx hyperframes remove-background avatar.mp4 -o transparent.webm --device cpu
|
||||
npx hyperframes remove-background subject.mp4 -o transparent.webm --device cpu
|
||||
```
|
||||
|
||||
- **Opt into CUDA** by setting `HYPERFRAMES_CUDA=1` and providing a GPU-enabled `onnxruntime-node` build (the bundled build is CPU + CoreML only, to keep the install small for the 99% of users who don't have a GPU):
|
||||
|
||||
```bash Terminal
|
||||
HYPERFRAMES_CUDA=1 npx hyperframes remove-background avatar.mp4 -o transparent.webm --device cuda
|
||||
HYPERFRAMES_CUDA=1 npx hyperframes remove-background subject.mp4 -o transparent.webm --device cuda
|
||||
```
|
||||
|
||||
Run `npx hyperframes remove-background --info` to see what providers are detected on your machine and which one `auto` would pick.
|
||||
@@ -115,7 +115,7 @@ Run `npx hyperframes remove-background --info` to see what providers are detecte
|
||||
|
||||
The transparent WebM behaves like any other video element. The two patterns you'll use most:
|
||||
|
||||
**Avatar over a background image:**
|
||||
**Subject over a background image:**
|
||||
|
||||
```html
|
||||
<div style="position: relative; width: 1920px; height: 1080px;">
|
||||
@@ -131,22 +131,104 @@ The transparent WebM behaves like any other video element. The two patterns you'
|
||||
</div>
|
||||
```
|
||||
|
||||
**Avatar over a HyperFrames scene:**
|
||||
**Subject over a HyperFrames scene:**
|
||||
|
||||
```html
|
||||
<!-- scene contents (text, animations, etc.) -->
|
||||
<div class="title-card">Welcome</div>
|
||||
|
||||
<!-- avatar layered on top -->
|
||||
<video src="transparent.webm" autoplay muted loop playsinline class="avatar"></video>
|
||||
<!-- subject layered on top -->
|
||||
<video src="transparent.webm" autoplay muted loop playsinline class="subject"></video>
|
||||
```
|
||||
|
||||
The avatar inherits the composition's frame rate and timeline — it plays through once during the scene's duration, so match the source clip length to the scene length when possible. If the scene is longer than the clip, `loop` handles it.
|
||||
The cutout inherits the composition's frame rate and timeline — it plays through once during the scene's duration, so match the source clip length to the scene length when possible. If the scene is longer than the clip, `loop` handles it.
|
||||
|
||||
<Tip>
|
||||
When rendering a composition that contains a `<video>` element, the renderer reads the source via ffmpeg internally. Transparent WebMs are decoded with the alpha plane preserved.
|
||||
</Tip>
|
||||
|
||||
## Compositing patterns and pitfalls
|
||||
|
||||
The cutout webm is a **re-encoded copy** of the source mp4's RGB — the matter pipeline decodes the source to raw RGB, runs segmentation, and re-encodes to VP9 with alpha. That choice has consequences depending on what you put behind it.
|
||||
|
||||
### The three patterns
|
||||
|
||||
| Pattern | Behind the cutout | Result |
|
||||
|---|---|---|
|
||||
| **Cutout over a different scene** *(most common)* | Static image, gradient, animated bg, or unrelated footage | Clean. The cutout is the only source of the subject — no doubling, no edge halo. Use any `--quality`. |
|
||||
| **Cutout over its own source mp4** *(text-behind-subject, talking-head with overlays)* | The same mp4 the cutout was generated from | Two RGB sources for the same person. At default `--quality balanced` (crf 18) the doubling is barely visible; at `--quality fast` (crf 30) you'll see a slight color shift / soft edge on the silhouette. Use `--quality best` (crf 12) for hero shots. |
|
||||
| **Cutout over different footage of the same subject** | Another take of the same person | Looks like two overlapping people. Avoid — re-shoot or re-cut the source. |
|
||||
|
||||
### Text-behind-subject: the recommended layout
|
||||
|
||||
Putting a headline *behind* a presenter so their silhouette occludes the text:
|
||||
|
||||
```html
|
||||
<!-- z=1 base mp4: full lobby + presenter, plays the whole scene -->
|
||||
<video
|
||||
id="cf-base"
|
||||
data-start="0" data-duration="6" data-media-start="0" data-track-index="0"
|
||||
src="presenter.mp4"
|
||||
muted playsinline
|
||||
></video>
|
||||
|
||||
<!-- z=2 headline -->
|
||||
<h1 id="cf-headline" style="position:absolute;top:50%;left:50%;
|
||||
transform:translate(-50%,-50%); z-index:2;
|
||||
color:#fff; text-shadow:0 6px 32px rgba(0,0,0,.55);
|
||||
clip-path:inset(0 0 100% 0); font-size:220px; font-weight:900;">
|
||||
MAKE IT IN HYPERFRAMES
|
||||
</h1>
|
||||
|
||||
<!-- z=3 cutout: same source, alpha around presenter, hidden until the cut.
|
||||
The wrapper carries the opacity, NOT the <video> itself. -->
|
||||
<div class="cutout-wrap" style="position:absolute;inset:0;z-index:3;opacity:0">
|
||||
<video
|
||||
id="cf-cutout"
|
||||
data-start="0" data-duration="6" data-media-start="0" data-track-index="1"
|
||||
src="presenter.webm"
|
||||
muted playsinline
|
||||
></video>
|
||||
</div>
|
||||
```
|
||||
|
||||
```js
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
const CUT = 3.3;
|
||||
|
||||
// Reveal the headline early
|
||||
tl.to("#cf-headline", { clipPath: "inset(0 0 0% 0)", duration: 0.6, ease: "expo.out" }, 0.25);
|
||||
|
||||
// At the cut, flip the cutout wrapper visible — silhouette punches through the headline
|
||||
tl.set(".cutout-wrap", { opacity: 1 }, CUT);
|
||||
|
||||
// Sentinel: extend timeline to the composition's full duration so the renderer
|
||||
// doesn't bail past the last meaningful tween.
|
||||
tl.set({}, {}, 6);
|
||||
```
|
||||
|
||||
### Two non-obvious rules
|
||||
|
||||
**1. Wrap the cutout video in a non-timed `<div>` and animate the wrapper, not the video.**
|
||||
|
||||
The framework forces `opacity: 1` on any element with `data-start`/`data-duration` while it's "active" — that's how it controls clip visibility. CSS `opacity: 0` on the video element is silently overwritten by the framework's clip lifecycle, so an opacity tween on the video element won't do anything. Wrap the video in a `<div>` that has no `data-*` attributes; the wrapper is owned entirely by your CSS/GSAP.
|
||||
|
||||
**2. Both videos start at `data-start="0"` and decode in sync from t=0.**
|
||||
|
||||
It's tempting to "late-mount" the cutout (`data-start="3.3"` to match the cut). Don't — Chrome does a seek + decoder warm-up at mount, which can land one frame off the base mp4 at the cut moment. With both videos mounted from t=0 and the cutout's wrapper opacity-animated, both decoders advance the same way and stay frame-accurate.
|
||||
|
||||
### Quality preset and color match
|
||||
|
||||
When the cutout is overlaid on its own source mp4, the encoder's CRF directly affects how visible the doubling is at edges:
|
||||
|
||||
| `--quality` | CRF | File size (12s @ 1080p) | When to use |
|
||||
|---|---|---|---|
|
||||
| `fast` | 30 | ~2 MB | Cutout sits over an unrelated background and file size matters |
|
||||
| `balanced` *(default)* | 18 | ~6 MB | Recommended for text-behind-subject and any pattern that overlays on the source |
|
||||
| `best` | 12 | ~12 MB | Hero shots, masters, or anything you'll re-encode downstream |
|
||||
|
||||
The encoder also writes BT.709 + limited-range color metadata so Chrome's YUV→RGB pipeline matches the source mp4's. Without those tags, the cutout would render slightly differently from the underlying mp4 even at lossless quality (visible red/skin shift).
|
||||
|
||||
## What u²-net_human_seg is and isn't good for
|
||||
|
||||
The model is purpose-built for **portrait / human matting**. It excels when:
|
||||
@@ -166,7 +248,7 @@ If your use case hits one of these, see the alternatives below.
|
||||
|
||||
## Alternatives — when the built-in command isn't the right tool
|
||||
|
||||
The CLI ships **one model on purpose** — the one that's MIT-licensed, runs everywhere, and produces production-quality output for HeyGen-style avatar workflows. The list below leads with **free, open-source tools** that pair naturally with HyperFrames. Each entry calls out the actual catch — license, install effort, hardware needs — so you can pick the right one for your situation. Full benchmarks are in the [matting eval](https://www.heygenverse.com/a/0dd5a431-1832-4858-862d-de7fb7d02654).
|
||||
The CLI ships **one model on purpose** — the one that's MIT-licensed, runs everywhere, and produces production-quality output for person/portrait video. The list below leads with **free, open-source tools** that pair naturally with HyperFrames. Each entry calls out the actual catch — license, install effort, hardware needs — so you can pick the right one for your situation. Full benchmarks are in the [matting eval](https://www.heygenverse.com/a/0dd5a431-1832-4858-862d-de7fb7d02654).
|
||||
|
||||
### Free, open-source CLIs and libraries
|
||||
|
||||
@@ -208,7 +290,7 @@ ffmpeg -i frames-%04d.png -c:v libvpx-vp9 \
|
||||
|
||||
### How to choose
|
||||
|
||||
- **Avatars / portraits, web playback, MIT-clean** → use the built-in `hyperframes remove-background` (this is what it's tuned for).
|
||||
- **Person / portrait video, web playback, MIT-clean** → use the built-in `hyperframes remove-background` (this is what it's tuned for).
|
||||
- **Non-human subject** (product, animal, object) → `rembg` with `isnet-general-use`.
|
||||
- **Maximum portrait quality, especially hair** → `BiRefNet` via Python.
|
||||
- **Long video where edge flicker would be visible**, GPL is OK → `RVM`.
|
||||
@@ -259,7 +341,7 @@ The decoded `frame0.png` should be RGBA and have non-trivial alpha values.
|
||||
The pipeline auto-falls-back to CPU if CoreML fails to bind, with a warning. If you want to skip the CoreML attempt entirely, force CPU:
|
||||
|
||||
```bash Terminal
|
||||
npx hyperframes remove-background avatar.mp4 -o transparent.webm --device cpu
|
||||
npx hyperframes remove-background subject.mp4 -o transparent.webm --device cpu
|
||||
```
|
||||
|
||||
### The alpha mask has rough or jagged edges
|
||||
|
||||
@@ -367,6 +367,7 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_
|
||||
|------|-------------|
|
||||
| `--output, -o` | Output path. Format inferred from extension: `.webm` (default), `.mov`, `.png` |
|
||||
| `--device` | Execution provider: `auto` (default), `cpu`, `coreml`, `cuda` |
|
||||
| `--quality` | WebM encoder preset: `fast` (crf 30, smallest), `balanced` (crf 18, default), `best` (crf 12, near-lossless). Higher quality keeps the cutout's RGB closer to the source mp4 — important when overlaying the cutout on its own source for text-behind-subject effects. Ignored for `.mov` / `.png`. |
|
||||
| `--info` | Print detected execution providers and exit (no render) |
|
||||
| `--json` | Output result as JSON |
|
||||
|
||||
|
||||
@@ -159,10 +159,16 @@ async function postprocess(
|
||||
}
|
||||
|
||||
// lanczos3 keeps soft edges; nearest leaves visible jaggies on hair.
|
||||
// Sharp upcasts the single-channel raw input to a 3-channel buffer during
|
||||
// resize, so the output is laid out as RGB-interleaved (R0,G0,B0,R1,G1,B1,...)
|
||||
// even though all three channels carry the same grayscale value. Force the
|
||||
// output back to single channel with toColourspace("b-w") so we can index
|
||||
// it linearly as a mask.
|
||||
const fullMask = await sharp(maskBuf, {
|
||||
raw: { width: INPUT_SIZE, height: INPUT_SIZE, channels: 1 },
|
||||
})
|
||||
.resize(width, height, { kernel: "lanczos3", fit: "fill" })
|
||||
.toColourspace("b-w")
|
||||
.raw()
|
||||
.toBuffer();
|
||||
|
||||
|
||||
@@ -46,6 +46,35 @@ describe("background-removal/pipeline — buildEncoderArgs", () => {
|
||||
expect(args[args.length - 1]).toBe("/tmp/out.webm");
|
||||
});
|
||||
|
||||
it("webm preset tags BT.709 colorspace + limited range", () => {
|
||||
// Without these tags, ffmpeg's RGB→YUV conversion uses the BT.601 default,
|
||||
// and Chrome's YUV→RGB pass on the resulting webm produces a different
|
||||
// RGB triple than the source mp4 (visible color shift on overlay). Pin
|
||||
// BT.709 limited-range so the cutout matches modern Rec.709 sources.
|
||||
const args = buildEncoderArgs("webm", 1920, 1080, 30, "/tmp/out.webm");
|
||||
const csIdx = args.indexOf("-colorspace");
|
||||
expect(csIdx).toBeGreaterThan(-1);
|
||||
expect(args[csIdx + 1]).toBe("bt709");
|
||||
const rangeIdx = args.indexOf("-color_range");
|
||||
expect(rangeIdx).toBeGreaterThan(-1);
|
||||
expect(args[rangeIdx + 1]).toBe("tv");
|
||||
});
|
||||
|
||||
it("webm quality presets map to crf 30/18/12", () => {
|
||||
const fast = buildEncoderArgs("webm", 1920, 1080, 30, "/tmp/o.webm", "fast");
|
||||
const balanced = buildEncoderArgs("webm", 1920, 1080, 30, "/tmp/o.webm", "balanced");
|
||||
const best = buildEncoderArgs("webm", 1920, 1080, 30, "/tmp/o.webm", "best");
|
||||
const crf = (args: string[]) => args[args.indexOf("-crf") + 1];
|
||||
expect(crf(fast)).toBe("30");
|
||||
expect(crf(balanced)).toBe("18");
|
||||
expect(crf(best)).toBe("12");
|
||||
});
|
||||
|
||||
it("webm default quality is balanced (crf 18)", () => {
|
||||
const args = buildEncoderArgs("webm", 1920, 1080, 30, "/tmp/o.webm");
|
||||
expect(args[args.indexOf("-crf") + 1]).toBe("18");
|
||||
});
|
||||
|
||||
it("mov preset emits ProRes 4444 + yuva444p10le", () => {
|
||||
const args = buildEncoderArgs("mov", 1920, 1080, 30, "/tmp/out.mov");
|
||||
expect(args).toContain("prores_ks");
|
||||
|
||||
@@ -20,11 +20,28 @@ import { type Device, type ModelId } from "./manager.js";
|
||||
|
||||
export type OutputFormat = "webm" | "mov" | "png";
|
||||
|
||||
export const QUALITY_CRF = {
|
||||
fast: 30,
|
||||
balanced: 18,
|
||||
best: 12,
|
||||
} as const;
|
||||
|
||||
export type Quality = keyof typeof QUALITY_CRF;
|
||||
|
||||
export const QUALITIES = Object.keys(QUALITY_CRF) as readonly Quality[];
|
||||
|
||||
export const DEFAULT_QUALITY: Quality = "balanced";
|
||||
|
||||
export const isQuality = (v: unknown): v is Quality =>
|
||||
typeof v === "string" && (QUALITIES as readonly string[]).includes(v);
|
||||
|
||||
export interface RenderOptions {
|
||||
inputPath: string;
|
||||
outputPath: string;
|
||||
device?: Device;
|
||||
model?: ModelId;
|
||||
/** Encoder CRF preset for `.webm`. See `QUALITY_CRF`. Ignored for `.mov`/`.png`. */
|
||||
quality?: Quality;
|
||||
onProgress?: (event: ProgressEvent) => void;
|
||||
}
|
||||
|
||||
@@ -100,6 +117,7 @@ export function buildEncoderArgs(
|
||||
height: number,
|
||||
fps: number,
|
||||
outputPath: string,
|
||||
quality: Quality = DEFAULT_QUALITY,
|
||||
): string[] {
|
||||
const base = [
|
||||
"-y",
|
||||
@@ -123,7 +141,7 @@ export function buildEncoderArgs(
|
||||
"-b:v",
|
||||
"0",
|
||||
"-crf",
|
||||
"30",
|
||||
String(QUALITY_CRF[quality]),
|
||||
"-deadline",
|
||||
"good",
|
||||
"-row-mt",
|
||||
@@ -132,6 +150,19 @@ export function buildEncoderArgs(
|
||||
"0",
|
||||
"-pix_fmt",
|
||||
"yuva420p",
|
||||
// Tag the output as BT.709 limited range so browsers use the same
|
||||
// YUV→RGB matrix the source video was encoded with. Without these tags
|
||||
// ffmpeg's default RGB→YUV conversion is BT.601, which causes a visible
|
||||
// color shift (red/skin tones in particular) when the matted overlay is
|
||||
// composited over the original mp4.
|
||||
"-colorspace",
|
||||
"bt709",
|
||||
"-color_primaries",
|
||||
"bt709",
|
||||
"-color_trc",
|
||||
"bt709",
|
||||
"-color_range",
|
||||
"tv",
|
||||
"-metadata:s:v:0",
|
||||
"alpha_mode=1",
|
||||
"-an",
|
||||
@@ -250,9 +281,20 @@ async function runPipeline(
|
||||
});
|
||||
const decoderExit = waitForExit(decoder, "ffmpeg decoder", () => decoderStderr);
|
||||
|
||||
const encoder = spawn("ffmpeg", buildEncoderArgs(format, width, height, fps || 30, outputPath), {
|
||||
stdio: ["pipe", "ignore", "pipe"],
|
||||
});
|
||||
const encoder = spawn(
|
||||
"ffmpeg",
|
||||
buildEncoderArgs(
|
||||
format,
|
||||
width,
|
||||
height,
|
||||
fps || 30,
|
||||
outputPath,
|
||||
options.quality ?? DEFAULT_QUALITY,
|
||||
),
|
||||
{
|
||||
stdio: ["pipe", "ignore", "pipe"],
|
||||
},
|
||||
);
|
||||
let encoderStderr = "";
|
||||
encoder.stderr?.on("data", (d: Buffer) => {
|
||||
encoderStderr += d.toString();
|
||||
|
||||
@@ -4,6 +4,7 @@ import { existsSync } from "node:fs";
|
||||
import * as clack from "@clack/prompts";
|
||||
import { c } from "../ui/colors.js";
|
||||
import { isDevice, DEVICES } from "../background-removal/manager.js";
|
||||
import { DEFAULT_QUALITY, QUALITIES, isQuality } from "../background-removal/pipeline.js";
|
||||
import type { Example } from "./_examples.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
@@ -23,6 +24,14 @@ export const examples: Example[] = [
|
||||
"Force CPU (skip CoreML/CUDA)",
|
||||
"hyperframes remove-background avatar.mp4 -o transparent.webm --device cpu",
|
||||
],
|
||||
[
|
||||
"Smaller file at the cost of color match (text-behind-subject won't blend as cleanly)",
|
||||
"hyperframes remove-background avatar.mp4 -o transparent.webm --quality fast",
|
||||
],
|
||||
[
|
||||
"Visually-lossless WebM (master / re-encode source)",
|
||||
"hyperframes remove-background avatar.mp4 -o transparent.webm --quality best",
|
||||
],
|
||||
["Show detected providers without rendering", "hyperframes remove-background --info"],
|
||||
];
|
||||
|
||||
@@ -48,6 +57,11 @@ export default defineCommand({
|
||||
description: `Execution provider: ${DEVICES.join(", ")}`,
|
||||
default: "auto",
|
||||
},
|
||||
quality: {
|
||||
type: "string",
|
||||
description: `Encoder quality preset for .webm output: ${QUALITIES.join(", ")} (default: ${DEFAULT_QUALITY}). Higher quality = closer color match when overlaying on the source mp4, larger file. Ignored for .mov / .png.`,
|
||||
default: DEFAULT_QUALITY,
|
||||
},
|
||||
info: {
|
||||
type: "boolean",
|
||||
description: "Print detected execution providers and exit (no render)",
|
||||
@@ -81,6 +95,12 @@ export default defineCommand({
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
if (!isQuality(args.quality)) {
|
||||
console.error(
|
||||
c.error(`Invalid --quality '${String(args.quality)}'. Use: ${QUALITIES.join(", ")}.`),
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const inputPath = resolve(args.input);
|
||||
const outputPath = resolve(args.output);
|
||||
@@ -95,6 +115,7 @@ export default defineCommand({
|
||||
inputPath,
|
||||
outputPath,
|
||||
device: args.device,
|
||||
quality: args.quality,
|
||||
onProgress: (event) => {
|
||||
if (event.kind === "info") {
|
||||
spin?.message(event.message);
|
||||
|
||||
@@ -115,6 +115,7 @@ export {
|
||||
parseImageElements,
|
||||
extractVideoFramesRange,
|
||||
extractAllVideoFrames,
|
||||
resolveProjectRelativeSrc,
|
||||
getFrameAtTime,
|
||||
createFrameLookupTable,
|
||||
FrameLookupTable,
|
||||
|
||||
@@ -12,6 +12,7 @@ import { downloadToTemp, isHttpUrl } from "../utils/urlDownloader.js";
|
||||
import { DEFAULT_CONFIG, type EngineConfig } from "../config.js";
|
||||
import { runFfmpeg } from "../utils/runFfmpeg.js";
|
||||
import { unwrapTemplate } from "../utils/htmlTemplate.js";
|
||||
import { resolveProjectRelativeSrc } from "./videoFrameExtractor.js";
|
||||
import type { AudioElement, AudioTrack, MixResult } from "./audioMixer.types.js";
|
||||
|
||||
export type { AudioElement, AudioTrack, MixResult } from "./audioMixer.types.js";
|
||||
@@ -325,13 +326,10 @@ export async function processCompositionAudio(
|
||||
}
|
||||
try {
|
||||
let srcPath = element.src;
|
||||
// Use isAbsolute() rather than startsWith("/"). On Windows, absolute paths
|
||||
// like "C:\…" are not detected by the latter, so we'd re-join them under
|
||||
// baseDir and produce duplicated, nonexistent paths.
|
||||
if (!isAbsolute(srcPath) && !isHttpUrl(srcPath)) {
|
||||
const fromCompiled = compiledDir ? join(compiledDir, srcPath) : null;
|
||||
srcPath =
|
||||
fromCompiled && existsSync(fromCompiled) ? fromCompiled : join(baseDir, srcPath);
|
||||
// Same browser-vs-filesystem path semantics as videos — see
|
||||
// resolveProjectRelativeSrc in videoFrameExtractor for the full why.
|
||||
srcPath = resolveProjectRelativeSrc(element.src, baseDir, compiledDir);
|
||||
}
|
||||
|
||||
if (isHttpUrl(srcPath)) {
|
||||
|
||||
@@ -139,12 +139,43 @@ export function buildEncoderArgs(
|
||||
else args.push("-global_quality", String(quality));
|
||||
break;
|
||||
}
|
||||
|
||||
// Same B-frame story as the SW branch below — nvenc emits B-frames
|
||||
// by default (qsv via b_strategy, vaapi too), and the negative-DTS
|
||||
// freeze hits the same downstream players. The unconditional
|
||||
// `-avoid_negative_ts make_zero` near the bottom of this function
|
||||
// covers the mux level, but we belt-and-suspenders the encoder too
|
||||
// so even tools that consume the chunk file directly (without going
|
||||
// through our mux step) play correctly. videotoolbox doesn't accept
|
||||
// `-bf` so it's skipped — videotoolbox h264 also doesn't emit
|
||||
// negative DTS in practice on macOS Sonoma+.
|
||||
if (
|
||||
codec === "h264" &&
|
||||
(gpuEncoder === "nvenc" || gpuEncoder === "qsv" || gpuEncoder === "vaapi")
|
||||
) {
|
||||
args.push("-bf", "0");
|
||||
if (gpuEncoder === "qsv") {
|
||||
args.push("-b_strategy", "0");
|
||||
}
|
||||
}
|
||||
} else {
|
||||
const encoderName = codec === "h264" ? "libx264" : "libx265";
|
||||
args.push("-c:v", encoderName, "-preset", preset);
|
||||
if (bitrate) args.push("-b:v", bitrate);
|
||||
else args.push("-crf", String(quality));
|
||||
|
||||
// Disable B-frames. Standard h264 with B-frames produces negative DTS
|
||||
// at the start of the stream (the first B-frame's decode order is
|
||||
// "before" the first I-frame's presentation time). VS Code's video
|
||||
// preview, several browser <video> pipelines, and some HW decoders
|
||||
// freeze on the first frame when DTS is negative, so audio plays alone.
|
||||
// -bf 0 makes PTS == DTS at every frame, eliminating the issue at the
|
||||
// source. Quality cost is ~5–10% larger files at the same CRF — a
|
||||
// worthwhile trade for "the file plays everywhere".
|
||||
if (codec === "h264") {
|
||||
args.push("-bf", "0");
|
||||
}
|
||||
|
||||
// Encoder-specific params: anti-banding + color space tagging.
|
||||
// aq-mode=3 redistributes bits to dark flat areas (gradients).
|
||||
// For HDR x265 paths we additionally embed BT.2020 + transfer + HDR static
|
||||
@@ -239,6 +270,8 @@ export function buildEncoderArgs(
|
||||
args.push("-pix_fmt", pixelFormat);
|
||||
}
|
||||
|
||||
args.push("-avoid_negative_ts", "make_zero");
|
||||
|
||||
args.push("-y", outputPath);
|
||||
return args;
|
||||
}
|
||||
@@ -510,6 +543,9 @@ export async function muxVideoWithAudio(
|
||||
} else {
|
||||
args.push("-c:a", "aac", "-b:a", "192k", "-movflags", "+faststart");
|
||||
}
|
||||
// PTS bases can diverge during mux and reintroduce negative DTS. See
|
||||
// buildEncoderArgs for the full reasoning on why that breaks playback.
|
||||
args.push("-avoid_negative_ts", "make_zero");
|
||||
args.push("-shortest", "-y", outputPath);
|
||||
|
||||
const processTimeout = config?.ffmpegProcessTimeout ?? DEFAULT_CONFIG.ffmpegProcessTimeout;
|
||||
|
||||
@@ -388,8 +388,14 @@ export async function initializeSession(session: CaptureSession): Promise<void>
|
||||
|
||||
await applyVideoMetadataHints(page, session.options.videoMetadataHints);
|
||||
|
||||
// Wait for all video elements to have loaded metadata (dimensions + duration)
|
||||
// Without this, frame 0 captures videos at their 300x150 default size.
|
||||
// Wait for all video elements to have decoded their CURRENT frame, not
|
||||
// just metadata. readyState >= 2 (HAVE_CURRENT_DATA) means a frame is
|
||||
// actually rasterized and ready to paint — at >= 1 (HAVE_METADATA) we
|
||||
// only know the dimensions, and the first <video> screenshot can come
|
||||
// back as a black/blank rectangle. This bites compositions with two
|
||||
// <video> elements of different codecs (h264 mp4 + VP9 webm) where the
|
||||
// faster decoder lets the readiness check pass while the slower one
|
||||
// hasn't painted, producing a black "first frame" for the slower clip.
|
||||
// skipReadinessVideoIds excludes natively-extracted videos (e.g. HDR HEVC
|
||||
// sources) whose frames come from ffmpeg out-of-band. videoMetadataHints
|
||||
// supply intrinsic dimensions for skipped videos whose layout depends on
|
||||
@@ -397,12 +403,12 @@ export async function initializeSession(session: CaptureSession): Promise<void>
|
||||
const skipIdsLiteral = JSON.stringify(session.options.skipReadinessVideoIds ?? []);
|
||||
const videosReady = await pollPageExpression(
|
||||
page,
|
||||
`(() => { const skip = new Set(${skipIdsLiteral}); const vids = Array.from(document.querySelectorAll("video")).filter(v => !skip.has(v.id)); return vids.length === 0 || vids.every(v => v.readyState >= 1); })()`,
|
||||
`(() => { const skip = new Set(${skipIdsLiteral}); const vids = Array.from(document.querySelectorAll("video")).filter(v => !skip.has(v.id)); return vids.length === 0 || vids.every(v => v.readyState >= 2); })()`,
|
||||
pageReadyTimeout,
|
||||
);
|
||||
if (!videosReady) {
|
||||
throw new Error(
|
||||
`[FrameCapture] video metadata not ready after ${pageReadyTimeout}ms. Video elements must load metadata before capture starts.`,
|
||||
`[FrameCapture] video first frame not decoded after ${pageReadyTimeout}ms. Video elements must reach readyState >= 2 (HAVE_CURRENT_DATA) before capture starts.`,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -484,16 +490,13 @@ export async function initializeSession(session: CaptureSession): Promise<void>
|
||||
|
||||
await applyVideoMetadataHints(page, session.options.videoMetadataHints);
|
||||
|
||||
// Wait for all video elements to have loaded metadata (dimensions + duration).
|
||||
// Without this, frame 0 captures videos at their 300x150 default size.
|
||||
// See screenshot-mode comment above for why skipReadinessVideoIds and
|
||||
// videoMetadataHints are paired.
|
||||
// Same readyState contract as the screenshot path above (>= 2 / HAVE_CURRENT_DATA).
|
||||
const beginframeSkipIdsLiteral = JSON.stringify(session.options.skipReadinessVideoIds ?? []);
|
||||
const videoDeadline =
|
||||
Date.now() + (session.config?.playerReadyTimeout ?? DEFAULT_CONFIG.playerReadyTimeout);
|
||||
while (Date.now() < videoDeadline) {
|
||||
const videosReady = await page.evaluate(
|
||||
`(() => { const skip = new Set(${beginframeSkipIdsLiteral}); const vids = Array.from(document.querySelectorAll("video")).filter(v => !skip.has(v.id)); return vids.length === 0 || vids.every(v => v.readyState >= 1); })()`,
|
||||
`(() => { const skip = new Set(${beginframeSkipIdsLiteral}); const vids = Array.from(document.querySelectorAll("video")).filter(v => !skip.has(v.id)); return vids.length === 0 || vids.every(v => v.readyState >= 2); })()`,
|
||||
);
|
||||
if (videosReady) break;
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
|
||||
@@ -221,12 +221,33 @@ export function buildStreamingArgs(
|
||||
else args.push("-global_quality", String(quality));
|
||||
break;
|
||||
}
|
||||
|
||||
// Mirror SW branch: GPU h264 paths emit B-frames by default (nvenc, qsv,
|
||||
// vaapi) and produce the same negative-DTS freeze for downstream players.
|
||||
// See chunkEncoder.buildEncoderArgs for the full explanation.
|
||||
if (
|
||||
codec === "h264" &&
|
||||
(gpuEncoder === "nvenc" || gpuEncoder === "qsv" || gpuEncoder === "vaapi")
|
||||
) {
|
||||
args.push("-bf", "0");
|
||||
if (gpuEncoder === "qsv") {
|
||||
args.push("-b_strategy", "0");
|
||||
}
|
||||
}
|
||||
} else {
|
||||
const encoderName = codec === "h264" ? "libx264" : "libx265";
|
||||
args.push("-c:v", encoderName, "-preset", preset);
|
||||
if (bitrate) args.push("-b:v", bitrate);
|
||||
else args.push("-crf", String(quality));
|
||||
|
||||
// Mirrors chunkEncoder: disable B-frames for h264 so PTS == DTS, no
|
||||
// negative DTS at stream start. Without this, files freeze on the
|
||||
// first frame in VS Code preview, several browsers, and some HW
|
||||
// decoders. See chunkEncoder.buildEncoderArgs for the full reasoning.
|
||||
if (codec === "h264") {
|
||||
args.push("-bf", "0");
|
||||
}
|
||||
|
||||
// Encoder-specific params: anti-banding + color space tagging.
|
||||
// For HDR, getHdrEncoderColorParams also emits the SMPTE ST 2086
|
||||
// mastering-display and CTA-861.3 MaxCLL/MaxFALL SEI messages —
|
||||
@@ -313,6 +334,10 @@ export function buildStreamingArgs(
|
||||
args.push("-pix_fmt", pixelFormat);
|
||||
}
|
||||
|
||||
// Belt-and-suspenders against negative DTS at stream start. See chunkEncoder
|
||||
// for the full explanation; same playback compatibility class.
|
||||
args.push("-avoid_negative_ts", "make_zero");
|
||||
|
||||
args.push("-y", outputPath);
|
||||
return args;
|
||||
}
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync } from "node:fs";
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
readdirSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { createHash } from "node:crypto";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
@@ -9,6 +17,9 @@ import {
|
||||
parseImageElements,
|
||||
extractAllVideoFrames,
|
||||
createFrameLookupTable,
|
||||
resolveProjectRelativeSrc,
|
||||
codecMayHaveAlpha,
|
||||
decoderForCodec,
|
||||
type VideoElement,
|
||||
type ExtractedFrames,
|
||||
} from "./videoFrameExtractor.js";
|
||||
@@ -23,6 +34,111 @@ import { runFfmpeg } from "../utils/runFfmpeg.js";
|
||||
// synthesized VFR fixture.
|
||||
const HAS_FFMPEG = spawnSync("ffmpeg", ["-version"]).status === 0;
|
||||
|
||||
// Codec-based alpha defaulting replaces tag-based detection (the
|
||||
// alpha_mode/ALPHA_MODE case bug — see ffprobe.test.ts for the regression
|
||||
// pin on that). The extractor uses these helpers for two decisions:
|
||||
// 1. whether to force the alpha-aware decoder (libvpx-vp9 for VP9, libvpx
|
||||
// for VP8)
|
||||
// 2. whether to default the cached frame format to PNG (with alpha) vs JPG
|
||||
// The "default to capable" trade is small file-size growth on opaque VP9
|
||||
// content for correctness on alpha-having content even when the sidecar tag
|
||||
// is missing or muxed with the wrong case.
|
||||
describe("codec alpha capability", () => {
|
||||
it("flags VP9, VP8, and ProRes as alpha-capable", () => {
|
||||
expect(codecMayHaveAlpha("vp9")).toBe(true);
|
||||
expect(codecMayHaveAlpha("VP9")).toBe(true);
|
||||
expect(codecMayHaveAlpha("vp8")).toBe(true);
|
||||
expect(codecMayHaveAlpha("prores")).toBe(true);
|
||||
});
|
||||
|
||||
it("does not flag h264 / h265 / mpeg4 (no alpha in their bitstreams)", () => {
|
||||
expect(codecMayHaveAlpha("h264")).toBe(false);
|
||||
expect(codecMayHaveAlpha("h265")).toBe(false);
|
||||
expect(codecMayHaveAlpha("hevc")).toBe(false);
|
||||
expect(codecMayHaveAlpha("mpeg4")).toBe(false);
|
||||
});
|
||||
|
||||
it("treats undefined / empty input as non-alpha", () => {
|
||||
expect(codecMayHaveAlpha(undefined)).toBe(false);
|
||||
expect(codecMayHaveAlpha("")).toBe(false);
|
||||
});
|
||||
|
||||
it("returns the alpha-aware decoder name for VP9 and VP8", () => {
|
||||
expect(decoderForCodec("vp9")).toBe("libvpx-vp9");
|
||||
expect(decoderForCodec("VP9")).toBe("libvpx-vp9");
|
||||
expect(decoderForCodec("vp8")).toBe("libvpx");
|
||||
});
|
||||
});
|
||||
|
||||
// Regression: a long-standing footgun where `<video src="../assets/foo">`
|
||||
// inside a sub-composition silently dropped the video from extraction. The
|
||||
// browser's URL resolver clamps `..` at the served origin's root (so the
|
||||
// page renders fine in the studio), but `path.join(projectDir, "../assets/foo")`
|
||||
// normalizes to <parentOfProjectDir>/assets/foo, which doesn't exist —
|
||||
// extraction skipped, no frame injection, rendered output shows the video's
|
||||
// first decoded frame for the whole clip duration. The resolver now mirrors
|
||||
// browser semantics by clamping any traversal that escapes the project root.
|
||||
describe("resolveProjectRelativeSrc — sub-composition path clamping", () => {
|
||||
let tmp: string;
|
||||
|
||||
beforeAll(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), "hf-resolver-"));
|
||||
mkdirSync(join(tmp, "project", "assets"), { recursive: true });
|
||||
writeFileSync(join(tmp, "project", "assets", "foo.mp4"), "");
|
||||
});
|
||||
afterAll(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("returns the literal join when the file exists at projectDir/src", () => {
|
||||
const projectDir = join(tmp, "project");
|
||||
expect(resolveProjectRelativeSrc("assets/foo.mp4", projectDir)).toBe(
|
||||
join(projectDir, "assets/foo.mp4"),
|
||||
);
|
||||
});
|
||||
|
||||
it("clamps a leading `../` so `../assets/foo.mp4` resolves to assets/foo.mp4", () => {
|
||||
const projectDir = join(tmp, "project");
|
||||
expect(resolveProjectRelativeSrc("../assets/foo.mp4", projectDir)).toBe(
|
||||
join(projectDir, "assets/foo.mp4"),
|
||||
);
|
||||
});
|
||||
|
||||
it("clamps multiple leading `../../../` segments", () => {
|
||||
const projectDir = join(tmp, "project");
|
||||
expect(resolveProjectRelativeSrc("../../../assets/foo.mp4", projectDir)).toBe(
|
||||
join(projectDir, "assets/foo.mp4"),
|
||||
);
|
||||
});
|
||||
|
||||
it("clamps mid-path traversal that escapes baseDir (not just leading `..`)", () => {
|
||||
// `assets/../../foo.mp4` collapses past projectDir via path.join — this
|
||||
// case used to silently escape; the resolver now strips embedded `..`
|
||||
// segments and re-anchors at the project root.
|
||||
const projectDir = join(tmp, "project");
|
||||
expect(resolveProjectRelativeSrc("assets/../../assets/foo.mp4", projectDir)).toBe(
|
||||
join(projectDir, "assets/foo.mp4"),
|
||||
);
|
||||
});
|
||||
|
||||
it("returns the (non-existent) base-dir path on miss so callers get a stable error message", () => {
|
||||
const projectDir = join(tmp, "project");
|
||||
expect(resolveProjectRelativeSrc("../assets/missing.mp4", projectDir)).toBe(
|
||||
join(projectDir, "../assets/missing.mp4"),
|
||||
);
|
||||
});
|
||||
|
||||
it("prefers compiled-dir over base-dir when the file exists in both", () => {
|
||||
const projectDir = join(tmp, "project");
|
||||
const compiledDir = join(tmp, "compiled");
|
||||
mkdirSync(join(compiledDir, "assets"), { recursive: true });
|
||||
writeFileSync(join(compiledDir, "assets", "foo.mp4"), "");
|
||||
expect(resolveProjectRelativeSrc("assets/foo.mp4", projectDir, compiledDir)).toBe(
|
||||
join(compiledDir, "assets/foo.mp4"),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseVideoElements", () => {
|
||||
it("parses videos without an id or data-start attribute", () => {
|
||||
const videos = parseVideoElements('<video src="clip.mp4"></video>');
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
|
||||
import { spawn } from "child_process";
|
||||
import { existsSync, mkdirSync, readdirSync, rmSync } from "fs";
|
||||
import { isAbsolute, join } from "path";
|
||||
import { isAbsolute, join, posix, resolve, sep } from "path";
|
||||
import { parseHTML } from "linkedom";
|
||||
import { extractMediaMetadata, type VideoMetadata } from "../utils/ffprobe.js";
|
||||
import {
|
||||
@@ -230,8 +230,17 @@ export async function extractVideoFramesRange(
|
||||
if (isHdr && isMacOS) {
|
||||
args.push("-hwaccel", "videotoolbox");
|
||||
}
|
||||
if (metadata.hasAlpha && metadata.videoCodec === "vp9") {
|
||||
args.push("-c:v", "libvpx-vp9");
|
||||
// Always force the alpha-aware decoder on codecs that can carry alpha. The
|
||||
// alternative — gating on `metadata.hasAlpha` — relies on tag detection that
|
||||
// has at least three known failure modes: case-sensitivity across ffmpeg
|
||||
// versions (`alpha_mode` vs `ALPHA_MODE`), missing tags from older muxers,
|
||||
// and mp4-as-webm rewraps that drop the sidecar. A wrong negative there
|
||||
// silently strips alpha during decode and the bug doesn't surface until
|
||||
// the rendered video is missing layers. Codec-based default has no such
|
||||
// ambiguity: libvpx-vp9 reads the alpha sidecar when present and decodes
|
||||
// normally when it isn't.
|
||||
if (codecMayHaveAlpha(metadata.videoCodec)) {
|
||||
args.push("-c:v", decoderForCodec(metadata.videoCodec));
|
||||
}
|
||||
args.push("-ss", String(startTime), "-i", videoPath, "-t", String(duration));
|
||||
|
||||
@@ -398,9 +407,31 @@ function resolveSegmentDuration(
|
||||
return sourceRemaining > 0 ? sourceRemaining : metadata.durationSeconds;
|
||||
}
|
||||
|
||||
/**
|
||||
* Codecs whose bitstream is allowed to carry an alpha channel. Default the
|
||||
* extraction path to PNG output for these regardless of `metadata.hasAlpha`
|
||||
* so a missed sidecar tag doesn't silently strip transparency. Opaque content
|
||||
* encoded in one of these codecs pays a small file-size cost on the cached
|
||||
* frames but stays correct on the rare case where alpha IS present and the
|
||||
* tag was missed.
|
||||
*/
|
||||
const ALPHA_CAPABLE_CODECS = new Set(["vp9", "vp8", "prores"]);
|
||||
|
||||
export function codecMayHaveAlpha(codec: string | undefined): boolean {
|
||||
return ALPHA_CAPABLE_CODECS.has((codec ?? "").toLowerCase());
|
||||
}
|
||||
|
||||
export function decoderForCodec(codec: string | undefined): string {
|
||||
const c = (codec ?? "").toLowerCase();
|
||||
if (c === "vp9") return "libvpx-vp9";
|
||||
if (c === "vp8") return "libvpx";
|
||||
return c;
|
||||
}
|
||||
|
||||
function resolveFrameFormat(metadata: VideoMetadata, requested?: "jpg" | "png"): CacheFrameFormat {
|
||||
if (requested) return requested;
|
||||
return metadata.hasAlpha ? "png" : "jpg";
|
||||
if (metadata.hasAlpha || codecMayHaveAlpha(metadata.videoCodec)) return "png";
|
||||
return "jpg";
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -459,6 +490,54 @@ async function convertVfrToCfr(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a relative `<video src>` to a filesystem path the way the browser
|
||||
* resolves it as a URL. Browsers clamp `..` segments at the served origin's
|
||||
* root; `path.join(projectDir, "../assets/foo")` does not. So a sub-comp
|
||||
* `<video src="../assets/foo">` loads in the page (browser clamps to
|
||||
* `<projectDir>/assets/foo`) but the filesystem-side resolver lands at
|
||||
* `<parentOfProjectDir>/assets/foo` — file missing, extraction skipped,
|
||||
* the rendered output shows the video's first frame for the whole clip.
|
||||
*
|
||||
* The clamp covers two escape patterns: leading `..` (`../assets/foo`) AND
|
||||
* mid-path escapes (`assets/../../foo`) that `path.join` collapses past the
|
||||
* project root silently. Both fall back to a project-rooted candidate that
|
||||
* strips traversal from the resolved path.
|
||||
*
|
||||
* Returns the first existing candidate, or the base-dir join on miss so
|
||||
* the caller's `existsSync` check produces a stable error path.
|
||||
*/
|
||||
export function resolveProjectRelativeSrc(
|
||||
src: string,
|
||||
baseDir: string,
|
||||
compiledDir?: string,
|
||||
): string {
|
||||
const fromCompiled = compiledDir ? join(compiledDir, src) : null;
|
||||
const fromBase = join(baseDir, src);
|
||||
const candidates: string[] = [];
|
||||
if (fromCompiled) candidates.push(fromCompiled);
|
||||
candidates.push(fromBase);
|
||||
// If the joined result escapes the project root (either via leading `..`
|
||||
// or mid-path traversal that path.join collapsed past baseDir), retry
|
||||
// with the basename re-anchored at the project root. This mirrors the
|
||||
// browser URL clamp without relying on a particular `..` shape.
|
||||
const baseAbs = resolve(baseDir);
|
||||
const fromBaseAbs = resolve(fromBase);
|
||||
if (!fromBaseAbs.startsWith(baseAbs + sep) && fromBaseAbs !== baseAbs) {
|
||||
// Normalize first (`assets/../../assets/foo.mp4` → `../assets/foo.mp4`)
|
||||
// then strip any remaining leading `..` segments. Stripping `..` from the
|
||||
// raw input would leave dangling siblings (`assets/../../assets/foo`
|
||||
// would become `assets/assets/foo` instead of `assets/foo`).
|
||||
const normalized = posix.normalize(src.replace(/\\/g, "/"));
|
||||
const stripped = normalized.replace(/^(\.\.\/)+/, "");
|
||||
if (stripped && stripped !== src && !stripped.startsWith("..")) {
|
||||
if (compiledDir) candidates.push(join(compiledDir, stripped));
|
||||
candidates.push(join(baseDir, stripped));
|
||||
}
|
||||
}
|
||||
return candidates.find(existsSync) ?? fromBase;
|
||||
}
|
||||
|
||||
export async function extractAllVideoFrames(
|
||||
videos: VideoElement[],
|
||||
baseDir: string,
|
||||
@@ -487,6 +566,9 @@ export async function extractAllVideoFrames(
|
||||
// Phase 1: Resolve paths and download remote videos
|
||||
const phase1Start = Date.now();
|
||||
const resolvedVideos: Array<{ video: VideoElement; videoPath: string }> = [];
|
||||
// Dedupe missing-src warnings: a composition with N <video> elements all
|
||||
// pointing at the same broken src should only print one warning, not N.
|
||||
const warnedSrcs = new Set<string>();
|
||||
for (const video of videos) {
|
||||
if (signal?.aborted) break;
|
||||
try {
|
||||
@@ -496,9 +578,7 @@ export async function extractAllVideoFrames(
|
||||
// baseDir and produce duplicated, nonexistent paths
|
||||
// (e.g. C:\tmp\hf-vfr-test-X\C:\tmp\hf-vfr-test-X\vfr_screen.mp4).
|
||||
if (!isAbsolute(videoPath) && !isHttpUrl(videoPath)) {
|
||||
const fromCompiled = compiledDir ? join(compiledDir, videoPath) : null;
|
||||
videoPath =
|
||||
fromCompiled && existsSync(fromCompiled) ? fromCompiled : join(baseDir, videoPath);
|
||||
videoPath = resolveProjectRelativeSrc(video.src, baseDir, compiledDir);
|
||||
}
|
||||
|
||||
if (isHttpUrl(videoPath)) {
|
||||
@@ -508,6 +588,19 @@ export async function extractAllVideoFrames(
|
||||
}
|
||||
|
||||
if (!existsSync(videoPath)) {
|
||||
// Loud: silent miss leaves the rendered video frozen at frame 0 with
|
||||
// no error in stdout — extremely confusing for authors. Dedupe by
|
||||
// src so 50 broken videos pointing at the same path don't spam.
|
||||
if (!warnedSrcs.has(video.src)) {
|
||||
warnedSrcs.add(video.src);
|
||||
process.stderr.write(
|
||||
`[hyperframes:render] WARNING: video src="${video.src}" ` +
|
||||
`could not be resolved on disk (looked for ${videoPath}). ` +
|
||||
`The rendered output will show this video's first frame for the entire clip duration. ` +
|
||||
`If your <video> lives inside a sub-composition, prefer project-root-relative paths ` +
|
||||
`(e.g. src="assets/foo.mp4") over "../assets/foo.mp4".\n`,
|
||||
);
|
||||
}
|
||||
errors.push({ videoId: video.id, error: `Video file not found: ${videoPath}` });
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -225,6 +225,46 @@ describe("ffprobe missing-binary fallback", () => {
|
||||
expect(meta.hasAlpha).toBe(true);
|
||||
});
|
||||
|
||||
// Regression: newer libavformat builds (and the output of `hyperframes
|
||||
// remove-background` itself) write the VP9-alpha sidecar tag as
|
||||
// `ALPHA_MODE` (uppercase). The lowercase-only check classified those
|
||||
// files as having no alpha, the producer extracted them as JPGs, and
|
||||
// the injected <img> overlays were fully opaque rectangles that hid
|
||||
// every static element below them on the z-stack. The bug was silent —
|
||||
// studio preview rendered correctly via native <video> playback while
|
||||
// production renders covered headlines and captions with the avatar.
|
||||
it("extractMediaMetadata detects ALPHA_MODE (uppercase) streams from newer ffmpeg builds", async () => {
|
||||
const { spawn } = createSpawnSpy([
|
||||
{
|
||||
kind: "exit",
|
||||
code: 0,
|
||||
stdout: JSON.stringify({
|
||||
streams: [
|
||||
{
|
||||
codec_type: "video",
|
||||
codec_name: "vp9",
|
||||
width: 320,
|
||||
height: 180,
|
||||
r_frame_rate: "30/1",
|
||||
avg_frame_rate: "30/1",
|
||||
pix_fmt: "yuv420p",
|
||||
tags: { ALPHA_MODE: "1" },
|
||||
},
|
||||
],
|
||||
format: { duration: "1.5" },
|
||||
}),
|
||||
},
|
||||
]);
|
||||
vi.resetModules();
|
||||
vi.doMock("child_process", () => ({ spawn }));
|
||||
|
||||
const { extractMediaMetadata: extractMediaMetadataMocked } = await import("./ffprobe.js");
|
||||
const meta = await extractMediaMetadataMocked("/tmp/alpha-uppercase.webm");
|
||||
|
||||
expect(meta.videoCodec).toBe("vp9");
|
||||
expect(meta.hasAlpha).toBe(true);
|
||||
});
|
||||
|
||||
it("extractMediaMetadata rethrows ffprobe-missing error for non-image files without fallback", async () => {
|
||||
const { spawn } = createSpawnSpy([{ kind: "missing" }]);
|
||||
vi.resetModules();
|
||||
|
||||
@@ -203,6 +203,21 @@ function extractStillImageMetadata(filePath: string): StillImageMetadata | null
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read an ffprobe tag case-insensitively. ffmpeg/libavformat versions disagree
|
||||
* on tag casing — VP9 alpha is `alpha_mode` in older builds and `ALPHA_MODE`
|
||||
* in newer ones; HDR tags vary similarly. Use this for any sidecar tag where
|
||||
* you want to be resilient across muxer versions.
|
||||
*/
|
||||
function readTagCI(tags: Record<string, string | undefined> | undefined, name: string): string {
|
||||
if (!tags) return "";
|
||||
const target = name.toLowerCase();
|
||||
for (const [key, value] of Object.entries(tags)) {
|
||||
if (key.toLowerCase() === target && typeof value === "string") return value;
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
function parseFrameRate(frameRateStr: string | undefined): number {
|
||||
if (!frameRateStr) return 0;
|
||||
const parts = frameRateStr.split("/");
|
||||
@@ -277,7 +292,7 @@ export async function extractMediaMetadata(filePath: string): Promise<VideoMetad
|
||||
: null;
|
||||
const colorSpace = ffprobeColorSpace ?? stillImageMeta?.colorSpace ?? null;
|
||||
const pixelFormat = videoStream.pix_fmt || "";
|
||||
const alphaMode = videoStream.tags?.alpha_mode || "";
|
||||
const alphaMode = readTagCI(videoStream.tags, "alpha_mode");
|
||||
const hasAlpha =
|
||||
/(^|[^a-z])yuva|rgba|argb|bgra|gbrap|gray[a-z0-9]*a/i.test(pixelFormat) || alphaMode === "1";
|
||||
|
||||
|
||||
@@ -650,7 +650,14 @@ async function runTestSuite(
|
||||
// Visual comparison (100 frames, 1 per 1% of video duration)
|
||||
logPretty("Comparing visual quality (100 checkpoints)...", "🔍");
|
||||
const videoMetadata = await extractMediaMetadata(renderedOutputPath);
|
||||
const videoDuration = videoMetadata.durationSeconds;
|
||||
const snapshotMetadata = await extractMediaMetadata(snapshotVideoPath);
|
||||
// Sample at the common duration. Container duration can drift between
|
||||
// rendered and snapshot when encoder/mux flags change (e.g. -avoid_negative_ts
|
||||
// can shift the first audio sample, extending reported duration without
|
||||
// changing video frame count). Using the rendered duration alone makes the
|
||||
// last checkpoint land on a frame index that may not exist in the snapshot,
|
||||
// which causes ffmpeg's PSNR filter to emit no `average:` line.
|
||||
const videoDuration = Math.min(videoMetadata.durationSeconds, snapshotMetadata.durationSeconds);
|
||||
|
||||
const visualCheckpoints: Array<{ time: number; psnr: number; passed: boolean }> = [];
|
||||
for (let i = 0; i < 100; i++) {
|
||||
|
||||
@@ -33,6 +33,7 @@ import {
|
||||
type EngineConfig,
|
||||
resolveConfig,
|
||||
extractAllVideoFrames,
|
||||
resolveProjectRelativeSrc,
|
||||
type ExtractedFrames,
|
||||
type ExtractionPhaseBreakdown,
|
||||
createFrameLookupTable,
|
||||
@@ -2312,13 +2313,14 @@ export async function executeRenderJob(
|
||||
if (job.config.hdrMode !== "force-sdr" && composition.videos.length > 0) {
|
||||
await Promise.all(
|
||||
composition.videos.map(async (v) => {
|
||||
let videoPath = v.src;
|
||||
if (!videoPath.startsWith("/")) {
|
||||
const fromCompiled = existsSync(join(compiledDir, videoPath))
|
||||
? join(compiledDir, videoPath)
|
||||
: join(projectDir, videoPath);
|
||||
videoPath = fromCompiled;
|
||||
}
|
||||
// Use the shared resolver so a `<video src="../assets/foo">` in a
|
||||
// sub-composition resolves the same way the browser would (see
|
||||
// resolveProjectRelativeSrc in videoFrameExtractor for the full
|
||||
// explanation). isAbsolute (not `startsWith("/")`) so Windows
|
||||
// absolute paths like `C:\...` skip the join correctly.
|
||||
const videoPath = isAbsolute(v.src)
|
||||
? v.src
|
||||
: resolveProjectRelativeSrc(v.src, projectDir, compiledDir);
|
||||
if (!existsSync(videoPath)) return;
|
||||
const meta = await extractMediaMetadata(videoPath);
|
||||
if (isHdrColorSpace(meta.colorSpace)) {
|
||||
|
||||
@@ -102,14 +102,14 @@ Compositions consume a flat array of word objects. The `id` field (`w0`, `w1`, .
|
||||
|
||||
## Background Removal (`remove-background`)
|
||||
|
||||
Remove the background from a video or image so it can sit as a transparent overlay in a composition (e.g. an avatar floating on a background plate).
|
||||
Remove the background from a video or image so the subject (typically a person — avatar, presenter, talking head) sits as a transparent overlay in a composition.
|
||||
|
||||
```bash
|
||||
npx hyperframes remove-background avatar.mp4 -o transparent.webm # default: VP9 alpha WebM
|
||||
npx hyperframes remove-background avatar.mp4 -o transparent.mov # ProRes 4444 (editing)
|
||||
npx hyperframes remove-background portrait.jpg -o cutout.png # single-image cutout
|
||||
npx hyperframes remove-background avatar.mp4 -o transparent.webm --device cpu
|
||||
npx hyperframes remove-background --info # detected providers
|
||||
npx hyperframes remove-background subject.mp4 -o transparent.webm # default: VP9 alpha WebM
|
||||
npx hyperframes remove-background subject.mp4 -o transparent.mov # ProRes 4444 (editing)
|
||||
npx hyperframes remove-background portrait.jpg -o cutout.png # single-image cutout
|
||||
npx hyperframes remove-background subject.mp4 -o transparent.webm --device cpu
|
||||
npx hyperframes remove-background --info # detected providers
|
||||
```
|
||||
|
||||
Uses `u2net_human_seg` (MIT). First run downloads ~168 MB of weights to `~/.cache/hyperframes/background-removal/models/`.
|
||||
@@ -124,6 +124,58 @@ Uses `u2net_human_seg` (MIT). First run downloads ~168 MB of weights to `~/.cach
|
||||
|
||||
Chrome decodes VP9 alpha natively, so the `.webm` plugs into a composition like any other muted-autoplay video — see the `hyperframes` skill for the `<video>` track conventions.
|
||||
|
||||
### Quality presets
|
||||
|
||||
`--quality fast|balanced|best` controls only the VP9 encoder's CRF — segmentation quality is fixed.
|
||||
|
||||
| Preset | CRF | When |
|
||||
| ---------- | --- | ----------------------------------------------------- |
|
||||
| `fast` | 30 | Iterating, smaller file, looser color match |
|
||||
| `balanced` | 18 | Default. Visually identical for most uses |
|
||||
| `best` | 12 | Master / final delivery. Largest file, tightest match |
|
||||
|
||||
### Compositing patterns — pick the right one
|
||||
|
||||
The cutout webm is a **re-encoded copy** of the source mp4's RGB. That choice has consequences depending on what you put behind it:
|
||||
|
||||
| Pattern | What's behind the cutout | Result |
|
||||
| -------------------------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Cutout over a different scene** (most common) | Static image, gradient, or unrelated video | Looks great. The cutout's RGB is the only source of the subject — no doubling, no edge halo. This is what `remove-background` is built for. |
|
||||
| **Cutout over its own source mp4** (text-behind-subject) | Same mp4 the cutout was generated from | Two RGB sources for the same person. At default `--quality balanced` (crf 18) the doubling is barely visible; at `--quality fast` (crf 30) you'll see a faint color shift / edge halo. Use `--quality best` (crf 12) for masters. |
|
||||
| **Cutout over a _different_ take of the same person** | Footage of the same subject | Will look like two separate people overlapping. Don't do this. |
|
||||
|
||||
**Text-behind-subject** (headline behind a presenter):
|
||||
|
||||
```html
|
||||
<video
|
||||
src="presenter.mp4"
|
||||
id="bg"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
<h1 id="headline" style="z-index:2; ...">MAKE IT IN HYPERFRAMES</h1>
|
||||
<div class="cutout-wrap" style="position:absolute;inset:0;z-index:3;opacity:0">
|
||||
<video
|
||||
src="presenter.webm"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-track-index="1"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
</div>
|
||||
```
|
||||
|
||||
Two key rules:
|
||||
|
||||
1. **Wrap the cutout video in a non-timed `<div>`** and animate the wrapper's opacity, not the video element's. The framework forces opacity:1 on active clips (any element with `data-start`/`data-duration`), so animating the video's opacity directly is silently overridden. The wrapper has no `data-*` attributes, so it's owned by your CSS/GSAP.
|
||||
2. **Both videos use `data-start="0"` and `data-media-start="0"`** so the framework decodes them in sync from t=0. Late-mounting the cutout (`data-start=3.3`) introduces a seek + warm-up that lands a frame off the base mp4 — visible as one frame of misalignment at the cut.
|
||||
|
||||
Then GSAP-flip the wrapper opacity at the cut: `tl.set(cutoutWrap, { opacity: 1 }, 3.3)`.
|
||||
|
||||
## TTS → Transcribe → Captions
|
||||
|
||||
When there's no pre-recorded voiceover, generate one and transcribe it back to get word-level timestamps for captions:
|
||||
|
||||
@@ -30,6 +30,79 @@ tl.to(
|
||||
tl.to("#pip-frame", { left: 40, duration: 0.6 }, 30);
|
||||
```
|
||||
|
||||
## Text Behind Subject (transparent webm overlay)
|
||||
|
||||
Put a headline _behind_ a presenter so their silhouette occludes the text. Requires a transparent cutout produced by `npx hyperframes remove-background presenter.mp4 -o presenter.webm`.
|
||||
|
||||
Three layers, plus one critical rule:
|
||||
|
||||
```html
|
||||
<!-- z=1 base — full opaque mp4 (lobby + presenter), always visible -->
|
||||
<video
|
||||
id="cf-base"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-media-start="0"
|
||||
data-track-index="0"
|
||||
src="presenter.mp4"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
|
||||
<!-- z=2 headline — visible the whole time -->
|
||||
<h1
|
||||
id="cf-headline"
|
||||
style="position:absolute;top:50%;left:50%;
|
||||
transform:translate(-50%,-50%); z-index:2; font-size:220px; font-weight:900;
|
||||
color:#fff; text-shadow:0 6px 32px rgba(0,0,0,.55); clip-path:inset(0 0 100% 0);"
|
||||
>
|
||||
MAKE IT IN HYPERFRAMES
|
||||
</h1>
|
||||
|
||||
<!-- z=3 cutout — same source, alpha around presenter, hidden until the cut -->
|
||||
<!-- WRAPPER has the opacity, NOT the video itself (see rule below). -->
|
||||
<div class="cutout-wrap" style="position:absolute;inset:0;z-index:3;opacity:0">
|
||||
<video
|
||||
id="cf-cutout"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-media-start="0"
|
||||
data-track-index="1"
|
||||
src="presenter.webm"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
</div>
|
||||
```
|
||||
|
||||
```js
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
const CUT = 3.3;
|
||||
|
||||
// Reveal headline early
|
||||
tl.to("#cf-headline", { clipPath: "inset(0 0 0% 0)", duration: 0.6, ease: "expo.out" }, 0.25);
|
||||
|
||||
// At the cut, flip the cutout wrapper visible — the presenter's silhouette
|
||||
// punches through the headline.
|
||||
tl.set(".cutout-wrap", { opacity: 1 }, CUT);
|
||||
|
||||
// Sentinel: extend timeline to the composition's full duration so the
|
||||
// renderer doesn't bail past the last meaningful tween.
|
||||
tl.set({}, {}, 6);
|
||||
|
||||
window.__timelines["cover-flip"] = tl;
|
||||
```
|
||||
|
||||
**Why a wrapper div, not opacity on the video itself?**
|
||||
|
||||
The framework forces `opacity: 1` on any element with `data-start`/`data-duration` while it's "active" — that's how it manages clip lifecycles. A CSS `opacity: 0` on the video element is silently overwritten. Wrap the video in a div with no `data-*` attributes; the wrapper is owned by your CSS/GSAP.
|
||||
|
||||
**Why both videos at `data-start="0"`?**
|
||||
|
||||
So both decode in sync from t=0. Late-mounting the cutout (`data-start=3.3`) makes Chrome do a seek + decoder warm-up at mount, which can land a frame off the base mp4 — visible as a one-frame jitter at the cut.
|
||||
|
||||
**Color match:** `remove-background` defaults to `--quality balanced` (crf 18) which keeps the cutout's RGB nearly identical to the source mp4 — minimal edge halo or color shift when overlaid. Use `--quality best` (crf 12) for hero shots; only drop to `--quality fast` (crf 30) when the cutout sits over a _different_ background and the size matters.
|
||||
|
||||
## Title Card with Fade
|
||||
|
||||
```html
|
||||
|
||||
Reference in New Issue
Block a user