mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
docs(remove-background): document compositing patterns and pitfalls
Skill (hyperframes-cli): three-pattern table (cutout-over-different-scene vs over-its-own-source vs over-different-take) + the two non-obvious rules (wrap video in non-timed div for opacity control, both videos data-start=0 for sync). Skill (hyperframes/patterns): worked text-behind-subject example. Docs: --quality flag, compositing pitfalls section, quality preset table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
James
@@ -147,6 +147,88 @@ The avatar inherits the composition's frame rate and timeline — it plays throu
|
||||
When rendering a composition that contains a `<video>` element, the renderer reads the source via ffmpeg internally. Transparent WebMs are decoded with the alpha plane preserved.
|
||||
</Tip>
|
||||
|
||||
## Compositing patterns and pitfalls
|
||||
|
||||
The cutout webm is a **re-encoded copy** of the source mp4's RGB — the matter pipeline decodes the source to raw RGB, runs segmentation, and re-encodes to VP9 with alpha. That choice has consequences depending on what you put behind it.
|
||||
|
||||
### The three patterns
|
||||
|
||||
| Pattern | Behind the cutout | Result |
|
||||
|---|---|---|
|
||||
| **Cutout over a different scene** *(most common)* | Static image, gradient, animated bg, or unrelated footage | Clean. The cutout is the only avatar source — no double-Brandon, no edge halo. Use any `--quality`. |
|
||||
| **Cutout over its own source mp4** *(text-behind-subject, talking-head with overlays)* | The same mp4 the cutout was generated from | Two RGB sources for the same person. At default `--quality balanced` (crf 18) the doubling is barely visible; at `--quality fast` (crf 30) you'll see a slight color shift / soft edge on the silhouette. Use `--quality best` (crf 12) for hero shots. |
|
||||
| **Cutout over different footage of the same subject** | Another take of the same person | Looks like two overlapping people. Avoid — re-shoot or re-cut the source. |
|
||||
|
||||
### Text-behind-subject: the recommended layout
|
||||
|
||||
Putting a headline *behind* a presenter so their silhouette occludes the text:
|
||||
|
||||
```html
|
||||
<!-- z=1 base mp4: full lobby + presenter, plays the whole scene -->
|
||||
<video
|
||||
id="cf-base"
|
||||
data-start="0" data-duration="6" data-media-start="0" data-track-index="0"
|
||||
src="avatar.mp4"
|
||||
muted playsinline
|
||||
></video>
|
||||
|
||||
<!-- z=2 headline -->
|
||||
<h1 id="cf-headline" style="position:absolute;top:50%;left:50%;
|
||||
transform:translate(-50%,-50%); z-index:2;
|
||||
color:#fff; text-shadow:0 6px 32px rgba(0,0,0,.55);
|
||||
clip-path:inset(0 0 100% 0); font-size:220px; font-weight:900;">
|
||||
MAKE IT IN HYPERFRAMES
|
||||
</h1>
|
||||
|
||||
<!-- z=3 cutout: same source, alpha around presenter, hidden until the cut.
|
||||
The wrapper carries the opacity, NOT the <video> itself. -->
|
||||
<div class="cutout-wrap" style="position:absolute;inset:0;z-index:3;opacity:0">
|
||||
<video
|
||||
id="cf-cutout"
|
||||
data-start="0" data-duration="6" data-media-start="0" data-track-index="1"
|
||||
src="avatar.webm"
|
||||
muted playsinline
|
||||
></video>
|
||||
</div>
|
||||
```
|
||||
|
||||
```js
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
const CUT = 3.3;
|
||||
|
||||
// Reveal the headline early
|
||||
tl.to("#cf-headline", { clipPath: "inset(0 0 0% 0)", duration: 0.6, ease: "expo.out" }, 0.25);
|
||||
|
||||
// At the cut, flip the cutout wrapper visible — silhouette punches through the headline
|
||||
tl.set(".cutout-wrap", { opacity: 1 }, CUT);
|
||||
|
||||
// Sentinel: extend timeline to the composition's full duration so the renderer
|
||||
// doesn't bail past the last meaningful tween.
|
||||
tl.set({}, {}, 6);
|
||||
```
|
||||
|
||||
### Two non-obvious rules
|
||||
|
||||
**1. Wrap the cutout video in a non-timed `<div>` and animate the wrapper, not the video.**
|
||||
|
||||
The framework forces `opacity: 1` on any element with `data-start`/`data-duration` while it's "active" — that's how it controls clip visibility. CSS `opacity: 0` on the video element is silently overwritten by the framework's clip lifecycle, so an opacity tween on the video element won't do anything. Wrap the video in a `<div>` that has no `data-*` attributes; the wrapper is owned entirely by your CSS/GSAP.
|
||||
|
||||
**2. Both videos start at `data-start="0"` and decode in sync from t=0.**
|
||||
|
||||
It's tempting to "late-mount" the cutout (`data-start="3.3"` to match the cut). Don't — Chrome does a seek + decoder warm-up at mount, which can land one frame off the base mp4 at the cut moment. With both videos mounted from t=0 and the cutout's wrapper opacity-animated, both decoders advance the same way and stay frame-accurate.
|
||||
|
||||
### Quality preset and color match
|
||||
|
||||
When the cutout is overlaid on its own source mp4, the encoder's CRF directly affects how visible the doubling is at edges:
|
||||
|
||||
| `--quality` | CRF | File size (12s @ 1080p) | When to use |
|
||||
|---|---|---|---|
|
||||
| `fast` | 30 | ~2 MB | Cutout sits over an unrelated background and file size matters |
|
||||
| `balanced` *(default)* | 18 | ~6 MB | Recommended for text-behind-subject and any pattern that overlays on the source |
|
||||
| `best` | 12 | ~12 MB | Hero shots, masters, or anything you'll re-encode downstream |
|
||||
|
||||
The encoder also writes BT.709 + limited-range color metadata so Chrome's YUV→RGB pipeline matches the source mp4's. Without those tags, the cutout would render slightly differently from the underlying mp4 even at lossless quality (visible red/skin shift).
|
||||
|
||||
## What u²-net_human_seg is and isn't good for
|
||||
|
||||
The model is purpose-built for **portrait / human matting**. It excels when:
|
||||
|
||||
@@ -367,6 +367,7 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_
|
||||
|------|-------------|
|
||||
| `--output, -o` | Output path. Format inferred from extension: `.webm` (default), `.mov`, `.png` |
|
||||
| `--device` | Execution provider: `auto` (default), `cpu`, `coreml`, `cuda` |
|
||||
| `--quality` | WebM encoder preset: `fast` (crf 30, smallest), `balanced` (crf 18, default), `best` (crf 12, near-lossless). Higher quality keeps the cutout's RGB closer to the source mp4 — important when overlaying the cutout on its own source for text-behind-subject effects. Ignored for `.mov` / `.png`. |
|
||||
| `--info` | Print detected execution providers and exit (no render) |
|
||||
| `--json` | Output result as JSON |
|
||||
|
||||
|
||||
@@ -124,6 +124,58 @@ Uses `u2net_human_seg` (MIT). First run downloads ~168 MB of weights to `~/.cach
|
||||
|
||||
Chrome decodes VP9 alpha natively, so the `.webm` plugs into a composition like any other muted-autoplay video — see the `hyperframes` skill for the `<video>` track conventions.
|
||||
|
||||
### Quality presets
|
||||
|
||||
`--quality fast|balanced|best` controls only the VP9 encoder's CRF — segmentation quality is fixed.
|
||||
|
||||
| Preset | CRF | When |
|
||||
| ---------- | --- | ----------------------------------------------------- |
|
||||
| `fast` | 30 | Iterating, smaller file, looser color match |
|
||||
| `balanced` | 18 | Default. Visually identical for most uses |
|
||||
| `best` | 12 | Master / final delivery. Largest file, tightest match |
|
||||
|
||||
### Compositing patterns — pick the right one
|
||||
|
||||
The cutout webm is a **re-encoded copy** of the source mp4's RGB. That choice has consequences depending on what you put behind it:
|
||||
|
||||
| Pattern | What's behind the cutout | Result |
|
||||
| -------------------------------------------------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Cutout over a different scene** (most common) | Static image, gradient, or unrelated video | Looks great. The cutout's RGB is the only avatar source — no double-Brandon, no edge halo. This is what `remove-background` is built for. |
|
||||
| **Cutout over its own source mp4** (text-behind-subject) | Same mp4 the cutout was generated from | Two RGB sources for the same Brandon. At default `--quality balanced` (crf 18) the doubling is barely visible; at `--quality fast` (crf 30) you'll see a faint color shift / edge halo. Use `--quality best` (crf 12) for masters. |
|
||||
| **Cutout over a _different_ take of the same person** | Footage of the same subject | Will look like two separate people overlapping. Don't do this. |
|
||||
|
||||
**Text-behind-subject** (headline behind a presenter):
|
||||
|
||||
```html
|
||||
<video
|
||||
src="brandon.mp4"
|
||||
id="bg"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
<h1 id="headline" style="z-index:2; ...">MAKE IT IN HYPERFRAMES</h1>
|
||||
<div class="cutout-wrap" style="position:absolute;inset:0;z-index:3;opacity:0">
|
||||
<video
|
||||
src="brandon.webm"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-track-index="1"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
</div>
|
||||
```
|
||||
|
||||
Two key rules:
|
||||
|
||||
1. **Wrap the cutout video in a non-timed `<div>`** and animate the wrapper's opacity, not the video element's. The framework forces opacity:1 on active clips (any element with `data-start`/`data-duration`), so animating the video's opacity directly is silently overridden. The wrapper has no `data-*` attributes, so it's owned by your CSS/GSAP.
|
||||
2. **Both videos use `data-start="0"` and `data-media-start="0"`** so the framework decodes them in sync from t=0. Late-mounting the cutout (`data-start=3.3`) introduces a seek + warm-up that lands a frame off the base mp4 — visible as one frame of misalignment at the cut.
|
||||
|
||||
Then GSAP-flip the wrapper opacity at the cut: `tl.set(cutoutWrap, { opacity: 1 }, 3.3)`.
|
||||
|
||||
## TTS → Transcribe → Captions
|
||||
|
||||
When there's no pre-recorded voiceover, generate one and transcribe it back to get word-level timestamps for captions:
|
||||
|
||||
@@ -30,6 +30,79 @@ tl.to(
|
||||
tl.to("#pip-frame", { left: 40, duration: 0.6 }, 30);
|
||||
```
|
||||
|
||||
## Text Behind Subject (transparent webm overlay)
|
||||
|
||||
Put a headline _behind_ a presenter so their silhouette occludes the text. Requires a transparent cutout produced by `npx hyperframes remove-background avatar.mp4 -o avatar.webm`.
|
||||
|
||||
Three layers, plus one critical rule:
|
||||
|
||||
```html
|
||||
<!-- z=1 base — full opaque mp4 (lobby + presenter), always visible -->
|
||||
<video
|
||||
id="cf-base"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-media-start="0"
|
||||
data-track-index="0"
|
||||
src="avatar.mp4"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
|
||||
<!-- z=2 headline — visible the whole time -->
|
||||
<h1
|
||||
id="cf-headline"
|
||||
style="position:absolute;top:50%;left:50%;
|
||||
transform:translate(-50%,-50%); z-index:2; font-size:220px; font-weight:900;
|
||||
color:#fff; text-shadow:0 6px 32px rgba(0,0,0,.55); clip-path:inset(0 0 100% 0);"
|
||||
>
|
||||
MAKE IT IN HYPERFRAMES
|
||||
</h1>
|
||||
|
||||
<!-- z=3 cutout — same source, alpha around presenter, hidden until the cut -->
|
||||
<!-- WRAPPER has the opacity, NOT the video itself (see rule below). -->
|
||||
<div class="cutout-wrap" style="position:absolute;inset:0;z-index:3;opacity:0">
|
||||
<video
|
||||
id="cf-cutout"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-media-start="0"
|
||||
data-track-index="1"
|
||||
src="avatar.webm"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
</div>
|
||||
```
|
||||
|
||||
```js
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
const CUT = 3.3;
|
||||
|
||||
// Reveal headline early
|
||||
tl.to("#cf-headline", { clipPath: "inset(0 0 0% 0)", duration: 0.6, ease: "expo.out" }, 0.25);
|
||||
|
||||
// At the cut, flip the cutout wrapper visible — Brandon's silhouette
|
||||
// punches through the headline.
|
||||
tl.set(".cutout-wrap", { opacity: 1 }, CUT);
|
||||
|
||||
// Sentinel: extend timeline to the composition's full duration so the
|
||||
// renderer doesn't bail past the last meaningful tween.
|
||||
tl.set({}, {}, 6);
|
||||
|
||||
window.__timelines["cover-flip"] = tl;
|
||||
```
|
||||
|
||||
**Why a wrapper div, not opacity on the video itself?**
|
||||
|
||||
The framework forces `opacity: 1` on any element with `data-start`/`data-duration` while it's "active" — that's how it manages clip lifecycles. A CSS `opacity: 0` on the video element is silently overwritten. Wrap the video in a div with no `data-*` attributes; the wrapper is owned by your CSS/GSAP.
|
||||
|
||||
**Why both videos at `data-start="0"`?**
|
||||
|
||||
So both decode in sync from t=0. Late-mounting the cutout (`data-start=3.3`) makes Chrome do a seek + decoder warm-up at mount, which can land a frame off the base mp4 — visible as a one-frame jitter at the cut.
|
||||
|
||||
**Color match:** `remove-background` defaults to `--quality balanced` (crf 18) which keeps the cutout's RGB nearly identical to the source mp4 — minimal edge halo or color shift when overlaid. Use `--quality best` (crf 12) for hero shots; only drop to `--quality fast` (crf 30) when the cutout sits over a _different_ background and the size matters.
|
||||
|
||||
## Title Card with Fade
|
||||
|
||||
```html
|
||||
|
||||
Reference in New Issue
Block a user