ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-10-30 09:16:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(guides): media pages titled by the task people search for (#6360)

Browse Source 
* docs(guides): media pages titled by the task people search for

* docs(guides): background fixes as a visible section; list color grading as an agreed deletion
This commit is contained in:
Ular Kimsanov
2026-10-26 01:45:25 +00:00
committed by GitHub
parent e48c8a0614
commit a4e6bc3254
14 changed files with 271 additions and 542 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs.json
+5 -1
View File
@@ -178,7 +178,6 @@
"guides/avatar-presenter",
"guides/voice-and-audio",
"guides/remove-background",
"guides/color-grading",
"guides/media-effects",
"guides/export-and-share"
]
@@ -1309,6 +1308,11 @@
"source": "/prompting/motion-graphics",
"destination": "/guides/motion-graphics#prompts-that-work",
"permanent": true
},
{
"source": "/guides/color-grading",
"destination": "/guides/media-effects#fix-and-grade-the-color",
"permanent": true
}
],
"integrations": {
docs/guides/color-grading.mdx
-189
View File
@@ -1,189 +0,0 @@
---
title: "Color grade images and footage"
sidebarTitle: "Color grading"
description: "Fix exposure and color on an image or video, shape a look with wheels and curves, apply a LUT, and check the result against real measurements."
---
import { DocsVideo } from "/snippets/docs-video.jsx";
Grading changes how a photo or video **looks** — brighter, warmer, moodier, more
or less colorful. It runs on one media element at a time. The file on disk never
changes, and text, captions, SVG and ordinary HTML stay on their own layers,
untouched.
<Frame caption="The same source before and after a restrained natural-portrait grade.">
<img
src="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/color-grading-before-after-v1.png"
alt="The same presenter frame before and after a restrained natural portrait grade"
/>
</Frame>
<DocsVideo
title="Correction, presets, curves, selective colour, scopes and a LUT on real footage"
src="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/color-grading-demo-v1.mp4"
poster="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/color-grading-demo-v1.jpg"
/>
Every control moves and the picture answers. The numbers on screen are measured
from the frame, not illustrative — shadows lifted 2% to 23%, median 28% to 41%.
## Correct first, then style
Get exposure, white balance, contrast and saturation believable first. Then shape
the tonal ranges or a single color, then a preset or a LUT, then grain, vignette
and film effects last. No preset rescues blown-out highlights, shadows with
nothing left in them, or the wrong shot.
## Try it in Studio
Select an image or video — on the canvas, in Layers, or on the timeline — and
open **Grade** in the Inspector.
Start with a preset. There are thirteen, from Neutral and Clean Studio to Night Lift, and Studio renders each as a thumbnail of **your** frame rather than
someone else's sample. **Strength** dials the chosen look between nothing and
full.
{/* VISUAL NEEDED (1): Studio's preset thumbnail grid — all thirteen Grade presets on one real frame, Neutral through Night Lift. */}
Hold the compare button in the panel header to flash back to the original. On
moving footage, scrub several places first — a grade that flatters one frame can
wreck the next.
## What each control does
| Control | Use it for |
| --- | --- |
| Preset and strength | Pick a whole look, then dial it back |
| Exposure, contrast, highlights, shadows, whites, blacks | Brightness and contrast, overall and per tonal range |
| Warmth, tint, vibrance, saturation | Take out a color cast, or push color further |
| Color wheels | Tint shadows, midtones and highlights separately |
| RGB curves | Redraw the brightness ramp — whole image, or red, green and blue one at a time |
| Hue curves | Pick one hue and move only it: shift, saturate or brighten |
| HSL selections | Key a band of hue, saturation and brightness, then correct only those pixels. Up to four, in order |
| Grain and vignette | Add texture, or darken the edges to pull the eye in |
| Custom LUT | Load a `.cube` file that maps every color to another |
{/* VISUAL NEEDED (2): color wheels, a bent RGB curve, a hue curve and one HSL
selection, each beside the frame it produced. */}
## Read the scopes, not your screen
Scopes measure the picture live, so you are not guessing from a monitor that may
be too bright. Studio draws four.
- **Histogram** — how much of the frame is dark, mid, or bright. Piled against
either end means detail is already gone.
- **Waveform** — brightness across the frame left to right, so you see *which
part* is blown out.
- **RGB parade** — that waveform split into red, green and blue. One channel
riding high is your color cast, named.
- **Vectorscope** — a circle of hues: direction is which color, distance from the
centre is how saturated. Skin tones land on one known line, so faces are quick
to check.
{/* VISUAL NEEDED (3): the four scopes side by side, on the same frame before
and after a correction. */}
## Compare candidate looks in one image
Instead of flipping between options, render them all onto one reference frame:
```bash
npx hyperframes grade-compare --for frame.png --grades grades.json
```
`--grades` takes a JSON array of `{ label, grading }` entries; `--for` takes an
image, or a video sampled at its first frame. The untouched frame leads as a cell
labelled `original` unless you pass `--no-baseline`. Up to sixteen cells land
four to a row in `grade-compare.png`, or wherever `--out` points. Swap `--grades`
for `--luts looks/a.cube,looks/b.cube` to compare LUT files the same way.
Phone photos keep their displayed orientation and aspect ratio in the sheet;
the source image file stays unchanged.
{/* VISUAL NEEDED (4): a real grade-compare.png sheet — the labelled grid with
`original` in the first cell. */}
## Or describe the problem to the agent
You do not need to name controls or invent values:
```text
The interview looks too dark and slightly cold.
Keep skin natural, recover the background enough to read, and avoid a filtered look.
```
The agent can list what exists, and measure a local source before changing it:
```bash
npx hyperframes media-treatment --capabilities --json
npx hyperframes media-treatment --selector '#interview' --analyze --json
```
`--analyze` reports the source's color metadata, where its brightness sits, what
is clipping, and a bounded correction to start from. For source-sensitive prompts
and worked before-and-afters, see
[Color grading and film effects](/prompting/color-grading).
## Reuse a grade
**Copy grade to** applies the current grade to other media, in this file or
across the project — a starting point, since the same numbers rarely suit two
shots. Project-wide copy refuses a relative LUT path, because that path means
something different from another composition's folder; stay in the current file,
or use a LUT reachable by URL or data URL.
## Use a LUT only when you know what it expects
A LUT is a lookup table: a file mapping every input color to an output color.
HyperFrames reads a 3D `.cube` file up to 64 points per side and blends it in at
an intensity you set.
Nothing normalizes your footage first — HyperFrames does not identify camera
profiles or run an ACES or OCIO pipeline, the color-management systems film
finishing uses. The safe case is a creative Rec.709 LUT, built for ordinary web
and broadcast video, which is also the only color space these controls work in.
LOG footage, shot deliberately flat and grey so it holds detail for grading,
needs the transform its camera expects or comes out wrong rather than stylized.
{/* VISUAL NEEDED (5): one frame under two or three .cube LUTs, plus the same
LUT at partial and full intensity. */}
## Where the grade is stored
Studio and the CLI write the result into `data-color-grading`, in named sections:
corrections under `adjust`, grain and vignette under `details`, stylized
treatments under `effects`.
```html
<video src="./interview.mp4" data-color-grading='{"preset":"skin-soft","intensity":0.7,"adjust":{"exposure":0.15},"effects":{"bloom":0.4}}'></video>
```
The nesting is not optional. A flat object such as `{"exposure":0.15}` renders
nothing at all; `npx hyperframes lint` catches it and names the section the
control belongs in.
## What it cannot do
Grading picks pixels by value, never by position: no face tracking, region
tracking, rotoscoping or masks — an HSL selection is a color qualifier, not a
shape. To treat one part of a frame, split it into its own media layer and grade
that layer, as in the [implementation reference](/reference/color-grading). It
targets `<video>`, `<img>` and `<canvas>` elements only, so a whole scene
including HTML text cannot be graded. A `<canvas>` is graded at its drawing-buffer
size (`width`/`height`) and re-read on every seek and playback frame.
SDR images and video are the supported case; 4K works at a higher preview and
render cost. An HDR source — iPhone HDR, HLG, Dolby Vision-style — gets an SDR
preview and a banner saying so: the render may stay HDR-tagged, but this is not
true HDR grading yet. Remote media needs permissive CORS headers and can vanish
before render time, so keep media in the project.
For blur, bloom, retro, print, glitch and art treatments, see
[Media effects](/guides/media-effects).
## Related topics
- [Apply media effects](/guides/media-effects)
- [Implement a grade in HTML](/reference/color-grading)
- [Deliver an HDR render](/guides/hdr)
docs/guides/export-and-share.mdx
+2 -2
View File
@@ -1,6 +1,6 @@
---
title: "Finish and share a video"
sidebarTitle: "Finish and share"
title: "Render a video to MP4"
sidebarTitle: "Render"
description: "Review the project, run its checks, render through an agent, Studio, or the CLI, and deliver the right file or link."
---
docs/guides/media-effects.mdx
+94 -72
View File
@@ -1,16 +1,13 @@
---
title: "Apply media effects"
sidebarTitle: "Media effects"
description: "Turn an image or video into ASCII, halftone print, tape damage, CRT, painted art, and more — without touching the original file."
title: "Add effects to images and video"
sidebarTitle: "Video effects"
description: "Give a photo or video a look (ASCII, halftone, tape damage, CRT, paint) or fix and grade its color, by asking or by hand. The original file never changes."
---
import { DocsVideo } from "/snippets/docs-video.jsx";
An effect changes how a photo or video **looks**, not what it is. The file on
disk never changes. You can adjust the effect later, animate it, or take it off
entirely.
Here is the whole range on real footage:
An effect changes how a picture looks, not what it is. The file stays untouched, so you can soften the effect,
animate it or take it off later.
<DocsVideo
title="Every media effect family, applied to real sources"
@@ -18,26 +15,28 @@ Here is the whole range on real footage:
poster="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/media-effects-showcase-v2.jpg"
/>
Eighteen effects, sixteen color looks and four LUT paths — all on a single media
layer, and all still editable afterwards.
*Eighteen effects, sixteen color looks and four LUT paths, each on one media layer and all still editable.*
## What each family is for
## Ask for a look
**Essentials — Blur, Pixelate, Bloom.**
Hide something, pull focus onto something else, or make bright areas glow.
Say what it should feel like; you don't need the effect names:
**Retro and glitch — Tape Damage, Film Artifacts, Scanlines, CRT Curvature,
Channel Separation, Digital Glitch, Chroma Softening.**
Make footage look like old tape, old film, an old monitor, or a broken signal.
The damage is built to behave like the real thing — tracking wobble, color
bleed and tearing — not random colored noise.
```text
Make the product clip feel like a clean two-ink editorial print. Keep the label readable.
The interview looks too dark and slightly cold. Keep skin natural, no filtered look.
Pixelate the email address in the screen recording from 0:04 to 0:07.
```
**Print — Halftone, Two-Ink Print, Ordered Dither, Mono Screen.**
Turn the picture into printed dots or a two-color poster.
Works the same in the Studio app and with your own agent. To copy a look from a video you like, add it as a
reference and say "grade it like this".
**Art — ASCII, Engraving, Crosshatch, Kuwahara Paint.**
Redraw the picture as text characters, engraved lines, hatching or paint. ASCII
alone has eight styles, not one switch.
## The effects
- **Essentials:** Blur, Pixelate, Bloom. Hide something, pull focus, make highlights glow.
- **Retro and glitch:** Tape Damage, Film Artifacts, Scanlines, CRT Curvature, Channel Separation, Digital Glitch,
Chroma Softening. Old tape, old film, old monitors, broken signals, with real tracking wobble and color bleed.
- **Print:** Halftone, Two-Ink Print, Ordered Dither, Mono Screen.
- **Art:** ASCII (eight styles), Engraving, Crosshatch, Kuwahara Paint.
<Frame caption="One source, four finishes: original, pixelated, two-ink print, and ASCII.">
<img
@@ -46,77 +45,100 @@ alone has eight styles, not one switch.
/>
</Frame>
For grain and darkened edges, use Grain and Vignette in
[Color grading](/guides/color-grading). For a HUD, light leak or flash, use a
[Catalog overlay](/catalog).
Commit to a strong look or leave it off: a half-strength ASCII reads as a mistake. For grain and vignette, use the
color controls below; for HUDs, light leaks and flashes, use a [Catalog overlay](/catalog).
## Try one in Studio
## Fix and grade the color
Select an image or video, then open **Effects** in the Inspector. Pick one
effect, adjust it, and play the clip all the way through — some effects only
look wrong once things move.
Correct first (exposure, white balance, contrast, saturation), then style. No look rescues blown highlights or the
wrong shot.
An effect covers the whole media layer. It does not follow a face, a screen or
an object. If only part of the frame should change, crop or mask that part into
its own layer first.
<Frame caption="The same source before and after a restrained natural-portrait grade.">
<img
src="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/color-grading-before-after-v1.png"
alt="The same presenter frame before and after a restrained natural portrait grade"
/>
</Frame>
Blur, Bloom and Kuwahara Paint are the expensive ones. Several of them stacked
across large layers will slow down both preview and rendering.
<DocsVideo
title="Correction, presets, curves, selective colour, scopes and a LUT on real footage"
src="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/color-grading-demo-v1.mp4"
poster="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/color-grading-demo-v1.jpg"
/>
## Or just ask the agent
## Adjust it by hand
Describe what you want it to look like. You do not need to know the control
names:
In the [browser editor](/studio/index), select an image or video and open **Effects** or **Grade** in the Inspector.
**Grade** starts with thirteen presets, each previewed on your own frame, and a **Strength** slider. Hold the compare
button to flash back to the original, and scrub a few places in moving footage: a grade that flatters one frame can
ruin the next.
```text
Make this product clip feel like a clean two-ink editorial print.
Keep the product shape and label readable.
```
| Control | Use it for |
| --- | --- |
| Preset and strength | Pick a whole look, then dial it back |
| Exposure, contrast, highlights, shadows, whites, blacks | Brightness and contrast, overall and per tonal range |
| Warmth, tint, vibrance, saturation | Take out a color cast, or push color further |
| Color wheels | Tint shadows, midtones and highlights separately |
| RGB curves | Redraw the brightness ramp — whole image, or red, green and blue one at a time |
| Hue curves | Pick one hue and move only it: shift, saturate or brighten |
| HSL selections | Key a band of hue, saturation and brightness, then correct only those pixels. Up to four, in order |
| Grain and vignette | Add texture, or darken the edges to pull the eye in |
| Custom LUT | Load a `.cube` file that maps every color to another |
The agent can look up what is available before it changes anything:
The scopes measure the picture so you aren't judging by your screen: the **histogram** shows lost detail at either
end, the **waveform** shows where across the frame, the **RGB parade** names a color cast, and the **vectorscope**
puts skin tones on one known line.
**Copy grade to** reuses a grade on other clips, as a starting point; the same numbers rarely suit two shots.
## LUTs
A LUT (a `.cube` file, up to 64 points per side) maps every color to another, blended in at the strength you set.
Use creative Rec.709 LUTs. HyperFrames doesn't convert camera LOG footage first, so a LOG clip needs its camera's own
transform or comes out wrong.
## Limits
- An effect or grade covers the whole layer. It doesn't follow a face or an object: to treat part of the frame, put
that part on its own layer.
- It applies to images, video and canvas, not to text or HTML.
- Standard video only. HDR sources preview in SDR and the effects don't carry into true HDR.
- Blur, Bloom and Kuwahara Paint are slow; several stacked on big layers slow preview and render.
- Keep media inside the project. A file on another server needs the right CORS headers and can vanish before render.
<Accordion title="For agents and code: the commands and the stored format">
List what exists, measure a source, then apply. `--dry-run` shows the change, `--clear` removes it.
```bash
npx hyperframes media-treatment --capabilities --json
npx hyperframes media-treatment --capability twoInkPrint --json
npx hyperframes media-treatment --selector '#interview' --analyze --json
npx hyperframes media-treatment --selector '#product' --grading '{"preset":"two-ink-print"}' --apply --json
```
Then apply it:
Compare several looks on one frame (up to sixteen, the original first):
```bash
npx hyperframes media-treatment \
--selector '#product' \
--grading '{"preset":"two-ink-print"}' \
--apply --json
npx hyperframes grade-compare --for frame.png --grades grades.json
```
Add `--dry-run` to see the change without writing it, `--analyze` to measure the
source first, or `--clear` to take an effect off. This writes the same
`data-color-grading` data that Studio reads, so the two never disagree.
Phone photos keep their displayed orientation and aspect ratio in the sheet; the source image stays unchanged.
## Keeping it useful
Everything lands in `data-color-grading`, in named sections: `adjust` for corrections, `details` for grain and
vignette, `effects` for treatments. A flat object such as `{"exposure":0.15}` renders nothing, and
`npx hyperframes lint` says which section it belongs in.
Use blur or pixelation when you actually mean to hide or reveal something. If
you pick a strong look, commit to it — a half-strength ASCII just reads as a
mistake. Compare against the original, and take the effect off when it starts
competing with the story.
```html
<video src="./interview.mp4" data-color-grading='{"preset":"skin-soft","intensity":0.7,"adjust":{"exposure":0.15},"effects":{"bloom":0.4}}'></video>
```
## Two limits worth knowing
The full format is in the [HTML schema](/reference/html-schema#color-grading-and-media-effects) and the
[color grading reference](/reference/color-grading).
**HDR.** Effects run on the standard (SDR) pipeline. A native HDR layer keeps
its HDR pixels through a separate path, so these effects will not appear on it.
Convert to SDR if the effect has to survive into the final file.
**Remote files.** Keep media inside the project. A file loaded from another
server only works if that server sends the right CORS headers, and it can
disappear between now and render time.
The [Color grading guide](/guides/color-grading) covers correction, curves,
selective color and LUTs. The
[HTML schema](/reference/html-schema#color-grading-and-media-effects) has the
exact data format.
</Accordion>
## Related topics
- [Color grade images and footage](/guides/color-grading)
- [Use images and video](/guides/video-components)
- [Browse ready-made visual effects](/catalog)
- [Color, effects, 3D and generated art: the words to use](/prompting/color-grading)
- [Browse ready-made effects in the Catalog](/catalog)
docs/guides/media.mdx
+1 -1
View File
@@ -88,7 +88,7 @@ in the archived version.
<Card title="Remove a background" icon="person-rays" href="/guides/remove-background">
Isolate a person from a real image or clip.
</Card>
<Card title="Color grading" icon="palette" href="/guides/color-grading">
<Card title="Color grading" icon="palette" href="/guides/media-effects#fix-and-grade-the-color">
Correct footage or establish a deliberate look.
</Card>
<Card title="Media effects" icon="wand-magic-sparkles" href="/guides/media-effects">
docs/guides/remove-background.mdx
+50 -111
View File
@@ -1,13 +1,48 @@
---
title: "Remove a background"
description: "Turn footage of a person or a portrait image into transparent media you can layer over a HyperFrames scene."
title: "Remove a video background"
description: "Cut a person or any object out of a clip or picture so titles can pass behind them: in the Studio app with a right-click, or free on your computer with one command."
sidebarTitle: "Remove background"
---
import { DocsVideo } from "/snippets/docs-video.jsx";
Use background removal when a person needs to appear over designed text,
graphics, or another scene. HyperFrames runs the included human-segmentation
model locally; it does not upload the source or require an API key.
Cut a person out so titles can pass behind them, lift an object out of a shot, or replace what's behind. The basic
cut-out runs on your computer: no upload, no account, no cost.
<DocsVideo
title="A presenter cut out of a real room, shown against the original and magnified at the hard edges"
src="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/remove-background-demo-v3.mp4"
poster="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/remove-background-demo-v1.jpg"
/>
*A presenter cut out of a real room, with the hard edges (hair, a shoulder a plant overlapped) magnified.*
## In the Studio app
Right-click a video or picture in the preview:
- **Cut out object here** lifts out whatever you clicked, a person or a thing. On a video it follows that object
through the clip. Shift-click adds to the selection; on a picture, **B** brushes the edges.
- **Remove background** keeps the person and drops the rest. It's free and runs on your Mac, made for people on plain
backgrounds. With a HeyGen API key you can choose a cloud cut through HeyGen instead, paid from your API balance.
Or just ask: "put the title behind her", "remove the cup from the table", "put her on a rainy street at night".
## On the command line
```bash
npx hyperframes remove-background subject.mp4 -o subject.webm
```
The first run downloads the model; it needs FFmpeg. Pick the output by where it goes: `.webm` for HyperFrames and the
web, `.mov` (ProRes 4444) for Premiere, Resolve or Final Cut, `.png` for a single photo.
For text behind the person, make the cut-out and a background plate with a person-shaped hole in one pass, then
stack: plate, your text or graphic, cut-out on top.
```bash
npx hyperframes remove-background subject.mp4 -o subject.webm --background-output plate.webm
```
<Frame caption="The same real frame before and after the local model. The softer hair and hand edges are the model’s actual limit on this source.">
<img
@@ -16,110 +51,13 @@ model locally; it does not upload the source or require an API key.
/>
</Frame>
## When it works, and when it won't
<DocsVideo
title="A presenter cut out of a real room, shown against the original and magnified at the hard edges"
src="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/remove-background-demo-v3.mp4"
poster="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/remove-background-demo-v1.jpg"
/>
The free model is made for people: a clear subject, steady framing, some contrast with the background. Expect trouble
with a held object (a dark mic or phone can vanish), fine hair over a busy background, and fast motion. For objects,
use **Cut out object here** in the app instead.
Every frame is real CLI output. The magnified panels sit on the hair edge and on
a shoulder a plant used to overlap, because those are where matting usually fails.
## Create the transparent file
Make sure FFmpeg is installed, then run:
```bash
npx hyperframes remove-background subject.mp4 -o subject.webm
```
The first run downloads the model. Later runs reuse it.
Choose the output by destination:
| Output | Use it for |
| ------------------- | ---------------------------------------------------------------- |
| Transparent `.webm` | A subject layered inside a web or HyperFrames composition |
| Transparent `.mov` | A ProRes 4444 round trip through Premiere, Resolve, or Final Cut |
| Transparent `.png` | A single portrait image |
```bash
npx hyperframes remove-background subject.mp4 -o subject.mov
npx hyperframes remove-background portrait.jpg -o portrait.png
```
## Use the cutout
Import the transparent file into the project and place it above the
background, text, or graphic it should cover. Then check the complete clip,
especially hair, hands, fast movement, and the first and last frames.
If you animate the subject, animate a wrapper around the video. This keeps the
transparent media and the composition’s timing behavior together.
## Compositing patterns and pitfalls
Keep the media element stable and animate a wrapper when the cutout must move,
resize, or reveal. This avoids mixing media playback with the transform that
controls the composition.
When a cutout and its original background must remain frame-aligned, give both
media elements the same `data-start` and playback offset. Hide or reveal them
through wrappers instead of mounting one source later. Then scrub the cut point
and check the rendered frame, not only normal playback.
## Put something behind the subject
For text-behind-subject or another three-layer effect, you can create both the
foreground cutout and an inverse-alpha background plate in one pass:
```bash
npx hyperframes remove-background subject.mp4 \
-o subject.webm \
--background-output plate.webm
```
The plate keeps the original surroundings and leaves a transparent
person-shaped hole. It is not an inpainted empty background. Place an opaque
graphic or scene beneath the hole, then place `subject.webm` above it.
`hyperframes check` judges text behind the cutout by the cutout's own alpha at
each sample time. A word partly behind the subject is reported as info with the
share it hides; a word 80% hidden, or two neighbouring words each half hidden,
is an error. You do not need `data-layout-allow-occlusion` for this layering.
## Know when it will work well
The included model is designed for people. It works best with a clear human
subject, stable framing, and reasonable contrast from the background.
Use another segmentation or masking tool when:
- the subject is a product, animal, or other object;
- the person holds something, especially a dark microphone, phone, card, or
instrument: the model can drop all or part of it;
- fine hair crosses a busy background;
- frame-to-frame edge stability must meet high-end VFX standards;
- you need the original background reconstructed after removing the person.
Background removal is preprocessing. Run it once, keep the transparent result
in the project, and reuse it.
## Balance quality and speed
`--device auto` chooses the best available local provider. Inspect the detected
provider with:
```bash
npx hyperframes remove-background --info
```
For WebM output, `--quality balanced` is the default. Use `best` when the
cutout sits directly over its original source and edge color must match; use
`fast` for a smaller file over an unrelated background.
## Common problems
## Fixes, quality and speed
| Problem | What to check |
| --------------------------------- | ----------------------------------------------------------------------------------------------- |
@@ -130,11 +68,12 @@ cutout sits directly over its original source and edge color must match; use
| The “background plate” has a hole | Expected: it is inverse alpha, not an inpainted clean plate. |
| Part of a held object is missing | The model matches people, not what they hold. Use another matting tool for that shot. |
The [CLI reference](/packages/cli#remove-background) lists every flag and
supported input.
`--quality best` for a cut-out over its own original footage, `fast` for a smaller file over something unrelated;
`--info` shows whether a fast local processor was found. To move the cut-out, animate a wrapper around the video,
not the video itself. Every flag is in the [CLI reference](/packages/cli#remove-background).
## Related topics
- [Add an AI avatar presenter](/guides/avatar-presenter)
- [Use images and video](/guides/video-components)
- [Add an avatar presenter](/guides/avatar-presenter)
- [Apply media effects without changing the source file](/guides/media-effects)
- [Apply media effects](/guides/media-effects)
docs/guides/video-components.mdx
+63 -61
View File
@@ -1,14 +1,13 @@
---
title: "Use images and video"
title: "Add images and video clips"
sidebarTitle: "Images and video"
description: "Import, place, replace, trim, and review images and footage in a HyperFrames project."
description: "Put a picture or a clip into a HyperFrames video, then place, trim, crop or swap it: ask the agent, drag it in, or write two attributes."
---
import { DocsVideo } from "/snippets/docs-video.jsx";
Images and footage should enter the project once, remain easy to identify, and
behave the same in preview and the final render.
Put a picture or a clip into your video three ways: ask the agent, drag it into the editor, or write it in the HTML.
All three change the same file, so you can mix them.
<DocsVideo
title="The five basic moves: import, place, crop, trim, and swap the source"
@@ -16,67 +15,24 @@ behave the same in preview and the final render.
poster="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/media-basics-demo-v1.jpg"
/>
Each step adds one attribute and shows the result. The last swaps the source
file; the crop, timing and placement around it stay exactly as they were.
*Import, place, crop, trim, then swap the file: the crop, timing and placement stay as they were.*
## Import a file
## Ask the agent
In Studio:
1. Open **Assets**.
2. Choose **Import media**, or drop the file into the panel.
3. Preview it.
4. Drag it to the timeline.
Studio accepts common image formats, SVG, MP4, WebM, and MOV. Keep the source
inside the project before the final render.
## Place it deliberately
- Set the start and duration by watching the surrounding edit.
- Choose a crop that preserves the important content.
- Keep source audio only when it belongs in the mix.
- Avoid stretching low-resolution media beyond what it can support.
- Use a wrapper when animating the size or position of video; this keeps frame capture reliable.
For a source-level change, ask the agent in terms of the visible result:
Name the file and say what should happen, in terms of what you see:
```text
Use assets/product-tour.mp4 from 00:04 to 00:10.
Crop around the editor canvas, keep the source muted, and place it beneath
the existing caption layer.
Use assets/product-tour.mp4 from 00:04 to 00:10, muted.
Crop around the editor canvas and place it under the captions.
```
## Change or replace the source
The agent copies the file into the project if it isn't there yet. In the Studio app, add the file to your message
with **+** or drop it into the chat.
Studio’s Media section shows the current project path and controls Fit,
Position, playback, and audio. Assets imports files and adds new layers. It does
not currently provide a one-click source replacement in that Inspector.
## Drag it in
To keep the existing layer, ask the agent to change its `src`, or edit the
element in Source. Then recheck the crop, clip length, transition, and any
motion tied to the old dimensions.
Replacing a file should not require rebuilding the whole scene. If the new
content also changes the story or several scenes, ask the agent to make the
broader revision.
## Avoid fragile sources
Temporary signed URLs, private browser sessions, and render-time network
requests can disappear or fail in another environment. Copy required media into
the project, keep filenames stable, and run `npx hyperframes check` before
rendering.
Continue to [Use Assets and Catalog](/studio/assets-and-blocks) for the Studio
controls, or [Media effects](/guides/media-effects) when the source needs a
non-destructive visual treatment.
## The same edits in Studio
Everything above is done by editing attributes by hand. Studio writes the
same attributes when you drag, trim or reframe — it is the other way into
one file, not a second copy of the project.
In the browser editor (`npx hyperframes preview`), open **Assets**, choose **Import media** or drop the file into the
panel, then drag it onto the timeline. Images, SVG, MP4, WebM and MOV work.
<Frame caption="Current Studio, changing the real project: inspect the source, change Fit and Position, crop on the canvas, then import another local asset.">
<video
@@ -90,9 +46,55 @@ one file, not a second copy of the project.
/>
</Frame>
## Write it in the HTML
A picture on screen for 3 seconds, and a muted clip that plays 6 seconds of its source starting at second 4:
```html
<img class="clip" src="assets/logo.svg"
data-start="0" data-duration="3" data-track-index="1" />
<video class="clip" src="assets/product-tour.mp4" muted playsinline
data-start="3" data-duration="6" data-media-start="4" data-track-index="0"></video>
```
`class="clip"` lets the timeline show and hide it; `data-start` and `data-duration` say when; `data-media-start`
trims the start of the source. Keep a clip's sound with `data-has-audio="true"` and set its level with
`data-volume`. To move or scale a video, animate a wrapper around it, not the video itself. Every attribute is in
the [HTML schema](/reference/html-schema).
## Swap the file
Ask the agent to use the new file, or change `src`. Then check the crop, the length and anything that moved with
the old picture. If the new file changes the story, ask for that bigger change instead.
## Keep it working everywhere
Copy every file into the project and give it a clear name (`pricing-page-scroll.mp4`). Links that expire or need a
login break the render on another computer. `npx hyperframes check` finds missing and late-loading files.
Before you export, watch once and check the first and last frame of each clip, the crop at the final size, and
that nothing looks stretched.
## More media tasks
<CardGroup cols={2}>
<Card title="Avatar presenter" icon="user-video" href="/guides/avatar-presenter">
Generate or reuse a presenter clip and compose around it.
</Card>
<Card title="Voice, music, and sound" icon="waveform" href="/guides/voice-and-audio">
Build an understandable audio mix.
</Card>
<Card title="Remove a background" icon="person-rays" href="/guides/remove-background">
Isolate a person from a real image or clip.
</Card>
<Card title="Effects and color grading" icon="wand-magic-sparkles" href="/guides/media-effects">
Apply a visual treatment, correct footage, or set a deliberate look.
</Card>
</CardGroup>
## Related topics
- [Use Assets and Catalog in Studio](/studio/assets-and-blocks)
- [Apply a media effect](/guides/media-effects)
- [Remove a human background](/guides/remove-background)
- [Use Assets and Catalog in the browser editor](/studio/assets-and-blocks)
- [Add effects to images and video](/guides/media-effects)
- [Remove a video background](/guides/remove-background)
docs/guides/voice-and-audio.mdx
+46 -99
View File
@@ -1,22 +1,13 @@
---
title: "Use voice, music, sound, and captions"
sidebarTitle: "Voice, sound, and captions"
description: "Build an understandable audio mix, transcribe real speech, and turn it into readable captions."
title: "Add voiceover, music, sound effects and captions"
sidebarTitle: "Voice, music and captions"
description: "Ask for narration, music, sound effects and captions in plain words: what's free, what HeyGen adds, and how to check the mix."
---
import { DocsVideo } from "/snippets/docs-video.jsx";
Give each audio layer one job. Add another layer only when it helps the viewer
understand or feel the video.
| Layer | Main job |
| ------------- | --------------------------------------------------- |
| Voiceover | Carry the explanation or story |
| Source audio | Preserve speech or sound from existing footage |
| Music | Shape pace, tone, and structure |
| Sound effects | Reinforce a specific action or transition |
| Captions | Make spoken words readable and searchable on screen |
Ask for sound the way you'd brief an editor. The agent finds or makes each piece, places it on the timeline and
mixes it, with the music dipping under the voice.
<DocsVideo
title="Narration, music ducking under it, and effects landing on the beat"
@@ -24,76 +15,52 @@ understand or feel the video.
poster="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/voice-audio-demo-v2.jpg"
/>
Turn the sound on for this one — the audio is the subject. You can watch the
music duck as narration arrives and see each effect land against the waveform.
*Sound on: narration, the music ducking under it, and effects landing with the action.*
## Create a voiceover
## Ask for it
Give the agent the approved wording and useful voice direction:
In the Studio app or your own agent, in one message or one at a time:
```text
Generate the approved SCRIPT.md as a warm, direct voiceover.
Natural pace, no announcer energy. Keep the product-name pronunciation exact.
Add a warm, unhurried narration of this script: [paste script]
Put a calm piano bed under it, quieter while she speaks.
A soft whoosh on each scene change, nothing else.
Captions: two lines at most, key words in the brand color.
```
The project can keep the script, audio, and word-level transcript separately.
That lets you regenerate the voice without discarding the timing and caption
work around it.
Quote the script when the words are approved; otherwise give the points and let the agent write it. Name what you
don't want, too: "no music" and "no sound effects" are followed exactly.
Listen before building the final edit. Fix wording, pronunciation, or delivery
in the script or voice direction rather than cutting around a bad read.
## Free, or HeyGen quality
## Transcribe recorded speech
| | Free, on your computer | With HeyGen |
| --- | --- | --- |
| **Voice** | Kokoro voices, in several languages | HeyGen voices and your own cloned voice |
| **Music** | Tracks you add, or local generation if you install it | A 10,000-track catalog |
| **Sound effects** | A bundled set, plus Openverse | The same |
| **Captions** | Local transcription with your own agent (Parakeet or Whisper) | The Studio app transcribes through HeyGen |
Transcription creates timed words from an audio or video file. Start with the
automatic engine: it uses Parakeet when installed and falls back to Whisper.
The Studio app uses HeyGen when you add a HeyGen API key (Home says **Add a HeyGen API key to make voiceovers, avatars
and clips**), and a local voice when you don't. With your own agent, sign in to the HeyGen CLI or set an API key. The
words to use for each are on [Voice, music and sound](/prompting/media-and-audio).
```bash
npx hyperframes transcribe interview.mp4
```
## Use your own recording
Pass the spoken language when you know it. This filters non-target speech and
lets the Whisper fallback choose a multilingual model when needed.
Add the file and say what it is ("this is the narration", "keep the interview audio"). To get timed words from it
yourself, transcription runs locally and costs nothing:
```bash
npx hyperframes transcribe interview.mp4 --language es
```
For difficult English audio, force the larger English Whisper model. For an
unknown language, use multilingual `large-v3` and let Whisper detect it.
Pass `--language` when the speech isn't English. It also imports an existing `.srt` or `.vtt`. For a talking-head clip that needs
captions or a tighter cut, see [Captions and recuts](/guides/captions-and-recuts).
```bash
npx hyperframes transcribe interview.mp4 --engine whisper --model medium.en
npx hyperframes transcribe interview.mp4 --engine whisper --model large-v3
```
## Get captions right
Larger models take longer and do not remove the need to read the result.
The command can also import an existing `.srt`, `.vtt`, or supported transcript
JSON file:
```bash
npx hyperframes transcribe subtitles.srt
```
## Turn the transcript into captions
1. Read the complete transcript.
2. Correct names, product terms, numbers, punctuation, and obvious recognition
errors.
3. Group words into short phrases that break on meaning.
4. Open [Captions in Studio](/studio/captions) to inspect the rhythm and adjust
placement, scale, or rotation. Make durable wording, timing, style, and
animation changes in the caption source or ask the agent to update it.
5. Watch once with sound for synchronization and once muted for readability.
Captions should follow what was said. They should not cover a face, product
control, or another important visual. Use emphasis on the few words that carry
the point instead of making every word compete.
For an existing talking-head clip, use [Captions or
recuts](/guides/captions-and-recuts) to choose between plain captions, designed
graphic overlays, and an actual change to the spoken edit.
Read the transcript before it goes on screen: names, product terms and numbers are where recognition slips. Keep
lines short and broken on meaning, keep them off faces and buttons, and emphasize the two or three words that carry
the point. To adjust them by hand, open [Captions](/studio/captions) in the browser editor.
## Repeat music inside a slot
@@ -109,43 +76,23 @@ speed and rate automation apply to the repeated audio. Looping videos with
`data-has-audio="true"` repeat their sound too, including any silence before
the next cycle.
## Mix for understanding
## Check the mix
- Keep voice clear above music.
- Duck music under important speech instead of reducing the whole track
equally. A [voiceover carve](/studio/voiceover-carve)
does this properly: it takes only the bands the voice occupies out of the
music, so the bed keeps its low end and its top instead of going limp for the
whole voiceover.
- Use sound effects for meaningful events, not every movement.
- Avoid cutting words, breaths, or reverb tails accidentally at scene
boundaries.
- Let a deliberate silence remain silent.
- Listen on ordinary headphones and laptop speakers.
Watch once with sound and once muted, then listen once without watching.
When narration drives the project, time visual changes to its real transcript
rather than estimated scene lengths. When music drives the project, cut to its analyzed structure (see
[music to video](/guides/music-to-video)).
- The voice stays clear through the loudest part of the music.
- Each effect lands with the thing it belongs to.
- Nothing cuts off a word or a breath at a scene change.
- The music ends on purpose, not at the end of the file.
Music can start after narration begins. When the media-use skill creates a
ducking lane for that clip, its opening gain matches the speech at that point
in the composition. Starting during an attack or release continues that ramp
from the correct level; completed speech does not create a new dip.
To dip the music only where the voice sits instead of turning it all down, use a
[voiceover carve](/studio/voiceover-carve).
## Review the complete sound pass
- The correct voice and source recordings are used.
- Names, claims, and captions are accurate.
- Speech stays clear through the loudest musical section.
- Effects arrive with the action they support.
- The opening does not surprise the listener with excessive volume.
- Music and ambience finish intentionally instead of stopping at the file edge.
Render a review file and listen without watching once. Audio problems are often
easier to notice when the visuals are not competing for attention.
Music can start after narration begins: the dip then starts at the right level for that moment, and speech that has
already ended doesn't make a new dip.
## Related topics
- [Add captions or repackage talking-head footage](/guides/captions-and-recuts)
- [Edit captions in Studio](/studio/captions)
- [Finish, render, and share](/guides/export-and-share)
- [Add captions to a video with AI](/guides/captions-and-recuts)
- [Make a video to music](/guides/music-to-video)
- [Voice, music and sound: the words to use](/prompting/media-and-audio)
docs/packages/cli.mdx
+1 -1
View File
@@ -607,7 +607,7 @@ whether it reused a file, never your intent text or paths — and honors the sam
opt-outs as [`telemetry`](#telemetry).
See [Voice, sound, and captions](/guides/voice-and-audio) and
[Color grading](/guides/color-grading) for the workflows around these assets, and
[Color grading](/guides/media-effects#fix-and-grade-the-color) for the workflows around these assets, and
[Authentication](/guides/authentication) for HeyGen credentials.
### `beats`
docs/prompting/color-grading.mdx
+2 -2
View File
@@ -7,7 +7,7 @@ import { DocsVideo } from "/snippets/docs-video.jsx";
Every chapter so far has built the image. This one changes what the image appears to have been *recorded on*. A grade is not decoration applied at the end; it is a claim about the medium — that this came off tape, or a press, or a tube, or a camera with a particular stock in it. Viewers read that claim instantly and they read it whether or not you meant it.
Grading attaches to a single `<img>` or `<video>` through a `data-color-grading` payload. It never touches your text, your cards, or your captions — only the pixels of the media element it is on. The full contract lives in the [Color Grading guide](/guides/color-grading); this chapter is about what to *ask for*, and about the three ways these prompts go wrong.
Grading attaches to a single `<img>` or `<video>` through a `data-color-grading` payload. It never touches your text, your cards, or your captions — only the pixels of the media element it is on. The full contract lives in the [Color Grading guide](/guides/media-effects#fix-and-grade-the-color); this chapter is about what to *ask for*, and about the three ways these prompts go wrong.
## The source has to have something to lose
@@ -206,7 +206,7 @@ Two caveats worth knowing before you promise a client a ramp:
## Next steps
- [Color Grading guide](/guides/color-grading) — the full contract, every key and bound
- [Color Grading guide](/guides/media-effects#fix-and-grade-the-color) — the full contract, every key and bound
- [Catalog: overlays and effects](/catalog) — HUD, flash, light leak, freeze-frame dressing
- [VFX and liquid glass](/prompting/vfx-and-liquid-glass) — the canvas-pipeline end of the catalog
docs/prompting/media-and-audio.mdx
+1 -1
View File
@@ -163,7 +163,7 @@ for you. Naming the layout you want still helps.
<Note>
Two mechanics the skills handle automatically, from the [Remove
background guide](/guides/remove-background#compositing-patterns-and-pitfalls):
background guide](/guides/remove-background#on-the-command-line):
- A cutout that reveals into view goes inside a non-timed `<div>`, and the
*wrapper* is what gets animated. HyperFrames owns clip visibility, so
docs/reference/color-grading.mdx
+1 -1
View File
@@ -7,7 +7,7 @@ description: "Persist, automate, animate, and spatially isolate media color grad
Use this reference when Studio controls are not enough: you need to write a
grade into HTML, apply it from automation, animate a supported property, or
limit an effect to a composited region. For visual correction and everyday
Studio use, start with [Color grading](/guides/color-grading).
Studio use, start with [Color grading](/guides/media-effects#fix-and-grade-the-color).
## Persist a grade
docs/reference/html-schema.mdx
+1 -1
View File
@@ -174,7 +174,7 @@ machine-readable surface instead of hard-coding an old effect list.
The grading pipeline applies to real `<img>` and `<video>` elements. It does
not grade a complete HTML scene. Use Studio or
[`media-treatment`](/packages/cli#media-treatment) for normal authoring; see
[Color Grading](/guides/color-grading) and [Media Effects](/guides/media-effects)
[Color Grading](/guides/media-effects#fix-and-grade-the-color) and [Media Effects](/guides/media-effects)
for the workflow and current SDR/HDR boundary.
## Relative timing
scripts/check-no-main-deletions.mjs
+4
View File
@@ -741,6 +741,10 @@ const FALLOW_PRUNE_REASON =
"fallow 3 reuses one audit snapshot per checkout and the hook prunes with `fallow audit-cache prune`, replacing this script";
export const ALLOWED_DELETIONS = new Map([
[
"docs/guides/color-grading.mdx",
"color grading is now the 'Fix and grade the color' section of the media effects guide; the old URL 301s there (docs.json)",
],
[
"docs/public/images/studio-app/export.jpg",
"the Studio app docs show Export as a recorded demo with its own poster; nothing links this still any more",