mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
WaterrrForever
2efbfd4758
* docs(prompting): correct workflow one-liners against skill contracts general-video leads with its positive identity and companion mode; faceless-explainer keys on invented visuals instead of TTS; talking-head-recut uses the 'graphic overlays' trigger term; motion-graphics gains its input side and overlay output; music-to-video stops implying images are required. * docs(prompting): make vocabulary video grids readable Replace the 4-5 column table hack with a 3-column CSS grid, switch demo clips to autoplay muted loops (no black poster frame, no player chrome over tiny videos), and align cells at 16:9. * docs(prompting): document the opening interview and run-shape questions The guide taught prompt shapes but never prepared readers for the conversation that follows: the intent interview, the two run-shape questions (storyboard, automation vs companion), the just-build-it skip, and BRIEF.md as the resumable artifact. Add that section to the overview, a disambiguation note on the storyboards page, and free up 'companion' as a reserved term in media-and-audio. * docs(guides): make BRIEF.md the pipeline's Step 3 artifact Step 3 (Strategy & Messaging) listed no output while describing exactly what BRIEF.md now captures. Name the artifact in the step table, project tree, step body, gate, and iterating list, and fix SCRIPT.md's step label in the tree (Step 4, not 3). * docs(quickstart): realign the setup surface with the skills catalog The quickstart drifted from docs/guides/skills.mdx, CLAUDE.md, and the prompting overview — it had never been updated when those surfaces were: - `--full-depth` on both install commands, with the reason inline. Without it `skills add` fetches the skills.sh registry blob, which lags `main` by hours, so a reader following the quickstart installs stale skills. - `check` in the `/hyperframes-cli` row, and a validate step in the manual dev loop, which went preview → render with no gate at all. The prompting overview calls `check` "the step people skip and regret" and states both `lint` and `check` must pass before rendering. - `/hyperframes-keyframes` in the core-skills table (8 rows → 9). - `/figma` in the optional-workflow list (10 → 11). * docs(skills): close the catalog drift class and complete the music-to-video input Follow-up on the two review nits from #2872. `/music-to-video`'s SKILL.md names three inputs — an audio file, a video to pull audio from, or a track generated from a mood brief. Every compressed copy of that description carried only the first two, and the third is the one that makes "a complete video needs zero assets" true. Fixed on all eight surfaces that state it, so no surface is now more correct than its siblings: the prompting overview and quickstart setup tables, docs/guides/skills.mdx, the README catalog, root CLAUDE.md + AGENTS.md, both CLI project templates, and the router's own routes/music-to-video.md Input line (whose Interview must-haves already listed all three). The drift was structural, not accidental: the sync set declared in docs/guides/skills.mdx and in CLAUDE.md's "Skill catalog maintenance" named four surfaces and never the two setup tables, so those two were free to rot while the declared four stayed correct. Both declarations now name them, and both say the set applies to a *changed contract* — a reworded description — not only to an added or renamed skill. skills-manifest.json regenerated for the touched route file. * docs(claude): point the routing-surface rule at routes/, not the moved stubs Item 3 of "Skill catalog maintenance" still sent readers to `references/workflow-catalog.md` for a workflow's input/output/trigger contract and `references/route-briefs.md` for its interview entry. Both are now "moved" stubs — the contract and the interview entry live together in `references/routes/<workflow>.md`, one read per candidate route. Same failure class the previous commit fixed at item 1: a maintenance rule outliving the layout it describes. Swept the tree for other pointers at the two stubs; there are none, so this closes it rather than fixing one instance.
143 lines
12 KiB
Plaintext
143 lines
12 KiB
Plaintext
---
|
||
title: Media and audio
|
||
description: "Ask for the voiceover, music, sound, captions, cutouts, and assets a composition needs — with the precise, unambiguous phrasing the media pipeline acts on."
|
||
---
|
||
|
||
By now your video moves and reads right; this level gives it a voice. HyperFrames owns media *playback*; a sibling media pipeline resolves everything else — voice, music, sound effects, images, icons, logos, captions, and background removal. You reach all of it by describing what the composition needs, and the agent resolves each need to a frozen local file. The craft here is precision: vague media asks ("add some music," "no sound") are the ones that come back wrong, because the pipeline does exactly what the words say.
|
||
|
||
## Voiceover (TTS)
|
||
|
||
Text-to-speech runs locally through Kokoro — no API key needed — with a HeyGen TTS upsell behind it. Describe the content and the agent picks a fitting voice, or name the voice, tone, and speed directly:
|
||
|
||
> Generate narration for this script with a professional female voice.
|
||
|
||
> Add TTS voiceover, British male voice, at 1.1× speed.
|
||
|
||
The [Vocabulary](/prompting/vocabulary#text-to-speech-voices) table maps content types to Kokoro voices (for example `af_heart` / `af_nova` for a product demo, `am_adam` / `bf_emma` for a tutorial, `af_sky` / `am_michael` for marketing). Name one directly if you already know it; otherwise describe the read and let the agent choose.
|
||
|
||
- ❌ `add a voice`
|
||
- ✅ `warm, unhurried female narration of the quoted script` — tone and pace are what actually change the delivery
|
||
|
||
## Background music
|
||
|
||
Music resolves from a large catalog by mood, and it should almost always sit *under* the narration, not compete with it. Give the mood **and** a loudness target — the pipeline can duck and normalize to a level, so an explicit target lands a mix instead of a guess:
|
||
|
||
> Add subtle electronic BGM, kept under −18 dB so it stays beneath the voiceover.
|
||
|
||
> Upbeat tech-launch music bed at a low level, ducking under narration.
|
||
|
||
- ❌ `add background music` — you'll get a full-volume track fighting the VO
|
||
- ✅ `subtle background music, ducked ~12 dB under the voice` — a mix instruction the pipeline can execute
|
||
|
||
<Tip>
|
||
A stated loudness target ("under −18 dB," "ducked under the voice") is the difference between music that supports the piece and music that buries it. When there's narration, always say the bed goes under it.
|
||
</Tip>
|
||
|
||
## Sound effects
|
||
|
||
SFX resolve from a small bundled library plus the catalog. Cue them to specific moments — a transition, a stamp-in, an impact — rather than sprinkling them:
|
||
|
||
> Add a whoosh on each of the three scene transitions.
|
||
|
||
> Put a soft click on the button press at 0:04.
|
||
|
||
## Pace reveals to the narration
|
||
|
||
Once a video has a voice, the voice is the clock — the next instruction that matters is telling the agent that on-screen elements land **on their spoken cues** — the stat appears as the narrator says it, not at some independent time the builder eyeballed. Without this, narration and visuals drift into two parallel tracks that happen to share a file:
|
||
|
||
> VO-paced reveals: each scene's elements land on their spoken cues; secondary elements keep resolving while the narrator is mid-thought; the scene is complete just as the narration moves on.
|
||
|
||
The capstone film applies exactly this rule to every region — its Direction block reads:
|
||
|
||
> VO-paced reveals: each region's elements land on their spoken cues as the camera arrives; secondary elements keep resolving while the camera is present; the region is complete just as the camera accelerates away.
|
||
|
||
Two practical notes: the agent gets word timings for free (the narration is transcribed with per-word timestamps, the same machinery behind [captions](#captions-and-transcription)), so "on its spoken cue" is a real, executable instruction — and the inverse rule matters just as much: the narration never waits for the visuals. Pace the camera and reveals to the voice, not the voice to the animation.
|
||
|
||
## Captions and transcription
|
||
|
||
Captions come from word-level timestamps. When you generate a voiceover, the timing comes with it; for existing footage, transcription produces the timing (Parakeet by default, with a whisper.cpp fallback). Scaffolding a project from a source video can generate captions from its audio directly.
|
||
|
||
> Transcribe the narration and add karaoke-style captions synced to it.
|
||
|
||
> Generate captions from `assets/interview.mp4` and style them hype, scale-pop.
|
||
|
||
Caption *look* is its own vocabulary (tone, size, per-word emphasis) — see [Captions catalog](/prompting/captions-catalog) for the styles. This page is about producing the timed text; that page is about styling it.
|
||
|
||
## Background removal (transparent cutouts)
|
||
|
||
The `remove-background` command mattes a subject out of a video or image locally and hands you a transparent WebM you can drop into any scene as a `<video>`:
|
||
|
||
> Remove the background from `assets/presenter.mp4` and float the subject over the scene.
|
||
|
||
One caveat is load-bearing: the built-in model is **purpose-built for people** — head-and-shoulders or full-body, reasonably stable framing, a background that contrasts with the subject. It returns a mostly-empty mask on **non-human subjects** (products, animals, objects). If you need to cut out a product, say so — the agent should route to a different tool rather than run the person model and get nothing.
|
||
|
||
- ❌ `remove the background from this product shot` with the built-in command — the human-matting model can't see it
|
||
- ✅ `matte the presenter out of assets/talk.mp4` (person) — or, for a product, flag that it's a non-human subject so a different matter is used
|
||
|
||
The [Remove background guide](/guides/remove-background) covers the person-only caveat, the two-layer plate for text-behind-subject, and alternatives for objects and hair-fine mattes.
|
||
|
||
## Video-in-video and picture-in-picture
|
||
|
||
Layering footage — a talking head over a scene, a subject in front of a headline, PiP inset — is a compositing prompt. Two grounded rules keep it frame-accurate, and the agent applies them for you, but naming the layout you want helps:
|
||
|
||
> Put the transparent presenter cutout in the bottom-right, over the chart scene.
|
||
|
||
> Layer the headline *behind* the presenter so their silhouette occludes the text.
|
||
|
||
<Note>
|
||
Two mechanics the workflow skills handle automatically (from the [Remove background guide](/guides/remove-background#compositing-patterns-and-pitfalls)): a cutout that reveals into view is wrapped in a non-timed `<div>` and the *wrapper* is animated (the framework forces `opacity: 1` on timed clips, so animating the video directly does nothing); and both the base video and the cutout mount at `data-start="0"` so their decoders stay in sync at the cut. You rarely need to say this — but it's why "late-mounting" a PiP clip can land a frame off.
|
||
</Note>
|
||
|
||
## Bring any footage
|
||
|
||
You don't need to pre-convert supplied footage before naming it in a prompt. If a clip's codec doesn't play back cleanly in a browser — HEVC (H.265) is the common case, straight off an iPhone or a screen recorder — the framework probes the asset and builds a bounded H.264 proxy automatically, cached under `.transcode-cache/`. `preview`, `play`, Studio, and published player pages use the proxy for playback; a render always decodes the original file, so nothing about final quality or color is touched. `hyperframes lint` also flags the asset at info level (`hevc_preview_codec`) so you know a proxy is in play, and it's optional — `--no-proxy` per command, or `media.autoProxy: false` in `hyperframes.json` project-wide. The same mechanism covers alpha-channel sources too (ProRes 4444, alpha WebM proxy to VP9+Opus WebM instead of being refused), so a transparent cutout in a hostile codec isn't a blocker either.
|
||
|
||
None of this changes how you phrase the ask: name the footage by path like any other supplied asset, and describe the composition you want built from it.
|
||
|
||
> Build a short picture-in-picture piece from `source-hevc.mp4` — inset it bottom-right over a full-bleed background scene, with a soft rounded border.
|
||
|
||
<video controls muted loop playsinline preload="metadata" src="https://static.heygen.ai/hyperframes-oss/docs/images/prompting/proxy-footage.mp4#t=0.1" style={{ borderRadius: "0.5rem", marginTop: "0.75rem" }}></video>
|
||
*Rendered from the prompt above, unedited — the source clip is a plain H.265/HEVC file; render decoded it directly via FFmpeg, while preview would have used the automatic H.264 proxy.*
|
||
|
||
See the [Rendering guide](/guides/rendering#input-video-codecs) for the mechanics — proxy generation, caching, and which codecs it covers.
|
||
|
||
## The supplied-assets rule
|
||
|
||
For any asset you already have, an explicit path is the instruction that removes the most ambiguity. The agent will search when you describe an asset, but a path removes every ambiguity about *which* file — and for your own brand assets, it's the only way to guarantee the right one:
|
||
|
||
- ❌ `use my logo`
|
||
- ✅ `use assets/logo.svg`
|
||
|
||
This matters even when resolution would otherwise work: brand and entity assets should point at *your* file, not a resolved lookalike. (Third-party logos are a separate case — the pipeline pulls official marks from a logo cascade and never hand-redraws them, so "add the LinkedIn logo" is fine; "add my company's logo" needs a path.)
|
||
|
||
## Say what "no sound" actually means
|
||
|
||
The most common audio mistake is a negative that means less than you think. "No narration" removes the voiceover — it does **not** silence music or sound effects. If you want genuine silence, say so:
|
||
|
||
- ❌ `no narration` when you mean a completely silent video — music and SFX can still be added
|
||
- ✅ `no audio at all` — the unambiguous way to ask for silence
|
||
|
||
This mirrors the negatives discipline in [Anatomy](/prompting/anatomy): close the gap explicitly, because the engine acts on the literal words.
|
||
|
||
## Related
|
||
|
||
<CardGroup cols={2}>
|
||
<Card title="Vocabulary" href="/prompting/vocabulary">Voice names, caption tones, and audio-reactive mappings</Card>
|
||
<Card title="Captions catalog" href="/prompting/captions-catalog">Styling the timed text this page produces</Card>
|
||
<Card title="Remove background guide" href="/guides/remove-background">The matting command, its person-only caveat, and alternatives</Card>
|
||
<Card title="Video components" href="/guides/video-components">Installable overlays, captions, and effects</Card>
|
||
</CardGroup>
|
||
|
||
<Note>
|
||
**Capstone thread** — the [Level 7 film](/prompting/capstone)'s Material region runs this chapter's entire pipeline on one clip: generated footage → HEVC auto-proxy → background removal mid-scene → word-synced captions from the clip's own transcription, with the clip's audio ducking the BGM (cut from the film, below).
|
||
</Note>
|
||
|
||
This is the clause in the [full capstone prompt](/prompting/capstone#the-full-prompt-verbatim) that buys the piece — prompt language you can lift for your own video:
|
||
|
||
> […] a real talking-head clip (generate a short clip of a person speaking one neutral line via the media pipeline's avatar video generation […] **transcode it to HEVC `hvc1`** so the automatic proxy subsystem carries preview) sits as a clip on the wire. The order of operations IS the story: as the camera arrives and BEFORE the person speaks, the framework mattes the footage — **the background peels away via background removal** […] THEN they speak, and the main **keywords of their own line — derived from the clip's transcription — land word-synced** […] The clip's own audio ducks the BGM briefly; the VO resumes as the camera pulls away.
|
||
|
||
<video controls muted loop playsinline preload="metadata" src="https://static.heygen.ai/hyperframes-oss/docs/images/prompting/capstone-region-material.mp4#t=0.1" style={{ borderRadius: "0.5rem", marginTop: "0.75rem" }}></video>
|
||
*That clause, rendered — the region cut from the finished film.*
|
||
|
||
*Next: [Design systems and brand](/prompting/design-systems) — pointing the agent at a source of brand truth instead of describing a vibe.*
|