mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-04 21:17:30 +00:00
Xuanru LiandJerrai
1df31d8717
* fix(audio): a sound is voice only with a voice-range pitch, judged on the whole clip Two ways the kind guess called a non-voice sound voice: - A short hit, click or whoosh can have a speech-shaped envelope. classify() now also measures the share of 40 ms frames with a 70-500 Hz fundamental (YIN, moved from the CLI into core) and calls a sound voice only when that share reaches 0.15. Speech measured 0.22-0.43; impacts, clicks and whooshes 0. - A range query judged a clip's kind on the slice it covered, so 1.7 s of a music bed read as voice. A film clip's kind is now judged on all of the clip that plays. Even-paced speech (a 64 s TTS take) moves less at syllable rate and was called music; with no beat and a voiced share of 0.25 or more it is now voice. * chore(audio): keep each touched file's comment share at or under its base The Comments check failed on run.ts, targets.ts and audioAnalysis.ts because the PR added comment lines to files whose share may only go down. Drops the added doc comments; no code change. — Jerrai * fix(audio): preserve clip evidence and stereo pitch during analysis * fix(audio): bound and cache whole-clip kind analysis * fix(audio): retain native-rate kind evidence when available --------- Co-authored-by: Jerrai <jerrai@heygen.com>
114 lines
5.3 KiB
Plaintext
114 lines
5.3 KiB
Plaintext
---
|
||
title: "Add voiceover, music, sound effects and captions"
|
||
sidebarTitle: "Voice, music and captions"
|
||
description: "Ask for narration, music, sound effects and captions in plain words: what's free, what HeyGen adds, and how to check the mix."
|
||
---
|
||
|
||
import { DocsVideo } from "/snippets/docs-video.jsx";
|
||
|
||
Ask for sound the way you'd brief an editor. The agent finds or makes each piece, places it on the timeline and
|
||
mixes it, with the music dipping under the voice.
|
||
|
||
<DocsVideo
|
||
title="Narration, music ducking under it, and effects landing on the beat"
|
||
src="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/voice-audio-demo-v2.mp4"
|
||
poster="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/voice-audio-demo-v2.jpg"
|
||
/>
|
||
|
||
*Sound on: narration, the music ducking under it, and effects landing with the action.*
|
||
|
||
## Ask for it
|
||
|
||
In the Studio app or your own agent, in one message or one at a time:
|
||
|
||
```text
|
||
Add a warm, unhurried narration of this script: [paste script]
|
||
Put a calm piano bed under it, quieter while she speaks.
|
||
A soft whoosh on each scene change, nothing else.
|
||
Captions: two lines at most, key words in the brand color.
|
||
```
|
||
|
||
Quote the script when the words are approved; otherwise give the points and let the agent write it. Name what you
|
||
don't want, too: "no music" and "no sound effects" are followed exactly.
|
||
|
||
## Free, or HeyGen quality
|
||
|
||
| | Free, on your computer | With HeyGen |
|
||
| --- | --- | --- |
|
||
| **Voice** | Kokoro voices, in several languages | HeyGen voices and your own cloned voice |
|
||
| **Music** | Tracks you add, or local generation if you install it | A 10,000-track catalog |
|
||
| **Sound effects** | A bundled set, plus Openverse | The same |
|
||
| **Captions** | Local transcription with your own agent (Parakeet or Whisper) | The Studio app transcribes through HeyGen |
|
||
|
||
The Studio app uses HeyGen when you add a HeyGen API key (Home says **Add a HeyGen API key to make voiceovers, avatars
|
||
and clips**), and a local voice when you don't. With your own agent, sign in to the HeyGen CLI or set an API key. The
|
||
words to use for each are on [Voice, music and sound](/prompting/media-and-audio).
|
||
|
||
## Use your own recording
|
||
|
||
Add the file and say what it is ("this is the narration", "keep the interview audio"). To get timed words from it
|
||
yourself, transcription runs locally and costs nothing:
|
||
|
||
```bash
|
||
npx hyperframes transcribe interview.mp4 --language es
|
||
```
|
||
|
||
Pass `--language` when the speech isn't English. It also imports an existing `.srt` or `.vtt`. For a talking-head clip that needs
|
||
captions or a tighter cut, see [Captions and recuts](/guides/captions-and-recuts).
|
||
|
||
## Get captions right
|
||
|
||
Read the transcript before it goes on screen: names, product terms and numbers are where recognition slips. Keep
|
||
lines short and broken on meaning, keep them off faces and buttons, and emphasize the two or three words that carry
|
||
the point. To adjust them by hand, open [Captions](/studio/assets-and-blocks#edit-captions) in the browser editor.
|
||
|
||
## Repeat music inside a slot
|
||
|
||
Use the native `loop` attribute to repeat a short track for a longer slot:
|
||
|
||
```html
|
||
<audio id="music" src="music.wav" data-start="0" data-duration="30" loop></audio>
|
||
```
|
||
|
||
The track repeats until the 30-second slot ends in preview and export. If you
|
||
set `data-media-start`, every repeat starts at that source in-point. Playback
|
||
speed and rate automation apply to the repeated audio. Looping videos with
|
||
`data-has-audio="true"` repeat their sound too, including any silence before
|
||
the next cycle.
|
||
|
||
## Check the mix
|
||
|
||
Watch once with sound and once muted, then listen once without watching.
|
||
|
||
- The voice stays clear through the loudest part of the music.
|
||
- Each effect lands with the thing it belongs to.
|
||
- Nothing cuts off a word or a breath at a scene change.
|
||
- The music ends on purpose, not at the end of the file.
|
||
|
||
The audio analysis gives each sound a first guess of `voice`, `music` or `sfx`.
|
||
Voice needs both a speech-like level pattern and voice-range pitch, so a hit
|
||
with a speech-like tail is less likely to be mistaken for narration. These are
|
||
guesses, not transcript labels. When you inspect a time range, kind uses the
|
||
whole source span played by the clip; loudness, hits and the other measurements
|
||
still describe the selected range. A short slice of a music bed therefore keeps
|
||
the bed’s kind. Repeated range queries reuse that guess until the source or clip
|
||
changes. If the whole clip cannot be analyzed, the result warns that kind uses
|
||
only the selected range; the range’s other measurements are still returned.
|
||
See [audio analysis](/packages/cli#audio) for the command and output.
|
||
|
||
To dip the music only where the voice sits instead of turning it all down, use a
|
||
[voiceover carve](/studio/audio-effects#make-music-sit-under-narration).
|
||
|
||
Music can start after narration begins: the dip then starts at the right level for that moment, and speech that has
|
||
already ended doesn't make a new dip.
|
||
|
||
<Card title="Practice it: Sound" icon="graduation-cap" href="https://hyperframes.heygen.com/lessons/sound">
|
||
Free 20-minute lesson with a short video, a quiz and a brief to try. Say the sound in your brief: the music's mood, the voice, and where the music starts and stops.
|
||
</Card>
|
||
|
||
## Related topics
|
||
|
||
- [Add captions to a video with AI](/guides/captions-and-recuts)
|
||
- [Make a video to music](/guides/music-to-video)
|
||
- [Voice, music and sound: the words to use](/prompting/media-and-audio)
|