mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
Vance Ingalls
0eca7b1e0a
* docs(audio): document the audio effects system The effects feature had no documentation at all — the only mentions anywhere in docs/ were changelog entries. This adds the three pages the color-grading feature already has, one per audience, and closes a gap in the existing audio guide. - prompting/audio-effects — a new Level 5 chapter on asking for a mix in symptoms rather than in filters, with the voiceover carve as the headline, the level-before-depth check, groups, and the three requests that have no honest answer (de-essing, noise removal, tone matching). - studio/audio-effects — the rack, presets by symptom, the carve module, groups, the deliberately asymmetric mute/solo, automation lanes, and troubleshooting. - reference/audio-effects — the contract: all four attributes, every effect and parameter range, why some parameters cannot be automated, the 19 presets, the five jobs and five one-knob profiles, carve semantics, the group model, the render bus, preview/render parity, and the three lint rules. Also points the existing "duck music under important speech" advice in guides/voice-and-audio at the carve, which does that properly and was never named there. Facts verified against source rather than the shipped skill prose: the panel section is "Audio FX" (the neighbouring "Effects" section is visual effects, and an earlier draft of this page named the wrong one), MAX_AUTOMATION_POINTS is 512, fromPreset carries a preset id rather than a boolean, the leveller targets the track's own 80th percentile, group mute reaches the render while solo never leaves Studio, and the CLI carve currently writes clip ids. No screenshots in this pass, and no placeholders for them either — the pages carry concrete markup and tables instead. Screenshots of the rack, the carve module, and a group row would each earn their place later. mint validate and mint broken-links both pass. * docs(audio): give audio its own Studio group, split by task Review feedback: the audio page did not belong in Studio / Edit. That group is already one task per page — canvas, timeline, animation, captions — and a single page covering the rack, the carve, groups, and automation lanes was four tasks bundled together and dropped in beside them. Studio now has an Audio group holding four task pages: - Effects and presets — the rack, presets by symptom, adding single effects in a working order, the one-knob controls, Even Out Levels - Voiceover carve — its own page, because it is the feature people come for - Groups, mute, and solo — including why mute reaches the export and solo cannot - Automation lanes — drawing envelopes, the shape menu, and which parameters cannot move at all Not a new top-level tab: the tabs here are audience-scoped (Guides, Studio, Catalog, Developers), so a feature tab would be the only one of its kind and would strand the prompting chapter out of its Level 5 sequence and the reference page out of Developers. Repointed the deep link in guides/voice-and-audio at the carve's own page. Lane interactions verified in source before documenting: the right-click menu offers Ramp up, Ramp down, Swell, Dip, and Simplify (which needs three points). Deliberately not documented: "clicking a lane label reveals it in the rack", which is not on main. mint validate and mint broken-links both pass. * docs(audio): fix the four review blockers All four verified in source before fixing; the review was right on every count. **The registry was incomplete.** `pitchshift` ships at `audioFx.ts:509-535` (`semitones` −12–12, `mix` 0–1, worklet-backed so neither automatable) and was missing entirely — sixteen effects, not fifteen. It also joins the worklet list, so five effects expose no automatable parameters rather than four. **Three presets were missing.** `chipmunk`, `giant`, and `monster` ship at `audioFxPresets.ts:336-357`, all built on `pitchshift`. Twenty-two presets, and Character holds ten. Fixed in both the reference table and the Studio list. **The copyable markup contradicted the warning above it.** The page said `carve.mjs` only finds double-quoted attributes and then gave three single-quoted examples — copying the chain example would make a later carve miss the existing chain and overwrite it. All three are now double-quoted with `"`, each followed by its unescaped reading so it stays legible. **The attribute table over-claimed.** `data-audio-group` is a plain id, not JSON, and is ignored on `<video>`; the other three also live on `<hf-audio-group>` for a group. The table now carries shape and valid host per attribute. **The automation contract was wrong on two axes.** A clip lane's `t` is clip-relative but a GROUP lane's is composition time, because a group has no `data-start` (`webAudioTransport.ts:337-342`, `audioMixer.ts:1311-1344`) — both pages now split the two clocks. And `volume` is not 0–1: the ceiling is `MAX_AUDIO_GAIN`, +12 dB or about 3.981 (`audioGain.ts:8-9`), so a boosting lane is valid and documented. **Current-main drift.** #3416 is merged, so the CLI now records the voices' shared group when it is safe and falls back to clip ids when that group contains the bed or a music/SFX member. Documented, including why neither refusal shows up on the run that writes it, and rebased onto main. mint validate and mint broken-links both pass. * docs(audio): name the real add-menu family, and finish propagating pitchshift Second review round. All three findings were my own incomplete propagation — I corrected the reference for `pitchshift` last round and left the reader-facing pages behind it. **The add-menu family is `Time`, not `Space`.** `propertyPanelFxAddMenu.tsx:22-28` labels the four groups Filters / Dynamics / Non-linear / Time, and the time group holds pitchshift, delay, chorus, phaser, and reverb. The Studio page sent readers looking for a group that does not exist. It is now a table naming the family and its contents, and the reference's "Time — space and width" heading is retitled, since that description stopped covering the family the moment pitch shift joined it. Also from the same file: the menu offers the named jobs in place of a bare `peaking`, because picking `peaking` is picking a machine and leaving the real decision — which range — for afterwards. Worth saying on the task page. **Pitch shift was missing from both no-automation lists** that a reader actually follows — `studio/audio-automation` and, unflagged but the same defect, the prompting chapter. Five worklet effects in all four places now. Called out explicitly on the Studio page, because a rising pitch is exactly the thing someone reaches for a lane to do, and the lane will not report that it cannot. **Narrowed the group-metadata sentence.** "The other three are JSON, and on a group they live on `<hf-audio-group>`" swept in `data-fx-carve`, contradicting the table directly above it. Only `data-fx-chain` and `data-automation` are group metadata. mint validate and mint broken-links both pass.
133 lines
5.2 KiB
Plaintext
133 lines
5.2 KiB
Plaintext
---
|
|
title: "Use voice, music, sound, and captions"
|
|
sidebarTitle: "Voice, sound, and captions"
|
|
description: "Build an understandable audio mix, transcribe real speech, and turn it into readable captions."
|
|
---
|
|
|
|
import { DocsVideo } from "/snippets/docs-video.jsx";
|
|
|
|
Give each audio layer one job. Add another layer only when it helps the viewer
|
|
understand or feel the video.
|
|
|
|
| Layer | Main job |
|
|
| ------------- | --------------------------------------------------- |
|
|
| Voiceover | Carry the explanation or story |
|
|
| Source audio | Preserve speech or sound from existing footage |
|
|
| Music | Shape pace, tone, and structure |
|
|
| Sound effects | Reinforce a specific action or transition |
|
|
| Captions | Make spoken words readable and searchable on screen |
|
|
|
|
|
|
<DocsVideo
|
|
title="Narration, music ducking under it, and effects landing on the beat"
|
|
src="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/voice-audio-demo-v2.mp4"
|
|
poster="https://static.heygen.ai/hyperframes-oss/docs/images/showcase/voice-audio-demo-v2.jpg"
|
|
/>
|
|
|
|
Turn the sound on for this one — the audio is the subject. You can watch the
|
|
music duck as narration arrives and see each effect land against the waveform.
|
|
|
|
## Create a voiceover
|
|
|
|
Give the agent the approved wording and useful voice direction:
|
|
|
|
```text
|
|
Generate the approved SCRIPT.md as a warm, direct voiceover.
|
|
Natural pace, no announcer energy. Keep the product-name pronunciation exact.
|
|
```
|
|
|
|
The project can keep the script, audio, and word-level transcript separately.
|
|
That lets you regenerate the voice without discarding the timing and caption
|
|
work around it.
|
|
|
|
Listen before building the final edit. Fix wording, pronunciation, or delivery
|
|
in the script or voice direction rather than cutting around a bad read.
|
|
|
|
## Transcribe recorded speech
|
|
|
|
Transcription creates timed words from an audio or video file. Start with the
|
|
automatic engine: it uses Parakeet when installed and falls back to Whisper.
|
|
|
|
```bash
|
|
npx hyperframes transcribe interview.mp4
|
|
```
|
|
|
|
Pass the spoken language when you know it. This filters non-target speech and
|
|
lets the Whisper fallback choose a multilingual model when needed.
|
|
|
|
```bash
|
|
npx hyperframes transcribe interview.mp4 --language es
|
|
```
|
|
|
|
For difficult English audio, force the larger English Whisper model. For an
|
|
unknown language, use multilingual `large-v3` and let Whisper detect it.
|
|
|
|
```bash
|
|
npx hyperframes transcribe interview.mp4 --engine whisper --model medium.en
|
|
npx hyperframes transcribe interview.mp4 --engine whisper --model large-v3
|
|
```
|
|
|
|
Larger models take longer and do not remove the need to read the result.
|
|
|
|
The command can also import an existing `.srt`, `.vtt`, or supported transcript
|
|
JSON file:
|
|
|
|
```bash
|
|
npx hyperframes transcribe subtitles.srt
|
|
```
|
|
|
|
## Turn the transcript into captions
|
|
|
|
1. Read the complete transcript.
|
|
2. Correct names, product terms, numbers, punctuation, and obvious recognition
|
|
errors.
|
|
3. Group words into short phrases that break on meaning.
|
|
4. Open [Captions in Studio](/studio/captions) to inspect the rhythm and adjust
|
|
placement, scale, or rotation. Make durable wording, timing, style, and
|
|
animation changes in the caption source or ask the agent to update it.
|
|
5. Watch once with sound for synchronization and once muted for readability.
|
|
|
|
Captions should follow what was said. They should not cover a face, product
|
|
control, or another important visual. Use emphasis on the few words that carry
|
|
the point instead of making every word compete.
|
|
|
|
For an existing talking-head clip, use [Captions or
|
|
recuts](/guides/captions-and-recuts) to choose between plain captions, designed
|
|
graphic overlays, and an actual change to the spoken edit.
|
|
|
|
## Mix for understanding
|
|
|
|
- Keep voice clear above music.
|
|
- Duck music under important speech instead of reducing the whole track
|
|
equally. A [voiceover carve](/studio/voiceover-carve)
|
|
does this properly: it takes only the bands the voice occupies out of the
|
|
music, so the bed keeps its low end and its top instead of going limp for the
|
|
whole voiceover.
|
|
- Use sound effects for meaningful events, not every movement.
|
|
- Avoid cutting words, breaths, or reverb tails accidentally at scene
|
|
boundaries.
|
|
- Let a deliberate silence remain silent.
|
|
- Listen on ordinary headphones and laptop speakers.
|
|
|
|
When narration drives the project, time visual changes to its real transcript
|
|
rather than estimated scene lengths. When music drives the project, use the
|
|
[music-video workflow](/guides/music-to-video) and its analyzed structure.
|
|
|
|
## Review the complete sound pass
|
|
|
|
- The correct voice and source recordings are used.
|
|
- Names, claims, and captions are accurate.
|
|
- Speech stays clear through the loudest musical section.
|
|
- Effects arrive with the action they support.
|
|
- The opening does not surprise the listener with excessive volume.
|
|
- Music and ambience finish intentionally instead of stopping at the file edge.
|
|
|
|
Render a review file and listen without watching once. Audio problems are often
|
|
easier to notice when the visuals are not competing for attention.
|
|
|
|
## Related topics
|
|
|
|
- [Add captions or repackage talking-head footage](/guides/captions-and-recuts)
|
|
- [Edit captions in Studio](/studio/captions)
|
|
- [Finish, render, and share](/guides/export-and-share)
|