* fix(audio): a sound is voice only with a voice-range pitch, judged on the whole clip
Two ways the kind guess called a non-voice sound voice:
- A short hit, click or whoosh can have a speech-shaped envelope. classify() now also
measures the share of 40 ms frames with a 70-500 Hz fundamental (YIN, moved from the
CLI into core) and calls a sound voice only when that share reaches 0.15. Speech
measured 0.22-0.43; impacts, clicks and whooshes 0.
- A range query judged a clip's kind on the slice it covered, so 1.7 s of a music bed
read as voice. A film clip's kind is now judged on all of the clip that plays.
Even-paced speech (a 64 s TTS take) moves less at syllable rate and was called music;
with no beat and a voiced share of 0.25 or more it is now voice.
* chore(audio): keep each touched file's comment share at or under its base
The Comments check failed on run.ts, targets.ts and audioAnalysis.ts because the
PR added comment lines to files whose share may only go down. Drops the added
doc comments; no code change.
— Jerrai
* fix(audio): preserve clip evidence and stereo pitch during analysis
* fix(audio): bound and cache whole-clip kind analysis
* fix(audio): retain native-rate kind evidence when available
---------
Co-authored-by: Jerrai <jerrai@heygen.com>
* perf(studio): music beat analysis no longer freezes the page or runs during playback
* fix(core): media-treatment --analyze finishes on long HDR clips
Grade analysis decoded the clip from the start and kept every frame until the
fps filter had emitted five samples. Past 50 s the filter rate floors at 0.1
fps, so it always decoded the first 40 s at full rate and resolution: a
3-minute 4K60 10-bit HLG clip timed out at 15 s, and every long clip was
graded from its first 40 s only.
Each sample now seeks to the middle of one fifth of the clip and decodes one
frame. The timeout applies per sample and its error names the sample time and
HYPERFRAMES_ANALYZE_TIMEOUT_MS.
* refactor(core): name the keyframe seek instead of commenting it
* fix(core): count one frame per grade sample on ffmpeg 8
* feat(cli): audio answers beats, downbeats, bpm, sections and describe through a provider ladder
Beats and downbeats try api, apple-mu, beat-this, then signal; sections try api, apple-mu,
then signal; describe tries api, then apple-sound. bpm is read off whichever provider placed
the beats. Each provider says whether it can run here and why not, and the ladder warns
when it passes one over.
The signal rung is a TypeScript tracker: log-band spectral flux, autocorrelation tempo with
a log-normal prior, Ellis dynamic-programming beats with a windowed local tempo, a low-band
bar phase, and Foote novelty sections snapped to beats. It reports no steady beat when the
onsets do not repeat.
The api rung posts the audio to HYPERFRAMES_AUDIO_API_URL with HYPERFRAMES_AUDIO_API_KEY and
moves each answered time onto the nearest signal beat within a quarter beat, reporting what
moved and what did not. beat-this runs Beat This! from a configured Python and local weights
and never downloads. apple-sound runs macOS's sound classifier through a Swift helper.
apple-mu reports that Music Understanding needs macOS 27.
words are no longer accepted: they come from hyperframes transcribe.
* docs(cli): audio's beats, downbeats, bpm, sections, describe and their provider ladder
* feat(cli): beats tracks the music with the TypeScript beat tracker, without Chrome
hyperframes beats and Studio's beat detection now share analyzeMusicFromSignal: band energies,
then the spectral-flux tracker in audioRhythm.ts, then each beat's loudness for its strength.
The CLI decodes with the same ffmpeg path as hyperframes audio, so beats no longer launches
headless Chrome. Studio still decodes in the browser and passes every channel.
The old analyzer averaged bpm-detective's tempo with an energy-peak tempo it had only compared
after folding both into one octave, and phased a rigid grid from noisy peaks. On 22 real tracks
with exact script grids it found the tempo 18% of the time with beat F 0.437; this finds it
91% of the time with F 0.887, and refuses the other two rather than guess.
Below a tracker confidence of 0.2 no beats are reported, so ambient or rubato music gets no
invented grid and the CLI still writes nothing. bpm-detective, the energy-peak grid, the
headless analyzer and its prebuilt bundle are gone.
* fix(core): tempo reads 96 BPM off beats on 10 ms frames, not 95.2
The tracker places beats on 10 ms frames, so a 625 ms beat lands 620 or 630 ms apart and the
median interval reported 95.2 or 96.8 BPM. Tempo now averages the intervals within 10% of the
median: as robust to a skipped or doubled beat, without the frame-sized step. Both
hyperframes audio and hyperframes beats read it.
* fix(cli): remove redundant audio context comment
* fix(core): remove redundant tempo interval comment
* fix(ci): record beat analyzer removals and browser evidence
* fix(core): keep flat audio novelty in one section
* chore: keep PR capture evidence out of the source tree
* fix(cli): honor BPM provider pins before analysis
* feat(cli): audio answers beats, downbeats, bpm, sections and describe through a provider ladder
Beats and downbeats try api, apple-mu, beat-this, then signal; sections try api, apple-mu,
then signal; describe tries api, then apple-sound. bpm is read off whichever provider placed
the beats. Each provider says whether it can run here and why not, and the ladder warns
when it passes one over.
The signal rung is a TypeScript tracker: log-band spectral flux, autocorrelation tempo with
a log-normal prior, Ellis dynamic-programming beats with a windowed local tempo, a low-band
bar phase, and Foote novelty sections snapped to beats. It reports no steady beat when the
onsets do not repeat.
The api rung posts the audio to HYPERFRAMES_AUDIO_API_URL with HYPERFRAMES_AUDIO_API_KEY and
moves each answered time onto the nearest signal beat within a quarter beat, reporting what
moved and what did not. beat-this runs Beat This! from a configured Python and local weights
and never downloads. apple-sound runs macOS's sound classifier through a Swift helper.
apple-mu reports that Music Understanding needs macOS 27.
words are no longer accepted: they come from hyperframes transcribe.
* docs(cli): audio's beats, downbeats, bpm, sections, describe and their provider ladder
* fix(cli): remove redundant audio context comment
* fix(core): keep flat audio novelty in one section
* fix(cli): honor BPM provider pins before analysis
* feat(cli): hyperframes audio measures where sounds land, in film time
Adds @hyperframes/core/audio-analysis: an energy envelope, spectral-flux
and energy onsets, the moments a sound lands (onset, steepest rise, peak,
tail end), BS.1770 loudness with true peak, silence spans and a first
guess at sfx/voice/music, all pure functions of PCM. Each capability runs
through a provider ladder so model or service providers can join later.
The radix-2 FFT moves out of audioCarve so both share it.
`hyperframes audio` decodes with FFmpeg and analyzes files, or the film's
own audio and video clips by element, second, range or boundary, reading
only the part of each file that plays. Timeline rows now carry
`mediaStart`, moved by an in-point host's cut, so file time maps to film
time.
* feat(cli): audio placements carry the element's own data-start, local to its composition
* feat(cli): audio placements carry the element's own data-start, local to its composition
* feat(cli): audio places a file with --place-at, drops --boundary
--start becomes --place-at and --media-start becomes --skip-file-seconds,
which only goes with --place-at, refuses to skip the whole file, and warns
when the skipped part held the file's loudest moment. --boundary is gone:
--at and --range cover the sounds a cut touches.
* fix(cli): pass a sound file to ffprobe as a file, on every platform
ffprobe treated a name starting with a dash as an option because the argument
list never ended the flags. The path check compared a Windows drive path to a
POSIX string, and a new timeline comment raised that file's comment share.
* fix(cli): hear a reversed stereo channel, and warn when its peak was skipped
Averaging the channels cancelled a reversed one, so a real hit was reported
as silence. The skip warning also compared against the decode margin, so a
peak just before the kept window produced no warning.