From bb49858b8a4df6ae259e30cb2ce06bc77f86bbc5 Mon Sep 17 00:00:00 2001 From: Xuanru Li <157947275+xuanruli@users.noreply.github.com> Date: Thu, 15 Oct 2026 02:32:10 +0000 Subject: [PATCH] feat(cli): hyperframes audio measures where sounds land, in film time (#5602) * feat(cli): hyperframes audio measures where sounds land, in film time Adds @hyperframes/core/audio-analysis: an energy envelope, spectral-flux and energy onsets, the moments a sound lands (onset, steepest rise, peak, tail end), BS.1770 loudness with true peak, silence spans and a first guess at sfx/voice/music, all pure functions of PCM. Each capability runs through a provider ladder so model or service providers can join later. The radix-2 FFT moves out of audioCarve so both share it. `hyperframes audio` decodes with FFmpeg and analyzes files, or the film's own audio and video clips by element, second, range or boundary, reading only the part of each file that plays. Timeline rows now carry `mediaStart`, moved by an in-point host's cut, so file time maps to film time. * feat(cli): audio placements carry the element's own data-start, local to its composition * feat(cli): audio placements carry the element's own data-start, local to its composition * feat(cli): audio places a file with --place-at, drops --boundary --start becomes --place-at and --media-start becomes --skip-file-seconds, which only goes with --place-at, refuses to skip the whole file, and warns when the skipped part held the file's loudest moment. --boundary is gone: --at and --range cover the sounds a cut touches. * fix(cli): pass a sound file to ffprobe as a file, on every platform ffprobe treated a name starting with a dash as an option because the argument list never ended the flags. The path check compared a Windows drive path to a POSIX string, and a new timeline comment raised that file's comment share. * fix(cli): hear a reversed stereo channel, and warn when its peak was skipped Averaging the channels cancelled a reversed one, so a real hit was reported as silence. The skip warning also compared against the decode margin, so a peak just before the kept window produced no warning. --- docs/packages/cli.mdx | 65 +- packages/cli/README.md | 10 + .../cli/src/audio/analysis/analyze.test.ts | 54 ++ packages/cli/src/audio/analysis/analyze.ts | 150 +++++ packages/cli/src/audio/analysis/decode.ts | 103 +++ .../cli/src/audio/analysis/format.test.ts | 52 ++ packages/cli/src/audio/analysis/format.ts | 97 +++ packages/cli/src/audio/analysis/run.test.ts | 147 +++++ packages/cli/src/audio/analysis/run.ts | 177 +++++ .../cli/src/audio/analysis/targets.test.ts | 146 ++++ packages/cli/src/audio/analysis/targets.ts | 114 ++++ packages/cli/src/cli.ts | 1 + packages/cli/src/commands/audio.ts | 224 +++++++ packages/cli/src/help.ts | 1 + .../timeline/describeProject.inPoint.test.ts | 48 +- .../cli/src/timeline/describeProject.test.ts | 1 + packages/cli/src/timeline/describeProject.ts | 8 +- packages/cli/src/timeline/timeExpr.test.ts | 1 + packages/core/package-subpaths.json | 6 + packages/core/package.json | 10 + packages/core/src/audioAnalysis.test.ts | 255 +++++++ packages/core/src/audioAnalysis.ts | 623 ++++++++++++++++++ packages/core/src/audioAnalysisLadder.ts | 132 ++++ packages/core/src/audioCarve.ts | 37 +- packages/core/src/fft.ts | 35 + 25 files changed, 2430 insertions(+), 67 deletions(-) create mode 100644 packages/cli/src/audio/analysis/analyze.test.ts create mode 100644 packages/cli/src/audio/analysis/analyze.ts create mode 100644 packages/cli/src/audio/analysis/decode.ts create mode 100644 packages/cli/src/audio/analysis/format.test.ts create mode 100644 packages/cli/src/audio/analysis/format.ts create mode 100644 packages/cli/src/audio/analysis/run.test.ts create mode 100644 packages/cli/src/audio/analysis/run.ts create mode 100644 packages/cli/src/audio/analysis/targets.test.ts create mode 100644 packages/cli/src/audio/analysis/targets.ts create mode 100644 packages/cli/src/commands/audio.ts create mode 100644 packages/core/src/audioAnalysis.test.ts create mode 100644 packages/core/src/audioAnalysis.ts create mode 100644 packages/core/src/audioAnalysisLadder.ts create mode 100644 packages/core/src/fft.ts diff --git a/docs/packages/cli.mdx b/docs/packages/cli.mdx index c186b36622..3b65380cc7 100644 --- a/docs/packages/cli.mdx +++ b/docs/packages/cli.mdx @@ -20,17 +20,17 @@ npm install -g hyperframes ## Find your command -| You want to | Use | -| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Start a project | [`init`](#init), [`add`](#add), [`catalog`](#catalog) | -| Bring in source material | [`capture`](#capture), [`transcribe`](#transcribe), [`tts`](#tts), [`remove-background`](#remove-background), [`media-treatment`](#media-treatment), [`beats`](#beats), [`normalize-audio`](#normalize-audio) | -| Look at it, or share it | [`preview`](#preview), [`present`](#present-and-play), [`play`](#present-and-play), [`publish`](#publish), [`open`](#open-and-catch-up), [`catch-up`](#open-and-catch-up) | -| Find problems | [`lint`](#lint), [`check`](#check), [`snapshot`](#snapshot), [`keyframes`](#keyframes), [`compare`](#compare-and-grade-compare), [`grade-compare`](#compare-and-grade-compare) | -| Edit from the terminal | [`timeline`](#timeline), [`history`](#history) | -| Make a file | [`render`](#render), [`benchmark`](#benchmark) | -| Render somewhere else | [`cloud`](#hyperframes-cloud), [`lambda`](#hyperframes-lambda), [`cloudrun`](#hyperframes-cloudrun) | -| Fix your setup | [`doctor`](#doctor), [`browser`](#browser), [`upgrade`](#upgrade), [`docs`](#docs), [`info`](#info-and-compositions), [`compositions`](#info-and-compositions) | -| Connect other tools | [`auth`](#hyperframes-auth), [`skills`](#skills), [`figma`](#figma-and-events), [`telemetry`](#telemetry), [`feedback`](#feedback) | +| You want to | Use | +| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Start a project | [`init`](#init), [`add`](#add), [`catalog`](#catalog) | +| Bring in source material | [`capture`](#capture), [`transcribe`](#transcribe), [`tts`](#tts), [`remove-background`](#remove-background), [`media-treatment`](#media-treatment), [`beats`](#beats), [`audio`](#audio), [`normalize-audio`](#normalize-audio) | +| Look at it, or share it | [`preview`](#preview), [`present`](#present-and-play), [`play`](#present-and-play), [`publish`](#publish), [`open`](#open-and-catch-up), [`catch-up`](#open-and-catch-up) | +| Find problems | [`lint`](#lint), [`check`](#check), [`snapshot`](#snapshot), [`keyframes`](#keyframes), [`compare`](#compare-and-grade-compare), [`grade-compare`](#compare-and-grade-compare) | +| Edit from the terminal | [`timeline`](#timeline), [`history`](#history) | +| Make a file | [`render`](#render), [`benchmark`](#benchmark) | +| Render somewhere else | [`cloud`](#hyperframes-cloud), [`lambda`](#hyperframes-lambda), [`cloudrun`](#hyperframes-cloudrun) | +| Fix your setup | [`doctor`](#doctor), [`browser`](#browser), [`upgrade`](#upgrade), [`docs`](#docs), [`info`](#info-and-compositions), [`compositions`](#info-and-compositions) | +| Connect other tools | [`auth`](#hyperframes-auth), [`skills`](#skills), [`figma`](#figma-and-events), [`telemetry`](#telemetry), [`feedback`](#feedback) | `validate`, `inspect`, and `layout` still work but are [deprecated](#deprecated-validate-inspect-and-layout). Use `check` instead. @@ -504,6 +504,49 @@ Needs a local Chrome, the same one `render` uses. Run within a frame or two — a different headless-Chrome audio sample rate can shift a beat slightly. +### `audio` + +Measure how sound moves in time, so a cut or an animation can land on it. + +```bash +npx hyperframes audio sfx/whoosh.mp3 # one file, in file seconds +npx hyperframes audio sfx/hit.wav --place-at 4.2 # not in the film yet: where it lands if it starts at 4.2 s +npx hyperframes audio --element '#riser' # one clip of the film, in film seconds +npx hyperframes audio --at 12 # every sound playing at 12 s +npx hyperframes audio --range 10,14 --only onsets # every hit between 10 and 14 s +``` + +Each sound reports: + +- **moments**: where it starts (RMS first reaches 20% of its peak), where it + rises fastest, its loudest point, and where its tail fades 40 dB under the peak. + A short impact accents on its peak; a slow whoosh accents on its steepest rise. +- **onsets**: every hit, with a strength relative to the strongest. +- **loudness**: integrated and short-term-max LUFS (ITU-R BS.1770, matching + FFmpeg's `ebur128` to 0.1 LU), sample and true peak, clipping. +- **silence**: quiet spans under `--silence-db` (−50 dBFS). +- **kind**: a first guess of `sfx`, `voice` or `music`, with the features behind it. +- **envelope**: the level over time, about 2,000 values per sound. + +A film query (`--element`, `--at`, `--range`) reads the project in +the current directory or `--dir`, finds the `