mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-17 15:24:19 +00:00
docs(plans): hand off the audio review fixes for a fresh session
Eleven of fifteen findings fixed on wa-19-review-fixes; the four left open are recorded with the analysis already done on each, including why the clip-before- volume-lane one was started and deliberately stopped. Also carries the cut list, the cleanup list, the carve-reconcile-wip undo regression, and the nine traps that cost time — the published-CLI-is-not-a-symlink one above all.
This commit is contained in:
@@ -0,0 +1,275 @@
|
||||
# Audio FX code-review fixes — session handoff
|
||||
|
||||
Written 2026-08-08 so a fresh session can continue without re-deriving anything.
|
||||
Everything here was verified in this worktree unless it says otherwise.
|
||||
|
||||
**Worktree:** `~/src/wt/hyperframes/webaudio-fx`
|
||||
**Working branch:** `wa-19-review-fixes` — 11 commits, **local only, never pushed**
|
||||
**Branched from:** `wa-18-lane-stretch` @ `4bcc7e188`
|
||||
|
||||
---
|
||||
|
||||
## 1. Where everything is
|
||||
|
||||
| Branch | State |
|
||||
| --- | --- |
|
||||
| `wa-18-lane-stretch` | **pushed**, matches remote. PR **#3058** open, ready for review, base `wa-17-lane-clipboard` |
|
||||
| `wa-19-review-fixes` | 11 commits, **local only**. The review fixes. |
|
||||
| `carve-reconcile-wip` | 1 commit (`7b57000a1`), **local only**. Working feature, but it breaks undo — see §5. |
|
||||
|
||||
Every other `wa-*` branch is in sync with its remote. The handoff §6 warning about
|
||||
needing to force-push 16 branches is **stale** — that was resolved; `wa-18` was a
|
||||
plain fast-forward.
|
||||
|
||||
### Test baselines (all green at `wa-19-review-fixes` HEAD)
|
||||
|
||||
```
|
||||
packages/studio 3661 passing, 18 todo, 1 file skipped
|
||||
packages/core 1657 passing
|
||||
packages/engine 733 passing, 3 skipped (src/services)
|
||||
```
|
||||
|
||||
`tsc`, `oxlint`, `oxfmt` clean across studio, core, engine, lint.
|
||||
|
||||
---
|
||||
|
||||
## 2. What the review was
|
||||
|
||||
`/code-review max` ran against the audio-FX **foundation** stack — NOT `wa-18`:
|
||||
|
||||
```
|
||||
#3019 → #3020 → #3021 → #3022 → #3014 → #3015 → #3016 → #3017
|
||||
origin/main … wa-5-automation-runtime-fixes, 6760 insertions
|
||||
```
|
||||
|
||||
Eight PRs, not the six that were asked for; splitting the closed
|
||||
`wa-1-fx-foundation` (#3013) into four is where the "6" came from. `#3036
|
||||
wa-13-fx-tail` was excluded.
|
||||
|
||||
**Review tip was `68ce91057`**, 13 commits below `wa-18` HEAD — so all the
|
||||
reviewer's line numbers are stale. Every fix below was re-verified against
|
||||
current code before being touched, and several had moved.
|
||||
|
||||
15 findings reported, ~10 more confirmed but cut by a cap, plus a cleanup list.
|
||||
Three candidates were refuted by the reviewer and dropped; one was already fixed
|
||||
upstack.
|
||||
|
||||
---
|
||||
|
||||
## 3. Fixed (11 commits on `wa-19-review-fixes`)
|
||||
|
||||
Each was verified against current code, fixed, given a test, and the test
|
||||
**falsified by deleting the fix**. Three ship without tests and say so in their
|
||||
commit message.
|
||||
|
||||
| Commit | What |
|
||||
| --- | --- |
|
||||
| `43c88961b` | Empty audio track no longer kills the whole render |
|
||||
| `714f662ed` | Browser lease taken inside the try that releases it (no test — resource ordering) |
|
||||
| `accac9984` | Phaser `in_gain`/`out_gain` lanes aimed at the trims, not the pinned wet/dry |
|
||||
| `b754e72d0` | Node ids follow their slot on an in-place chain update |
|
||||
| `f6f0e4cf2` | Volume lane sampled at clip-local time, matching the render |
|
||||
| `436db2506` | FX parameters can be typed; a bare blur no longer re-persists |
|
||||
| `59563eae0` | Lint rule stops blaming the wrong element in a chained timeline |
|
||||
| `007c94722` | Generated audio-FX runtime regenerated before core's tests |
|
||||
| `05d081c61` | Clip's FX graph disposed when it ends, not only when stopped |
|
||||
| `371920695` | FX rack locked while the carve is measuring |
|
||||
| `bfa27381e` | ffmpeg children given the signal that actually aborts them (no test — wiring) |
|
||||
|
||||
Two judgement calls worth knowing:
|
||||
|
||||
- **A guard was removed, not kept.** The review said `latest` (in
|
||||
`propertyPanelFxControls`) is never resynced. True, but the edit-flag fix in
|
||||
`436db2506` subsumes it — three sibling mutations failed, that one passed. Code
|
||||
no test can falsify was dropped rather than left in.
|
||||
- **`.gitignore` was not touched** (standing rule). That is why `007c94722`
|
||||
makes `test` regenerate the artifact instead of copying the sibling's
|
||||
`build && git diff --exit-code` gate, which needs the file tracked.
|
||||
|
||||
---
|
||||
|
||||
## 4. Remaining work
|
||||
|
||||
### 4a. The four still open from the top 15
|
||||
|
||||
**1. FX output clipped to ±1 before the volume lane ducks it.**
|
||||
`packages/engine/src/services/audioFxRender.ts:~136` clamps and quantises to
|
||||
int16 in `writeWav`; `audioMixer.ts:~936` then calls `applyVolumeEnvelopeToWav`,
|
||||
which re-reads that file and multiplies. So a chain that overshoots full scale is
|
||||
destructively clipped even when the envelope immediately pulls the track 12 dB
|
||||
down — audible distortion in the render that preview does not have.
|
||||
|
||||
*I started this one and stopped deliberately.* The fix is to apply the envelope
|
||||
to the float samples inside `applyAudioFxChain` before `writeWav` quantises.
|
||||
That means:
|
||||
- hoisting `envelopeKeyframes` (currently computed AFTER the FX call, `audioMixer.ts:~933`) above `applyAudioFxChain`
|
||||
- extending its options with the envelope + `trackStart` + base volume
|
||||
- setting `bakedEnvelope = true` so the mixer skips the second pass
|
||||
- extracting the incremental segment-cursor gain walk out of
|
||||
`audioVolumeEnvelope.ts:~107-116` so both paths share one implementation
|
||||
- keeping the ffmpeg-expression fallback intact for non-int16 WAVs
|
||||
|
||||
It changes rendered audio for every track with both a chain and a lane, so it
|
||||
wants its own session and a before/after listen.
|
||||
|
||||
**2. Base64 PCM round-tripped whole through one `page.evaluate`.**
|
||||
`audioFxRender.ts:~256`. ~184 MB per 3-minute stereo track across one CDP message
|
||||
pair. Verified ceilings: puppeteer-core sets `maxPayload: 256 MB` and
|
||||
`browserManager.ts` does not pass `pipe: true`, so stereo 48 kHz past **~8.7 min**
|
||||
exceeds the cap; V8's max string length is a second wall at ~17.5 min. Both raise
|
||||
`AudioFxRenderError`, fatal. Compounded by `processCompositionAudio`'s unbounded
|
||||
`Promise.all` holding N payloads at once. Also `Array.from(u8.subarray(...))`
|
||||
boxes each 32 KiB chunk for nothing — `apply` takes array-likes.
|
||||
|
||||
**3. `setRate` does not reschedule FX automation.**
|
||||
`webAudioTransport.ts:~263`. `attachElementFxChain` gets a frozen
|
||||
`{scheduledAt, elapsed, rate}` and commits ramps to absolute context time;
|
||||
`setRate` only assigns `playbackRate.value`. A lowpass sweeping over 10 clip-
|
||||
seconds, switched to 2× mid-play, eats 20 s of material in 10 s of wall clock
|
||||
while the sweep keeps its original schedule. The recovery path in
|
||||
`runtime/init.ts` (`stopAll()` + reschedule) only fires when
|
||||
`hasBoundedActiveSources()` is true — a project-level music bed with no
|
||||
`data-duration` is unbounded, so it never recovers. Second consequence:
|
||||
`timingNow()` advances `elapsed` with the stale rate, so any later chain edit
|
||||
re-aims the envelope at the wrong clip position.
|
||||
|
||||
**4. `powerSpectrum` allocates on the main thread.**
|
||||
`audioCarve.ts:~100` — two `Float64Array(4096)` inside the per-hop loop. A
|
||||
5-minute 48 kHz voiceover is ~461 MB of transient garbage and ~7030 FFTs in one
|
||||
uninterruptible block, called synchronously from the panel. `re` is fully
|
||||
overwritten each iteration and `im` only needs `.fill(0)`, so both hoist; Welch
|
||||
converges after a few hundred frames, so striding the hops gives the same
|
||||
spectrum for ~2% of the work. **Now copied into two more loops** —
|
||||
`audioCarve.ts:~320` and `~558` — so each needs its own spectrum check.
|
||||
|
||||
### 4b. Confirmed but cut by the reviewer's 15-cap
|
||||
|
||||
Rough severity order, file:line at the review tip:
|
||||
|
||||
- `audioFxGraph.ts:254`/`:316` — chorus + phaser `lfo.start()` at build time, not clip-relative → preview ≠ render phase
|
||||
- `audioFxWorklets.ts:247` — a rejected `addModule` is cached per-context forever while `readyContexts` is only written on success → all worklet effects permanently silent, no retry
|
||||
- `audioFxGraph.ts:159` — worklet `dispose` only disconnects while every `process()` returns `true` → abandoned processors never released
|
||||
- `audioFx.ts:754` — `Number(null)`/`Number("")` === 0 passes `Number.isFinite`, so a null param clamps to 0 instead of its declared default (compressor threshold 0 dB = never engages). `numberOrNull` at `audioAutomation.ts:141` already guards exactly this
|
||||
- `audioStage.ts:89` — `audioFailures: undefined` flattens the typed failure the stage comment says it exists to attach
|
||||
- `audioFxRender.ts:204` — per-element mkdtemp + page + `goto` + runtime re-injection
|
||||
- `.fallowrc.jsonc:662` — file-scoped health ignore also exempts `syncRuntimeMedia`; `:157` `ignoreExports` names two symbols that do not exist
|
||||
- `propertyPanelFxSection.tsx:287` — `key={type-index}` (PLAUSIBLE; likely inert now that the `latest` ref issue is fixed)
|
||||
|
||||
### 4c. Cleanup list
|
||||
|
||||
- `audioFxRender.ts:43` duplicates `parseWavLayout`
|
||||
- `propertyPanelFxControls.tsx:51` re-implements FlatSlider/FlatToggle/FlatSelectRow, losing their throttle, pointer capture, Escape-revert and design telemetry
|
||||
- `PropertyPanelFlat.tsx:549` reads via hardcoded `"fx-chain"` literals while writing via constants
|
||||
- `audioMixer.ts:328` re-rolls curve sampling at 30/s vs the scheduler's 100/s while `sampleAutomationCurve` names it as a consumer
|
||||
- `build-audio-fx-runtime.ts` is a near-verbatim clone of `build-position-edits-render.ts`
|
||||
- `audioFxGraph.ts:279`/`:352` four hand-rolled wet/dry shells whose dispose lists have already drifted
|
||||
- `audioFxGraph.ts:448` enabled-node filter restated 3× while `enabledAudioFxNodes()` goes unused
|
||||
- `audioFxGraph.ts:494` dead `shapeOf` recomputation on every observer tick
|
||||
- `PropertyPanelFlat.tsx:624` a `doc!` non-null assertion plus an `as` cast laundering `undefined`; `audioAutomation.ts:122` a cast after the guard already narrowed — both against the repo's "avoid `any`/`as T`" rule and the standing no-`!` rule
|
||||
|
||||
---
|
||||
|
||||
## 5. `carve-reconcile-wip` — working, but parked
|
||||
|
||||
`7b57000a1`, branched off `wa-18` HEAD. Makes a deleted voice re-measure its bed
|
||||
**wherever the author is looking**, rather than only when the bed's Audio FX
|
||||
section is opened (which is what ships on `wa-18`).
|
||||
|
||||
Contains:
|
||||
- `carveAnalysis.ts` — the carve lifted out of the panel as a computation over plain data, so it runs without a DOM. The panel calls it too.
|
||||
- `fxCarve` mirrored onto `TimelineElement` beside `fxChain`/`automation`
|
||||
- `useCarveReconcile` — watches the timeline store, writes all three attributes in one `patch-element`. Debounced 800 ms so an undo's flicker is not read as a deletion; single-flighted per carve so its own write cannot loop the decode. 11 tests, each falsified.
|
||||
- The panel's own prune effect removed (one owner, or the two race through different write paths and 409 the save queue).
|
||||
|
||||
**Verified working:** deleting a voice with the bed's panel never opened pruned
|
||||
the sources and re-measured the bands (400 Hz −8.2 → −8.33 dB, 1 kHz −6.68 →
|
||||
−7.14).
|
||||
|
||||
**Why it is parked:** it kills undo. After the watcher's `patch-element` write,
|
||||
pressing Undo issues **no HTTP request at all** and the file is untouched — the
|
||||
button is enabled, the click just does nothing. Measured, in contrast to the same
|
||||
flow before the change, where undo restored the clip. The write goes around the
|
||||
edit-history mechanism and leaves the stack unusable; the exact mechanism was
|
||||
never pinned down.
|
||||
|
||||
Likely next move: read the file, patch, then `recordEdit` the before/after so the
|
||||
history stack stays coherent — at the cost of the reconcile becoming its own undo
|
||||
step (two presses to get the clip back). Worth deciding which is wanted first.
|
||||
|
||||
Also: the 800 ms settle is a **guess, not a measurement** — undo broke before the
|
||||
real store-resync gap could be observed.
|
||||
|
||||
---
|
||||
|
||||
## 6. Traps that cost time this session
|
||||
|
||||
1. **`hyperframes` on PATH is the PUBLISHED npm package**, v0.7.99 at
|
||||
`~/.bun/install/global/node_modules/hyperframes` — a real directory, NOT a
|
||||
symlink to this repo. `hyperframes preview` serves the released studio bundle
|
||||
and none of this branch's work. The memory note saying it is symlinked is
|
||||
**wrong for this machine**. To see local studio changes:
|
||||
```bash
|
||||
cd packages/studio && bun run dev # → http://localhost:5190
|
||||
ln -sfn ~/src/recap-stitch packages/studio/data/projects/recap-stitch
|
||||
# open http://localhost:5190/#project/recap-stitch
|
||||
```
|
||||
2. **The generated audio-FX runtime goes stale silently.** Editing
|
||||
`packages/core/stubs/audio-fx-runtime-entry.ts` has NO effect on engine render
|
||||
tests until `bun run build:audio-fx-runtime` runs in `packages/core`. This cost
|
||||
a confusing falsification round. `007c94722` makes core's `test` regenerate it,
|
||||
but the engine's tests do not.
|
||||
3. **`rtk` wraps git and grep and sometimes eats output.** `git diff --name-only`
|
||||
returned nothing; `grep -n` reported "0 matches" on a file that matched. Use
|
||||
`/usr/bin/git` and `/usr/bin/grep` when output looks wrong. `--include=*.ts`
|
||||
globs fail in this shell — quote them or use `/usr/bin/grep -r`.
|
||||
4. **NUL bytes got written into source twice** by heredoc/Write round-trips — a
|
||||
literal space inside a template literal became `\x00`, which made ripgrep skip
|
||||
the file as binary and report zero matches. Check with
|
||||
`python3 -c "print(open(F,'rb').read().count(b'\x00'))"` if a search
|
||||
inexplicably finds nothing.
|
||||
5. **happy-dom has no `OfflineAudioContext`**, so `analyse()` returns before the
|
||||
decode and any test of the "while analysing" window never opens it. Stub both
|
||||
`fetch` and `OfflineAudioContext` — see the test in `371920695`.
|
||||
6. **React ignores a plain `input.value = x`** in tests. Use the native setter
|
||||
(`Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, "value").set`)
|
||||
then dispatch `input`. See `propertyPanelFxControls.test.tsx`.
|
||||
7. **The `fallow` pre-commit gate already fails on this stack** — every commit
|
||||
here and on `wa-18` used `--no-verify`, as the previous 49 did.
|
||||
8. **Two vitest runs racing on this machine** produce spurious 5 s timeouts in
|
||||
unrelated PropertyPanel tests. Do not chase them; re-run alone.
|
||||
9. **The timeline store does not refresh after a delete** — the deleted clip stays
|
||||
rendered until a page reload. Independent of this work, but it gates when
|
||||
store-driven reconciliation fires.
|
||||
|
||||
---
|
||||
|
||||
## 7. Commands that work here
|
||||
|
||||
```bash
|
||||
# tests — use the package's own binary, NOT bunx vitest
|
||||
cd packages/studio && ./node_modules/.bin/vitest run
|
||||
cd packages/core && ./node_modules/.bin/vitest run
|
||||
cd packages/engine && ./node_modules/.bin/vitest run src/services/
|
||||
|
||||
# after ANY change to core types or the FX stub that studio/engine read
|
||||
bun run --filter @hyperframes/core build
|
||||
cd packages/core && bun run build:audio-fx-runtime
|
||||
|
||||
bunx tsc --noEmit -p packages/studio # and -p packages/core, -p packages/engine, -p packages/lint
|
||||
bunx oxlint packages/studio/src packages/core/src packages/engine/src
|
||||
bunx oxfmt <files>
|
||||
|
||||
# commits need --no-verify (§6.7)
|
||||
```
|
||||
|
||||
## 8. Fixture
|
||||
|
||||
`~/src/recap-stitch/index.html` — four narration clips (`narration-1..4`) on
|
||||
tracks 11–14, `music-bed` carving all four, `sfx-explosion` on track 9.
|
||||
`hyperframes lint`: 0 errors, 3 pre-existing warnings.
|
||||
|
||||
Backups: `index.html.bak` (pre-split original), `index.html.pre-shared-rows`
|
||||
(before the shared-row work — note this one has the clips on separate tracks).
|
||||
|
||||
To exercise **shared automation lane rows**, point all four narrations at one
|
||||
`data-track-index`; they are currently one per row.
|
||||
Reference in New Issue
Block a user