ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-11-07 15:27:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
5,605 Commits 1,574 Branches 630 Tags
sync/hyperframes-codegen-c38641ae
Commit Graph
4 Commits
Author SHA1 Message Date
Jerrai e1925d4317 feat(audio): composition master bus (data-fx-chain on the root) (#5420) 
* feat(audio): composition master bus (data-fx-chain on the root)

A clip can belong to only one audio group, so there was no way to put one
effect, such as a limiter, across the whole mix. The data-fx-chain on the
top-level composition root now runs once over the sum of every clip and every
group, after each group's own chain and fader. No new attribute: a sub-composition
root's data-fx-chain was never applied to anything and still is not.

Render: when a master chain exists, mixAudioTracks writes a float WAV instead
of AAC, applyAudioFxChain runs the chain over it, and ffmpeg encodes AAC last,
so the chain reads unclipped samples. An unreadable chain is a mix failure, as
for a group. Preview: the transport splices the chain between the clips' sum and
the monitor, re-anchored per play generation and watched for edits.

Only the top-level root is read. No master automation: the root's data-automation
is not read as the bus's. No lint rule: data-fx-chain is valid on clips, groups
and roots, and the one place it does nothing (a sub-composition root) has always
been silent.

— Jerrai

* fix(core): preview routes every audible video through the master bus

A root data-fx-chain is attached only inside the Web Audio graph, but a plainly
audible <video> joined that graph only for its own chain, group, automation or a
volume over 1. So preview limited the music and left the talking head dry, and a
composition whose only audio is video got no master chain at all, while export
sums every audible element into the master. A master chain on the top-level
composition root is now a reason to join, the same as a video's own chain.
Elements under a root with no chain keep native output. Sub-composition roots
are still ignored.

— Jerrai

* fix(audio): master bus review round 2 (temp dir, tests, docs, Studio meter)

- audioMixer: a fatal FX error in the master pass or a group bus left the
  work directory (the full float sum, about 1.4 GB an hour) on disk. The
  final mix now runs in try/finally and the group rethrow removes it first.
- Tests: the "once over the sum" case now uses a limiter then a gain, so a
  per-clip chain (-10.4) and a doubled one (-18) both fail it; a new case
  proves the sum stays float until the chain reads it; the fader case uses
  a group volume of 0.5 so the fader's position is exercised; the work-dir
  removal is covered for both paths.
- Docs: "Three rules exist" (the master-bus lint rule is gone), a bad master
  chain stops the render with audio_processing_failed, a sub-composition
  file opened on its own has its root chain as the master bus, and a seek
  clears a group or master limiter.
- Studio: TimelineElement.masterChain carries the root's data-fx-chain, so
  the program meter shows for a video-only composition whose only audio
  routing is the master chain, as preview now plays it.
- Comment blocks over four lines cut; webAudioTransport.setRate split so
  its CRAP score is under the Fallow threshold.

— Jerrai

* fix(audio): master chain reaches tracks the render adds; honest plain-limiter copy

- Master chain: it was carried on each parsed AudioElement, so a track the
  producer adds itself (appendAutoDetectedVideoAudio, the browser-discovered
  audio in probeStage) never had it, and a composition whose only audio was
  such a track exported with no master chain, no ceiling and no warning. It is
  now a composition property: parseMasterFxChain(html) at compile, carried on
  CompiledComposition and CompositionMetadata, passed to processCompositionAudio
  by runAudioStage (in-process and distributed plan). AudioElement.masterFxChain
  is removed so there is one source. Each forward (compile stage, local
  orchestrator, distributed plan) has a test that fails when it is dropped.
- Plain limiter: it follows the level with no lookahead and Level after is
  applied past the ceiling, so it cannot promise "nothing gets past" or "never
  clips". Studio copy (description, panel text, knob label, summary) and the
  hyperframes-audio skill now say a fast or inter-sample peak can pass it and
  point to the true-peak limiter for a delivery limit. A copy test guards it.

— Jerrai

* fix(producer): keep master bus comments within the ratchet; run the plan test with the ffmpeg tests

Drop three one-line field comments that raised their files' comment share,
and list plan.masterBus.test.ts with the integration tests because plan()
reads the ffmpeg version.

— Jerrai

* chore(ci): repin the producer test-classification source in the reachability map

The reachability map pins the hash of test-classification.mjs, which now
lists one more integration test. The map's producer lane command already
covers every test under src, in both the unit and integration matrix lanes.

— Jerrai
 2026-10-11 19:07:00 +00:00
Jerrai 3746703ea7 feat(core): truepeak, a lookahead true-peak limiter FX (#5419) 
* feat(core): add a lookahead true-peak limiter FX (truepeak)

The existing limiter is an envelope follower with no lookahead and no
inter-sample detection, so a -14 dB ceiling can leave the true peak several dB
over. truepeak estimates the true peak on a 4x polyphase interpolator, takes the
minimum required gain over a lookahead window and smooths it with a boxcar of
the same length, so the gain is already down when the peak leaves the delay
line. Release is exponential and can only lower the gain further; channels
share one gain.

The offline render lengthens the OfflineAudioContext by the chain's latency and
drops the lead-in, so a limited clip stays sample-aligned with its source.
Preview plays the lookahead (about 3.3 ms at the default).

— Jerrai

* docs(audio): truepeak ceiling is a 4x estimate; measure it with an ideal interpolator

The unit test's meter was a 96-tap Kaiser sinc, which rolls off near Nyquist and
reads full-band noise about 0.5 dB low, so the PR reported a 0.3 dB broadband
overshoot. Against an ideal 16x interpolation (zero-padded FFT, cross-checked with
a 16x soxr resample) hot full-band white noise ends 1.2 to 1.7 dB over the ceiling
(median 1.4 over 28 runs of 10 to 30 s). ffmpeg's ebur128 true peak is also a 4x
meter and reads the same files 0.1 dB over.

- The test meter is now the FFT interpolation. Tolerances are what it measures on
  each fixed signal: tones 0.01 dB (they land on the ceiling), the dense program
  0.5 dB (reads 0.445, the old 0.4 passed only because the meter read low), and a
  new full-band noise case at -1, -6 and -14 dBTP with 1.35 dB (reads 1.09 to 1.31).
- audio-effects.mdx and the hyperframes-audio skill say the ceiling is a 4x
  estimate, give the measured overshoot and recommend a 2 dB margin under a hard
  delivery limit.
- skills-manifest.json regenerated.

— Jerrai

* fix(core): truepeak UI copy no longer promises a hard ceiling

The registry description and the rack summary said the true peak is never above
the ceiling. The limiter holds a 4x estimate, and full-band noise can land up to
about 1.7 dB above it, as the docs now say. The summary reads "held near", the
description names the estimate and the overshoot, and the rack blurb says "holds
... to a ceiling". No skill file changed, so the skills manifest is unchanged.

— Jerrai

* fix(core): Studio truepeak copy states the 4x-estimate margin

The ceiling control was labelled "Never exceed" and the module said the mix
"must not clip", so a Studio user never saw that dense full-band content can end
up to about 1.7 dB above the ceiling. The label is now "Target ceiling" with a
hint that names the 4x estimate and the headroom to leave, the blurb says
"toward a target ceiling", the rack summary says peaks are pulled down toward the
value, and the registry hint no longer says "highest true peak allowed". The
audio skill's "hard dBTP ceiling" wording is gone and the skills manifest is
regenerated. A copy test keeps the promise out.

— Jerrai

* fix(core): truepeak clears on a seek, and the review nits (fallow, comments)

- Unexport TRUE_PEAK_OVERSAMPLE, TRUE_PEAK_DETECTOR_DELAY and
  truePeakLookaheadSamples: nothing imports them (Fallow audit).
- A persistent group bus keeps its chain across a seek, so the limiter's
  delay line and held gain carried over. The worklet takes __hfReset, the
  graph exposes FxChainHandle.reset(), and reanchor() calls it.
- The processor-loading harness is shared by the truepeak and worklet tests
  (it was a 12-line clone); the test FFT no longer repeats audioCarve's
  bit-reversal; two test bodies are under the CRAP threshold.
- Comment blocks cut to four lines or fewer (Comments check).
- Docs state the preview delay in ms.

— Jerrai
 2026-10-10 18:28:21 +00:00
Vance IngallsandClaude Opus 5.5 3c299acc7f feat(core): audible videos in preview web audio, audio groups, meters and carve (2/8) (#4814) 
* feat(core): add isAudibleVideoElement predicate and audible media selector

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(runtime): route processed audible videos through web audio in preview

An audible <video> joins the Web Audio schedule, hidden-toggle and
group-capture passes when it needs a gain el.volume cannot express: a
boost above unity, an fx chain, automation, or an audio group. A plain
audible video at unity stays on native output. The clock master stays
audio-only; a video whose capture fails stays on native output (never
decoded, never failed closed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(audio): let audible videos join audio groups in preview and render

resolveAudioGroups, audioGroupOf, group render ids, the engine mixer and
the audio_group_no_members lint rule accept an audible <video>. A hidden
group drops its video members from the mix; muted videos stay inert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(editing): offer the audio fx section on audible videos

EditableElementFacts gains hasAudio; studio and sdk derive it from muted and data-has-audio.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(studio): show audio meters when a project has an audible video

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(studio): offer audible videos as carve voices

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(studio): let audible video clips join audio groups

Drops the video refusal; grouping and group audition count audible videos.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(engine): honor hidden audio groups for auto-detected video audio

Muted videos that declare data-has-audio are no longer mixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(studio): keep authored src, loop and honest status when keeping cutout sound

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(core): treat only data-has-audio=true as audible in runtime and lint

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(core): trim the web audio route and group-key comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(runtime): capture ramped clips so a group's bus carries them in preview

A rate lane skipped Web Audio entirely, so a ramped video or audio clip in a
group played native output without the group's fader, mute or FX while the
export applied them. Only the decoded-buffer fallback cannot follow a rate
curve; the media-element capture can, so the skip now guards that fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(studio): show meters only for videos Web Audio carries

A plain audible video at unity stays on native output, which the meters
cannot read, so it no longer advertises them. The runtime's join rule and the
meter gate now share audibleVideoNeedsWebAudio (core/audible-video).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-02 02:28:30 +00:00
Vance Ingalls 0eca7b1e0a docs(audio): document the audio effects system (#3420) 
* docs(audio): document the audio effects system

The effects feature had no documentation at all — the only mentions anywhere in
docs/ were changelog entries. This adds the three pages the color-grading feature
already has, one per audience, and closes a gap in the existing audio guide.

- prompting/audio-effects — a new Level 5 chapter on asking for a mix in
  symptoms rather than in filters, with the voiceover carve as the headline, the
  level-before-depth check, groups, and the three requests that have no honest
  answer (de-essing, noise removal, tone matching).
- studio/audio-effects — the rack, presets by symptom, the carve module, groups,
  the deliberately asymmetric mute/solo, automation lanes, and troubleshooting.
- reference/audio-effects — the contract: all four attributes, every effect and
  parameter range, why some parameters cannot be automated, the 19 presets, the
  five jobs and five one-knob profiles, carve semantics, the group model, the
  render bus, preview/render parity, and the three lint rules.

Also points the existing "duck music under important speech" advice in
guides/voice-and-audio at the carve, which does that properly and was never
named there.

Facts verified against source rather than the shipped skill prose: the panel
section is "Audio FX" (the neighbouring "Effects" section is visual effects, and
an earlier draft of this page named the wrong one), MAX_AUTOMATION_POINTS is 512,
fromPreset carries a preset id rather than a boolean, the leveller targets the
track's own 80th percentile, group mute reaches the render while solo never
leaves Studio, and the CLI carve currently writes clip ids.

No screenshots in this pass, and no placeholders for them either — the pages
carry concrete markup and tables instead. Screenshots of the rack, the carve
module, and a group row would each earn their place later.

mint validate and mint broken-links both pass.

* docs(audio): give audio its own Studio group, split by task

Review feedback: the audio page did not belong in Studio / Edit. That group is
already one task per page — canvas, timeline, animation, captions — and a single
page covering the rack, the carve, groups, and automation lanes was four tasks
bundled together and dropped in beside them.

Studio now has an Audio group holding four task pages:

- Effects and presets — the rack, presets by symptom, adding single effects in a
  working order, the one-knob controls, Even Out Levels
- Voiceover carve — its own page, because it is the feature people come for
- Groups, mute, and solo — including why mute reaches the export and solo cannot
- Automation lanes — drawing envelopes, the shape menu, and which parameters
  cannot move at all

Not a new top-level tab: the tabs here are audience-scoped (Guides, Studio,
Catalog, Developers), so a feature tab would be the only one of its kind and
would strand the prompting chapter out of its Level 5 sequence and the reference
page out of Developers.

Repointed the deep link in guides/voice-and-audio at the carve's own page.

Lane interactions verified in source before documenting: the right-click menu
offers Ramp up, Ramp down, Swell, Dip, and Simplify (which needs three points).
Deliberately not documented: "clicking a lane label reveals it in the rack",
which is not on main.

mint validate and mint broken-links both pass.

* docs(audio): fix the four review blockers

All four verified in source before fixing; the review was right on every count.

**The registry was incomplete.** `pitchshift` ships at `audioFx.ts:509-535`
(`semitones` −12–12, `mix` 0–1, worklet-backed so neither automatable) and was
missing entirely — sixteen effects, not fifteen. It also joins the worklet list,
so five effects expose no automatable parameters rather than four.

**Three presets were missing.** `chipmunk`, `giant`, and `monster` ship at
`audioFxPresets.ts:336-357`, all built on `pitchshift`. Twenty-two presets, and
Character holds ten. Fixed in both the reference table and the Studio list.

**The copyable markup contradicted the warning above it.** The page said
`carve.mjs` only finds double-quoted attributes and then gave three
single-quoted examples — copying the chain example would make a later carve miss
the existing chain and overwrite it. All three are now double-quoted with
`&quot;`, each followed by its unescaped reading so it stays legible.

**The attribute table over-claimed.** `data-audio-group` is a plain id, not JSON,
and is ignored on `<video>`; the other three also live on `<hf-audio-group>` for
a group. The table now carries shape and valid host per attribute.

**The automation contract was wrong on two axes.** A clip lane's `t` is
clip-relative but a GROUP lane's is composition time, because a group has no
`data-start` (`webAudioTransport.ts:337-342`, `audioMixer.ts:1311-1344`) — both
pages now split the two clocks. And `volume` is not 0–1: the ceiling is
`MAX_AUDIO_GAIN`, +12 dB or about 3.981 (`audioGain.ts:8-9`), so a boosting lane
is valid and documented.

**Current-main drift.** #3416 is merged, so the CLI now records the voices'
shared group when it is safe and falls back to clip ids when that group contains
the bed or a music/SFX member. Documented, including why neither refusal shows
up on the run that writes it, and rebased onto main.

mint validate and mint broken-links both pass.

* docs(audio): name the real add-menu family, and finish propagating pitchshift

Second review round. All three findings were my own incomplete propagation — I
corrected the reference for `pitchshift` last round and left the reader-facing
pages behind it.

**The add-menu family is `Time`, not `Space`.** `propertyPanelFxAddMenu.tsx:22-28`
labels the four groups Filters / Dynamics / Non-linear / Time, and the time group
holds pitchshift, delay, chorus, phaser, and reverb. The Studio page sent readers
looking for a group that does not exist. It is now a table naming the family and
its contents, and the reference's "Time — space and width" heading is retitled,
since that description stopped covering the family the moment pitch shift joined
it.

Also from the same file: the menu offers the named jobs in place of a bare
`peaking`, because picking `peaking` is picking a machine and leaving the real
decision — which range — for afterwards. Worth saying on the task page.

**Pitch shift was missing from both no-automation lists** that a reader actually
follows — `studio/audio-automation` and, unflagged but the same defect, the
prompting chapter. Five worklet effects in all four places now. Called out
explicitly on the Studio page, because a rising pitch is exactly the thing
someone reaches for a lane to do, and the lane will not report that it cannot.

**Narrowed the group-metadata sentence.** "The other three are JSON, and on a
group they live on `<hf-audio-group>`" swept in `data-fx-carve`, contradicting the
table directly above it. Only `data-fx-chain` and `data-automation` are group
metadata.

mint validate and mint broken-links both pass.
 2026-08-22 05:49:35 -07:00