Commit Graph
9 Commits
Author SHA1 Message Date
Michael Wang c4439890cf fix(cli): escape audio and media-use output, pin model files (#8013)
* fix(cli): audio and media-use print control characters from project files as visible escapes

hyperframes audio, audio --read and media-use resolve (including
--candidates) printed file names, element ids, transcript words, model
descriptions and media manifest fields as they were. They now pass
through terminalText, so a control character shows as a visible \xNN
escape. The media-use engine runs without the CLI's packages, so it
carries a small copy of the helper. --json output is unchanged.

* fix(engine): playlist check refuses drive-relative and other colon-prefixed entries

ffmpeg reads an HLS or ffconcat entry with a colon before its first
separator as a URL or a drive, and opens it apart from the list's
folder. The folder check read entries such as C:Videos\a.mp4 or c:x.mp4
as names inside the folder and approved them. It now refuses any entry,
raw or percent-decoded, whose colon comes before its first /, \, ? or #,
as it already refuses absolute and parent-folder paths. http and https
segments are still kept.

* fix(cli): background-removal and Kokoro TTS model files are pinned to a sha256

The u2net_human_seg weights and the Kokoro model and voices files were
downloaded from release URLs and used whatever bytes arrived, and any
cached copy was trusted because it existed. Each file now has a pinned
size and sha256. A cached copy is checked before it is loaded and
downloaded again when it does not match. A download that does not
match is discarded with a message that names the file.

* fix(engine): playlist check reads entries from the folder ffmpeg uses

ffmpeg resolves a list's entries against its path cut at the first ? or
#. A list at proj/h#z/list.m3u8 naming ../h#z/x.ts was judged against
proj/h#z and approved, while ffmpeg opened proj/../h#z/x.ts. Entries are
now judged against the same cut folder. A list in a folder such as
"Take #2" still passes when its entries resolve inside that folder.

* fix(cli): a pinned model's cached copy is checked and removed under the download lock

ensurePinnedFile removed a cached copy that failed its check before the
download lock was taken, so a concurrent run could delete a file another
run had just published. downloadFile now takes a keepExisting check and
runs it under the lock: a copy that passes is kept, one that fails is
removed and downloaded again. The mismatch error now says the file does
not match its expected checksum, and that a retry helps only if the
download was interrupted or corrupted.

* fix(cli): audio escapes each field's line breaks, and media-use escapes recipe and stats output

The audio formatters escaped their output line by line, so a newline
inside a src, warning or transcript still printed as a real line break.
Each output line is now escaped whole with terminalText, which shows a
newline inside a field as \x0a. media-use resolve also escapes its
recipe lines and its --stats lines, which print recipe names, paths and
missed intents.

* fix(cli): media-use error messages show control characters as visible escapes

resolve printed error messages as they were, and some name project or
cache data, such as the known recipe names when a recipe is not found.
Every error: line now passes through terminalText, as the success lines
already do.

* ci: the edit accuracy filter includes the CLI model file check

The CLI's pinned model download now imports whisper/modelFile.ts, so a
change to it must run the edit accuracy bench too.

* test(cli): pinned model tests use private temp folders and rename like downloadFile

The tests wrote fixtures under fixed names in the shared temp folder,
and their download stand-in checked for a file before replacing it.
Each test now works in its own mkdtemp folder, and the stand-in asks
keepExisting directly and renames the part file into place, as
downloadFile does.

* test(cli): the media-use terminalText copy is checked against core's

The media-use engine keeps its own copy of terminalText. A CLI test now
compares the two over U+0000 to U+00A0, so a change to either fails
until both match. keepsDest moves above downloadFile's doc comment,
which belongs to downloadFile.
2026-11-17 07:16:03 +00:00
Michael Wang 9701f0639c fix(cli): the audio caches never write or read through links in .hyperframes (#7825)
* fix(cli): the audio caches never write or read through links in .hyperframes

The read-audio cache, its stems, mix and descriptions, and the audio-kind cache
now use only real folders under the project's .hyperframes. A link or a file in
place of a cache folder stops the write with a clear message. Every cache file is
written to an exclusive temp file with a random name and renamed into place, so a
link at a temp or final name is replaced, never written through. ffmpeg writes the
voice-free mix into a private staging folder. Stem downloads use the new
downloadToFile replaceLink option. Cached files are read only when they are plain
files, and a stem list may only name files in its own folder.

* fix(cli): the kind cache reads no entry through a linked cache folder

readKindCache now checks that .hyperframes and audio-kind are real folders before
it reads an entry, as writeKindCache already does before it writes one.

* test(cli): read the replaced download before checking its type and mode

The replaceLink test now reads the file's bytes and lstat result once each, so
no read follows a check on the same path.
2026-11-15 17:31:16 +00:00
JerraiandJerrai b37db11225 feat(cli): audio --read --layer reads a one-layer file whole, no separation (#7345)
* feat(cli): audio --read --layer music|sfx|voice reads a one-layer file whole

A caller that knows a file holds a single layer can declare it: the file is
read whole as that layer with no stem separation and nothing uploaded.
Without the flag the read is unchanged. An unknown layer is rejected.

Consumer: heygen-com/experiment-framework#66322. Proposal OK'd by Rory
(C0BEH487Q7R, 2026-11-07).

Co-Authored-By: Jerrai

* chore(cli): drop added comments from audio read layer

Co-Authored-By: Jerrai
2026-11-07 03:25:46 +00:00
JerraiandXuanru Li acd5c30e60 feat(cli): read audio stems through your HeyGen account (#6993)
* feat(cli): audio --read organises the reading by stem

Measures where each stem is heard, leaves the drums' recurring hits to the
music while it plays, names the stem of each remaining effect, writes a
voice-free mix next to the stems, and describes each audible stem and the
whole recording with Gemini.

* feat(cli): audio --read reads stems as vocals, drums, bass and melodic

Piano, guitar and other are mixed into melodic.wav next to the stems; levels,
spans, descriptions, effects and pieces are read per group, and a group heard
for under two seconds or 5% of the recording is left out of the reading.

* feat(cli): audio --read describes the six stems and writes a BGM prompt from them

Stems are read one by one again; a stem heard under two seconds or 5% of the
recording is still left out. Each stem's description gives the times of its
changes and names the effects it hears. The overall and the BGM prompt are
written from those descriptions and the measured spans, not by listening to
the mix again; the BGM prompt orders the arrangement without seconds.

* fix(cli): audio --read writes no BGM prompt for a recording without music

Foley and effects in a rhythm are not music; the reading says so instead of a
prompt that would make a remix generate a track the reference never had.

* fix(cli): audio --read prompts name no example sounds

* fix(cli): describe audio through HeyGen and keep effects measured

* docs(cli): document measured effect events without sound kinds

* fix(cli): clear audio description review checks

Use portable stem path assertions and simplify response validation.
Keep auth-derived URLs out of cache hashes and trim redundant comments.

— Jerrai

---------

Co-authored-by: Xuanru Li <157947275+xuanruli@users.noreply.github.com>
2026-11-02 18:42:05 +00:00
Xuanru LiandJerrai 1df31d8717 fix(audio): a sound is voice only with a voice-range pitch, judged on the whole clip (#6653)
* fix(audio): a sound is voice only with a voice-range pitch, judged on the whole clip

Two ways the kind guess called a non-voice sound voice:

- A short hit, click or whoosh can have a speech-shaped envelope. classify() now also
  measures the share of 40 ms frames with a 70-500 Hz fundamental (YIN, moved from the
  CLI into core) and calls a sound voice only when that share reaches 0.15. Speech
  measured 0.22-0.43; impacts, clicks and whooshes 0.
- A range query judged a clip's kind on the slice it covered, so 1.7 s of a music bed
  read as voice. A film clip's kind is now judged on all of the clip that plays.

Even-paced speech (a 64 s TTS take) moves less at syllable rate and was called music;
with no beat and a voiced share of 0.25 or more it is now voice.

* chore(audio): keep each touched file's comment share at or under its base

The Comments check failed on run.ts, targets.ts and audioAnalysis.ts because the
PR added comment lines to files whose share may only go down. Drops the added
doc comments; no code change.

— Jerrai

* fix(audio): preserve clip evidence and stereo pitch during analysis

* fix(audio): bound and cache whole-clip kind analysis

* fix(audio): retain native-rate kind evidence when available

---------

Co-authored-by: Jerrai <jerrai@heygen.com>
2026-10-28 12:48:08 +00:00
Miguel Ángel cfef528d3a fix: phone videos with an undecodable first audio track check, preview and render (#6551)
* fix: phone videos with an undecodable first audio track check, preview and render

* fix(engine): a rate-lane clip bakes its ramp from the decodable audio track

* test: phone-clip tests end ffprobe options before the input
2026-10-27 13:09:02 +00:00
Xuanru LiandJerrai cf04452c0d fix(cli): audio --read reads music onsets, effect peaks, pauses and voice against music (#6546)
* fix(cli): audio --read starts music where its sound is sustained

A run of the bed is split where sound fills less than half of the surrounding
2 s, so a short tone and scattered clicks that bridge into the music no longer
pull its start earlier. Sparse stretches are left to effects.

* fix(cli): audio --read finds effects in each stem and levels them from the mix

A sound ends where the next one starts, so a run of kicks is not one long
riser. Effects are searched in the drums and in the bed outside music, each on
its own, so a click inside a tone is still found. Each effect's peak is read
from the decoded mix in dBFS, and the music reference is taken next to the
effect rather than under it, where its own leakage sits.

* fix(cli): audio --read measures the recording through each pause

Pauses carry the whole recording's level so the read can say what keeps
playing while the voice stops.

* feat(cli): audio --read prints peaks, what plays in pauses, voice vs music

Effect lines print the absolute peak and name the reference of the relative
figure; pause lines keep the pause kind and list the effects, music and
ambience under them; sentences over music print voice and music loudness and
which is louder. Read schema version 4.

* fix(cli): audio --read finds bed effects on the whole bed, not a blanked one

Blanking the bed wherever music or ambience played made every blanked
stretch that ended while sound was still audible look like a sound rising
out of silence. Fades and quiet bridges then printed as dings, hits and
whooshes. The bed is now searched whole, and each source carries the
spans it plays music or ambience in. A sound whose onset-to-peak overlaps
one of those spans belongs to it, so it is not an effect. The drums and
the bare mix carry no spans.

— Jerrai

---------

Co-authored-by: Jerrai <jerrai@heygen.com>
2026-10-27 11:08:00 +00:00
Xuanru Li 87baff8842 fix(cli): audio --read reports noise beds as ambience, not music (#6536)
* fix(cli): audio --read reports noise beds as ambience, not music

A stretch of the non-voice stems counted as music whenever it was above
-55 dB, so room tone and engine rumble separated into "other" printed as
a music piece. A stretch is now music only with a beat, clear tonal chroma
or more than one leading instrument; otherwise it is ambience with its
level in dBFS, called quiet under -45 dBFS, and breaks lie only between
two runs of music. The header lists only stems that carry sound, with
their level, instead of the separator's six fixed names.

* refactor(cli): audio --read gate carries its meaning in names, not comments

* fix(cli): keep ambience out of downstream music measurements

* fix(cli): measure music beat evidence within each stretch
2026-10-27 09:44:40 +00:00
Xuanru Li 2c1d9d50d2 feat(cli): read a whole recording as voice, effect and music lines (#6056)
* feat(cli): read a whole recording as voice, effect and music lines

`hyperframes audio <file> --read` turns an episode or reference video into
time-stamped lines an agent can read or grep: sentences with rate, pitch and
ending; pauses; sound effects grouped by identical sound, with band, attack,
tail and level over the music; and the music bed split at its cut-outs into
pieces that start, resume, return or are new.

Effects and music come from stems separated by the service at
HYPERFRAMES_AUDIO_STEMS_URL (vocals, drums, bass, guitar, piano, other).
Without stems, effects are read from the mix outside speech; with neither
stems nor a transcript, voice and effects are not read, and each case is
reported in a warning. Results are cached per file content.

* refactor(cli): name the service error parse instead of commenting it

* fix(cli): call HeyGen's audio services with the CLI's HeyGen credential

Stems for `audio --read` and the `api` analysis provider only ran when
HYPERFRAMES_AUDIO_STEMS_URL / HYPERFRAMES_AUDIO_API_URL and
HYPERFRAMES_AUDIO_API_KEY were set, so someone with HEYGEN_API_KEY or a saved
`hyperframes auth login --api-key` still got "not set" and had to know the
internal route.

Both now default to /v3/audio/stems and /v3/audio/analysis on the HeyGen API
(HEYGEN_API_URL when set) with the credential the other HeyGen commands use.
The URL variables still point at another service, which then needs its own
HYPERFRAMES_AUDIO_API_KEY, so the HeyGen credential never goes to a host it was
not issued for.

* fix(cli): keep audio on the machine unless --quality best is asked for

`hyperframes audio` defaulted to --quality best, so anyone with a HeyGen
credential uploaded their audio, and could be billed, for an ordinary beats or
--read run. tts, transcribe and remove-background never leave the machine
unasked.

The default is now local for both measuring and --read. best is unchanged and
still uploads to /v3/audio/analysis and /v3/audio/stems. When a capability goes
unanswered because a service was left out, or --read reads without stems, the
warning says --quality best would add it.

* fix(cli): bound stem transfers and preserve recording timestamps

* fix(cli): an empty audio service URL keeps the service off

HyperFrames Desktop blanks HYPERFRAMES_AUDIO_API_URL and its key while
the analysis route is behind a flag, but gives runs HEYGEN_API_KEY and
HEYGEN_API_URL for its gateway. Falling back to that credential would let a
run that passes --quality best reach the route anyway. An empty URL now turns
the service off, whatever credential is present.

* fix(ci): capture package name in release-age exclusion

(cherry picked from commit f9c6ba3573)

* fix(cli): resend the stems request with its key until the service answers

The stems service answers once the stems are made, which can outlast the
connection, and answers 409 request_in_progress to a resend while the first
request runs. A failed job's reason is failure_message.

* fix(cli): retry incomplete stem separations

* test: validate audio mock route handlers before dispatch

* test: trace premature Windows CLI runner exits

* test: remove temporary Windows runner diagnostics
2026-10-22 13:22:38 +00:00