mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-10-30 09:16:55 +00:00
Miguel Ángel
5cbef29d3a
* feat(cli): download Parakeet with models install parakeet and transcribe with it `hyperframes models install parakeet` installs the sherpa-onnx runtime from npm and the Parakeet TDT 0.6B v3 int8 model from Hugging Face, pinned to a revision. Every file is checked against its pinned size and sha256 under a temp name before it is renamed into place. Re-running it is a no-op. On Linux with glibc older than 2.32 it refuses up front. `transcribe --engine auto|parakeet` now prefers that engine when it is installed, then an installed parakeet-mlx runner, then whisper. Audio is cut into about 60 s windows at the quietest 100 ms near each cut, and decoding runs in a child process. If the child cannot load the runtime or crashes, transcribe falls back with one stderr line saying why. The JSON output and transcript.json shape are unchanged. * fix(skills): report the real word count and engine from transcribe fallbacks The media-use transcribe script read `words` from the CLI's transcript, which is a flat array, so its whisper path always reported an undefined word count. The embedded-captions script labelled every CLI transcript whisper.cpp even when `--engine auto` ran Parakeet; it now uses the engine the CLI reports. * fix(cli): pin the sherpa-onnx native package to the runtime's exact version * test(cli): type the sherpa install test doubles * fix(cli): transcribe audio with no speech as 0 words instead of failing Parakeet writes an empty word list for a music bed, a silent intro or a very short clip, and loading it failed with "Unrecognized JSON transcript format". An empty list is now read as a transcript with no words, as whisper's empty output already is. A non-empty list still has to look like words. * fix(cli): re-decode a Parakeet window that skipped the start of its speech At some window lengths the model silently drops the first clause: on a looped speech clip, 4 of 167 lengths between 55 and 60 s, and a 61.91 s window, lost their first 9 words. A window with a token gap of at least 1 s that is as loud as speech is decoded again with 0.5 s of leading silence, and the result with more tokens wins. Clean windows are unchanged. Decoding with 1.5 s of overlap before each cut was measured too and was worse on 573 s of audio (2.62% WER against 0.98%), so it is not used. Empty audio no longer makes a zero-length window, which aborted the decoder. * fix(cli): stop Parakeet on Ctrl-C and name the repair when its install breaks - Ctrl-C during a Parakeet transcribe now stops with exit 130 instead of starting whisper, and a SIGTERM to the CLI also stops its decode child. - Ctrl-C during models install parakeet aborts the download and the npm install, keeps no partial file, and exits 130. A staging dir left by a killed install is swept by the next run. - When Parakeet fails under --engine auto, the warning leads with the Parakeet error and names hyperframes models install parakeet. If the fallback fails too, the Parakeet error is reported rather than whisper_unavailable, so --optional no longer hides a broken install. - An explicit --engine parakeet no longer falls back: it fails with the Parakeet error and the repair command. - The install refuses up front on systems with no native package (naming the six supported ones) and on musl, instead of a raw npm error. - 24-bit and float WAV go through ffmpeg, since sherpa-onnx reads only 16-bit PCM. The decode timeout scales with the audio length. * docs(cli): point Parakeet setup at models install parakeet The transcribe docs and the media-use references named parakeet-mlx as the only way to get Parakeet. They now name hyperframes models install parakeet, and the CLI reference gets a models section. * test(cli): cover the npm cancel signal and decode timeout, and fix a doubled period The Parakeet failure line printed "failed.. To repair" when the decoder's own message ended in a period. Tests now also pin that Ctrl-C reaches the npm install and that the decode timeout scales with the audio length. * fix(cli): report unreadable audio as an input error, not a Parakeet failure Prepare the WAV once before the Parakeet fallback and hand the same file to whisper, so ffmpeg runs once. Ctrl-C while ffmpeg runs now ends as a cancel (exit 130) instead of starting whisper. * fix(cli): recover a clause Parakeet drops after a pause by decoding the gap alone Each loud gap without tokens is decoded on its own with 0.5 s either side, and only its new tokens are spliced in. The padded whole-window retry stays as the fallback. A window with no tokens at all is no longer decoded twice. * perf(cli): size the decode timeout from the WAV header instead of reading the file * docs(cli): say a broken Parakeet install still fails transcribe --optional * fix(cli): treat Ctrl-C during audio preparation as a cancel and drop the partial WAV ffmpeg traps SIGINT and exits 255 rather than dying by the signal, so that status now counts as a cancel unless the timeout fired. A failed or stopped ffmpeg removes the temp WAV it was writing. The decode timeout now sizes a prepared WAV from its file size, so a large header can no longer hide its length. * fix(cli): splice a re-decoded gap by whole words so its neighbours are never doubled The gap decode splits edge words into other tokens than the window decode did. Match neighbours as whole words within 0.3 s of each gap edge, add only whole words that start inside the gap, and decode the gap behind 0.5 s of silence like the padded retry. * fix(cli): count a cancel only when ffmpeg says it received a signal ffmpeg 7 and later also exit 255 when a file is not permitted (EPERM), so exit 255 alone read a permission error as a cancel. ffmpeg's stderr is now captured, a cancel needs its received-signal line, and any other failure reports ffmpeg's last line. The output path is passed to the shared runner explicitly. * fix(cli): keep a spliced gap word that has no leading space The filter dropped such a word; it is now kept and given its word boundary. * refactor(cli): trim three long comments in the Parakeet decoder and installer * fix(cli): repair a Parakeet runtime missing its native binary, and pick Parakeet by language The runtime dir is per platform and arch. Installed now needs the runtime manifest and the native sherpa-onnx.node; a half-installed dir is removed before npm runs, since install() keeps any dir holding the runtime manifest. The decode worker flattens a multi-line loader error onto its prefixed line and stops between windows once orphaned. transcribe opens its cancellation scope before audio prep, so Ctrl-C under --json cleans up. auto uses Parakeet only for its 25 languages and only where the host can run it; --engine parakeet with another language fails. * fix(cli): sweep reused-pid staging, name failed model downloads, keep ffmpeg's cause sweepStaleStaging also removes a staging dir older than 6 hours whose pid is alive, since containers reuse pids. A failed model download names the file, the host and a retry. runFfmpeg keeps ffmpeg's stderr on its error, so transcribe shows "Output file does not contain any stream" for a video without audio, and reads a cancel only from ffmpeg's own signal line. * fix(skills): make media-use --engine whisper run whisper and report the engine that ran The whisper path now passes --engine whisper when asked, or --engine auto, and reads the engine from the CLI's --json line, so a CLI Parakeet run is no longer labelled whisper. Docs state the language rule, the real repair behaviour and the platforms Parakeet runs on. * test(cli): pin that a long Parakeet decoder error is kept whole up to a bound * fix(cli): report a broken Parakeet runtime with the repair instead of skipping it * refactor(cli): name the staging age limit instead of commenting it * fix(skills): let media-use --engine parakeet use the CLI Parakeet when parakeet-mlx is absent * fix(cli): repair a Parakeet runtime that fails to load, and read Ctrl-C after whisper models install parakeet now loads the runtime in a child process, the way the decode worker does, and reinstalls it when that fails, so a missing shared library, a corrupt binary or missing runtime JS is repaired. It reports changed: false when the runtime loads. transcribe keeps its cheap manifest check. After a failure, transcribe lets the event loop turn once before reading the Ctrl-C scope, since a signal that landed during the synchronous whisper fallback is only handled then. * fix(cli): read Ctrl-C from the whisper child, and re-check the Parakeet runtime after a repair A whisper fallback killed by SIGINT or SIGTERM is a cancel (exit 130), the same rule decode and runFfmpeg use; the loop-turn wait is gone. The runtime probe treats a child stopped by Ctrl-C as a cancel and keeps the runtime; a reinstall is probed again and fails with the loader's error if it still does not load. media-use keeps the CLI's last lines, stdout included, when hyperframes transcribe fails. * fix(cli): give the user-stop rule one owner, SIGHUP included, and end a cancelled install at 130 stoppedByCancelSignal in renderCancellation.ts uses the scope's own signal list (SIGINT, SIGTERM, SIGHUP) and is false whenever Node stopped the child (a code, killed, or a spawn error). decode, runFfmpeg, the runtime probe and the whisper fallback all route through it, so a probe killed by SIGHUP keeps the runtime. isWhisperTimeoutError now keys on ETIMEDOUT only, which Node sets on execFileSync's own timeout, so a SIGTERM reaches the cancel rule. A cancelled models install sets exit 130 and returns instead of failing, and the probe says what went wrong (timed out, crashed, no output). * refactor(cli): let the stop-rule helper name carry its meaning
67 lines
4.2 KiB
Markdown
67 lines
4.2 KiB
Markdown
# Credits
|
|
|
|
## Prior art
|
|
|
|
HyperFrames was inspired by prior work in the browser-based video rendering space.
|
|
In particular, we want to acknowledge:
|
|
|
|
- **[Remotion](https://www.remotion.dev)** pioneered the approach of using a
|
|
headless browser + FFmpeg `image2pipe` pipeline to turn web primitives into
|
|
deterministic video in the JavaScript ecosystem. Several of HyperFrames'
|
|
architectural ideas — ordered async barriers for parallel frame capture,
|
|
multi-host port availability probing for dev servers, and the broader shape
|
|
of a "render HTML to video" CLI — were informed by studying how Remotion
|
|
approaches these problems.
|
|
|
|
All code in this repository is independently implemented and distributed
|
|
under the [Apache 2.0 License](LICENSE). HyperFrames is not affiliated with
|
|
Remotion.
|
|
|
|
## Thanks
|
|
|
|
Thanks also to the authors and maintainers of the open-source projects
|
|
HyperFrames builds on, including Puppeteer, FFmpeg, GSAP, Hono, and the
|
|
broader Node.js ecosystem.
|
|
|
|
## Third-party licenses
|
|
|
|
- **[mediabunny](https://github.com/Vanilagy/mediabunny)** — media toolkit used
|
|
in the studio for fast metadata extraction from file headers. Licensed under
|
|
the [Mozilla Public License 2.0 (MPL-2.0)](https://mozilla.org/MPL/2.0/).
|
|
- **[Parakeet TDT 0.6B v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3)** by NVIDIA — the
|
|
speech recognition model `hyperframes models install parakeet` downloads, in the int8 ONNX export
|
|
from [csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8](https://huggingface.co/csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8).
|
|
Licensed under [Creative Commons Attribution 4.0 (CC-BY-4.0)](https://creativecommons.org/licenses/by/4.0/).
|
|
It runs on **[sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)**, Apache-2.0.
|
|
- The seven 3D-motion catalog pieces (`canopy-part-title`, `glass-shard-title`,
|
|
`code-slice-hero`, `frost-sequence-camera-orbit`, `cuboid-carousel`,
|
|
`orbit-card`, `wireframe-portal-title`) are contributed with their author's
|
|
permission under this repository's license. What they vendor or load, by upstream:
|
|
- **[three.js](https://threejs.org)**, MIT. Vendored as r185 in `frost-sequence-camera-orbit`,
|
|
`cuboid-carousel` and `orbit-card` (`Three-LICENSE.txt` beside the copy); bundled as r181 inside
|
|
`glass-shard-title`'s `glass-main.js`; loaded from the jsDelivr CDN by `canopy-part-title`
|
|
(0.170.0) and `wireframe-portal-title` (0.181.2).
|
|
- **[GSAP](https://gsap.com)** 3.14.2, under the [GSAP Standard License](https://gsap.com/standard-license)
|
|
(not an OSI open-source licence). Vendored in `code-slice-hero`, `cuboid-carousel`,
|
|
`frost-sequence-camera-orbit` and `orbit-card` with `GSAP-NOTICE.txt`; loaded from the CDN by
|
|
`glass-shard-title`, `canopy-part-title` and `wireframe-portal-title`.
|
|
- **[three-mesh-bvh](https://github.com/gkjohnson/three-mesh-bvh)** 0.9.14 and
|
|
**[opentype.js](https://github.com/opentypejs/opentype.js)** 2.x (range in the block's
|
|
`source/package.json`), both MIT, bundled into the frost block's `frost.js`.
|
|
- **[Clipper](https://sourceforge.net/projects/jsclipper/)** 6.4.2 (JavaScript port of Angus
|
|
Johnson's Clipper), Boost Software License 1.0: build-time source of `frost-sequence-camera-orbit`
|
|
and loaded from the CDN as `clipper-lib` by `wireframe-portal-title`.
|
|
- **[d3-delaunay](https://github.com/d3/d3-delaunay)** 6.0.4, ISC, loaded from the CDN by
|
|
`glass-shard-title`.
|
|
- **[Geist](https://github.com/vercel/geist-font)**, **[Archivo](https://github.com/Omnibus-Type/Archivo)**
|
|
and **[Cormorant Garamond](https://github.com/CatharsisFonts/Cormorant)**, SIL Open Font License 1.1,
|
|
vendored with their licence text; `canopy-part-title` (Gelasio) and `cuboid-carousel` (Inter) load
|
|
their fonts from Google Fonts at run time.
|
|
- **[Ferndale Studio 01](https://polyhaven.com/a/ferndale_studio_01)** HDR from Poly Haven, CC0
|
|
(`glass-shard-title`); it needs no licence text.
|
|
- Origin to be confirmed with the author: `canopy-part-title/assets/leaf-surface-color.webp` and
|
|
`leaf-surface-normal.webp`, `glass-shard-title/assets/matcap-1.png`,
|
|
`frost-sequence-camera-orbit/assets/shards-atlas.png`,
|
|
`frost-sequence-camera-orbit/assets/textures/ice-inclusions-generated.png` and
|
|
`frost-sequence-camera-orbit/assets/textures/bluenoise64.png`.
|