mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-13 15:49:53 +00:00
fix(media-use): repoint the dead videogen tier, demote past unusable local models (#3509)
* fix(media-use): repoint the dead videogen tier, demote past unusable models
`LOCAL_MODELS.videogen`'s `large` tier named `dgrauet/ltx-2.3-mlx-bf16`, which
returns HTTP 401 and cannot be downloaded at all. It was not a dormant entry:
`rankedByPreference` sorts by descending `needs.ramMB` when no `rank` is set,
so the largest fitting tier is tried FIRST by design. Any machine clearing
32 GB *available* RAM selected the dead entry, `ltxVideoGenerate` caught the
failure and returned a bare `null`, and since `ltx.local` is last in
`["heygen.video", "ltx.local"]` and network providers are skipped under
`--local-only` (`registry.mjs:206`), local video generation failed outright
instead of falling back to the tier that works.
It survived review because the table landed with "live verification on a 24GB
M-series Mac" - and a 24 GB machine cannot select a 32 GB tier, so that entry
was unreachable on the only machine that validated it. The unit fixtures
inherit the same ceiling (`fittingSpecs` is 20000MB), so every existing test
exercised the medium tier alone.
Two changes:
1. Repoint to `dgrauet/ltx-2.3-mlx-q8` (reachable) and correct `sizeMB` from
45000 to 28800. Measured against the HF API: the q8 repo totals 87.5 GB,
and the registry's own targeted `--include` subset is 28.76 GB. That
matches the sibling q4 entry's convention (`sizeMB: 20000` vs a measured
19.48 GB subset), so 45000 was wrong under either reading. `--low-ram` is
added because the entry's own note calls it required at this tier's 32 GB
floor, and the invoke omitted it.
2. A repoint alone is one bad URL from a repeat, so add the missing recovery.
`selectModelLadder` returns every fitting model best-first;
`selectModel`'s pick is now defined as that list's head. All three sites
that previously selected exactly one model and failed terminally walk the
ladder instead, demoting past a tier that cannot run here - gated weights,
runner off PATH, an OOM at a tier that nominally fits:
- `ltx-video-provider.mjs` (videogen, the reported failure)
- `mflux-provider.mjs` (imagegen - same shape, and its 32 GB/64 GB tiers
are equally unverifiable on a 24 GB machine)
- `local-run.mjs` (tts/asr/upscale - `fish-speech` missing should still
get you Kokoro)
Every demotion is logged rather than silent, so a quietly smaller model is
never mistaken for the tier the machine nominally qualified for.
Also fixes the `install` string both videogen entries share: it ended at
`uv sync --all-extras`, which leaves the entry point in `.venv/bin`, so the
"`ltx-2-mlx` not on PATH" hint named a command that following the instruction
would not put on PATH.
The q8 tier is NOT live-verified - no 32 GB+ Apple Silicon machine was
available - and its notes say so. Shipping it unverified is safe precisely
because of change 2: a wrong tier now costs one failed attempt, not the whole
local path.
- Rames Jusso
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(media-use): report the real videogen download size, disclose it, discard failed partials
Addresses review feedback on #3509 (CHANGES_REQUESTED at 90df164a), plus the
follow-on ask to tell the user what a download costs before they accept it.
1. `sizeMB` described a targeted `--include` subset that no run ever gets.
Both videogen invokes pass a repo id to `--model`, and upstream
`resolve_model_dir()` (`ltx_pipelines_mlx/utils/_orchestration.py:35-40`)
calls `snapshot_download(repo)` with no `allow_patterns`, so the full repo
lands regardless of what was pre-fetched. Corrected to measured repo
totals: q8 87500 (87,511,991,375 B) and q4 59700 (59,686,429,583 B). q4 was
wrong the same way at 20000, so both are fixed together rather than leaving
one convention on each side.
My earlier claim that 28800 "matches the sibling q4 entry's convention" was
wrong in the way that matters: the convention itself described a subset the
runner does not honor. The file's own comment already said "blind
snapshot-downloads the lot (60 GB q4, 88 GB q8)" three lines above the
fields that contradicted it, and the original report measured it too ("the
q4 cache ended at 56 GB and q8 at 82 GB"), which reconciles exactly once
read as GiB: 59.69 GB = 55.6 GiB, 87.51 GB = 81.5 GiB. So the download is
the complete repo both times, not a partial fetch.
Removed the `--include` recipe rather than repairing it: it is ineffective
(the runner refetches at generate time) and insufficient (`--two-stage` is
"dev model + CFG at half-res, upscale, distilled LoRA refine" per upstream's
own help text, so it needs transformer-dev AND transformer-distilled AND
spatial_upscaler_x2; `--distilled` needs an upscaler too). The q4 tier
verified on a 24 GB Mac only worked BECAUSE the download is unfiltered.
2. Nothing told the user what they were agreeing to before a tool started
pulling tens of GB. `describeDownload()` in `specs.mjs` names the size and
the directory the weights land in, and checks free space with `statfs`
against that directory rather than cwd, since the weights do not land in
cwd. A tier that will not fit is still offered, with a plain statement that
it will not fit: hiding it would make a machine that could free up space
look like it has no large tier. Unknown free space reports as unknown, not
as zero. Wired into both providers' install hints, the `runLocalModel`
install payload (now carrying `sizeMB`), and `describeModelLadder`.
3. Each retry attempt mints its own timestamped temp path, so a partial
artifact from a failed tier was orphaned rather than overwritten, and a
lower tier then succeeding hid it. Both providers discard the partial before
demoting, guarded so a file that cannot be removed never masks the generate
failure it came from. Video is the material case: a partial mp4 is large.
`local-run.mjs` is deliberately unchanged here. Its `out` is caller-provided
and identical across attempts, so a partial is overwritten rather than
orphaned, and unlinking a path the caller named would be a footgun. The rule
the two providers follow is: clean up what you allocate.
Tests: 553/553 across `skills/**/*.test.mjs` (+15). New coverage pins the
cleanup (failed tier's partial removed, returned artifact survives, one discard
per attempt on the all-fail path, an unremovable partial still surfaces the
real failure) and the disclosure (cache-dir precedence, statfs walk-up to the
deepest existing ancestor, unknown-vs-zero, and the will-not-fit wording).
Every new guard mutation-tested: removing any one of them turns tests red.
- Rames Jusso
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
James Russo
co-authored by
Claude Opus 5
Claude Opus 5
parent
ee64c3b116
commit
8392e84a18
@@ -12,6 +12,10 @@
|
||||
// (quality that is NOT size, e.g. ASR), else by RAM footprint (the quality
|
||||
// proxy for generation). No fit -> recommend the CLI/cloud path.
|
||||
//
|
||||
// selectModelLadder() returns EVERY fitting model in that same order. Callers
|
||||
// that can retry walk it so ONE unusable entry (gated weights, a missing
|
||||
// binary, an OOM) demotes to the next tier instead of killing the local path.
|
||||
//
|
||||
// Picks reflect the 2026 research pass, verified live where noted.
|
||||
|
||||
export const CAPABILITIES = ["tts", "asr", "upscale", "videogen", "imagegen"];
|
||||
@@ -102,41 +106,49 @@ const MODELS = {
|
||||
},
|
||||
],
|
||||
videogen: [
|
||||
// 2026-07 X research pass + live verification on a 24GB M-series Mac.
|
||||
// 2026-07 X research pass + live verification on a 24GB M-series Mac -
|
||||
// which reaches the q4 tier only: a 24GB machine cannot select the 32GB
|
||||
// entry below it, so that tier's claims stay unverified until someone
|
||||
// runs it on a 32GB+ machine.
|
||||
// The Mac-local video story is LTX 2.3 on MLX via dgrauet/ltx-2-mlx (the
|
||||
// pipeline these weights were converted for; also powers Phosphene).
|
||||
// Wan 2.x MLX exists only as A14B conversions (too large for consumer
|
||||
// unified memory); revisit when a 5B Wan MLX conversion lands.
|
||||
// IMPORTANT: download the weights with a targeted include list first;
|
||||
// pointing tools at the repo blind snapshot-downloads all 60 GB:
|
||||
// hf download dgrauet/ltx-2.3-mlx-q4 --include \
|
||||
// transformer-distilled-1.1.safetensors connector.safetensors \
|
||||
// "vae_*.safetensors" audio_vae.safetensors vocoder.safetensors "*.json"
|
||||
// IMPORTANT: sizeMB below is the FULL repo, because that is what a run
|
||||
// actually downloads. Both invokes pass a repo id to `--model`, and
|
||||
// upstream resolve_model_dir() (ltx_pipelines_mlx/utils/_orchestration.py)
|
||||
// calls snapshot_download(repo) with no allow_patterns - so the whole repo
|
||||
// lands regardless of what you pre-fetched. A targeted `hf download
|
||||
// --include` subset used to be documented here; it was removed because it
|
||||
// is both ineffective (the runner refetches the rest at generate time) and
|
||||
// insufficient (--two-stage needs transformer-dev AND transformer-distilled
|
||||
// AND the x2 spatial upscaler; --distilled needs an upscaler too). The q4
|
||||
// tier verified below only worked BECAUSE the download is unfiltered.
|
||||
{
|
||||
id: "ltx-2.3-mlx-q4",
|
||||
tier: "medium",
|
||||
sizeMB: 20000, // distilled subset; gemma-3-12b-4bit text encoder adds ~7GB
|
||||
sizeMB: 59700, // full repo, measured 59.69GB; gemma-3-12b-4bit text encoder adds ~7GB
|
||||
needs: { ramMB: 16384, gpu: true },
|
||||
wordTimestamps: false,
|
||||
install:
|
||||
"git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras",
|
||||
'git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras && export PATH="$PWD/.venv/bin:$PATH"',
|
||||
invoke:
|
||||
"ltx-2-mlx generate --prompt {prompt} --distilled --low-ram --model dgrauet/ltx-2.3-mlx-q4 --width {w} --height {h} --frames {frames} --frame-rate 24 --output {out}",
|
||||
notes:
|
||||
"LTX 2.3 int4 on MLX. Verified on 24GB unified: 512x320 x 33 frames in ~19 min cold (incl. text-encoder download), t2v with audio. Dims must be multiples of 64. i2v, retake/extend, keyframe interpolation supported.",
|
||||
},
|
||||
{
|
||||
id: "ltx-2.3-mlx-bf16",
|
||||
id: "ltx-2.3-mlx-q8",
|
||||
tier: "large",
|
||||
sizeMB: 45000,
|
||||
sizeMB: 87500, // full repo, measured 87.51GB
|
||||
needs: { ramMB: 32768, gpu: true },
|
||||
wordTimestamps: false,
|
||||
install:
|
||||
"git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras",
|
||||
'git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras && export PATH="$PWD/.venv/bin:$PATH"',
|
||||
invoke:
|
||||
"ltx-2-mlx generate --prompt {prompt} --two-stage --model dgrauet/ltx-2.3-mlx-bf16 --width {w} --height {h} --frames {frames} --frame-rate 24 --output {out}",
|
||||
"ltx-2-mlx generate --prompt {prompt} --two-stage --low-ram --model dgrauet/ltx-2.3-mlx-q8 --width {w} --height {h} --frames {frames} --frame-rate 24 --output {out}",
|
||||
notes:
|
||||
"Full-precision two-stage pipeline (upstream production default). 32GB with --low-ram block streaming; 64-128GB Macs for long/HD runs (the 25s multi-scene spots seen in the wild).",
|
||||
"LTX 2.3 int8 on MLX, two-stage (upstream production default; higher quality than the q4 distilled tier). Replaced dgrauet/ltx-2.3-mlx-bf16, which is gated (HTTP 401) and cannot be downloaded at all. Costs an 87.5GB download against q4's 59.7GB - a real tradeoff, not a rounding difference. --two-stage is dev model + CFG at half-res, upscale, then distilled LoRA refine (upstream's own help text), so it needs transformer-dev + transformer-distilled + spatial_upscaler_x2; the full snapshot carries all three. --low-ram matches this tier's 32GB floor (block streaming); 64-128GB Macs for long/HD runs. NOT live-verified on a 32GB+ machine - the q4 tier below is the verified one.",
|
||||
},
|
||||
],
|
||||
imagegen: [
|
||||
@@ -247,6 +259,23 @@ function rankedByPreference(table) {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Every local model for a capability this machine can actually run, best-first
|
||||
* (same ordering as selectModel, whose pick is this list's head).
|
||||
*
|
||||
* Callers that can retry should walk the whole list: a table entry can be
|
||||
* unusable for reasons no spec check can see - weights pulled or gated behind a
|
||||
* login, the runner missing from PATH, an OOM at a tier that nominally fits. On
|
||||
* a single-select call any one of those fails the entire local path, because the
|
||||
* cascade cannot tell "this model is broken" from "nothing here fits you".
|
||||
* Demoting to the next fitting tier is almost always what the user wanted.
|
||||
*/
|
||||
export function selectModelLadder(capability, specs, { preferTier } = {}) {
|
||||
const table = tableFor(capability);
|
||||
const pool = preferTier ? table.filter((m) => m.tier === preferTier) : table;
|
||||
return rankedByPreference(pool).filter((model) => meetsSpecs(model, specs));
|
||||
}
|
||||
|
||||
/**
|
||||
* Pick the best local model the machine can run for a capability: the
|
||||
* highest-footprint model that fits the available-RAM budget (and GPU/VRAM).
|
||||
@@ -255,10 +284,8 @@ function rankedByPreference(table) {
|
||||
*/
|
||||
export function selectModel(capability, specs, { preferTier } = {}) {
|
||||
const table = tableFor(capability);
|
||||
const pool = preferTier ? table.filter((m) => m.tier === preferTier) : table;
|
||||
for (const model of rankedByPreference(pool)) {
|
||||
if (meetsSpecs(model, specs)) return { model, tier: model.tier };
|
||||
}
|
||||
const [model] = selectModelLadder(capability, specs, { preferTier });
|
||||
if (model) return { model, tier: model.tier };
|
||||
const smallest = table.reduce((a, b) => (a.sizeMB <= b.sizeMB ? a : b));
|
||||
return {
|
||||
recommend: "cli",
|
||||
@@ -280,6 +307,7 @@ export function describeModelLadder(capability, specs) {
|
||||
id: model.id,
|
||||
tier: model.tier,
|
||||
needsRamMB: model.needs?.ramMB ?? 0,
|
||||
sizeMB: model.sizeMB,
|
||||
fits,
|
||||
reason: fits
|
||||
? `fits (needs ~${model.needs?.ramMB}MB, ${budget}MB available)`
|
||||
|
||||
Reference in New Issue
Block a user