mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-07 18:26:17 +00:00
* feat(media-use): color grading — grade/lut resolve, smart-grade, grade-compare CLI Add color grading to media-use as first-class resolve types plus a faithful comparison command. All local, offline, deterministic — no model, no GPU. - resolve -t grade / -t lut: produce a data-color-grading block (or a frozen .cube). Look cascade: core preset (no file) -> bundled .cube library -> parametric buildCube. Emitted .cube is Rec.709 and validated against core's colorLuts constraints (LUT_3D_SIZE <= 64) before it is frozen. - smart grade (grade --for <media>): ffmpeg signalstats -> adjust suggestion (exposure / contrast / white balance), surfaced with the measured evidence on stderr as a starting point; never auto-applied. - hyperframes grade-compare: renders N candidate grades onto a reference frame through the real runtime shader into one labeled comparison PNG, so an agent picks a look without opening Studio. Prepends an "original" baseline cell by default (--no-baseline to omit). Shares the headless-capture pipeline with snapshot via capture/captureCompositionFrame. - media-use SKILL: proactive "media opportunity pass" guidance (grounded signal -> offer, ask once, surface don't mutate). Verified: media-use 116/116, grade-compare 7/7, snapshot 9/9, lint + format clean, full build green, comparison renders end to end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * test(cli): narrow grade-compare baseline assertion off unknown-typed grading Assert the whole cell via toEqual instead of reaching into .grading.preset / .grading.lut on the unknown-typed field, keeping the test typecheck-clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(media-use): agent-authored LUTs via --params + validate --from cube; never-read-.cube guardrail - resolve -t lut / -t grade --params '<json>': build a parametric .cube from explicit params (bypassing the intent cascade), validate, and freeze in one step. --intent becomes the optional description. Lets an agent commit a look it computed itself. - --from <file.cube> now validates the ingested LUT for lut/grade types and rejects an invalid/oversized cube (no partial write) — the escape hatch for a LUT the agent generated with its own code. - SKILL.md: hard rule to never read a .cube body into context (~size^3 lines, zero legible signal) — inspect via grade-compare (see it) or cube-validate (ok/size), read the manifest description for meaning; plus both authoring paths and the parametric-vs-film-stock ceiling note. Verified: media-use 116/116, lint + format clean; smokes — --params builds a valid frozen cube, grade --params returns a lut block, bad JSON and an oversized --from cube are both rejected with no stray file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(cli): grade-compare validates referenced LUTs, warns on no-op cells, caps candidates Bug-bash follow-ups — grade-compare silently accepted bad input: - Validate LUT *content*, not just existence: each referenced .cube is parsed with core's parseCubeLut (now exported from @hyperframes/core) and rejected with a per-cell error ("LUT for \"<label>\" is not a valid .cube: ..."). A file that exists but isn't a valid cube no longer renders a silent no-op cell. - Warn on inactive cells: a grading that normalizes to inactive (e.g. a malformed {lut:12345}) emits a stderr warning naming the cell; the auto-prepended "original" baseline is intentionally inactive and stays silent. stdout remains valid JSON. - Cap candidates at 16 (excluding baseline): over-cap input renders the first N and reports {truncated:true, total:M} on stdout + a stderr note — no silent drop, no unbounded giant sheet. Verified: grade-compare 10/10; non-cube LUT → clear error; {lut:12345} → warning + ok; 20 cells → cells=17 truncated total=20; valid runs unchanged. Lint/format clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(cli): general `hyperframes compare` visual-variant primitive Generalize grade-compare's "render N variants → one labeled sheet → the agent looks and picks" loop into a standalone command that works on ANY variation (font, layout, motion, grade, whole compositions) — the tool never needs to know what differs. - `hyperframes compare <path...> [--at <sec>] [--labels a,b,c] [--out] [--cols] [--json]`: renders each agent-authored composition variant through the real runtime (captureCompositionFrame) and stitches one labeled comparison sheet + JSON ({ok, sheet, rendered, variants, truncated?/total?}). 2+ paths required; caps at 16 with loud truncation. It presents, it does not judge — choosing is the caller's job. - Factored the shared "render a labeled set → contact sheet" path so compare, grade-compare, and snapshot all sit on it (no duplication). grade-compare is now the first color-specific specialization of this primitive. - New pathArgs util + contactSheet test; hyperframes-cli SKILL documents compare as the agent's "see your own renders and choose" primitive. Verified: 26/26 across compare + grade-compare + snapshot + contactSheet (no regressions); compare renders 3 variants into one visibly-distinct labeled sheet; 2+-path error path clean; lint/format clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(ci): green the skills CI — skip ffmpeg tests when absent, oxfmt markdown The "Test: skills" CI job runs bare `node --test` with no ffmpeg on PATH (by design — skills tests are meant to be node-builtin-only). The grade-analyzer + smart-grade tests shell to ffmpeg and were failing there with ENOENT. Guard them to skip when ffmpeg isn't on PATH; they still run locally / where it is. Also oxfmt README.md + hyperframes/media-use SKILL.md (the whole-repo `oxfmt --check .` Format job caught markdown left unformatted by the rebase conflict resolution). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(ci): skip core-conformance test when tsx is unavailable The "Test: skills" CI job installs no deps, so the normalizeHfColorGrading conformance test (which imports core's TS via `node --import tsx`) failed there. Guard it to skip when tsx can't resolve; runs locally / in the deps-installed Test job. Completes the skills-CI greening (the ffmpeg guards handled the rest). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(cli): escape grade-compare src double-quotes (CodeQL XSS) + Windows-safe compare test - grade-compare built `<img src="...">` (double-quoted) with the single-quote escaper, leaving `"` unescaped — a `"` in the frame path could break out (CodeQL: incomplete HTML attribute sanitization). Use escapeXml for src. - compare label test hard-coded POSIX paths that can't match on Windows; assert the derived labels (the subject); path resolution is covered elsewhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * refactor(media-use): generate LUT library from params (drop committed .cube files) The 3 bundled .cube files were 733 lines each (2,199 total) and were themselves buildCube output — pure repo bloat. Replace with compact per-look params in luts/index.json, generated on resolve; add an optional `url` for future scanned LUTs to be CDN-hosted + downloaded on demand (freezeUrl) instead of committed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(media-use): serve library LUTs from CDN on-demand (static.heygen.ai/luts), params fallback Looks now carry a CDN `url` (hosted at s3://heygen-public/luts → static.heygen.ai/luts/<id>.cube); resolve downloads + validates + freezes on demand, like bgm/image. `params` stays as the deterministic offline fallback (--local-only, or if the download fails), so resolution is never blocked on the network. Provider prefers url, falls back to params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(media-use): address #2041 review — atomic LUT writes, compare telemetry, follow-ups - Atomic .cube writes: library provider (url + params) and the parametric generator now write to a .tmp path, validate, then rename, so a crash can never orphan an invalid .cube at the final path (was validate-after-write). - track("media_use_resolve") now emits provenance.via (url/params-fallback/params). - grade-compare + compare: --timeout flag (was hardcoded 5000) and a media_use_compare event (cells, truncated, total, render_ready_timed_out); openSettledCompositionPage now surfaces the render-ready timeout. - compare staging skips node_modules/.git; --for gets an upfront existence check. - Rec.709 luma comment; HYPERFRAMES_ANALYZE_TIMEOUT_MS override; measured note uses basename; LUT s3 hosting moved from index.json into luts/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
163 lines
7.1 KiB
JavaScript
163 lines
7.1 KiB
JavaScript
// Provider registry — the v2 contract.
|
|
//
|
|
// Each media type maps to an ORDERED list of provider entries. Providers are
|
|
// tried in order; the first to return a non-null result wins, which keeps
|
|
// resolution deterministic (same request -> same provider -> same file ->
|
|
// reproducible renders). heygen-CLI is always first for the types it serves.
|
|
//
|
|
// An entry exposes any of three capability methods — search / generate /
|
|
// process — plus { name }. media-use holds no keys; each external tool owns its
|
|
// own auth. Providers, by type:
|
|
// - heygen CLI: catalog + TTS, first for every type it serves (sub creds)
|
|
// - mflux: local FLUX-class image gen, spec-selected to the machine's RAM
|
|
// (free, private, offline once cached)
|
|
// - codex CLI: image gen on the user's ChatGPT sub — the better-quality upsell
|
|
// and the fallback when no local model fits
|
|
// - Kokoro (via the hyperframes CLI): local voiceover, free/private default
|
|
// for the voice type, ahead of the paid HeyGen TTS upsell
|
|
//
|
|
// Generation is local-first, cloud-upsell. `ctx.provider` forces one provider
|
|
// (e.g. "make an image with codex").
|
|
|
|
import { bgmProvider } from "./bgm-provider.mjs";
|
|
import { sfxProvider } from "./sfx-provider.mjs";
|
|
import { imageProvider, iconProvider } from "./image-provider.mjs";
|
|
import { brandProvider } from "./brand-provider.mjs";
|
|
import {
|
|
svglSearch,
|
|
simpleIconsSearch,
|
|
githubAvatarSearch,
|
|
faviconSearch,
|
|
} from "./logo-provider.mjs";
|
|
import { heygenTtsGenerate } from "./voice-provider.mjs";
|
|
import { localTtsGenerate } from "./tts-local-provider.mjs";
|
|
import { codexImageGenerate } from "./codex-provider.mjs";
|
|
import { mfluxImageGenerate } from "./mflux-provider.mjs";
|
|
|
|
// Provider markers: `network` = hits a remote service (skipped by --local-only).
|
|
// `paid` = costs wallet credits (documentation for the agent's cost judgment,
|
|
// X4: agent-initiated paid should confirm). HeyGen catalog SEARCH is free;
|
|
// HeyGen TTS now costs credits, so it is the paid upsell behind local Kokoro.
|
|
const A = (name, caps) => ({ name, ...caps }); // local, free
|
|
const N = (name, caps) => ({ name, network: true, ...caps }); // remote, free
|
|
const P = (name, caps) => ({ name, network: true, paid: true, ...caps }); // remote, paid
|
|
|
|
// heygen-CLI first (and currently only). All remote providers are skipped by --local-only.
|
|
const REGISTRY = {
|
|
bgm: [N("heygen.audio.sounds", { search: bgmProvider.search })],
|
|
sfx: [N("heygen.audio.sounds", { search: sfxProvider.search })],
|
|
image: [
|
|
N("heygen.asset.search", { search: imageProvider.search }),
|
|
// Catalog miss -> generate. Local first (best FLUX-class model the machine's
|
|
// RAM can run, spec-selected; free, private, kept under --local-only), then
|
|
// the codex CLI on the user's ChatGPT sub as the better-quality upsell and
|
|
// the fallback when no local model fits.
|
|
A("mflux.local", { generate: mfluxImageGenerate }),
|
|
N("codex.image_gen", { generate: codexImageGenerate }),
|
|
],
|
|
icon: [N("heygen.asset.search", { search: iconProvider.search })],
|
|
logo: [
|
|
// Official brand marks. Tiers verified by a 54-brand stress test (100%
|
|
// cascade hit); HeyGen asset search is deliberately absent — it returns
|
|
// generic look-alike icons for brand queries. All free, all network →
|
|
// --local-only leaves only the cache rungs.
|
|
N("svgl", { search: svglSearch }),
|
|
N("simple-icons", { search: simpleIconsSearch }),
|
|
N("github.avatar", { search: githubAvatarSearch }),
|
|
N("favicon.ddg", { search: faviconSearch }),
|
|
],
|
|
voice: [
|
|
// Local Kokoro first (free, private, on-device via the hyperframes CLI, kept
|
|
// under --local-only), then HeyGen TTS as the higher-quality paid upsell and
|
|
// the fallback when Kokoro is not set up.
|
|
A("kokoro.local", { generate: localTtsGenerate }),
|
|
P("heygen.tts", { generate: heygenTtsGenerate }),
|
|
],
|
|
brand: [
|
|
// Local design spec, not heygen — reads frame.md / design.md tokens.
|
|
A("design_spec", { search: brandProvider.search }),
|
|
],
|
|
grade: [
|
|
// Local deterministic cascade handled by resolve.mjs so grade records can
|
|
// carry an inline block as well as an optional frozen .cube file.
|
|
A("color_grade.local", { search: async () => null, generate: async () => null }),
|
|
],
|
|
lut: [
|
|
// Lower-level local LUT generation/freezing path handled by resolve.mjs.
|
|
A("cube_lut.local", { search: async () => null, generate: async () => null }),
|
|
],
|
|
};
|
|
|
|
function listFor(type) {
|
|
const list = REGISTRY[type];
|
|
if (!list) throw new Error(`unknown media type: ${type}`);
|
|
return list;
|
|
}
|
|
|
|
/** Ordered providers for a type. */
|
|
export function getProviders(type) {
|
|
return listFor(type);
|
|
}
|
|
|
|
/** All declared media types. */
|
|
export function listTypes() {
|
|
return Object.keys(REGISTRY);
|
|
}
|
|
|
|
/** Provider names available for a type, in cascade order (for --provider validation). */
|
|
export function providerNamesFor(type) {
|
|
return listFor(type).map((p) => p.name);
|
|
}
|
|
|
|
/**
|
|
* Does an override token (full name like "codex.image_gen" or a prefix like
|
|
* "codex") match any provider declared for the type? Same match rule as
|
|
* runProviders, so validation and dispatch never disagree.
|
|
*/
|
|
export function providerMatches(type, want) {
|
|
return providerNamesFor(type).some((n) => n === want || n.startsWith(`${want}.`));
|
|
}
|
|
|
|
/**
|
|
* Back-compat shim for the v1 single-provider API. Returns the first declared
|
|
* provider for the type (tagged with `type`); throws for an unknown type.
|
|
* Kept for v1 callers only — new code should use getProviders/runCapability.
|
|
*/
|
|
export function getProvider(type) {
|
|
const first = listFor(type)[0] || {};
|
|
return { ...first, type };
|
|
}
|
|
|
|
/**
|
|
* Run a capability across an explicit ordered provider list. Tries each in
|
|
* order, returns the first non-null result, skips providers that don't expose
|
|
* the capability. Pure over its input — the unit-testable core of the cascade.
|
|
*
|
|
* Offline guard: a `network` provider is skipped when `ctx.localOnly` is set —
|
|
* unconditionally, even under a `ctx.provider` override. --local-only is a hard
|
|
* safety flag: it must never make a network call. Forcing a network provider
|
|
* while offline yields a clean miss (the caller explains the conflict), never a
|
|
* silent network request.
|
|
* Provider override: `ctx.provider` (a full name like "codex.image_gen" or a
|
|
* prefix like "codex") pins resolution to matching providers only — this is how
|
|
* a user "make an image WITH codex" forces the upsell instead of taking the
|
|
* free-first default.
|
|
*/
|
|
export async function runProviders(providers, capability, intent, ctx) {
|
|
const want = ctx?.provider;
|
|
for (const p of providers) {
|
|
if (want && p.name !== want && !p.name.startsWith(`${want}.`)) continue;
|
|
if (p.network && ctx?.localOnly) continue; // --local-only wins, even over --provider
|
|
const fn = p[capability];
|
|
if (typeof fn !== "function") continue;
|
|
const res = await fn(intent, ctx);
|
|
if (res) return res;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
/** Run a capability over the providers for a type (deterministic, heygen-first). */
|
|
export async function runCapability(type, capability, intent, ctx) {
|
|
return runProviders(getProviders(type), capability, intent, ctx);
|
|
}
|