mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-04 07:19:52 +00:00
* feat(media-use): color grading — grade/lut resolve, smart-grade, grade-compare CLI Add color grading to media-use as first-class resolve types plus a faithful comparison command. All local, offline, deterministic — no model, no GPU. - resolve -t grade / -t lut: produce a data-color-grading block (or a frozen .cube). Look cascade: core preset (no file) -> bundled .cube library -> parametric buildCube. Emitted .cube is Rec.709 and validated against core's colorLuts constraints (LUT_3D_SIZE <= 64) before it is frozen. - smart grade (grade --for <media>): ffmpeg signalstats -> adjust suggestion (exposure / contrast / white balance), surfaced with the measured evidence on stderr as a starting point; never auto-applied. - hyperframes grade-compare: renders N candidate grades onto a reference frame through the real runtime shader into one labeled comparison PNG, so an agent picks a look without opening Studio. Prepends an "original" baseline cell by default (--no-baseline to omit). Shares the headless-capture pipeline with snapshot via capture/captureCompositionFrame. - media-use SKILL: proactive "media opportunity pass" guidance (grounded signal -> offer, ask once, surface don't mutate). Verified: media-use 116/116, grade-compare 7/7, snapshot 9/9, lint + format clean, full build green, comparison renders end to end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * test(cli): narrow grade-compare baseline assertion off unknown-typed grading Assert the whole cell via toEqual instead of reaching into .grading.preset / .grading.lut on the unknown-typed field, keeping the test typecheck-clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(media-use): agent-authored LUTs via --params + validate --from cube; never-read-.cube guardrail - resolve -t lut / -t grade --params '<json>': build a parametric .cube from explicit params (bypassing the intent cascade), validate, and freeze in one step. --intent becomes the optional description. Lets an agent commit a look it computed itself. - --from <file.cube> now validates the ingested LUT for lut/grade types and rejects an invalid/oversized cube (no partial write) — the escape hatch for a LUT the agent generated with its own code. - SKILL.md: hard rule to never read a .cube body into context (~size^3 lines, zero legible signal) — inspect via grade-compare (see it) or cube-validate (ok/size), read the manifest description for meaning; plus both authoring paths and the parametric-vs-film-stock ceiling note. Verified: media-use 116/116, lint + format clean; smokes — --params builds a valid frozen cube, grade --params returns a lut block, bad JSON and an oversized --from cube are both rejected with no stray file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(cli): grade-compare validates referenced LUTs, warns on no-op cells, caps candidates Bug-bash follow-ups — grade-compare silently accepted bad input: - Validate LUT *content*, not just existence: each referenced .cube is parsed with core's parseCubeLut (now exported from @hyperframes/core) and rejected with a per-cell error ("LUT for \"<label>\" is not a valid .cube: ..."). A file that exists but isn't a valid cube no longer renders a silent no-op cell. - Warn on inactive cells: a grading that normalizes to inactive (e.g. a malformed {lut:12345}) emits a stderr warning naming the cell; the auto-prepended "original" baseline is intentionally inactive and stays silent. stdout remains valid JSON. - Cap candidates at 16 (excluding baseline): over-cap input renders the first N and reports {truncated:true, total:M} on stdout + a stderr note — no silent drop, no unbounded giant sheet. Verified: grade-compare 10/10; non-cube LUT → clear error; {lut:12345} → warning + ok; 20 cells → cells=17 truncated total=20; valid runs unchanged. Lint/format clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(cli): general `hyperframes compare` visual-variant primitive Generalize grade-compare's "render N variants → one labeled sheet → the agent looks and picks" loop into a standalone command that works on ANY variation (font, layout, motion, grade, whole compositions) — the tool never needs to know what differs. - `hyperframes compare <path...> [--at <sec>] [--labels a,b,c] [--out] [--cols] [--json]`: renders each agent-authored composition variant through the real runtime (captureCompositionFrame) and stitches one labeled comparison sheet + JSON ({ok, sheet, rendered, variants, truncated?/total?}). 2+ paths required; caps at 16 with loud truncation. It presents, it does not judge — choosing is the caller's job. - Factored the shared "render a labeled set → contact sheet" path so compare, grade-compare, and snapshot all sit on it (no duplication). grade-compare is now the first color-specific specialization of this primitive. - New pathArgs util + contactSheet test; hyperframes-cli SKILL documents compare as the agent's "see your own renders and choose" primitive. Verified: 26/26 across compare + grade-compare + snapshot + contactSheet (no regressions); compare renders 3 variants into one visibly-distinct labeled sheet; 2+-path error path clean; lint/format clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(ci): green the skills CI — skip ffmpeg tests when absent, oxfmt markdown The "Test: skills" CI job runs bare `node --test` with no ffmpeg on PATH (by design — skills tests are meant to be node-builtin-only). The grade-analyzer + smart-grade tests shell to ffmpeg and were failing there with ENOENT. Guard them to skip when ffmpeg isn't on PATH; they still run locally / where it is. Also oxfmt README.md + hyperframes/media-use SKILL.md (the whole-repo `oxfmt --check .` Format job caught markdown left unformatted by the rebase conflict resolution). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(ci): skip core-conformance test when tsx is unavailable The "Test: skills" CI job installs no deps, so the normalizeHfColorGrading conformance test (which imports core's TS via `node --import tsx`) failed there. Guard it to skip when tsx can't resolve; runs locally / in the deps-installed Test job. Completes the skills-CI greening (the ffmpeg guards handled the rest). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(cli): escape grade-compare src double-quotes (CodeQL XSS) + Windows-safe compare test - grade-compare built `<img src="...">` (double-quoted) with the single-quote escaper, leaving `"` unescaped — a `"` in the frame path could break out (CodeQL: incomplete HTML attribute sanitization). Use escapeXml for src. - compare label test hard-coded POSIX paths that can't match on Windows; assert the derived labels (the subject); path resolution is covered elsewhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * refactor(media-use): generate LUT library from params (drop committed .cube files) The 3 bundled .cube files were 733 lines each (2,199 total) and were themselves buildCube output — pure repo bloat. Replace with compact per-look params in luts/index.json, generated on resolve; add an optional `url` for future scanned LUTs to be CDN-hosted + downloaded on demand (freezeUrl) instead of committed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(media-use): serve library LUTs from CDN on-demand (static.heygen.ai/luts), params fallback Looks now carry a CDN `url` (hosted at s3://heygen-public/luts → static.heygen.ai/luts/<id>.cube); resolve downloads + validates + freezes on demand, like bgm/image. `params` stays as the deterministic offline fallback (--local-only, or if the download fails), so resolution is never blocked on the network. Provider prefers url, falls back to params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(media-use): address #2041 review — atomic LUT writes, compare telemetry, follow-ups - Atomic .cube writes: library provider (url + params) and the parametric generator now write to a .tmp path, validate, then rename, so a crash can never orphan an invalid .cube at the final path (was validate-after-write). - track("media_use_resolve") now emits provenance.via (url/params-fallback/params). - grade-compare + compare: --timeout flag (was hardcoded 5000) and a media_use_compare event (cells, truncated, total, render_ready_timed_out); openSettledCompositionPage now surfaces the render-ready timeout. - compare staging skips node_modules/.git; --for gets an upfront existence check. - Rec.709 luma comment; HYPERFRAMES_ANALYZE_TIMEOUT_MS override; measured note uses basename; LUT s3 hosting moved from index.json into luts/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
233 lines
7.8 KiB
JavaScript
233 lines
7.8 KiB
JavaScript
import { existsSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
|
import { join } from "node:path";
|
|
import { allocateId } from "./manifest.mjs";
|
|
import { freezeUrl } from "./freeze.mjs";
|
|
import { tokenOverlap } from "./match.mjs";
|
|
import { buildCube } from "./cube-build.mjs";
|
|
import { validateCube, validateCubeFile } from "./cube-validate.mjs";
|
|
|
|
const SKILL_DIR = join(import.meta.dirname, "..", "..");
|
|
const LUT_DIR = join(SKILL_DIR, "luts");
|
|
const LUT_INDEX = join(LUT_DIR, "index.json");
|
|
export const LIBRARY_LUT_OFFLINE_CODE = "MEDIA_USE_LIBRARY_LUT_OFFLINE";
|
|
|
|
// Mirrored from packages/core/src/colorGrading.ts HfColorGradingPresetId.
|
|
// Keep this list in lockstep with the core runtime contract.
|
|
export const CORE_PRESET_IDS = [
|
|
"neutral",
|
|
"natural-lift",
|
|
"fresh-pop",
|
|
"warm-daylight",
|
|
"clean-studio",
|
|
"skin-soft",
|
|
"food-pop",
|
|
"night-lift",
|
|
"muted-editorial",
|
|
"vintage-wash",
|
|
"mono-clean",
|
|
"mono-fade",
|
|
"warm-clean",
|
|
"cool-clean",
|
|
"soft-boost",
|
|
"bright-pop",
|
|
"deep-contrast",
|
|
];
|
|
|
|
const PRESET_SYNONYMS = {
|
|
neutral: ["neutral", "identity", "none", "ungraded", "natural base"],
|
|
"natural-lift": ["natural lift", "natural light", "gentle lift", "soft natural"],
|
|
"fresh-pop": ["fresh pop", "fresh", "bright fresh", "clean colorful"],
|
|
"warm-daylight": [
|
|
"warm daylight",
|
|
"warm natural light",
|
|
"golden daylight",
|
|
"sunlit",
|
|
"warm sunny",
|
|
],
|
|
"clean-studio": ["clean studio", "studio clean", "cool studio", "product studio"],
|
|
"skin-soft": ["skin soft", "soft skin", "portrait soft", "beauty skin"],
|
|
"food-pop": ["food pop", "food vibrant", "appetizing", "restaurant color"],
|
|
"night-lift": ["night lift", "night", "low light lift", "city night"],
|
|
"muted-editorial": ["muted editorial", "editorial muted", "magazine muted"],
|
|
"vintage-wash": ["vintage wash", "vintage", "retro wash", "aged film"],
|
|
"mono-clean": ["mono clean", "black white clean", "monochrome clean"],
|
|
"mono-fade": ["mono fade", "black white fade", "faded monochrome"],
|
|
"warm-clean": ["warm clean", "clean warm", "warm product"],
|
|
"cool-clean": ["cool clean", "clean cool", "cool crisp"],
|
|
"soft-boost": ["soft boost", "soft bright", "gentle boost"],
|
|
"bright-pop": ["bright pop", "bright punchy", "vivid bright"],
|
|
"deep-contrast": ["deep contrast", "high contrast punchy", "punchy contrast", "bold contrast"],
|
|
};
|
|
|
|
function presetCandidates() {
|
|
return CORE_PRESET_IDS.map((id) => ({
|
|
kind: "preset",
|
|
preset: id,
|
|
synonyms: PRESET_SYNONYMS[id] ?? [],
|
|
text: [id, ...(PRESET_SYNONYMS[id] ?? [])].join(" "),
|
|
}));
|
|
}
|
|
|
|
export function readBundledLutIndex() {
|
|
if (!existsSync(LUT_INDEX)) return [];
|
|
const parsed = JSON.parse(readFileSync(LUT_INDEX, "utf8"));
|
|
const entries = Array.isArray(parsed) ? parsed : parsed.looks;
|
|
if (!Array.isArray(entries)) return [];
|
|
return entries.map((entry) => {
|
|
const params =
|
|
entry.params && typeof entry.params === "object" && !Array.isArray(entry.params)
|
|
? entry.params
|
|
: null;
|
|
const url = typeof entry.url === "string" && entry.url.trim() ? entry.url.trim() : null;
|
|
return {
|
|
id: String(entry.id),
|
|
description: String(entry.description ?? entry.id),
|
|
tags: Array.isArray(entry.tags) ? entry.tags.map(String) : [],
|
|
intensity: Number.isFinite(Number(entry.intensity)) ? Number(entry.intensity) : 1,
|
|
...(params && { params }),
|
|
...(url && { url }),
|
|
};
|
|
});
|
|
}
|
|
|
|
function libraryCandidates() {
|
|
return readBundledLutIndex().map((entry) => ({
|
|
kind: "library",
|
|
...entry,
|
|
text: [entry.id, entry.description, ...entry.tags].join(" "),
|
|
}));
|
|
}
|
|
|
|
export function matchColorLook(intent) {
|
|
const normalized = String(intent ?? "")
|
|
.trim()
|
|
.toLowerCase()
|
|
.replace(/\s+/g, " ");
|
|
for (const candidate of presetCandidates()) {
|
|
if (candidate.preset === normalized) {
|
|
return { kind: "preset", preset: candidate.preset, score: 99 };
|
|
}
|
|
}
|
|
|
|
const candidates = [...presetCandidates(), ...libraryCandidates()]
|
|
.map((candidate, index) => ({
|
|
...candidate,
|
|
index,
|
|
score: tokenOverlap(intent, candidate.text),
|
|
}))
|
|
.filter((candidate) => candidate.score >= 2)
|
|
.sort((a, b) => b.score - a.score || a.index - b.index);
|
|
|
|
if (candidates.length === 0) return null;
|
|
const best = candidates[0];
|
|
if (best.kind === "preset") {
|
|
return { kind: "preset", preset: best.preset, score: best.score };
|
|
}
|
|
return {
|
|
kind: "library",
|
|
id: best.id,
|
|
description: best.description,
|
|
tags: best.tags,
|
|
intensity: best.intensity,
|
|
...(best.params && { params: best.params }),
|
|
...(best.url && { url: best.url }),
|
|
score: best.score,
|
|
};
|
|
}
|
|
|
|
export function isLibraryLutOfflineMiss(err) {
|
|
return err?.code === LIBRARY_LUT_OFFLINE_CODE;
|
|
}
|
|
|
|
function libraryRecord(match, { id, localPath, fullPath, via }) {
|
|
return {
|
|
id,
|
|
localPath,
|
|
fullPath,
|
|
lut: { src: localPath, intensity: match.intensity },
|
|
source: "library",
|
|
description: match.description,
|
|
metadata: {
|
|
provider: "cube_lut.library",
|
|
provenance: {
|
|
look_id: match.id,
|
|
tags: match.tags,
|
|
via,
|
|
},
|
|
},
|
|
};
|
|
}
|
|
|
|
function assertValidCubeText(cube, label) {
|
|
const check = validateCube(cube);
|
|
if (!check.ok) throw new Error(`${label}: ${check.error}`);
|
|
}
|
|
|
|
function assertValidCubeFile(path, label) {
|
|
const check = validateCubeFile(path);
|
|
if (!check.ok) throw new Error(`${label}: ${check.error}`);
|
|
}
|
|
|
|
function offlineLibraryMiss(match) {
|
|
const err = new Error(`library LUT "${match.id}" is CDN-only and --local-only is set`);
|
|
err.code = LIBRARY_LUT_OFFLINE_CODE;
|
|
return err;
|
|
}
|
|
|
|
export async function freezeLibraryLut(match, { projectDir, type, localOnly = false }) {
|
|
if (!match || match.kind !== "library") {
|
|
throw new Error("freezeLibraryLut requires a library match");
|
|
}
|
|
|
|
// Prefer the CDN url so looks download on-demand (like bgm/image). Fall back
|
|
// to deterministic buildCube params when offline (--local-only) or if the
|
|
// download/validation fails, so resolution is never blocked on the network.
|
|
if (match.url && !localOnly) {
|
|
const { id, localPath } = allocateId(projectDir, type, ".cube");
|
|
const fullPath = join(projectDir, localPath);
|
|
const tmpPath = `${fullPath}.tmp`;
|
|
try {
|
|
// Download + validate at a .tmp path, then atomically rename. A crash
|
|
// (SIGKILL/OOM) between write and validate can't orphan an invalid .cube
|
|
// at the final path — only a validated cube is ever renamed into place.
|
|
await freezeUrl(match.url, tmpPath);
|
|
assertValidCubeFile(tmpPath, `downloaded library LUT ${match.id} failed validation`);
|
|
renameSync(tmpPath, fullPath);
|
|
return libraryRecord(match, { id, localPath, fullPath, via: "url" });
|
|
} catch (err) {
|
|
rmSync(tmpPath, { force: true });
|
|
if (!match.params) {
|
|
throw new Error(`failed to freeze library LUT ${match.id}: ${err.message}`);
|
|
}
|
|
// else: fall through to the params fallback below
|
|
}
|
|
}
|
|
|
|
if (match.params) {
|
|
const { id, localPath } = allocateId(projectDir, type, ".cube");
|
|
const fullPath = join(projectDir, localPath);
|
|
const tmpPath = `${fullPath}.tmp`;
|
|
try {
|
|
const cube = buildCube(match.params);
|
|
assertValidCubeText(cube, `invalid library LUT ${match.id}`);
|
|
// Write + validate at .tmp, then atomic rename — same no-orphan guarantee
|
|
// as the url path above.
|
|
writeFileSync(tmpPath, cube);
|
|
assertValidCubeFile(tmpPath, `invalid frozen LUT ${localPath}`);
|
|
renameSync(tmpPath, fullPath);
|
|
} catch (err) {
|
|
rmSync(tmpPath, { force: true });
|
|
throw err;
|
|
}
|
|
return libraryRecord(match, {
|
|
id,
|
|
localPath,
|
|
fullPath,
|
|
via: match.url ? "params-fallback" : "params",
|
|
});
|
|
}
|
|
|
|
if (match.url) throw offlineLibraryMiss(match); // url-only entry, offline
|
|
throw new Error(`misconfigured library LUT "${match.id}": expected params or url`);
|
|
}
|