mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-04 07:19:52 +00:00
* feat(media-use): color grading — grade/lut resolve, smart-grade, grade-compare CLI Add color grading to media-use as first-class resolve types plus a faithful comparison command. All local, offline, deterministic — no model, no GPU. - resolve -t grade / -t lut: produce a data-color-grading block (or a frozen .cube). Look cascade: core preset (no file) -> bundled .cube library -> parametric buildCube. Emitted .cube is Rec.709 and validated against core's colorLuts constraints (LUT_3D_SIZE <= 64) before it is frozen. - smart grade (grade --for <media>): ffmpeg signalstats -> adjust suggestion (exposure / contrast / white balance), surfaced with the measured evidence on stderr as a starting point; never auto-applied. - hyperframes grade-compare: renders N candidate grades onto a reference frame through the real runtime shader into one labeled comparison PNG, so an agent picks a look without opening Studio. Prepends an "original" baseline cell by default (--no-baseline to omit). Shares the headless-capture pipeline with snapshot via capture/captureCompositionFrame. - media-use SKILL: proactive "media opportunity pass" guidance (grounded signal -> offer, ask once, surface don't mutate). Verified: media-use 116/116, grade-compare 7/7, snapshot 9/9, lint + format clean, full build green, comparison renders end to end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * test(cli): narrow grade-compare baseline assertion off unknown-typed grading Assert the whole cell via toEqual instead of reaching into .grading.preset / .grading.lut on the unknown-typed field, keeping the test typecheck-clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(media-use): agent-authored LUTs via --params + validate --from cube; never-read-.cube guardrail - resolve -t lut / -t grade --params '<json>': build a parametric .cube from explicit params (bypassing the intent cascade), validate, and freeze in one step. --intent becomes the optional description. Lets an agent commit a look it computed itself. - --from <file.cube> now validates the ingested LUT for lut/grade types and rejects an invalid/oversized cube (no partial write) — the escape hatch for a LUT the agent generated with its own code. - SKILL.md: hard rule to never read a .cube body into context (~size^3 lines, zero legible signal) — inspect via grade-compare (see it) or cube-validate (ok/size), read the manifest description for meaning; plus both authoring paths and the parametric-vs-film-stock ceiling note. Verified: media-use 116/116, lint + format clean; smokes — --params builds a valid frozen cube, grade --params returns a lut block, bad JSON and an oversized --from cube are both rejected with no stray file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(cli): grade-compare validates referenced LUTs, warns on no-op cells, caps candidates Bug-bash follow-ups — grade-compare silently accepted bad input: - Validate LUT *content*, not just existence: each referenced .cube is parsed with core's parseCubeLut (now exported from @hyperframes/core) and rejected with a per-cell error ("LUT for \"<label>\" is not a valid .cube: ..."). A file that exists but isn't a valid cube no longer renders a silent no-op cell. - Warn on inactive cells: a grading that normalizes to inactive (e.g. a malformed {lut:12345}) emits a stderr warning naming the cell; the auto-prepended "original" baseline is intentionally inactive and stays silent. stdout remains valid JSON. - Cap candidates at 16 (excluding baseline): over-cap input renders the first N and reports {truncated:true, total:M} on stdout + a stderr note — no silent drop, no unbounded giant sheet. Verified: grade-compare 10/10; non-cube LUT → clear error; {lut:12345} → warning + ok; 20 cells → cells=17 truncated total=20; valid runs unchanged. Lint/format clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(cli): general `hyperframes compare` visual-variant primitive Generalize grade-compare's "render N variants → one labeled sheet → the agent looks and picks" loop into a standalone command that works on ANY variation (font, layout, motion, grade, whole compositions) — the tool never needs to know what differs. - `hyperframes compare <path...> [--at <sec>] [--labels a,b,c] [--out] [--cols] [--json]`: renders each agent-authored composition variant through the real runtime (captureCompositionFrame) and stitches one labeled comparison sheet + JSON ({ok, sheet, rendered, variants, truncated?/total?}). 2+ paths required; caps at 16 with loud truncation. It presents, it does not judge — choosing is the caller's job. - Factored the shared "render a labeled set → contact sheet" path so compare, grade-compare, and snapshot all sit on it (no duplication). grade-compare is now the first color-specific specialization of this primitive. - New pathArgs util + contactSheet test; hyperframes-cli SKILL documents compare as the agent's "see your own renders and choose" primitive. Verified: 26/26 across compare + grade-compare + snapshot + contactSheet (no regressions); compare renders 3 variants into one visibly-distinct labeled sheet; 2+-path error path clean; lint/format clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(ci): green the skills CI — skip ffmpeg tests when absent, oxfmt markdown The "Test: skills" CI job runs bare `node --test` with no ffmpeg on PATH (by design — skills tests are meant to be node-builtin-only). The grade-analyzer + smart-grade tests shell to ffmpeg and were failing there with ENOENT. Guard them to skip when ffmpeg isn't on PATH; they still run locally / where it is. Also oxfmt README.md + hyperframes/media-use SKILL.md (the whole-repo `oxfmt --check .` Format job caught markdown left unformatted by the rebase conflict resolution). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(ci): skip core-conformance test when tsx is unavailable The "Test: skills" CI job installs no deps, so the normalizeHfColorGrading conformance test (which imports core's TS via `node --import tsx`) failed there. Guard it to skip when tsx can't resolve; runs locally / in the deps-installed Test job. Completes the skills-CI greening (the ffmpeg guards handled the rest). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(cli): escape grade-compare src double-quotes (CodeQL XSS) + Windows-safe compare test - grade-compare built `<img src="...">` (double-quoted) with the single-quote escaper, leaving `"` unescaped — a `"` in the frame path could break out (CodeQL: incomplete HTML attribute sanitization). Use escapeXml for src. - compare label test hard-coded POSIX paths that can't match on Windows; assert the derived labels (the subject); path resolution is covered elsewhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * refactor(media-use): generate LUT library from params (drop committed .cube files) The 3 bundled .cube files were 733 lines each (2,199 total) and were themselves buildCube output — pure repo bloat. Replace with compact per-look params in luts/index.json, generated on resolve; add an optional `url` for future scanned LUTs to be CDN-hosted + downloaded on demand (freezeUrl) instead of committed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(media-use): serve library LUTs from CDN on-demand (static.heygen.ai/luts), params fallback Looks now carry a CDN `url` (hosted at s3://heygen-public/luts → static.heygen.ai/luts/<id>.cube); resolve downloads + validates + freezes on demand, like bgm/image. `params` stays as the deterministic offline fallback (--local-only, or if the download fails), so resolution is never blocked on the network. Provider prefers url, falls back to params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(media-use): address #2041 review — atomic LUT writes, compare telemetry, follow-ups - Atomic .cube writes: library provider (url + params) and the parametric generator now write to a .tmp path, validate, then rename, so a crash can never orphan an invalid .cube at the final path (was validate-after-write). - track("media_use_resolve") now emits provenance.via (url/params-fallback/params). - grade-compare + compare: --timeout flag (was hardcoded 5000) and a media_use_compare event (cells, truncated, total, render_ready_timed_out); openSettledCompositionPage now surfaces the render-ready timeout. - compare staging skips node_modules/.git; --for gets an upfront existence check. - Rec.709 luma comment; HYPERFRAMES_ANALYZE_TIMEOUT_MS override; measured note uses basename; LUT s3 hosting moved from index.json into luts/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
182 lines
5.7 KiB
JavaScript
182 lines
5.7 KiB
JavaScript
const DEFAULT_SIZE = 33;
|
|
const MAX_SIZE = 64;
|
|
|
|
function clamp(value, min, max) {
|
|
if (!Number.isFinite(value)) return 0;
|
|
return Math.min(max, Math.max(min, value));
|
|
}
|
|
|
|
function clampUnit(value) {
|
|
return clamp(value, 0, 1);
|
|
}
|
|
|
|
function readParam(params, key, min, max) {
|
|
return clamp(Number(params?.[key] ?? 0), min, max);
|
|
}
|
|
|
|
function luma([r, g, b]) {
|
|
// Rec.709 luma weightings (matches the color space the grading runtime uses).
|
|
return r * 0.2126 + g * 0.7152 + b * 0.0722;
|
|
}
|
|
|
|
function smoothstep(edge0, edge1, value) {
|
|
const t = clampUnit((value - edge0) / (edge1 - edge0));
|
|
return t * t * (3 - 2 * t);
|
|
}
|
|
|
|
function applyLiftGain(color, params) {
|
|
const y = luma(color);
|
|
const blacks = readParam(params, "blacks", -1, 1);
|
|
const shadows = readParam(params, "shadows", -1, 1);
|
|
const highlights = readParam(params, "highlights", -1, 1);
|
|
const whites = readParam(params, "whites", -1, 1);
|
|
const shadowMask = 1 - smoothstep(0.18, 0.62, y);
|
|
const highlightMask = smoothstep(0.38, 0.82, y);
|
|
const offset =
|
|
blacks * 0.08 + shadows * 0.12 * shadowMask + highlights * 0.12 * highlightMask + whites * 0.08;
|
|
return color.map((channel) => clampUnit(channel + offset));
|
|
}
|
|
|
|
function applyExposure(color, params) {
|
|
const exposure = readParam(params, "exposure", -2, 2);
|
|
const gain = 2 ** exposure;
|
|
const lift = Math.max(0, exposure) * 0.015;
|
|
return color.map((channel) => clampUnit(channel * gain + lift));
|
|
}
|
|
|
|
function applyContrast(color, params) {
|
|
const contrast = readParam(params, "contrast", -1, 1);
|
|
if (contrast === 0) return color;
|
|
const factor = 1 + contrast * 1.2;
|
|
return color.map((channel) => clampUnit(0.5 + (channel - 0.5) * factor));
|
|
}
|
|
|
|
function applyWhiteBalance(color, params) {
|
|
const temperature = readParam(params, "temperature", -1, 1);
|
|
const tint = readParam(params, "tint", -1, 1);
|
|
const redScale = 1 + temperature * 0.28 + tint * 0.08;
|
|
const greenScale = 1 - Math.abs(tint) * 0.1 - tint * 0.08;
|
|
const blueScale = 1 - temperature * 0.28 + tint * 0.08;
|
|
return [
|
|
clampUnit(color[0] * redScale),
|
|
clampUnit(color[1] * greenScale),
|
|
clampUnit(color[2] * blueScale),
|
|
];
|
|
}
|
|
|
|
function applySplitTone(color, params) {
|
|
const split = params?.splitTone;
|
|
if (!split) return color;
|
|
const intensity = clampUnit(Number(split.intensity ?? 0));
|
|
if (intensity === 0) return color;
|
|
const balance = clampUnit(Number(split.balance ?? 0.5));
|
|
const y = luma(color);
|
|
const shadowMask = 1 - smoothstep(balance - 0.25, balance + 0.2, y);
|
|
const highlightMask = smoothstep(balance - 0.2, balance + 0.25, y);
|
|
const shadows = Array.isArray(split.shadows) ? split.shadows : [0, 0, 0];
|
|
const highlights = Array.isArray(split.highlights) ? split.highlights : [0, 0, 0];
|
|
return color.map((channel, i) =>
|
|
clampUnit(
|
|
channel +
|
|
Number(shadows[i] ?? 0) * shadowMask * intensity +
|
|
Number(highlights[i] ?? 0) * highlightMask * intensity,
|
|
),
|
|
);
|
|
}
|
|
|
|
function applySaturation(color, params) {
|
|
const saturation = readParam(params, "saturation", -1, 1);
|
|
const vibrance = readParam(params, "vibrance", -1, 1);
|
|
if (saturation === 0 && vibrance === 0) return color;
|
|
const y = luma(color);
|
|
const currentSat = Math.max(
|
|
Math.abs(color[0] - y),
|
|
Math.abs(color[1] - y),
|
|
Math.abs(color[2] - y),
|
|
);
|
|
const vibranceWeight = 1 - clampUnit(currentSat * 2);
|
|
const factor = clamp(1 + saturation + vibrance * vibranceWeight, 0, 2.5);
|
|
return color.map((channel) => clampUnit(y + (channel - y) * factor));
|
|
}
|
|
|
|
function applyParams(color, params) {
|
|
let out = applyLiftGain(color, params);
|
|
out = applyExposure(out, params);
|
|
out = applyContrast(out, params);
|
|
out = applyWhiteBalance(out, params);
|
|
out = applySplitTone(out, params);
|
|
out = applySaturation(out, params);
|
|
return out;
|
|
}
|
|
|
|
function formatNumber(value) {
|
|
return clampUnit(value).toFixed(6);
|
|
}
|
|
|
|
export function buildCube(params = {}, size = DEFAULT_SIZE) {
|
|
if (!Number.isInteger(size) || size < 2 || size > MAX_SIZE) {
|
|
throw new Error(`LUT size must be an integer from 2 to ${MAX_SIZE}`);
|
|
}
|
|
const lines = [
|
|
`TITLE "media-use parametric grade"`,
|
|
"DOMAIN_MIN 0 0 0",
|
|
"DOMAIN_MAX 1 1 1",
|
|
`LUT_3D_SIZE ${size}`,
|
|
];
|
|
const denom = size - 1;
|
|
for (let b = 0; b < size; b++) {
|
|
for (let g = 0; g < size; g++) {
|
|
for (let r = 0; r < size; r++) {
|
|
const out = applyParams([r / denom, g / denom, b / denom], params);
|
|
lines.push(`${formatNumber(out[0])} ${formatNumber(out[1])} ${formatNumber(out[2])}`);
|
|
}
|
|
}
|
|
}
|
|
return `${lines.join("\n")}\n`;
|
|
}
|
|
|
|
export function paramsFromIntent(intent) {
|
|
const text = String(intent ?? "").toLowerCase();
|
|
const params = {};
|
|
let matched = false;
|
|
|
|
if (/\b(warm|golden|sunlit|sunny)\b/.test(text)) {
|
|
params.temperature = 0.18;
|
|
matched = true;
|
|
} else if (/\b(cool|blue|icy|crisp)\b/.test(text)) {
|
|
params.temperature = -0.16;
|
|
matched = true;
|
|
}
|
|
|
|
if (/\b(cinematic|film|movie)\b/.test(text)) {
|
|
params.contrast = 0.08;
|
|
params.saturation = 0.04;
|
|
matched = true;
|
|
}
|
|
if (/\b(punchy|contrast|dramatic|bold)\b/.test(text)) {
|
|
params.contrast = Math.max(params.contrast ?? 0, 0.22);
|
|
matched = true;
|
|
}
|
|
if (/\b(bright|airy|lift)\b/.test(text)) {
|
|
params.exposure = 0.16;
|
|
params.shadows = 0.08;
|
|
matched = true;
|
|
}
|
|
if (/\b(dark|moody|low-key)\b/.test(text)) {
|
|
params.exposure = -0.12;
|
|
params.contrast = Math.max(params.contrast ?? 0, 0.12);
|
|
matched = true;
|
|
}
|
|
if (/\b(vibrant|saturated|colorful)\b/.test(text)) {
|
|
params.saturation = Math.max(params.saturation ?? 0, 0.16);
|
|
params.vibrance = 0.12;
|
|
matched = true;
|
|
}
|
|
if (/\b(muted|desaturated|washed)\b/.test(text)) {
|
|
params.saturation = Math.min(params.saturation ?? 0, -0.16);
|
|
matched = true;
|
|
}
|
|
|
|
return matched ? params : null;
|
|
}
|