mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-02 20:18:35 +00:00
feat(skill): w2h-verify script + Step 6 DoD wiring — tooling-based enforcement
Three rounds of text-based enforcement plateaued. A third agent debrief showed the same failures: 9% asset usage (vs ≥30% floor), shader transitions declared in STORYBOARD.md but not in shipping index.html, SFX timestamps drifted up to 12.4s, animation-map skipped, MP4 not rendered, honest-disclosure section omitted from final summary. The pattern is clear: language-only enforcement is selectively interpretable by the agent under ship pressure. Move enforcement into tooling — facts the agent can't fudge. **New script: `skills/website-to-hyperframes/scripts/w2h-verify.mjs`** Pure file-analysis script (no shell spawns). Computes six checks and prints a PASS/FAIL/INFO table: 1. Asset usage — assets referenced in compositions ÷ assets captured; target ≥30%. Tested against videos/huly-v3: caught 6/74 = 8% FAIL. 2. Shader transitions consistency — STORYBOARD.md-declared shaders vs index.html. Longest-name matching to avoid substring false positives (cross-warp-morph not double-counted as cross-warp). Tested: caught 6 declared / 1 present / 5 missing. 3. SFX timestamp drift — parses STORYBOARD.md table rows for `sfx/X.mp3` + time-with-`s`, parses index.html <audio data-start>, flags drift >0.5s. Tested: caught 12.4s drift on click.mp3 that the agent debrief didn't even mention. 4. animation-map.json existence — explicit file check. 5. Rendered MP4 existence — scans project root, output/, renders/. 6. Required artifacts — STORYBOARD.md, DESIGN.md, SCRIPT.md, index.html all present. Exit code: 0 (all pass) or 1 (one or more fail). The script's output becomes the Step 6 deliverable — paste verbatim into the user-facing summary. **Skill update: `step-6-validate.md`** Adds `w2h-verify report` to the DoD checklist with the rule: paste the FULL output verbatim into the final summary. Cherry-picking rows, substituting adjectives for percentages, or omitting FAIL lines is explicitly forbidden. If a row says FAIL, either fix it and re-run until PASS or include the FAIL line verbatim in "What I did NOT verify" with a one-sentence reason. Test run against the project that prompted this: ``` SUMMARY: 1 PASS · 4 FAIL · 1 INFO - Asset usage: FAIL 6/74 (8%) target ≥30% - Shader transitions: FAIL 6 declared, 1 present, 5 missing - SFX timestamps: FAIL 3 drifted >0.5s (max 12.4s) - animation-map.json: FAIL missing - Rendered MP4: INFO no .mp4 found - Required artifacts: PASS ``` The agent could selectively ignore "the WCAG warnings are false positives." The agent cannot selectively ignore a line that says `6/74 (8%) — target ≥30%`. 2 files changed (+ 1 new script, ~330 lines). Format checks pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
5594a8286c
commit
190f1ec71a
@@ -18,10 +18,46 @@ Score each item 1–5. If any item scores below 3, fix it before continuing. **D
|
||||
[ ] Brand assets actually visible → for each beat, name which captured SVG / illustration / screenshot is on screen and at what timestamp. If a beat shows zero captured assets, justify why.
|
||||
[ ] Audio duration matches video ±0.5s → paste both numbers
|
||||
[ ] animation-map.json generated → run `node <repo-root>/skills/hyperframes/scripts/animation-map.mjs <project-dir>`; confirm every beat has events listed and no bbox/flag warnings
|
||||
[ ] w2h-verify report → run `node <repo-root>/skills/website-to-hyperframes/scripts/w2h-verify.mjs <project-dir>`; paste the FULL output (every row, every percent) verbatim into your final user-facing summary — see "w2h-verify — the source of truth" below
|
||||
[ ] Audio + motion verification done → see "Audio + motion verification" below; played the full preview, confirmed SFX lands at storyboard timestamps
|
||||
[ ] Critic sub-agent run → paste its single biggest quality gap finding, verbatim
|
||||
```
|
||||
|
||||
### w2h-verify — the source of truth
|
||||
|
||||
The skill ran for months on agents reading "REQUIRED" and skipping anyway. The verify script ends that — it computes facts the agent cannot fudge:
|
||||
|
||||
- **Asset usage %** (assets referenced in compositions ÷ assets captured)
|
||||
- **Shader transitions consistency** (shaders declared in STORYBOARD.md vs shaders present in index.html)
|
||||
- **SFX timestamp drift** (storyboard `t=X.Xs` vs index.html `data-start=X.X`)
|
||||
- **animation-map.json existence**
|
||||
- **Rendered MP4 existence**
|
||||
- **Required artifacts present** (STORYBOARD.md, DESIGN.md, SCRIPT.md, index.html)
|
||||
|
||||
Run it as the LAST gate in your DoD pass, after fixing everything else:
|
||||
|
||||
```bash
|
||||
node <repo-root>/skills/website-to-hyperframes/scripts/w2h-verify.mjs <project-dir>
|
||||
```
|
||||
|
||||
(Locate the repo root from a project subdirectory: `find / -path '*/skills/website-to-hyperframes/scripts/w2h-verify.mjs' -maxdepth 12 2>/dev/null | head -1`.)
|
||||
|
||||
**The script's output is the deliverable.** Paste the entire report — the table, the percentages, the FAIL lines — verbatim into your final user-facing summary, in the "What I verified" / "What I did NOT verify" section. The user will read it directly. You don't get to summarize, simplify, or omit rows.
|
||||
|
||||
**If any row says FAIL:**
|
||||
|
||||
- Either fix the underlying issue and re-run until the row says PASS
|
||||
- Or include the FAIL row verbatim in your final summary's "What I did NOT verify" section with a one-sentence explanation of why you chose not to fix it
|
||||
|
||||
**Forbidden:**
|
||||
|
||||
- Hand-writing your own verification summary that doesn't match the script's output
|
||||
- Cherry-picking which rows to include
|
||||
- Replacing percentages with adjectives ("most assets used" instead of "8%")
|
||||
- Running the script, seeing FAIL, and not mentioning it
|
||||
|
||||
The script's exit code is 0 (all pass) or 1 (one or more fail). If you ship with exit=1, the user knows from the report exactly what they're getting.
|
||||
|
||||
### Per-beat file read
|
||||
|
||||
This is what verification means now: you open each `compositions/beat-N.html` and read it top-to-bottom against DESIGN.md and STORYBOARD.md. Step 5 already required this once before advancing here — repeat it here as the final check, in case fixes during Step 5 introduced new problems.
|
||||
|
||||
@@ -0,0 +1,359 @@
|
||||
#!/usr/bin/env node
|
||||
// w2h-verify.mjs — verification report for a website-to-hyperframes project.
|
||||
//
|
||||
// Outputs facts the agent cannot fudge: asset usage %, storyboard-vs-build
|
||||
// consistency, artifact existence. Result becomes the Step 6 deliverable —
|
||||
// paste verbatim into the final user-facing summary.
|
||||
//
|
||||
// This script does PURE FILE ANALYSIS — it does not spawn lint/inspect.
|
||||
// Run those separately via the CLI and include their summaries alongside.
|
||||
//
|
||||
// Usage:
|
||||
// node skills/website-to-hyperframes/scripts/w2h-verify.mjs <project-dir>
|
||||
//
|
||||
// Exit codes:
|
||||
// 0 = all gates pass
|
||||
// 1 = one or more gates failed (final summary must disclose)
|
||||
// 2 = script error (project-dir not found, etc.)
|
||||
|
||||
import { readFile, readdir } from "node:fs/promises";
|
||||
import { join, resolve, basename } from "node:path";
|
||||
import { existsSync } from "node:fs";
|
||||
|
||||
const PROJECT_DIR = resolve(process.argv[2] || ".");
|
||||
|
||||
// Thresholds — adjust here, not by interpretation.
|
||||
const ASSET_USAGE_MIN_PCT = 30;
|
||||
const SFX_DRIFT_TOLERANCE_S = 0.5;
|
||||
|
||||
const ASSET_EXTS = new Set([".jpg", ".jpeg", ".png", ".svg", ".webp", ".gif", ".mp4", ".webm"]);
|
||||
const ASSET_EXCLUDE_PATTERNS = [/favicon/i, /apple-touch-icon/i];
|
||||
|
||||
const SHADER_NAMES = [
|
||||
"cross-warp-morph",
|
||||
"cross-warp",
|
||||
"cinematic-zoom",
|
||||
"gravitational-lens",
|
||||
"glitch",
|
||||
"light-leak",
|
||||
"flash-through-white",
|
||||
"whip-pan",
|
||||
"domain-warp",
|
||||
"thermal-bloom",
|
||||
"swirl",
|
||||
"ridged-noise",
|
||||
"sdf-reveal",
|
||||
"chromatic-aberration",
|
||||
"ripple",
|
||||
];
|
||||
|
||||
// ─── Main ────────────────────────────────────────────────────────────────────
|
||||
|
||||
async function main() {
|
||||
if (!existsSync(PROJECT_DIR)) {
|
||||
console.error(`✗ Project directory not found: ${PROJECT_DIR}`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const results = [];
|
||||
results.push(await checkAssetUsage());
|
||||
results.push(await checkShaderTransitionsConsistency());
|
||||
results.push(await checkSfxTimestampConsistency());
|
||||
results.push(await checkAnimationMapExists());
|
||||
results.push(await checkMp4Exists());
|
||||
results.push(await checkStoryboardArtifactExists());
|
||||
|
||||
printReport(results);
|
||||
|
||||
const anyFail = results.some((r) => r.status === "FAIL");
|
||||
process.exit(anyFail ? 1 : 0);
|
||||
}
|
||||
|
||||
// ─── Checks ──────────────────────────────────────────────────────────────────
|
||||
|
||||
async function checkAssetUsage() {
|
||||
const captureDir = join(PROJECT_DIR, "capture", "assets");
|
||||
if (!existsSync(captureDir)) {
|
||||
return {
|
||||
name: "Asset usage",
|
||||
status: "INFO",
|
||||
detail: "No capture/assets/ directory — capture may not have run",
|
||||
};
|
||||
}
|
||||
|
||||
const captured = await listAssetFiles(captureDir);
|
||||
const compositionsDir = join(PROJECT_DIR, "compositions");
|
||||
const referencedSet = new Set();
|
||||
|
||||
const filesToScan = [join(PROJECT_DIR, "index.html")];
|
||||
if (existsSync(compositionsDir)) {
|
||||
const compFiles = (await readdir(compositionsDir)).filter((f) => f.endsWith(".html"));
|
||||
for (const f of compFiles) filesToScan.push(join(compositionsDir, f));
|
||||
}
|
||||
|
||||
for (const f of filesToScan) {
|
||||
if (!existsSync(f)) continue;
|
||||
const content = await readFile(f, "utf-8");
|
||||
for (const asset of captured) {
|
||||
if (content.includes(asset.path) || content.includes(basename(asset.path))) {
|
||||
referencedSet.add(asset.path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const total = captured.length;
|
||||
const used = referencedSet.size;
|
||||
const pct = total > 0 ? Math.round((used * 100) / total) : 0;
|
||||
const pass = pct >= ASSET_USAGE_MIN_PCT;
|
||||
|
||||
return {
|
||||
name: "Asset usage",
|
||||
status: pass ? "PASS" : "FAIL",
|
||||
detail: `${used}/${total} (${pct}%) — target ≥${ASSET_USAGE_MIN_PCT}%`,
|
||||
extra: pass
|
||||
? null
|
||||
: `Unused brand assets are sitting in capture/assets/. The brand isn't visually present at ${pct}%.`,
|
||||
};
|
||||
}
|
||||
|
||||
async function checkShaderTransitionsConsistency() {
|
||||
const storyboardPath = join(PROJECT_DIR, "STORYBOARD.md");
|
||||
const indexPath = join(PROJECT_DIR, "index.html");
|
||||
if (!existsSync(storyboardPath) || !existsSync(indexPath)) {
|
||||
return {
|
||||
name: "Shader transitions",
|
||||
status: "INFO",
|
||||
detail: "STORYBOARD.md or index.html missing — cannot check",
|
||||
};
|
||||
}
|
||||
|
||||
const storyboard = await readFile(storyboardPath, "utf-8");
|
||||
const index = await readFile(indexPath, "utf-8");
|
||||
|
||||
// Match longest names first so "cross-warp-morph" doesn't double-count as "cross-warp" too.
|
||||
const sortedNames = [...SHADER_NAMES].sort((a, b) => b.length - a.length);
|
||||
let sbScratch = storyboard;
|
||||
const declared = [];
|
||||
for (const name of sortedNames) {
|
||||
if (sbScratch.includes(name)) {
|
||||
declared.push(name);
|
||||
sbScratch = sbScratch.split(name).join(""); // strip all matches before next probe
|
||||
}
|
||||
}
|
||||
if (declared.length === 0) {
|
||||
return {
|
||||
name: "Shader transitions",
|
||||
status: "PASS",
|
||||
detail: "STORYBOARD declared none — no shader transitions expected",
|
||||
};
|
||||
}
|
||||
|
||||
const present = declared.filter((name) => index.includes(name));
|
||||
const missing = declared.filter((name) => !index.includes(name));
|
||||
const pass = missing.length === 0;
|
||||
|
||||
return {
|
||||
name: "Shader transitions",
|
||||
status: pass ? "PASS" : "FAIL",
|
||||
detail: `STORYBOARD declared ${declared.length}, ${present.length} present in index.html, ${missing.length} missing`,
|
||||
extra: pass
|
||||
? null
|
||||
: `Missing from build: ${missing.join(", ")}. STORYBOARD.md and index.html disagree.`,
|
||||
};
|
||||
}
|
||||
|
||||
async function checkSfxTimestampConsistency() {
|
||||
const storyboardPath = join(PROJECT_DIR, "STORYBOARD.md");
|
||||
const indexPath = join(PROJECT_DIR, "index.html");
|
||||
if (!existsSync(storyboardPath) || !existsSync(indexPath)) {
|
||||
return { name: "SFX timestamps", status: "INFO", detail: "STORYBOARD.md or index.html missing" };
|
||||
}
|
||||
|
||||
const storyboard = await readFile(storyboardPath, "utf-8");
|
||||
const index = await readFile(indexPath, "utf-8");
|
||||
|
||||
const sfxRefs = [];
|
||||
for (const line of storyboard.split("\n")) {
|
||||
const fileMatch = line.match(/sfx\/([\w-]+\.mp3)/);
|
||||
const timeMatch = line.match(/\|\s*(\d+(?:\.\d+)?)s?\s*\|/);
|
||||
if (fileMatch && timeMatch) {
|
||||
sfxRefs.push({ file: fileMatch[1], storyboardT: parseFloat(timeMatch[1]) });
|
||||
}
|
||||
}
|
||||
|
||||
if (sfxRefs.length === 0) {
|
||||
return { name: "SFX timestamps", status: "INFO", detail: "No SFX entries detected in STORYBOARD.md" };
|
||||
}
|
||||
|
||||
const indexSfx = new Map();
|
||||
const audioRegex =
|
||||
/<audio[^>]*src=["'](?:[^"']*\/)?sfx\/([\w-]+\.mp3)["'][^>]*?data-start=["']([0-9.]+)["']/g;
|
||||
let m;
|
||||
while ((m = audioRegex.exec(index)) !== null) {
|
||||
indexSfx.set(m[1], parseFloat(m[2]));
|
||||
}
|
||||
|
||||
const drifts = [];
|
||||
const missing = [];
|
||||
for (const ref of sfxRefs) {
|
||||
if (!indexSfx.has(ref.file)) {
|
||||
missing.push(ref.file);
|
||||
continue;
|
||||
}
|
||||
const indexT = indexSfx.get(ref.file);
|
||||
const drift = Math.abs(indexT - ref.storyboardT);
|
||||
if (drift > SFX_DRIFT_TOLERANCE_S) {
|
||||
drifts.push({ file: ref.file, storyboardT: ref.storyboardT, indexT, drift });
|
||||
}
|
||||
}
|
||||
|
||||
const pass = missing.length === 0 && drifts.length === 0;
|
||||
return {
|
||||
name: "SFX timestamps",
|
||||
status: pass ? "PASS" : "FAIL",
|
||||
detail: `${sfxRefs.length} SFX in STORYBOARD · ${indexSfx.size} in index.html · ${missing.length} missing · ${drifts.length} drifted >${SFX_DRIFT_TOLERANCE_S}s`,
|
||||
extra: pass
|
||||
? null
|
||||
: [
|
||||
...missing.map((f) => `MISSING in index.html: ${f}`),
|
||||
...drifts.map(
|
||||
(d) =>
|
||||
`DRIFT: ${d.file} storyboard=${d.storyboardT}s index=${d.indexT}s drift=${d.drift.toFixed(2)}s`,
|
||||
),
|
||||
].join("\n "),
|
||||
};
|
||||
}
|
||||
|
||||
async function checkAnimationMapExists() {
|
||||
const path = join(PROJECT_DIR, "animation-map.json");
|
||||
if (existsSync(path)) {
|
||||
return { name: "animation-map.json", status: "PASS", detail: "exists" };
|
||||
}
|
||||
return {
|
||||
name: "animation-map.json",
|
||||
status: "FAIL",
|
||||
detail: "missing — run `node <repo>/skills/hyperframes/scripts/animation-map.mjs <project-dir>`",
|
||||
};
|
||||
}
|
||||
|
||||
async function checkMp4Exists() {
|
||||
const candidates = [PROJECT_DIR, join(PROJECT_DIR, "output"), join(PROJECT_DIR, "renders")];
|
||||
for (const dir of candidates) {
|
||||
if (!existsSync(dir)) continue;
|
||||
try {
|
||||
const files = await readdir(dir);
|
||||
if (files.some((f) => f.endsWith(".mp4"))) {
|
||||
return {
|
||||
name: "Rendered MP4",
|
||||
status: "PASS",
|
||||
detail: `found .mp4 in ${dir.replace(PROJECT_DIR, ".")}`,
|
||||
};
|
||||
}
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
return {
|
||||
name: "Rendered MP4",
|
||||
status: "INFO",
|
||||
detail:
|
||||
"no .mp4 found — preview-only delivery; if claiming verified motion, render is required (Path 2 of audio+motion verification)",
|
||||
};
|
||||
}
|
||||
|
||||
async function checkStoryboardArtifactExists() {
|
||||
const required = ["STORYBOARD.md", "DESIGN.md", "SCRIPT.md", "index.html"];
|
||||
const missing = required.filter((f) => !existsSync(join(PROJECT_DIR, f)));
|
||||
if (missing.length === 0) {
|
||||
return { name: "Required artifacts", status: "PASS", detail: required.join(", ") };
|
||||
}
|
||||
return {
|
||||
name: "Required artifacts",
|
||||
status: "FAIL",
|
||||
detail: `missing: ${missing.join(", ")}`,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Helpers ─────────────────────────────────────────────────────────────────
|
||||
|
||||
async function listAssetFiles(dir) {
|
||||
const out = [];
|
||||
async function walk(d) {
|
||||
let entries;
|
||||
try {
|
||||
entries = await readdir(d, { withFileTypes: true });
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
for (const e of entries) {
|
||||
const full = join(d, e.name);
|
||||
if (e.isDirectory()) {
|
||||
await walk(full);
|
||||
continue;
|
||||
}
|
||||
const ext = e.name.slice(e.name.lastIndexOf(".")).toLowerCase();
|
||||
if (!ASSET_EXTS.has(ext)) continue;
|
||||
if (ASSET_EXCLUDE_PATTERNS.some((re) => re.test(e.name))) continue;
|
||||
const rel = full.replace(PROJECT_DIR + "/", "");
|
||||
out.push({ path: rel, name: e.name });
|
||||
}
|
||||
}
|
||||
await walk(dir);
|
||||
return out;
|
||||
}
|
||||
|
||||
// ─── Report ──────────────────────────────────────────────────────────────────
|
||||
|
||||
function printReport(results) {
|
||||
const cols = { name: 26, status: 8, detail: 60 };
|
||||
const line = "─".repeat(cols.name + cols.status + cols.detail + 6);
|
||||
|
||||
console.log("");
|
||||
console.log(`w2h-verify · ${PROJECT_DIR}`);
|
||||
console.log(line);
|
||||
console.log(
|
||||
"Check".padEnd(cols.name) + " │ " + "Status".padEnd(cols.status) + " │ " + "Detail",
|
||||
);
|
||||
console.log(line);
|
||||
|
||||
for (const r of results) {
|
||||
const symbol = r.status === "PASS" ? "✓" : r.status === "FAIL" ? "✗" : "·";
|
||||
console.log(
|
||||
r.name.padEnd(cols.name) +
|
||||
" │ " +
|
||||
`${symbol} ${r.status}`.padEnd(cols.status) +
|
||||
" │ " +
|
||||
(r.detail || ""),
|
||||
);
|
||||
if (r.extra) {
|
||||
console.log(
|
||||
"".padEnd(cols.name) +
|
||||
" │ " +
|
||||
"".padEnd(cols.status) +
|
||||
" │ " +
|
||||
r.extra.split("\n").join("\n" + " ".repeat(cols.name + cols.status + 6 + 4)),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(line);
|
||||
const pass = results.filter((r) => r.status === "PASS").length;
|
||||
const fail = results.filter((r) => r.status === "FAIL").length;
|
||||
const info = results.filter((r) => r.status === "INFO").length;
|
||||
console.log(`SUMMARY: ${pass} PASS · ${fail} FAIL · ${info} INFO`);
|
||||
|
||||
if (fail > 0) {
|
||||
console.log("");
|
||||
console.log("Step 6 NOT done. Fix FAIL items, OR include this report verbatim in your final");
|
||||
console.log("summary's \"What I did NOT verify\" section so the user knows what's broken.");
|
||||
} else {
|
||||
console.log("");
|
||||
console.log("All gates pass. Paste this report into your final user-facing summary as evidence.");
|
||||
}
|
||||
console.log("");
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
console.error("w2h-verify script error:", e);
|
||||
process.exit(2);
|
||||
});
|
||||
Reference in New Issue
Block a user