Files
hyperframes/skills/remotion-to-hyperframes/references/eval.md
T
WaterrrForeverandMiguel Angel Simon Sierra f2358315e5 feat(skills): re-land reverse any video on the HyperFrames plan, with clearer credit wording (#6833)
* fix(cli,skills): say a step's credits are its total, and keep cost out of retries

An out-of-credits refusal named what the step needs, and an agent read
that as the shortfall. It now says the number is the step's total.

After its last try, an agent told the person what was spent and what
another try would cost. The HeyGen reference now says not to.

* fix(cli): plan avatar credits from the HyperFrames price rows

An avatar's credits came from its old API price, about five times what
the plan charges, so --max-credits refused calls that fit and a top-up
asked for too much. They now use the Avatar IV photo row plus the
general avatar video row, 950 credits a minute.

* fix(cli): pin the avatar price test and keep pricing under the comment share

* fix(cli): plan avatar seconds on the Avatar IV row alone

The general avatar row is only checked at admission and billed on
seconds left after Avatar IV, so adding it to every second made a 15 s
script plan at 238 instead of 200 and a 220 limit refuse a call that fits.

* fix(cli): the credits link opens the web plan page, which is live

* test(cli): pin the production credits link too

* chore(skills): regenerate the skills manifest after merging main

* docs(cli): keep the person measure's doc comments to four lines

* docs(cli): keep the measure doc comments to four lines

* docs(cli): keep the measure comment lines to 120 characters

---------

Co-authored-by: Miguel Angel Simon Sierra <miguel.sierra@heygen.com>
2026-10-31 00:47:54 +00:00

4.3 KiB

Eval: how to validate a translation end-to-end

Every translation should be measured. The skill ships a source linter and a tiered test corpus; the CLI's diff command scores the result. Together they gate translation quality.

The tools

Tool Input Output
scripts/lint_source.py Remotion source dir or file JSON findings + exit code (0 clean, 1 has blockers)
npx hyperframes diff A B original MP4, HF MP4 per-frame SSIM, worst cell per frame, cuts, worst.jpg sheet; exit 1 on fail

Run them in this order: lint → render → diff → (if fail) read the sheet.

Per-fixture flow

# 1. Lint the source — blockers mean stop
python3 ../../scripts/lint_source.py ./remotion-src/src/

# 2. Generate any binary assets (T2+T3 only)
[ -f setup.sh ] && ./setup.sh

# 3. Render Remotion baseline
cd remotion-src && npm install && npm run render
# -> remotion-src/out/baseline.mp4

# 4. Render HF translation
cd .. && node ../../../packages/cli/dist/cli.js render hf-src/ --output hf.mp4
# -> hf.mp4

# 5. SSIM diff, gated on the tier's mean threshold only
mkdir -p diff && node ../../../packages/cli/dist/cli.js diff ./remotion-src/out/baseline.mp4 ./hf.mp4 \
  --min-mean 0.95 --min-p05 0 --min-cell 0 --out ./diff --json > diff/summary.json
# -> diff/summary.json, diff/worst.jpg, diff/ssim.log

Before rendering, npx hyperframes diff <baseline.mp4> hf-src --at 0.5,2,4.1 scores the project at chosen times with no render.

Reading diff/summary.json

{
  "pass": true,
  "ssim": { "frames": 90, "mean": 0.974, "p05": 0.972, "min": 0.971, "cell_p05": 0.93 },
  "worst": [
    { "t": 1.2, "frame": 36, "ssim": 0.971, "cell": { "roi": [480, 270, 480, 270], "ssim": 0.88 } }
  ],
  "cuts": { "reference": [1.5], "yours": [1.5], "missing": [], "extra": [], "pass": true },
  "duration": { "reference": 3, "yours": 3, "pass": true },
  "sheet": "diff/worst.jpg"
}
Field What it tells you
ssim.mean average SSIM across all frames; the headline number
ssim.p05 5th percentile: most frames sit above it
ssim.cell_p05 each frame's worst cell of a 4x4 grid; a wrong element shows here, not in mean
worst the lowest-scoring moments, each with the cell (in original pixels) that failed
cuts hard cuts on both sides; a missing or extra one is a sequencing problem
pass every gate held: --min-mean, --min-p05, --min-cell, cuts and duration

Threshold rule of thumb

Set the threshold ~0.02 below measured p05:

  • Real translation regressions drop mean by 0.05+ — caught.
  • Encoder/font drift between CI runs is bounded at ~0.01 — not caught.

If a calibration run's measured mean is far above your initial threshold guess, don't tighten the threshold to fit. Leave headroom — fixtures re-rendered on different hardware will drift.

When the diff fails

  1. Look at diff/worst.jpg first. The worst moments side by side, each with its failing cell boxed, show whether the failure is structural (wrong scene durations, missing element) or cosmetic (different font weight, slight timing skew).
  2. Check diff/ssim.log. Per-frame SSIM tells you which frames failed. Cluster of bad frames in the middle of a scene = animation problem; bad frames at scene boundaries = sequencing problem.
  3. Re-read the relevant reference. timing.md for spring/easing issues, sequencing.md for scene boundary issues, media.md for asset loading issues.

CI integration

The fixtures are not yet wired into CI (packages/producer/tests/ runs inside Docker; the skill corpus needs the same). PR 7 of the stack adds the orchestrator that runs all four tiers and emits an aggregated pass report. For now, evaluate by hand per fixture.