ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-13 07:40:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(skills): remotion-to-hyperframes SKILL.md + orchestrator (7/7)

Browse Source 
The leaf PR. Replaces the placeholder SKILL.md from PR 1 with the real
5-step workflow that loads the per-topic references on demand
(skill-creator's progressive-disclosure pattern), and adds a top-level
orchestrator that runs every tier and reports a pass/fail summary.

SKILL.md changes:
  - Frontmatter unchanged from PR 1 (already covers the trigger phrases
    and out-of-scope cases)
  - Body rewritten as a 5-step workflow:
      1. Lint (load escape-hatch.md if blockers)
      2. Plan (load api-map.md, then per-topic references on demand)
      3. Generate (HF index.html with paused GSAP timeline)
      4. Validate (render_diff.sh against per-tier threshold)
      5. Document gaps (TRANSLATION_NOTES.md if needed)
  - Includes a "Source contains -> Load reference" table so the agent
    only loads the references the source actually needs
  - Documents the validated baseline numbers (T1 0.974, T2 0.985,
    T3 0.953, T4 8/8) so reviewers can reproduce
  - Calls out the critical Remotion encoder config (PNG + BT.709) that
    avoids the ~0.05 SSIM hit from yuvj420p vs yuv420p

Orchestrator (assets/test-corpus/run.sh):
  - Iterates tier-1-* through tier-4-* directories
  - T1-T3: setup -> lint -> npm install (lazy) -> render Remotion ->
           render HF -> SSIM diff at the fixture's expected threshold ->
           generate strip on failure
  - T4: validate.sh (lint-only)
  - Emits run-report.json with per-tier pass/fail and aggregate counts
  - Accepts a single-tier argument for fast iteration: ./run.sh tier-1-title-card

Validated end-to-end on a clean checkout:
    ▶ tier-1-title-card → mean SSIM 0.9739 (≥ 0.95) ✓
    ▶ tier-2-multi-scene → mean SSIM 0.985292 (≥ 0.95) ✓
    ▶ tier-3-data-driven → mean SSIM 0.952941 (≥ 0.9) ✓
    ▶ tier-4-escape-hatch → 8/8 cases ✓
    passed 4/4, failed 0, skipped 0

Closes the 7-PR stack: scaffold, eval harness, 4 tiers of corpus,
references, and now the SKILL.md body that ties everything together.
This commit is contained in:
James
2026-04-28 00:00:10 +00:00
parent 890f305cd1
commit b7769b2364
3 changed files with 330 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/remotion-to-hyperframes/SKILL.md
+80 -11
View File
@@ -7,26 +7,95 @@ description: Translate a Remotion (React-based) video composition into a HyperFr
## Overview
Translate Remotion (React-based) video compositions into HyperFrames (HTML + GSAP) compositions. Most Remotion idioms have direct HyperFrames equivalents — the translation is mechanical for ~80% of typical compositions. This skill encodes the mapping and guards against the lossy 20%.
Translate Remotion (React-based) video compositions into HyperFrames (HTML + GSAP) compositions. Most Remotion idioms have direct HyperFrames equivalents — the translation is mechanical for ~80% of typical compositions. This skill encodes the mapping and guards against the lossy 20% by refusing to translate patterns that don't fit HF's seek-driven model and recommending the runtime interop pattern from [PR #214](https://github.com/heygen-com/hyperframes/pull/214) instead.
The skill ships with a **tiered test corpus** (T1–T4, 4 fixtures total) that grades translations against measured SSIM thresholds. Don't translate without running the eval — a translation that "looks right" but renders 0.05 SSIM lower than the validated baseline is silently wrong.
## Workflow
1. **Lint the source.** Run the source-lint script against the Remotion project to surface any patterns that can't translate cleanly (React state hooks, async metadata, third-party React components). If the source uses any blocker pattern, recommend the runtime interop escape hatch (PR #214 pattern) instead of attempting a translation.
### Step 1: Lint the source
2. **Scaffold the translation.** Generate a HyperFrames HTML skeleton from the Remotion source — `Composition` props become `data-*` attributes on the root `#stage` div, `<Sequence>` wrappers become elements with `data-start` / `data-duration` / `data-track-index`, `<AbsoluteFill>` becomes `<div style="position:absolute;inset:0">`. Leave timing-sensitive and easing-sensitive sections marked for refinement.
Run [`scripts/lint_source.py`](scripts/lint_source.py) over the Remotion source directory. The lint detects patterns that can't translate cleanly:
3. **Refine timing and easing.** Convert each `useCurrentFrame`-driven `interpolate` / `spring` call into an equivalent paused GSAP tween on the composition timeline. This is the part where translation correctness matters most — easing curves and stagger timing are what readers notice.
- **Blockers** (refuse + recommend interop): `useState`, `useReducer`, `useEffect`/`useLayoutEffect` with non-empty deps, async `calculateMetadata`, third-party React UI libraries (MUI, Chakra, Mantine, antd, shadcn, Radix, NextUI).
- **Warnings** (translate after dropping the construct): `@remotion/lambda` config, `delayRender`, `useCallback`, `useMemo`, custom hooks.
- **Info** (translate with note): `staticFile`, `interpolateColors`.
4. **Validate by frame-diff.** Render both the original Remotion composition and the translated HyperFrames composition, then compute per-frame SSIM. Threshold-based pass/fail tells the user which scenes are visually correct and which need another pass.
If any blocker fires, **stop**. Read [`references/escape-hatch.md`](references/escape-hatch.md) and surface the recommendation message. Warnings don't stop translation — drop the offending construct in step 3 and note the gap in `TRANSLATION_NOTES.md`. `@remotion/lambda` config is the canonical warning case: the skill drops the import + `renderMediaOnLambda(...)` calls but translates the rest of the composition.
5. **Document the gaps.** Any Remotion features that didn't translate (custom React subcomponents requiring manual rewrite, library transitions without a HyperFrames equivalent, etc.) get listed in a `TRANSLATION_NOTES.md` next to the output so the user can finish them or decide to use the runtime interop instead.
### Step 2: Plan the translation
Read [`references/api-map.md`](references/api-map.md) — the index of every Remotion API and its HF equivalent or per-topic reference. Identify which topic references you'll need based on what the source uses:
| Source contains | Load reference |
| ------------------------------------------------------------------------- | --------------------------------------------- |
| `Composition`, `defaultProps`, `schema`, `calculateMetadata` | [`parameters.md`](references/parameters.md) |
| `Sequence`, `Series`, `Loop`, `AbsoluteFill`, `Freeze` | [`sequencing.md`](references/sequencing.md) |
| `useCurrentFrame`, `interpolate`, `spring`, `Easing`, `interpolateColors` | [`timing.md`](references/timing.md) |
| `Audio`, `Video`, `Img`, `IFrame`, `staticFile`, `delayRender` | [`media.md`](references/media.md) |
| `TransitionSeries`, `@remotion/transitions` | [`transitions.md`](references/transitions.md) |
| `@remotion/lottie` | [`lottie.md`](references/lottie.md) |
| `@remotion/google-fonts/<Family>`, `Font.loadFont`, `@font-face` | [`fonts.md`](references/fonts.md) |
Don't load all of them — load only what the specific source needs.
### Step 3: Generate the HF composition
Emit `index.html` with:
- Root `<div id="stage">` carrying the composition's `data-composition-id`, `data-start="0"`, `data-duration` (in seconds), `data-fps`, `data-width`, `data-height`, plus one `data-*` per scalar prop.
- A flat list of scene divs with `data-start` / `data-duration` / `data-track-index`.
- Inline `<style>` for layout; CSS sets the `from` state of every animated property.
- A single `<script>` tag at the bottom containing one paused `gsap.timeline({paused: true})`. Every Remotion `useCurrentFrame()` derivation becomes a tween on this timeline at the right offset.
- `window.__timelines["<composition-id>"] = tl;` registers the timeline with HF's runtime.
Custom React subcomponents inline as repeated HTML using the prop interface as the template (see [`parameters.md`](references/parameters.md) for the per-instance `data-*` pattern).
### Step 4: Validate
Run the eval harness — [`references/eval.md`](references/eval.md) for the full guide. Quick path:
```bash
# Render Remotion baseline (after npm install in the fixture)
cd remotion-src && npx remotion render <CompositionId> out/baseline.mp4
# Render HF translation
cd ../hf-src && npx hyperframes render --output ../hf.mp4
# SSIM diff
../../scripts/render_diff.sh ./remotion-src/out/baseline.mp4 ./hf.mp4 ./diff
```
Threshold: ~0.02 below `p05` of the source's complexity tier (see `eval.md`'s validated thresholds table). If the diff fails, run [`scripts/frame_strip.sh`](scripts/frame_strip.sh) to see _which_ frames diverged, then re-read the relevant timing/sequencing/media reference.
**Critical**: both renders must use matching pixel format. Set `Config.setVideoImageFormat("png")` + `Config.setColorSpace("bt709")` in the Remotion source's `remotion.config.ts` — otherwise the diff measures encoder differences (~0.05 SSIM hit), not translation fidelity.
### Step 5: Document gaps
Anything that didn't translate cleanly (volume ramps dropped, custom presentations approximated, fonts substituted) gets a `TRANSLATION_NOTES.md` written next to the HF output. See [`references/limitations.md`](references/limitations.md) for the format.
## What this skill explicitly does NOT do
- **Translate React state machines.** Remotion compositions that drive animation via `useState` + `useEffect` are not deterministic frame-capture targets in HyperFrames' model; recommend the runtime interop escape hatch.
- **Translate `@remotion/lambda` configuration.** HyperFrames is single-machine today; Lambda-specific code drops with a note.
- **Run Remotion's render pipeline alongside HyperFrames.** That's the runtime interop pattern from [PR #214](https://github.com/heygen-com/hyperframes/pull/214) — a separate problem with a separate (and existing) solution.
- **Translate React state machines.** Compositions that drive animation via `useState` + `useEffect` are not deterministic frame-capture targets in HyperFrames' seek-driven model. Recommend the runtime interop pattern.
- **Run Remotion's render pipeline alongside HyperFrames.** That's the runtime interop pattern from [PR #214](https://github.com/heygen-com/hyperframes/pull/214) — a separate solution for compositions that fail this skill's lint.
## Status
(`@remotion/lambda` is _not_ a blocker — Lambda config is deployment, not animation. The skill drops it as a warning and translates the rest. See [`references/escape-hatch.md`](references/escape-hatch.md).)
Skill scaffold landed; eval harness, test corpus, and translation references are added in subsequent PRs in the stack. Until then, this skill should bow out and recommend the user hand-translate or use the runtime interop pattern.
## How to grade your own translation
Run the test corpus orchestrator:
```bash
./assets/test-corpus/run.sh
```
It runs T1, T2, T3 (render + diff) and T4 (lint validation), prints a per-tier pass/fail table, and emits an aggregate JSON report. Use this to verify the skill is working end-to-end on a clean checkout — and as a regression check after editing any reference.
Validated baseline (as of 2026-04-27):
| Tier | Composition shape | Mean SSIM | Threshold |
| ---- | ------------------------------------------- | --------- | --------- |
| T1 | single-element fade-in | 0.974 | 0.95 |
| T2 | multi-scene + spring + audio + image | 0.985 | 0.95 |
| T3 | data-driven, custom subcomponents, count-up | 0.953 | 0.90 |
| T4 | escape-hatch (8 lint cases) | 8/8 pass | n/a |