mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-02 20:18:35 +00:00
fix(skill): w2h enforcement round 2 — close the deeper shirking patterns
A second agent debrief (different session, with the prior enforcement edits
applied) revealed the most damning failure yet: the agent used 1 of 65
captured assets. They wrote their own "Asset Audit" table saying SKIP for
64 hero illustrations, brand SVGs, and signature visuals. The entire point
of capture is to USE the brand's assets — that gate was bypassed entirely.
The debrief also revealed 5 other patterns the prior enforcement missed.
This commit closes all 6.
**Pattern A — Asset Audit gate (step-3-storyboard.md)** — BIGGEST FIX
Adds a non-skippable Asset Audit section in Step 3 that requires viewing
every page of capture/assets/contact-sheet-*.jpg + svgs/contact-sheet-*.jpg,
pasting 5 distinctive assets per page (with descriptions of what's actually
pictured), and choosing USE/SKIP per asset with one-sentence justification
for each SKIP. Brand-defaults floor: at least one beat MUST use a captured
hero illustration/photograph/signature diagram — not just the logo. The
forbidden list explicitly calls out: reading asset-descriptions.md alone
without opening the contact sheets, and rebuilding signature graphics in
CSS when the brand's own SVG of that graphic is in capture/assets/.
**Pattern B — Auto mode scope (SKILL.md + step-2-brief.md)**
Clarifies that auto mode covers user-PREFERENCE gates (TTS provider,
voice, beat count, captions yes/no — where the agent decides on the
user's behalf) but NOT quality-VERIFICATION gates (Asset Audit, per-beat
HTML read, DoD checklist, honest disclosure). Adds explicit test for
distinguishing: if the answer changes the content of the video, it's a
preference; if the answer is "did the verification happen?", it's a
quality gate. The agent that skipped the captions question by reasoning
"auto mode says bias toward action" was misusing auto mode.
**Pattern C — 3-path audio/motion verification (step-6-validate.md)**
Replaces the prior "Path 1 or explicit deferred" with three explicit
paths: (1) Play preview in Playwright, (2) render low-res MP4 and read
frames at ≥5fps, or (3) explicit deferred disclosure with QUANTIFIED
coverage gap ("18/900 frames = 2% coverage"). The percentage in Path 3 is
mandatory — vague "deferred to user" was the loophole. Forbidden: claiming
"confirmed via snapshot" as audio/motion evidence; 18 PNGs from a 900-
frame video is 2% coverage, not verification.
**Pattern D — Sub-agent diagnoses are hypotheses (beat-builder-guide.md)**
When a sub-agent reports "this is a linter false positive" / "this is a
known bug", that is a HYPOTHESIS from one symptom — not a verified
finding. Before propagating the workaround to other beats, main agent
must EITHER read the source to confirm OR explicitly disclose the
unverified claim. The debrief showed the main agent applied beat-2's
"linter false positive" diagnosis to beat-4 without ever reading
packages/core/src/lint/utils.ts to confirm.
**Pattern E — Re-snapshot after parallel sub-agents (step-5-build.md)**
When sub-agents run in parallel, each snapshots a project where sibling
beats may not exist yet. Their snapshots at beat boundaries or during
shader transitions show the WRONG content (typically previous beat).
Required after all complete: a canonical project-wide snapshot via the
CLI — that's what Step 6's DoD uses. Sub-agents' intermediate snapshots
are sanity checks, not the deliverable.
**Pattern F — STORYBOARD.md must be updated when divergence accepted
(beat-builder-guide.md)**
When a sub-agent diverges from spec ("the real brand mark is lowercase
'huly' not uppercase 'HULY'") AND the main agent accepts the divergence,
the main agent MUST patch STORYBOARD.md to reflect reality. Otherwise
the spec lies and the next session reading it as ground truth gets the
wrong information. Examples covered: brand mark casing, cell size at
scale, SFX timing alignment.
6 files changed, +131/-11 (net +120 lines of enforcement).
Format checks pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
18444decde
commit
5594a8286c
@@ -14,7 +14,16 @@ Users say things like:
|
||||
- "Turn this website into a 15-second social ad for Instagram"
|
||||
- "Create a 30-second product tour from https://..."
|
||||
|
||||
The workflow has 7 steps. Each produces an artifact that gates the next. By default it's collaborative — gates marked 💬 stop and ask the user. If the user signals autonomous mode ("decide for me", "surprise me"), every 💬 gate is skipped; see step-2-brief.md for how that propagates.
|
||||
The workflow has 7 steps. Each produces an artifact that gates the next. By default it's collaborative — gates marked 💬 stop and ask the user. If the user signals autonomous mode ("decide for me", "surprise me"), 💬 user-preference gates are skipped; see step-2-brief.md for how that propagates.
|
||||
|
||||
**Autonomous mode is NOT "skip all gates."** Auto mode covers user-preference questions (TTS provider, voice, color emphasis, beat count, music yes/no, captions yes/no — where the agent decides on the user's behalf). It does NOT cover quality-verification gates. The following remain non-skippable in auto mode:
|
||||
|
||||
- Asset Audit (Step 3) — viewing contact sheets and justifying USE/SKIP for each asset
|
||||
- Per-beat HTML read (Step 5) — structured evidence block per beat
|
||||
- DoD checklist (Step 6) — including animation-map, per-warning WCAG verification, audio/motion playback
|
||||
- Honest disclosure section (Step 6) — "What I did NOT verify" must appear in your final summary
|
||||
|
||||
If you find yourself reasoning "auto mode says bias toward action, so I'll skip X" — and X is a verification gate, not a preference question — that reasoning is wrong. Bias toward action applies to deciding _what to build_, not to deciding _whether to verify_.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -156,6 +156,29 @@ FLAG: STORYBOARD.md beat 3 says "Row 1 transitions blue → orange at 3.5s" but
|
||||
|
||||
The main agent then confirms or corrects before Step 6 advances. Picking an interpretation silently means the build looks "fine" while diverging from intent — and the user only notices in motion.
|
||||
|
||||
### Sub-agent diagnoses are unverified claims, not facts
|
||||
|
||||
When a sub-agent reports "this is a linter false positive" / "this is a known bug" / "this attribute doesn't work as documented" — those are HYPOTHESES, not findings. Sub-agents diagnose from one symptom; they don't have repo-wide context.
|
||||
|
||||
Before propagating any sub-agent diagnosis (e.g., applying the same "workaround" to another beat, or telling the user "this is a known bug"), do ONE of:
|
||||
|
||||
1. **Verify by reading the source.** Open the file the sub-agent claims is buggy. Confirm the bug exists. Example: "I read `packages/core/src/lint/utils.ts:42` and confirmed the regex matches `url(\"data:image/svg+xml...\")` incorrectly. The workaround is to base64-encode the URI."
|
||||
2. **Disclose the unverified claim.** Don't suppress it — surface it. Example: "Sub-agent for beat 2 diagnosed `root_missing_composition_id` as a linter false positive on inline SVG data URIs. I applied the same workaround to beat 4 WITHOUT verifying the underlying claim. Worth filing as a regression against `packages/core/src/lint/utils.ts` to confirm."
|
||||
|
||||
**Forbidden:** silently adopting the workaround pattern and presenting "lint passes" as evidence. If the workaround came from an unverified diagnosis, "lint passes because the diagnosis was correct AND I worked around it" and "lint passes because the diagnosis was wrong but the workaround happened to make the symptom disappear" are both possible. Without verification, you don't know which. The next session inherits the workaround AND the unverified diagnosis.
|
||||
|
||||
### When you accept a sub-agent's divergence from spec — UPDATE the spec
|
||||
|
||||
If a sub-agent reports "I diverged from STORYBOARD.md because..." AND you accept the divergence, you MUST update STORYBOARD.md to reflect the actual implementation. Otherwise the spec lies about the artifact.
|
||||
|
||||
Examples:
|
||||
|
||||
- Sub-agent: "Storyboard says 'HULY' uppercase but the actual logo asset is lowercase 'huly'. I used lowercase." Accept → edit STORYBOARD.md beat N to say "huly" lowercase. Note the change inline.
|
||||
- Sub-agent: "Storyboard says cells at 56px but they read too small at 1920×1080. I used 96px." Accept → edit STORYBOARD.md beat N's cell size to 96px.
|
||||
- Sub-agent: "Storyboard says SFX at t=4.7s but the visual moment lands at t=5.2s; I aligned SFX to the visual." Accept → edit STORYBOARD.md SFX line to t=5.2s.
|
||||
|
||||
**Forbidden:** accepting the divergence silently and leaving the storyboard with the wrong spec. The next session reading STORYBOARD.md will trust it as ground truth. The spec is a contract; if you break the contract, update the contract.
|
||||
|
||||
---
|
||||
|
||||
## Continuous motion — the most important rule
|
||||
|
||||
@@ -118,7 +118,16 @@ When the user gives no creative direction, default to what the brand's visual id
|
||||
|
||||
With that minimum in hand, still write an ambitious storyboard. "Surprise me" means "impress me," not "play it safe." Go bold.
|
||||
|
||||
**Autonomous mode propagates.** When the user signals "surprise me" / "decide for me" / "just build it" here at Step 2, that signal kills every downstream 💬 gate too — Step 3's storyboard approval, Step 4's TTS provider choice, captions. Make all creative decisions yourself and present the finished video at the end. Do not ask four separate questions across four separate steps. Read the room once and commit.
|
||||
**Autonomous mode propagates — for user-preference gates only.** When the user signals "surprise me" / "decide for me" / "just build it" here at Step 2, that signal kills downstream user-preference 💬 gates: Step 3's storyboard approval, Step 4's TTS provider choice, music yes/no, captions yes/no. Make those creative decisions yourself and present the finished video at the end. Do not ask four separate questions across four separate steps. Read the room once and commit.
|
||||
|
||||
**Auto mode does NOT skip quality-verification gates.** These run regardless and must produce evidence in your final summary:
|
||||
|
||||
- Asset Audit (Step 3) — view contact sheets, justify USE/SKIP per asset
|
||||
- Per-beat HTML evidence block (Step 5)
|
||||
- DoD checklist (Step 6) — animation-map, per-warning WCAG verification, audio + motion playback (or explicit "deferred" disclosure)
|
||||
- "What I did NOT verify" disclosure (Step 6)
|
||||
|
||||
**Test for "preference vs quality gate":** if the answer changes the _content_ of the video (which voice? captions on? beat 3 cinematic or fast?), it's a preference — auto mode decides. If the answer is "did the verification happen?", it's a quality gate — auto mode does NOT apply. Reasoning "auto mode says bias toward action, so I'll skip the contact sheets" misuses auto mode.
|
||||
|
||||
### Specific direction
|
||||
|
||||
|
||||
@@ -139,6 +139,44 @@ There might be VFX blocks available (vfx-liquid-glass, vfx-iphone-device, vfx-sh
|
||||
|
||||
**Shader transitions — block name ≠ shader name.** When you run the commands above and see `domain-warp-dissolve` in `registry/blocks/`, the HyperShader runtime name is `domain-warp` (without "-dissolve"). After installing a block, open its showcase HTML (`compositions/<block-name>.html`) and find the actual shader name used in `HyperShader.init()`. That is what you put in the storyboard. Then delete the showcase file — it's a demo only and will pollute your compositions/ directory with lint warnings.
|
||||
|
||||
## Asset Audit — REQUIRED before writing beats (non-skippable)
|
||||
|
||||
The skill's #1 purpose is to USE the brand's captured assets — not rebuild them from CSS. Most of your beats should feature at least one captured asset: a hero illustration, a signature SVG, product photography, brand mark, distinctive graphic. **If your STORYBOARD.md ends with only the logo used, you have failed this step.**
|
||||
|
||||
**Why this gate exists:** Earlier sessions wrote their own "Asset Audit" that said SKIP for 60+ of 65 captured assets, used only the logo, and shipped a video that visually was indistinguishable from a generic dark-mode SaaS launch. The captured MetaBrain illustration, the GitHub-sync diagram, the knowledge-base hero — all left on the floor. The signature visuals that make a brand recognizable were absent. Don't repeat that.
|
||||
|
||||
**Required pre-storyboard procedure:**
|
||||
|
||||
1. **View every page of `capture/assets/contact-sheet-*.jpg`** AND every page of `capture/assets/svgs/contact-sheet-*.jpg`. These are the sheets generated by capture for this exact purpose. Open each page; scan cell-by-cell. Do not skim — you are looking for the brand's visual identity, frame by frame.
|
||||
|
||||
2. **For each contact sheet page, paste this block into STORYBOARD.md under an "Asset Audit" section:**
|
||||
|
||||
```
|
||||
Contact sheet: capture/assets/contact-sheet-1.jpg (page 1 of N)
|
||||
5 most visually distinctive assets I see (filename + one-sentence description of what the image shows):
|
||||
1. <filename>: <what's actually pictured — not the filename, the content>
|
||||
2. <filename>: <description>
|
||||
3. <filename>: <description>
|
||||
4. <filename>: <description>
|
||||
5. <filename>: <description>
|
||||
```
|
||||
|
||||
Repeat for every contact sheet page. The number of pages × 5 is your candidate asset pool.
|
||||
|
||||
3. **For each beat in STORYBOARD.md**, choose USE or SKIP for each candidate asset:
|
||||
- **USE** means the asset appears in the beat's HTML at build time (`<img src=...>`, inline SVG, `background-image: url(...)`).
|
||||
- **SKIP** requires a one-sentence reason explaining why this asset doesn't serve this beat. "Doesn't fit storyboard" is not a reason — name which storyboard moment failed to find a use for it.
|
||||
|
||||
4. **Brand-defaults floor:** at least ONE beat must use the brand's signature visual (hero illustration, hero photograph, or signature diagram — not the logo). If you've named 5+ candidate hero illustrations in step 2 above and zero of them appear in any beat, that is the failure mode this gate exists to catch.
|
||||
|
||||
**Forbidden:**
|
||||
|
||||
- Writing "SKIP" for every asset except the logo without per-asset justification
|
||||
- Reading `capture/extracted/asset-descriptions.md` (the text file) and making decisions from filenames alone, without opening the contact sheets
|
||||
- Concluding "I'll rebuild the GitHub-sync diagram in CSS" when the brand's own SVG of that diagram is sitting in `capture/assets/`. Use the real asset.
|
||||
|
||||
If your final beat list uses less than ~30% of relevant captured assets (relevant = anything except the favicon and tiny UI icons), revisit. The brand is visually carried by its own art; rebuilding it from divs erases what makes it recognizable.
|
||||
|
||||
### HTML-in-Canvas — plan for it here, build in Step 5
|
||||
|
||||
The `drawElementImage` Chrome API captures any live HTML/CSS as a GPU-accelerated texture at 60fps. This is HyperFrames' highest-impact capability — it lets you render captured product screenshots or UI through:
|
||||
|
||||
@@ -389,6 +389,22 @@ For every `.html` file in `compositions/`, confirm that `index.html` has a `data
|
||||
|
||||
**Captions stub rule:** Never create a `compositions/captions.html` with an empty transcript (`const script = [];`). If the VO/transcript step was skipped or failed, do not create the captions composition at all. An empty captions file that returns immediately is worse than no captions file — it silently does nothing and wastes a track slot.
|
||||
|
||||
### Parallel sub-agent snapshots are stale — re-snapshot after all complete
|
||||
|
||||
When you dispatch sub-agents in parallel (one per beat), each sub-agent snapshots a project where sibling beats may not exist yet. Their per-beat snapshots are valid for THEIR beat in isolation, but **any snapshot at a beat boundary or during a shader transition will show the wrong content** — typically the previous beat's content because the next beat hasn't been built.
|
||||
|
||||
Example: Beat 6's sub-agent took a snapshot at t=25.7s and saw Beat 1's content because Beat 5 didn't exist yet when Beat 6's sub-agent ran. The sub-agent reported this as "shader transition behavior showing previous scene" — a plausible-sounding but wrong diagnosis.
|
||||
|
||||
**Required after all sub-agents complete:**
|
||||
|
||||
```bash
|
||||
node /<repo-root>/packages/cli/dist/cli.js snapshot <project-dir> --frames <N>
|
||||
```
|
||||
|
||||
where N follows the snapshot formula: `max(beats × 3, ceil(duration_seconds / 2))`. This is the canonical snapshot that Step 6's DoD uses — not any individual sub-agent's intermediate snapshots.
|
||||
|
||||
Sub-agents' snapshots are still useful as per-beat sanity checks, but they are not the deliverable. The post-completion snapshot is. Don't skip it because "the sub-agents already snapshotted."
|
||||
|
||||
## 5. Read each beat HTML top-to-bottom — REQUIRED gate before Step 6
|
||||
|
||||
**This gate is non-skippable.** "I read it and it looks fine", "the sub-agent confirmed", "the snapshots look right" are NOT acceptable. Snapshots are 3 frames out of 300+ in motion — they hide everything that goes wrong between them.
|
||||
|
||||
@@ -173,27 +173,52 @@ If you cannot find any problems and want to score everything 4–5, you are not
|
||||
|
||||
Read every score. Fix anything below 3 before showing the user. If the CTA scores below 3, fix the CTA. Do not rationalize low scores as "the user can decide."
|
||||
|
||||
## Audio + motion verification — separate from snapshot verification
|
||||
## Audio + motion verification — three paths, pick one
|
||||
|
||||
Snapshots are silent stills. They prove what FRAMES look like, but they do NOT prove:
|
||||
Snapshots are silent stills. 18 PNG snapshots from a 30s 30fps video = 18/900 = 2% of frames. The other 98% — including all motion, all transitions, all audio — is unverified by snapshots alone. "Confirmed via snapshot" is not coverage.
|
||||
|
||||
- That audio fires at all
|
||||
- That SFX lands exactly when the visual moment lands (target: ±0.1s, hard limit: ±0.5s)
|
||||
- That narration syncs to per-beat content
|
||||
- That transitions feel right in motion
|
||||
You MUST do ONE of these three paths before declaring done:
|
||||
|
||||
After snapshot DoD items pass, **play the preview Studio URL in a browser** (or via Playwright if available). Don't scrub — actually play it from start to end at 1.0× speed. Then verify:
|
||||
### Path 1 (preferred): Play the preview
|
||||
|
||||
Open the Studio URL in a browser via Playwright (or another browser tool you have). Play start-to-end at 1.0× speed (NOT scrubbed). Confirm:
|
||||
|
||||
```
|
||||
[ ] Played full video front-to-back at 1.0× — actually played, not scrubbed
|
||||
[ ] For each SFX in STORYBOARD.md: sound lands at the visual moment within ±0.1s
|
||||
(Beat N SFX `<file>`: storyboard says t=<x>s → heard at t=<y>s → drift <z>s)
|
||||
[ ] Narration delivers the right line per beat (no off-by-one or missing lines)
|
||||
[ ] No moments where audio is present but visual is unintentionally mid-transition
|
||||
[ ] No moments where audio is present but visual is mid-transition unintentionally
|
||||
[ ] Audio audible and not clipped/peaked
|
||||
```
|
||||
|
||||
**If you cannot play the preview** (no Playwright in this session, CLI-only environment), state this explicitly: "Audio + motion verification deferred to user — no preview tool available in this session." That is honest disclosure. **Silently skipping this check while presenting a "looks good" verdict fails the gate.**
|
||||
### Path 2: Render a low-res MP4 and read it frame-by-frame
|
||||
|
||||
When Playwright isn't available, render at 540p (fast — ~30s for a 30s video) and read the MP4:
|
||||
|
||||
```bash
|
||||
node /<repo-root>/packages/cli/dist/cli.js render <project-dir> \
|
||||
--width 960 --height 540 --quality medium
|
||||
```
|
||||
|
||||
Then sample the resulting MP4 at minimum 5fps (use `ffmpeg -i <mp4> -r 5 frames/frame-%04d.png` if needed). Read those frames sequentially. For each SFX moment in STORYBOARD.md, find the corresponding frame and confirm the visual matches.
|
||||
|
||||
### Path 3 (last resort): Explicit deferred disclosure with quantified gap
|
||||
|
||||
If neither Path 1 nor Path 2 is possible in this session, your final summary MUST contain this verbatim:
|
||||
|
||||
```
|
||||
**Audio + motion verification: NOT POSSIBLE in this session.**
|
||||
- Snapshots cover: <N> frames out of <video_duration × fps> total (<percentage>% coverage)
|
||||
- NOT verified: motion between snapshots, SFX/visual timing alignment, shader transition smoothness, audio mix levels, narration sync to beats
|
||||
- Recommended user action: open the preview URL above and play start-to-end; flag anything that feels off
|
||||
```
|
||||
|
||||
**Forbidden everywhere:**
|
||||
|
||||
- "Confirmed via snapshot" or "snapshots look right" as audio/motion evidence
|
||||
- "Preview is running, looks good" without actually playing it
|
||||
- Path 3 disclosure that omits the quantified coverage gap (the percentage is mandatory)
|
||||
|
||||
## Preview (always do this)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user