mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-11 14:50:02 +00:00
feat: website capture pipeline + 7-step video production skill (#284)
* feat(cli): add website capture with AI-powered DESIGN.md generation Adds `hyperframes capture <url>` command that extracts a complete design system from any website, producing AI-agent-ready output: - Full-page screenshot (lazy-load aware, nav at top) - AI-generated DESIGN.md via Claude API (colors, typography, elevation, components, do's/don'ts) with programmatic asset catalog (136+ assets with HTML context annotations like img[src], css url(), link[rel=preload]) - CSS-purged compositions (87% size reduction via PurgeCSS) - HTML-prettified compositions (one-tag-per-line for AI readability) - CLAUDE.md + .cursorrules auto-generated for AI agent instructions - Asset deduplication (srcset variants) and tracking pixel filtering * feat(cli): add gemini 3.1 pro, playwright screenshots, replica refinement - switch to gemini 3.1 pro (gemini-3.1-pro-preview) with claude fallback - playwright for full-page screenshots (fixes puppeteer gradient/fixed bugs) - replica refinement loop: generate, screenshot, compare, fix - extract inline svgs (50 max, 10kb each) to assets/svgs/ - extract visible text in dom order for content accuracy - detect js libraries (gsap, three.js, scrolltrigger) via globals - improved asset catalog grouping and naming - reverse-engineered aura system prompt documentation - comprehensive session handoff doc Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update session handoff with slack research findings - key finding: team already wants DESIGN.md integration (James, Bin, Vance) - skills quality matters enormously - must invoke /hyperframes-compose - eval infrastructure exists (Abhay's dashboards, Teodora's 78-criteria guide) - templates at templates/ need study before finalizing skill - session handoff updated with critical next steps * refactor(cli): simplify capture pipeline, remove replica generator * feat(capture): add Lottie detection and WebGL shader extraction Captures Lottie animations via network interception and WebGL shader source via gl.shaderSource hooking during site crawl. Updates website-to-hyperframes skill with asset planning guidance, Lottie/shader reading instructions, and stronger creative direction for scene planning. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(capture): clean pipeline + shader-first creative workflow Capture pipeline: - Remove dead deps (puppeteer-extra, stealth plugin, duplicate devDeps) - Remove duplicate generateAgentPrompt() call (first lied about DESIGN.md) - Remove dead canvas-to-image code in htmlExtractor (post canvas removal) - Parallelize image downloads (batches of 5 via Promise.allSettled) - Fix pre-existing TS error (match[1] guard in font downloader) - Default capture output to captures/<hostname> Skill creative overhaul: - Add shader transition selection to creative director step (Step 4) - Add shader wiring instructions to engineer step (Step 5) - Replace 4-line energy modifiers with visual vocabulary table - Strip rigid scene-by-scene templates from video-recipes.md - Strip example fill data from scene plan tables - Add "read transition refs before planning" instruction - Add creative ambition language ("how the hell did they make this") Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add skill architecture redesign spec Comprehensive redesign of website-to-hyperframes skill and capture pipeline based on code review findings and Claude Code architecture research. Key changes: remove AI auto-generation, restructure skill into phases, embed shader boilerplate in scaffold, fix color format. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add implementation plan for skill architecture redesign 13-task plan covering: capture pipeline cleanup (remove AI generation, fix colors to HEX, add asset descriptions, shader-ready scaffold), skill restructuring (4 phases with artifact gates), and compose skill Visual Identity Gate upgrade. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(capture): remove AI auto-generation and SDK dependencies * fix(capture): convert extracted colors to HEX format * refactor(capture): remove AI key path, add asset descriptions generator * refactor(capture): update agent prompt, remove hasDesignMd, add asset descriptions * feat(capture): pre-wire shader transitions in index.html scaffold * chore: remove duplicate visual-styles.md (canonical is in hyperframes/) * refactor(skill): rewrite website-to-hyperframes as phase-based orchestrator * feat(skill): add Phase 1 understand reference * feat(skill): add Phase 2 design reference with full DESIGN.md schema * feat(skill): add Phase 3 creative direction reference * feat(skill): add Phase 4 build reference with inline shader example * feat(skill): upgrade Visual Identity Gate to produce full DESIGN.md * docs: update CLAUDE.md skill references for phase-based workflow * fix: address code review findings - Remove orphaned `false` argument in generateAgentPrompt call (critical: was shifting hasLottie, hasShaders, catalogedAssets parameters) - Add HSL color handling in rgbToHex via temp element resolution - Remove build artifact commit section from phase-4-build.md - Fix __GSAP_TIMELINE reference to __timelines Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(capture): regex double-escape + simplify scaffold + fix asset descriptions - Double-escape regex in tokenExtractor template literal (\s→\\s, \d→\\d, \(→\\() so browser receives valid regex patterns via page.evaluate() - Simplify index.html scaffold: scene slots + audio + timeline + comment pointing to shader-setup.md reference (no broken inline shader boilerplate) - Fix asset descriptions: use CatalogedAsset.contexts/notes instead of nonexistent htmlContext field Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: code review — 16 bugs, 7-step skill rewrite, cleanup Code fixes: - snapshot.ts: path traversal guard, browser leak (try/finally), div-by-zero for --frames 1, port bind error handling, rAF-based render settle - index.ts: remove invalid thinkingConfig for gemini-2.5-flash, fix Gemini batch/rate-limit comments, fix video preview viewport y-coordinate - tokenExtractor.ts: remove dead seen[si] dedup code - gsap.ts: index ALL classes for inline-style transform conflict detection Skill architecture rewrite (4-phase → 7-step): - Replace phase-1 through phase-4 with step-1 through step-7 - Add techniques.md (10 visual techniques with code patterns) - Fix /hyperframes-compose → /hyperframes (skill doesn't exist) - Fix captures/arc-browser reference → shader-setup.md (file doesn't exist) - Fix step-7 hardcoded captures/stripe path - Document Gemini API free/paid rate limits in step-1 Cleanup: - CLAUDE.md: restore from Stripe-capture overwrite, update 4-phase → 7-step - .gitignore: add PR #267 skills (hyperframes-animation-map, hyperframes-contrast) - Delete old phase-*.md, animation-recreation.md, tts-integration.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove dev artifacts, research docs, wrong lockfiles Remove files that shouldn't ship in this PR: - docs/research/ (aura analysis, prompt catalogs) - docs/session-*.md, docs/SESSION-HANDOFF.md (dev notes) - docs/superpowers/ planning and spec docs - pnpm-lock.yaml at root and cli (repo uses bun, not pnpm) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(CLAUDE.md): align with main — slim format, add website-to-hyperframes mention Main PR #283 removed the full skills table from CLAUDE.md and moved it to AGENTS.md. Align with that decision: use main's slim dev-focused format, fix pnpm→bun references, add one-line /website-to-hyperframes pointer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): add capture command to help groups The capture command was registered in cli.ts but missing from the help groups, so it wouldn't appear in `hyperframes --help`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: format skill reference files (oxfmt) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: regenerate bun.lock after rebase The lockfile was stale after rebasing onto main — bun install --frozen-lockfile failed in CI because new dependencies (google/genai, patchright, purgecss) weren't reflected in the lockfile. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review comments + improve capture quality Review fixes (16 comments from jrusso1020 + vanceingalls): - screenshotCapture: remove Playwright dep, use Puppeteer for all screenshots - screenshotCapture: dynamic screenshot count based on page height (30% overlap) - snapshot.ts: fix duration() function-vs-property bug, cross-platform path guard - htmlExtractor: fix code injection via parameterized evaluate - index.ts: video preview re-measures position after scroll, .env file loading - capture.ts: BLOCKED.md on timeout failures - gsap.ts: 5 inline-style lint tests added (all pass) - Remove Playwright, patchright deps; @google/genai to optionalDependencies - Gitignore: generic patterns instead of 20 hardcoded directories - Remove asset-sourcing.md, video-recipes.md (unused, duplicated guidance) Capture quality improvements (tested on 10+ websites): - Color extraction: canvas-based oklch/lab resolver, pixel sampling via elementFromPoint, broad sweep for accent colors, gradient/shadow extraction - Section detection: broadened selectors for div-based layouts, height cap to skip page-level wrappers, parent bg walkup for dark sites - Font downloads: cap 6 per family / 30 total (Cal.com: 306→30) - CTA detection: text pattern matching + nav context filtering - Heading text: innerText with whitespace normalization - Gemini captioning: maxOutputTokens 100→300, .env auto-loading - .env.example updated with GEMINI_API_KEY docs - TTS ranking: Kokoro first with Python 3.10+ note * fix: address PR review comments + improve capture quality Review round 2 fixes (jrusso1020 + vanceingalls): - verify/index.ts: add path traversal guard (relative + isAbsolute) - verify/index.ts: fix sections[i] undefined typecheck error (CI green) - index.ts: escape Lottie JSON with \u003c to prevent </script> breakout - step-4-storyboard: fix technique count contradiction (2-3 per beat, not across whole video) - step-6-build: perspective tilt uses gsap.set() instead of CSS transform (avoids GSAP overwrite conflict) - step-1-capture: reorder — command first, Gemini note after (zero-config is the default path, API key is optional enhancement) - step-7-validate: add tsx fallback for snapshot command - step-3-script: vary hook patterns, don't default to number every time - assetDownloader: exempt SVGs from 10KB minimum filter (company logos like Hubspot/Intel/DHL are 2-6KB; HeyGen capture: 13→75 assets) Note: adm-zip was NOT removed (reviewer #3) — it's still in packages/cli/package.json:30. The root package.json had patchright and purgecss removed, not adm-zip. Note: ANTHROPIC_API_KEY not restored in .env.example — grep confirms zero references in the entire codebase. The @anthropic-ai/sdk dependency was removed earlier in this branch. * refactor(capture): split index.ts (1175 to 566 lines) into modules Mechanical extraction, zero logic changes. New files: - mediaCapture.ts (345 lines): Lottie preview, video manifest/screenshots - contentExtractor.ts (314 lines): library detection, text, Gemini, asset descriptions - scaffolding.ts (135 lines): .env loading, project scaffold generation Also fixes false-positive BLOCKED.md with structural Cloudflare detection. Tested on 20 websites, pre/post output identical. * chore(capture): remove --split flow (splitter, verify, cssPurger, purgecss) The --split feature auto-generates compositions from captured HTML — a different approach from the /website-to-hyperframes skill workflow where agents build compositions from scratch using the storyboard. No skill file, no step reference, and no test session ever used --split. Removes 923 lines of unused code + purgecss dependency. Backed up to ~/Desktop/capture-split-backup/ for reference. * fix(security): add ssrf protection, lottie injection fix, oom guard - assetDownloader: add isPrivateUrl() guard blocking private IP ranges (127.x, 10.x, 172.16-31.x, 192.168.x, 169.254.x), cloud metadata endpoints, localhost, and non-HTTP schemes - mediaCapture: fix Lottie JSON injection by loading shell HTML first then passing animation data via parameterized page.evaluate() - index.ts: check Content-Length header before response.buffer() in Lottie network interception to avoid OOM on multi-GB responses * fix(capture): security fixes, timeout, sub-agent dispatch instructions Security (from miguel-heygen review): - assetDownloader: export isPrivateUrl() SSRF guard - htmlExtractor: add isPrivateUrl check before CSS fetch - mediaCapture: add isPrivateUrl check before Lottie fetch - mediaCapture: fix previewPage leak (try/finally) - mediaCapture: skip Lottie files > 2MB for preview (CDP limit) - contentExtractor: skip images > 4MB for Gemini captioning - index.ts: check Content-Length before response.buffer() (OOM guard) - snapshot.ts: register error handler before server.listen() Capture improvements: - Default timeout 30s to 120s (Shopify needs ~90s for Cloudflare) - step-6-build: sub-agent dispatch template with explicit rules: pass file PATHS not contents, use local fonts not Google Fonts, verify ../assets/ references after each beat * fix(capture): catalog before DOM mutation, networkidle2, faster Gemini Critical: asset cataloger now runs BEFORE extractHtml which converts img src to data URLs. Framer sites like heykuba.com went from 2 to 78 images. - networkidle2 instead of networkidle0 (unblocks SPAs with WebSockets) - Lazy-load wait: scroll to bottom, wait for img.complete - CSS background-image cataloging for Framer/Webflow - SVG naming: checks class, id, parent, inner text (not just aria-label) - Gemini batch 5->20, pause 12s->2s (paid tier: 2000 RPM, ~0.001/img) - maxOutputTokens 300->500, descriptions sorted captioned-first - Remove tsx fallback from step-1 (reviewer nit, published CLI has it) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ular Kimsanov
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
ebc12f7dc9
commit
87f4c77e2f
@@ -0,0 +1,162 @@
|
||||
# Step 6: Build Compositions
|
||||
|
||||
**Before building, fully re-read these files:**
|
||||
|
||||
- **DESIGN.md** — your color palette, fonts, components, and Do's/Don'ts. Every composition must use EXACT hex colors and font families from this file. If it says "white backgrounds" — use white, not dark.
|
||||
- **STORYBOARD.md** — the beat-by-beat plan you're executing. Each beat specifies assets, animations, transitions, and which techniques to use.
|
||||
- **`extracted/asset-descriptions.md`** — when the storyboard assigns an asset to a beat, re-read the description to understand what it shows and how to position/style it correctly.
|
||||
- **[techniques.md](techniques.md)** — code patterns for the 10 visual techniques. When the storyboard says "SVG path drawing" or "per-word kinetic typography" — read the code pattern from this file and adapt it.
|
||||
- **transcript.json** — word-level timestamps that drive scene durations.
|
||||
|
||||
**Split the work: spawn a sub-agent for each beat.** By this step your context is full of captured data, DESIGN.md, SCRIPT, STORYBOARD, and transcript. Building compositions on top of all that means the detailed rules below compete with thousands of tokens of prior work. Each sub-agent gets a fresh context focused on one beat — dramatically better output.
|
||||
|
||||
**How to dispatch each sub-agent:**
|
||||
|
||||
Pass file PATHS, not file contents. The #1 failure mode is reading an asset file and pasting its SVG/image data into the sub-agent prompt. The sub-agent then uses inline content instead of referencing the file on disk. Same with fonts — pass the local woff2 path, don't substitute Google Fonts.
|
||||
|
||||
```
|
||||
Build the composition for beat 1. Save to compositions/beat-1-hook.html.
|
||||
|
||||
STORYBOARD for this beat:
|
||||
[paste the beat section from STORYBOARD.md]
|
||||
|
||||
ASSETS — reference by path, do NOT read/inline the file contents:
|
||||
- Logo: <img src="../assets/favicon.svg"> (top-left, 40x40px)
|
||||
- Hero image: <img src="../assets/hero-bg.png"> (full-bleed background)
|
||||
- Noise texture: ../assets/noise.png (full-frame overlay, 3% opacity)
|
||||
|
||||
FONTS — use @font-face with the captured font files, NOT Google Fonts:
|
||||
@font-face { font-family: 'BrandFont'; src: url('../assets/fonts/BrandFont-Regular.woff2'); }
|
||||
|
||||
Read DESIGN.md for exact colors and Do's/Don'ts.
|
||||
Read techniques.md for animation code patterns.
|
||||
Invoke /hyperframes for composition structure rules.
|
||||
```
|
||||
|
||||
After each sub-agent finishes, verify the composition references `../assets/` — if it used inline SVGs or Google Fonts instead of the captured files, fix it before moving on.
|
||||
|
||||
Invoke the `/hyperframes` skill first — it has the rules for data attributes, timeline contracts, deterministic rendering, and layout. Everything below supplements those rules, not replaces them.
|
||||
|
||||
---
|
||||
|
||||
## Per-Composition Process
|
||||
|
||||
For each beat in the storyboard:
|
||||
|
||||
### 1. Read the beat's storyboard section
|
||||
|
||||
Know the mood, visual description, assets, animation choreography, transition, and SFX before writing any HTML.
|
||||
|
||||
### 2. Build the static end-state first
|
||||
|
||||
Position every element where it should be at its **most visible moment** — the frame where everything is fully entered and correctly placed. Write this as static HTML+CSS. No GSAP yet.
|
||||
|
||||
This is the "Layout Before Animation" principle from the compose skill. The CSS position is the ground truth. Animations describe the journey to and from it.
|
||||
|
||||
### 3. Verify the static layout
|
||||
|
||||
Look at it. Check:
|
||||
|
||||
- Are elements where the storyboard says they should be?
|
||||
- Are depth layers present (foreground / midground / background)?
|
||||
- Do any elements overlap unintentionally?
|
||||
- Are assets sized correctly? (hero images should fill 50-70% of frame, not sit at 100x100px)
|
||||
|
||||
### 4. Add entrance animations
|
||||
|
||||
Use `gsap.from()` — animate FROM offscreen/invisible TO the CSS position. The CSS position is where the element ends up.
|
||||
|
||||
### 5. Add mid-scene activity
|
||||
|
||||
Every visible element must have continuous motion. A still image on a still background is a JPEG with a progress bar.
|
||||
|
||||
| Element type | Mid-scene activity |
|
||||
| ---------------------- | ------------------------------------------------ |
|
||||
| Image / screenshot | Slow zoom (scale 1→1.03), slow pan, or Ken Burns |
|
||||
| Stat / number | Counter animates from 0 to target |
|
||||
| Logo grid | Subtle shimmer sweep, or gentle scale pulse |
|
||||
| Any persistent element | Subtle float (y ±4-6px, sine.inOut, yoyo) |
|
||||
|
||||
### 6. Add exit / transition
|
||||
|
||||
Check the storyboard's transition specification for this beat:
|
||||
|
||||
- **CSS transition**: implement the exit animation (e.g., `y:-150, blur:30px, 0.33s power2.in`). The next composition handles its own entry.
|
||||
- **Shader transition**: no exit animation needed — the shader handles the blend. Read `skills/hyperframes/references/transitions/shader-setup.md` for the full WebGL boilerplate and `skills/hyperframes/references/transitions/shader-transitions.md` for the fragment shader. Copy the FULL boilerplate — a simplified version produces black screens.
|
||||
- **Hard cut**: no exit animation. The scene simply ends.
|
||||
|
||||
For all CSS transition types and their GSAP implementations, read `skills/hyperframes/references/transitions/catalog.md`.
|
||||
|
||||
### 7. Asset cross-reference
|
||||
|
||||
Before self-review, verify you actually used the assets you planned to:
|
||||
|
||||
1. Open STORYBOARD.md and find this beat's asset assignments
|
||||
2. List every asset that was assigned to this beat
|
||||
3. Search the composition HTML for each filename (e.g., grep for "wave-fallback-desktop")
|
||||
4. If any assigned asset is missing from the HTML, add it now
|
||||
5. Check for the inline anti-pattern: if the HTML contains `<svg xmlns=` or `data:image/` but no `../assets/` references, the assets were inlined instead of referenced. Replace inline content with `<img src="../assets/filename.svg">`
|
||||
6. Check fonts: if the HTML uses `fonts.googleapis.com` but there are captured fonts in `assets/fonts/`, replace with `@font-face` pointing to the local files
|
||||
|
||||
This step catches the two most common failures: compositions ending up text-only, and assets being inlined instead of file-referenced.
|
||||
|
||||
### 8. Self-review
|
||||
|
||||
After building the composition, check WITH ACTUAL CODE:
|
||||
|
||||
- [ ] Asset cross-reference passed (step 7 above — every assigned asset is in the HTML)
|
||||
- [ ] Elements are where the storyboard says they should be (no misplacement)
|
||||
- [ ] No overlapping text (text covering text is always ugly)
|
||||
- [ ] Depth layers present (2+ layers minimum)
|
||||
- [ ] Every visible element has mid-scene activity (not just entrance + exit)
|
||||
- [ ] Font sizes above minimum (20px body text, 16px labels — sub-14px is unreadable after encoding)
|
||||
- [ ] No full-screen dark linear gradients (H.264 creates visible banding — use solid + localized radial glows)
|
||||
- [ ] Timeline registered: `window.__timelines["comp-id"] = tl`
|
||||
- [ ] Colors match DESIGN.md exactly (paste the HEX value, don't approximate)
|
||||
|
||||
**If `skills/hyperframes-animation-map/` is installed**, run it:
|
||||
|
||||
```bash
|
||||
node skills/hyperframes-animation-map/scripts/animation-map.mjs <composition-dir>
|
||||
```
|
||||
|
||||
Read the summaries. Fix every flag: offscreen, collision, invisible, pacing issues.
|
||||
|
||||
### 9. Move to the next composition
|
||||
|
||||
---
|
||||
|
||||
## Asset Presentation
|
||||
|
||||
Never embed a raw flat image. Every image must have motion treatment:
|
||||
|
||||
- **Perspective tilt**: use `gsap.set(el, { transformPerspective: 1200, rotationY: -8 })` + `box-shadow` — creates depth. Do NOT use CSS `transform: perspective(...)` as GSAP will overwrite it.
|
||||
- **Slow zoom (Ken Burns)**: GSAP `scale: 1` → `1.04` over beat duration — makes photos cinematic
|
||||
- **Device frame**: Wrap in a laptop/phone shape using CSS `border-radius` and `box-shadow`
|
||||
- **Floating UI**: Extract a key element and animate it at a different z-depth for parallax
|
||||
- **Scroll reveal**: Clip the image to a viewport window and animate `y` position
|
||||
|
||||
---
|
||||
|
||||
## Audio Wiring
|
||||
|
||||
In the root `index.html`:
|
||||
|
||||
- **Narration**: `<audio id="narration" src="narration.wav" data-start="0" data-duration="..." data-track-index="0" data-volume="1">`
|
||||
- **Underscore/music** (if storyboard specifies): `<audio id="underscore" src="underscore.mp3" data-start="0" data-duration="..." data-track-index="3" data-volume="0.15">`
|
||||
- **SFX** (if storyboard specifies): individual `<audio>` elements at specific `data-start` timestamps
|
||||
- **Captions** (optional — only if user requests): sub-composition on a parallel track. Skip unless explicitly asked for.
|
||||
|
||||
---
|
||||
|
||||
## Critical Rules
|
||||
|
||||
These exist because the capture engine is deterministic. Violations produce broken output.
|
||||
|
||||
- **No `repeat: -1`** — calculate exact repeats from beat duration
|
||||
- **No `Math.random()`** — use a seeded PRNG (mulberry32)
|
||||
- **Register every timeline**: `window.__timelines["comp-id"] = tl`
|
||||
- **Synchronous timeline construction** — no async/await wrapping timeline code
|
||||
- **Never use ANY CSS `transform` for centering** — not `translate(-50%, -50%)`, not `translateX(-50%)`, not `translateY(-50%)`. GSAP animates the `transform` property, which overwrites ALL CSS transforms including centering. The element flies offscreen. Use flexbox centering instead: `display:flex; align-items:center; justify-content:center` on a wrapper div. The linter catches this (`gsap_css_transform_conflict`) but only if you run it.
|
||||
- **Minimum font sizes**: 20px body, 16px labels
|
||||
- **No full-screen dark linear gradients** — H.264 banding
|
||||
Reference in New Issue
Block a user