ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-03 04:38:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
749 Commits 856 Branches 404 Tags
v0.5.5
Commit Graph
27 Commits
Author SHA1 Message Date
JamesandClaude Opus 4.7 d2ca45ef75 feat(cli): add remove-background command for transparent video 
Adds `hyperframes remove-background` — a local-AI subcommand that mattes a
video or image with the u2net_human_seg ONNX model and emits a transparent
WebM (VP9-alpha), ProRes 4444 .mov, or RGBA PNG. Drops directly into any
composition's <video> tag — no green screen, no API keys, no upload.

Auto-picks the fastest available execution provider via onnxruntime-node:
CoreML on Apple Silicon, CUDA when HYPERFRAMES_CUDA=1, CPU otherwise.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 04:17:51 +00:00
James Russo 4d0e262eb3 Merge pull request #511 from heygen-com/renovate/google-genai-1.x 
chore(deps): update dependency @google/genai to ^1.50.1
 2026-04-27 22:34:28 -07:00
Miguel Ángel c58b995cb0 refactor: use postcss for composition CSS scoping 2026-04-27 16:42:42 -04:00
renovate[bot] 2ed47eb076 chore(deps): update dependency @google/genai to ^1.50.1 2026-04-27 07:39:37 +00:00
Vance Ingalls 10d2725b54 perf(player): p0-1a perf test infra + composition-load smoke test (#399) 
## Summary

First slice of `P0-1` from the player perf proposal: lays the foundation for a player perf gate so later PRs can plug in fps / scrub / drift / parity scenarios without rebuilding infrastructure. Ships one smoke scenario (`03-load`, cold + warm composition load) to prove the gate end-to-end on real numbers.

## Why

There was no automated way to catch player perf regressions. Every perf concern in the existing proposal — composition load time, sustained FPS, scrub p95, mirror-clock drift, live-vs-seek parity — needs the same plumbing: a same-origin harness, a Puppeteer runner, a baseline file, a gate that emits structured results, and a CI workflow that runs the right scenarios on the right changes. Building that up-front in one reviewable PR lets every subsequent perf PR (`P0-1b`, `P0-1c`, and beyond) be a 100-line scenario file plus a baseline entry instead of re-litigating the framework.

## What changed

### Harness — `packages/player/tests/perf/server.ts`

- `Bun.serve` on a free port, single same-origin host for the player IIFE bundle, hyperframe runtime, GSAP from `node_modules`, and fixture HTML.
- Same-origin matters: cross-origin would force every probe through `postMessage`, hiding bugs and inflating numbers in ways production never sees. Tests should measure the path the studio editor actually takes.
- Routes:
  - `/player.js` → built IIFE bundle (rebuilt on demand).
  - `/vendor/runtime.js`, `/vendor/gsap.min.js` → resolved from `node_modules` so fixtures don't need to ship copies.
  - `/fixtures/*` → fixture HTML.

### Runner — `packages/player/tests/perf/runner.ts`

- `puppeteer-core` thin wrappers (`launchBrowser`, `loadHostPage`).
- Uses the system Chrome detected by `setup-chrome` in CI rather than the bundled puppeteer revision — keeps the action smaller, lets us pin Chrome version policy at the workflow level, and matches what users actually run.

### Gate — `packages/player/tests/perf/perf-gate.ts` + `baseline.json`

- Loads `baseline.json` (initial budgets: cold/warm comp load, fps, scrub p95 isolated/inline, drift max/p95) with a 10% `allowedRegressionRatio`.
- Per-metric direction (`lower-is-better` / `higher-is-better`) so the same evaluator handles latency and throughput.
- Returns a structured `GateReport` consumed by both the CLI (table output) and `metrics.json` (CI artifact).
- Two modes: `measure` (log only — used during the rollout) and `enforce` (fail the build) — flip per-metric once we trust the signal, without touching the harness.

### CLI orchestrator — `packages/player/tests/perf/index.ts`

- Parses `--mode` / `--scenarios` / `--runs` / `--fixture` in both space- and equals-separated form (so `--scenarios fps,scrub` and `--scenarios=fps,scrub` both work — matches what humans type and what GitHub Actions emits).
- Runs scenarios, runs the gate, and **always** writes `results/metrics.json` with schema version, git SHA, metrics, and gate rows — so failed runs are still investigable from the artifact alone.

### Fixture + smoke scenario

- `fixtures/gsap-heavy/index.html`: 200 stagger-animated tiles, no media. Heavy enough to make load time meaningful, light enough to be deterministic.
- `scenarios/03-load.ts`: cold + warm composition load. Measures from navigation start to player `ready` event, reports p95 across runs.

### CI — `.github/workflows/player-perf.yml`

- `paths-filter` on `player` / `core` / `runtime` — perf only runs when something that could move the needle actually changed.
- Sets up bun + node + chrome, runs perf in `measure` mode on a shard matrix (so future scenarios shard naturally), uploads `metrics.json` artifacts, and a summary job aggregates shard results into a single PR comment.

### Wiring

- `packages/player`: `puppeteer-core`, `gsap`, `@types/bun` devDeps; typecheck extended to cover the perf `tsconfig`; new `perf` script.
- Root `package.json`: `player:perf` workspace script so `bun run player:perf` runs the whole suite locally with the same flags CI uses.
- `.gitignore`: `packages/player/tests/perf/results/`.
- Separate `tests/perf/tsconfig.json` so test code doesn't pollute the package `rootDir` while still being typechecked.

## Test plan

- [x] Local: `bun run player:perf` passes — cold p95 ≈ 386 ms, warm p95 ≈ 375 ms, both well under the seeded baselines.
- [x] Typecheck, lint, format pass on the perf workspace.
- [x] Existing player unit tests (71/71) still green.
- [ ] First CI run after merge will be the real signal: confirms `setup-chrome` works on hosted runners, the shard matrix wires up, and `metrics.json` artifacts upload.

## Stack

Step `P0-1a` of the player perf proposal. The next two slices are content-only — they don't touch the harness:

- `P0-1b` (#400): adds `02-fps`, `04-scrub`, `05-drift` scenarios on a 10-video-grid fixture.
- `P0-1c` (#401): adds `06-parity` (live playback vs. synchronously-seeked reference, compared via SSIM).

Wiring this gate up first means each follow-up is a self-contained scenario file + baseline row + workflow shard.
 2026-04-22 18:04:05 -07:00
James Russo 4ce1792601 feat(codex-plugin): add OpenAI Codex plugin at repo root (#387) 2026-04-21 15:33:32 -07:00
Miguel Ángel 03c2158e0f ci: verify on windows-latest + fix cross-platform build bugs it surfaced (#342) 
* fix(cli): make build copy cross-platform and deterministic

* fix(core): keep rewritten asset URLs POSIX on Windows

* ci(windows): add render verification workflow

* ci(windows): load canary gsap from cdn

* build: use dependency-aware workspace ordering

* Revert "build: use dependency-aware workspace ordering"

This reverts commit 99bc2ffbdf.
 2026-04-20 04:35:55 +02:00
Vance Ingalls 5a3fde19d4 feat(engine): add HDR video output pipeline (#265) 
## Summary

Adds the ability to render HDR video output (H.265 10-bit, BT.2020) from HyperFrames compositions. When the renderer detects HDR source video, it automatically switches to the HDR output pipeline — no flags needed.

## What it does

- **Auto-detection** — Probes each video source with `ffprobe`. If any has bt2020/PQ/HLG color metadata, the output switches to H.265 10-bit with correct color tags. SDR-only compositions are unaffected (H.264, bt709).
- **HLG pass-through** — Native HLG pixels from FFmpeg extraction are piped directly to the encoder without conversion. This avoids brightness loss from HLG→linear→PQ conversion (which requires an OOTF system gamma we can't reliably apply).
- **Encoder HDR support** — Both chunk and streaming encoders accept HDR presets: `libx265`, `yuv420p10le`, BT.2020 color primaries, `hvc1` codec tag (required for Apple playback).
- **WebGPU HDR capture (gated)** — A complete WebGPU float16 readback pipeline is implemented and tested but gated behind headed Chrome (headless doesn't expose WebGPU). Ready for future use with WebGPU canvas content.
- **HDR utilities** — `detectTransfer()` (PQ vs HLG), `getHdrEncoderColorParams()`, `analyzeCompositionHdr()`. 15 unit tests.

## Key design decisions

| Decision | Why |
|----------|-----|
| No `--hdr` flag | SDR content encoded as HDR causes orange shift in browsers. Auto-detect eliminates this. |
| HLG pass-through (not HLG→PQ) | Conversion loses brightness without OOTF. Pass-through matches source exactly. |
| `hvc1` codec tag | Apple QuickTime requires `hvc1` (not `hev1`) for HEVC playback. |
| 1-hour streaming timeout | HDR capture at ~6fps needs more time than the default 10-minute FFmpeg timeout. |

## Files changed

| File | What changed |
|------|-------------|
| `packages/engine/src/utils/hdr.ts` | **NEW** — HDR detection, transfer types, encoder params (15 tests) |
| `packages/engine/src/services/hdrCapture.ts` | **NEW** — WebGPU readback, HLG conversion, PQ encode |
| `packages/engine/src/services/streamingEncoder.ts` | HDR presets, raw rgb48le input, color tags |
| `packages/engine/src/services/chunkEncoder.ts` | HDR presets, conditional color tags |
| `packages/producer/src/services/renderOrchestrator.ts` | Auto-detection loop, HDR pass-through capture path |

## How to test

Render a composition with an HDR video source. The output should be H.265 10-bit with HDR metadata visible in `ffprobe` (bt2020, arib-std-b67 or smpte2084). Plays correctly in QuickTime and on HDR displays.

## Stack position

**2 of 6** — Stacked on #258 (SDR/HDR normalization). Provides the encoder infrastructure that phases 1-5 build on.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-04-19 15:10:59 -07:00
Ular KimsanovandClaude Opus 4.6 87f4c77e2f feat: website capture pipeline + 7-step video production skill (#284) 
* feat(cli): add website capture with AI-powered DESIGN.md generation

Adds `hyperframes capture <url>` command that extracts a complete design
system from any website, producing AI-agent-ready output:

- Full-page screenshot (lazy-load aware, nav at top)
- AI-generated DESIGN.md via Claude API (colors, typography, elevation,
  components, do's/don'ts) with programmatic asset catalog (136+ assets
  with HTML context annotations like img[src], css url(), link[rel=preload])
- CSS-purged compositions (87% size reduction via PurgeCSS)
- HTML-prettified compositions (one-tag-per-line for AI readability)
- CLAUDE.md + .cursorrules auto-generated for AI agent instructions
- Asset deduplication (srcset variants) and tracking pixel filtering

* feat(cli): add gemini 3.1 pro, playwright screenshots, replica refinement

- switch to gemini 3.1 pro (gemini-3.1-pro-preview) with claude fallback
- playwright for full-page screenshots (fixes puppeteer gradient/fixed bugs)
- replica refinement loop: generate, screenshot, compare, fix
- extract inline svgs (50 max, 10kb each) to assets/svgs/
- extract visible text in dom order for content accuracy
- detect js libraries (gsap, three.js, scrolltrigger) via globals
- improved asset catalog grouping and naming
- reverse-engineered aura system prompt documentation
- comprehensive session handoff doc

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update session handoff with slack research findings

- key finding: team already wants DESIGN.md integration (James, Bin, Vance)
- skills quality matters enormously - must invoke /hyperframes-compose
- eval infrastructure exists (Abhay's dashboards, Teodora's 78-criteria guide)
- templates at templates/ need study before finalizing skill
- session handoff updated with critical next steps

* refactor(cli): simplify capture pipeline, remove replica generator

* feat(capture): add Lottie detection and WebGL shader extraction

Captures Lottie animations via network interception and WebGL shader
source via gl.shaderSource hooking during site crawl. Updates
website-to-hyperframes skill with asset planning guidance, Lottie/shader
reading instructions, and stronger creative direction for scene planning.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(capture): clean pipeline + shader-first creative workflow

Capture pipeline:
- Remove dead deps (puppeteer-extra, stealth plugin, duplicate devDeps)
- Remove duplicate generateAgentPrompt() call (first lied about DESIGN.md)
- Remove dead canvas-to-image code in htmlExtractor (post canvas removal)
- Parallelize image downloads (batches of 5 via Promise.allSettled)
- Fix pre-existing TS error (match[1] guard in font downloader)
- Default capture output to captures/<hostname>

Skill creative overhaul:
- Add shader transition selection to creative director step (Step 4)
- Add shader wiring instructions to engineer step (Step 5)
- Replace 4-line energy modifiers with visual vocabulary table
- Strip rigid scene-by-scene templates from video-recipes.md
- Strip example fill data from scene plan tables
- Add "read transition refs before planning" instruction
- Add creative ambition language ("how the hell did they make this")

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add skill architecture redesign spec

Comprehensive redesign of website-to-hyperframes skill and capture
pipeline based on code review findings and Claude Code architecture
research. Key changes: remove AI auto-generation, restructure skill
into phases, embed shader boilerplate in scaffold, fix color format.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add implementation plan for skill architecture redesign

13-task plan covering: capture pipeline cleanup (remove AI generation,
fix colors to HEX, add asset descriptions, shader-ready scaffold),
skill restructuring (4 phases with artifact gates), and compose skill
Visual Identity Gate upgrade.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(capture): remove AI auto-generation and SDK dependencies

* fix(capture): convert extracted colors to HEX format

* refactor(capture): remove AI key path, add asset descriptions generator

* refactor(capture): update agent prompt, remove hasDesignMd, add asset descriptions

* feat(capture): pre-wire shader transitions in index.html scaffold

* chore: remove duplicate visual-styles.md (canonical is in hyperframes/)

* refactor(skill): rewrite website-to-hyperframes as phase-based orchestrator

* feat(skill): add Phase 1 understand reference

* feat(skill): add Phase 2 design reference with full DESIGN.md schema

* feat(skill): add Phase 3 creative direction reference

* feat(skill): add Phase 4 build reference with inline shader example

* feat(skill): upgrade Visual Identity Gate to produce full DESIGN.md

* docs: update CLAUDE.md skill references for phase-based workflow

* fix: address code review findings

- Remove orphaned `false` argument in generateAgentPrompt call (critical:
  was shifting hasLottie, hasShaders, catalogedAssets parameters)
- Add HSL color handling in rgbToHex via temp element resolution
- Remove build artifact commit section from phase-4-build.md
- Fix __GSAP_TIMELINE reference to __timelines

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(capture): regex double-escape + simplify scaffold + fix asset descriptions

- Double-escape regex in tokenExtractor template literal (\s→\\s, \d→\\d, \(→\\()
  so browser receives valid regex patterns via page.evaluate()
- Simplify index.html scaffold: scene slots + audio + timeline + comment pointing
  to shader-setup.md reference (no broken inline shader boilerplate)
- Fix asset descriptions: use CatalogedAsset.contexts/notes instead of
  nonexistent htmlContext field

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: code review — 16 bugs, 7-step skill rewrite, cleanup

Code fixes:
- snapshot.ts: path traversal guard, browser leak (try/finally), div-by-zero
  for --frames 1, port bind error handling, rAF-based render settle
- index.ts: remove invalid thinkingConfig for gemini-2.5-flash, fix Gemini
  batch/rate-limit comments, fix video preview viewport y-coordinate
- tokenExtractor.ts: remove dead seen[si] dedup code
- gsap.ts: index ALL classes for inline-style transform conflict detection

Skill architecture rewrite (4-phase → 7-step):
- Replace phase-1 through phase-4 with step-1 through step-7
- Add techniques.md (10 visual techniques with code patterns)
- Fix /hyperframes-compose → /hyperframes (skill doesn't exist)
- Fix captures/arc-browser reference → shader-setup.md (file doesn't exist)
- Fix step-7 hardcoded captures/stripe path
- Document Gemini API free/paid rate limits in step-1

Cleanup:
- CLAUDE.md: restore from Stripe-capture overwrite, update 4-phase → 7-step
- .gitignore: add PR #267 skills (hyperframes-animation-map, hyperframes-contrast)
- Delete old phase-*.md, animation-recreation.md, tts-integration.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: remove dev artifacts, research docs, wrong lockfiles

Remove files that shouldn't ship in this PR:
- docs/research/ (aura analysis, prompt catalogs)
- docs/session-*.md, docs/SESSION-HANDOFF.md (dev notes)
- docs/superpowers/ planning and spec docs
- pnpm-lock.yaml at root and cli (repo uses bun, not pnpm)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(CLAUDE.md): align with main — slim format, add website-to-hyperframes mention

Main PR #283 removed the full skills table from CLAUDE.md and moved it
to AGENTS.md. Align with that decision: use main's slim dev-focused
format, fix pnpm→bun references, add one-line /website-to-hyperframes
pointer.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): add capture command to help groups

The capture command was registered in cli.ts but missing from
the help groups, so it wouldn't appear in `hyperframes --help`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: format skill reference files (oxfmt)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: regenerate bun.lock after rebase

The lockfile was stale after rebasing onto main — bun install
--frozen-lockfile failed in CI because new dependencies (google/genai,
patchright, purgecss) weren't reflected in the lockfile.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review comments + improve capture quality

Review fixes (16 comments from jrusso1020 + vanceingalls):
- screenshotCapture: remove Playwright dep, use Puppeteer for all screenshots
- screenshotCapture: dynamic screenshot count based on page height (30% overlap)
- snapshot.ts: fix duration() function-vs-property bug, cross-platform path guard
- htmlExtractor: fix code injection via parameterized evaluate
- index.ts: video preview re-measures position after scroll, .env file loading
- capture.ts: BLOCKED.md on timeout failures
- gsap.ts: 5 inline-style lint tests added (all pass)
- Remove Playwright, patchright deps; @google/genai to optionalDependencies
- Gitignore: generic patterns instead of 20 hardcoded directories
- Remove asset-sourcing.md, video-recipes.md (unused, duplicated guidance)

Capture quality improvements (tested on 10+ websites):
- Color extraction: canvas-based oklch/lab resolver, pixel sampling via
  elementFromPoint, broad sweep for accent colors, gradient/shadow extraction
- Section detection: broadened selectors for div-based layouts, height cap
  to skip page-level wrappers, parent bg walkup for dark sites
- Font downloads: cap 6 per family / 30 total (Cal.com: 306→30)
- CTA detection: text pattern matching + nav context filtering
- Heading text: innerText with whitespace normalization
- Gemini captioning: maxOutputTokens 100→300, .env auto-loading
- .env.example updated with GEMINI_API_KEY docs
- TTS ranking: Kokoro first with Python 3.10+ note

* fix: address PR review comments + improve capture quality

Review round 2 fixes (jrusso1020 + vanceingalls):
- verify/index.ts: add path traversal guard (relative + isAbsolute)
- verify/index.ts: fix sections[i] undefined typecheck error (CI green)
- index.ts: escape Lottie JSON with \u003c to prevent </script> breakout
- step-4-storyboard: fix technique count contradiction (2-3 per beat, not
  across whole video)
- step-6-build: perspective tilt uses gsap.set() instead of CSS transform
  (avoids GSAP overwrite conflict)
- step-1-capture: reorder — command first, Gemini note after (zero-config
  is the default path, API key is optional enhancement)
- step-7-validate: add tsx fallback for snapshot command
- step-3-script: vary hook patterns, don't default to number every time
- assetDownloader: exempt SVGs from 10KB minimum filter (company logos
  like Hubspot/Intel/DHL are 2-6KB; HeyGen capture: 13→75 assets)

Note: adm-zip was NOT removed (reviewer #3) — it's still in
packages/cli/package.json:30. The root package.json had patchright
and purgecss removed, not adm-zip.

Note: ANTHROPIC_API_KEY not restored in .env.example — grep confirms
zero references in the entire codebase. The @anthropic-ai/sdk dependency
was removed earlier in this branch.

* refactor(capture): split index.ts (1175 to 566 lines) into modules

Mechanical extraction, zero logic changes.

New files:
- mediaCapture.ts (345 lines): Lottie preview, video manifest/screenshots
- contentExtractor.ts (314 lines): library detection, text, Gemini, asset descriptions
- scaffolding.ts (135 lines): .env loading, project scaffold generation

Also fixes false-positive BLOCKED.md with structural Cloudflare detection.
Tested on 20 websites, pre/post output identical.

* chore(capture): remove --split flow (splitter, verify, cssPurger, purgecss)

The --split feature auto-generates compositions from captured HTML — a
different approach from the /website-to-hyperframes skill workflow where
agents build compositions from scratch using the storyboard.

No skill file, no step reference, and no test session ever used --split.
Removes 923 lines of unused code + purgecss dependency.

Backed up to ~/Desktop/capture-split-backup/ for reference.

* fix(security): add ssrf protection, lottie injection fix, oom guard

- assetDownloader: add isPrivateUrl() guard blocking private IP ranges
  (127.x, 10.x, 172.16-31.x, 192.168.x, 169.254.x), cloud metadata
  endpoints, localhost, and non-HTTP schemes
- mediaCapture: fix Lottie JSON injection by loading shell HTML first
  then passing animation data via parameterized page.evaluate()
- index.ts: check Content-Length header before response.buffer() in
  Lottie network interception to avoid OOM on multi-GB responses

* fix(capture): security fixes, timeout, sub-agent dispatch instructions

Security (from miguel-heygen review):
- assetDownloader: export isPrivateUrl() SSRF guard
- htmlExtractor: add isPrivateUrl check before CSS fetch
- mediaCapture: add isPrivateUrl check before Lottie fetch
- mediaCapture: fix previewPage leak (try/finally)
- mediaCapture: skip Lottie files > 2MB for preview (CDP limit)
- contentExtractor: skip images > 4MB for Gemini captioning
- index.ts: check Content-Length before response.buffer() (OOM guard)
- snapshot.ts: register error handler before server.listen()

Capture improvements:
- Default timeout 30s to 120s (Shopify needs ~90s for Cloudflare)
- step-6-build: sub-agent dispatch template with explicit rules:
  pass file PATHS not contents, use local fonts not Google Fonts,
  verify ../assets/ references after each beat

* fix(capture): catalog before DOM mutation, networkidle2, faster Gemini

Critical: asset cataloger now runs BEFORE extractHtml which converts img
src to data URLs. Framer sites like heykuba.com went from 2 to 78 images.

- networkidle2 instead of networkidle0 (unblocks SPAs with WebSockets)
- Lazy-load wait: scroll to bottom, wait for img.complete
- CSS background-image cataloging for Framer/Webflow
- SVG naming: checks class, id, parent, inner text (not just aria-label)
- Gemini batch 5->20, pause 12s->2s (paid tier: 2000 RPM, ~0.001/img)
- maxOutputTokens 300->500, descriptions sorted captioned-first
- Remove tsx fallback from step-1 (reviewer nit, published CLI has it)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-16 12:03:16 -07:00
James RussoandClaude Opus 4.6 b23b0751da fix(player): parent-frame media playback for mobile (#266) 
* fix(player): parent-frame media playback for mobile

Mobile browsers block media.play() inside iframes when the user
gesture happened in the parent frame — postMessage doesn't transfer
user activation (per the User Activation v2 spec).

## Problem

The player renders compositions in a sandboxed iframe. When a user
taps play in the parent frame, the player sends a postMessage to the
iframe's runtime, which calls audio.play(). On mobile, this fails
silently because the iframe has no user activation context.

## Solution

The player now extracts ALL timed media elements (audio/video with
data-start) from the iframe's DOM (same-origin access), creates
parent-frame copies, and disables the iframe originals. On play(),
parentMedia.play() runs synchronously in the gesture call stack,
satisfying mobile autoplay policy.

### Generic media handling

- Finds all `audio[data-start], video[data-start]` in the iframe
- Creates a parent-frame copy for each (Audio or Video element)
- Preserves data-start offsets for correct seek positioning
- Strips data-start from iframe elements so the runtime ignores them
- Falls back to iframe media for cross-origin iframes

### `audio-src` attribute

Convenience for the common single-narration case. When set, the
player starts preloading audio immediately — before the iframe loads.
This eliminates the loading delay that caused jittery playback.

### No active sync

Both parent media and the GSAP timeline are real-time systems. When
started simultaneously, they naturally stay within ~10ms — no drift
correction needed. Active sync with coarse granularity (50ms polling)
caused MORE jitter than it prevented via repeated audio seeks.

## CI

- Added unified `test` job replacing separate per-package test jobs
- Added root `test` script: `bun run --filter '*' test`
- New packages with test scripts are automatically included
- Added happy-dom for player DOM tests

## Tests

- 10 new tests for parent-frame media: preloading, play, pause,
  seek, muted/rate sync, cleanup, attribute changes
- All 21 player tests pass

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(shader-transitions): pass CI when no test files exist

Add --passWithNoTests to vitest run so the unified test job
doesn't fail on packages that have a test script but no test
files yet.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): update tests for new id field and GSAP lint rule

- normalize.test.ts: loadTranscript now assigns id fields (w0, w1, etc.)
  to SRT/VTT results and empty string for words-json passthrough
- lintProject.test.ts: add GSAP CDN script to validHtml() fixture to
  satisfy the missing_gsap_script lint rule added in core

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): add missing data-start/data-duration to validHtml fixture

The validHtml() test fixture was missing data-start and data-duration
attributes, triggering the root_composition_missing_data_start and
root_composition_missing_data_duration lint warnings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(ci): fetch LFS objects for producer test job

Producer regression tests compare rendered output against reference MP4
files stored in git LFS. Without lfs: true, checkout fetches pointer
files instead of actual videos, causing "moov atom not found" errors.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* ci: remove redundant test-producer job

The regression workflow already runs the same 28 producer fixtures
in a Docker container with prod-matching Chrome/fonts/ffmpeg, sharded
across 8 parallel matrix jobs with 40-min timeouts. The CI test-producer
job was a duplicate that ran on bare runners with worse determinism
and a 15-min timeout too short for all fixtures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-14 11:20:07 -07:00
Vance Ingalls cb3d94c2a5 feat: add @hyperframes/shader-transitions package (#251) 
## Summary

New `@hyperframes/shader-transitions` package that encapsulates WebGL shader transitions into a single `HyperShader.init()` call. Replaces ~200 lines of per-composition boilerplate that LLMs failed to wire correctly 60% of the time.

### API

```js
var tl = HyperShader.init({
  bgColor: "#0a0a1a",
  accentColor: "#6366f1",
  scenes: ["scene1", "scene2", "scene3", "scene4", "scene5"],
  transitions: [
    { time: 7.2, shader: "cross-warp-morph", duration: 0.7 },
    { time: 15.2, shader: "domain-warp", duration: 0.7 },
  ]
});
tl.from("#s1-title", { y: 50, opacity: 0, duration: 0.7 }, 0.3);
```

### What the library handles

- **13 shader programs**: domain-warp, ridged-burn, whip-pan, sdf-iris, ripple-waves, gravitational-lens, cinematic-zoom, chromatic-split, glitch, swirl-vortex, thermal-distortion, cross-warp-morph, light-leak
- **html2canvas** bundled as dependency (not CDN) — single script tag for CLI users
- **DOM-during-holds**: canvas hidden between transitions, GSAP animations play on live DOM
- **Async capture with pause/resume**: timeline pauses during capture, resumes after textures uploaded — prevents progress tween from running ahead
- **Accent color theming**: `accentColor` derives dark/mid/bright uniforms. Burns, glows, leaks match the composition palette
- **Graceful degradation**: falls back silently when WebGL unavailable

### Code quality (from 3 review agents)

- No `!` non-null assertions — all WebGL creation calls throw on failure
- Vertex shader compiled once, cached across all programs
- Uniform/attribute locations cached per program via WeakMap (not looked up every frame)
- Captured canvases freed after texture upload (8MB each)
- Single timeline creation (was creating two, discarding one)
- Shared `tickShader()` render callback (was copy-pasted)
- `.finally()` for DOM restore in capture (was duplicated in `.then`/`.catch`)
- `parseHex` validates input (was silently producing NaN on invalid hex)
- Dead `ND`/`CP` shader library exports removed

### Shader-compatible CSS rules (transitions.md)

6 rules for compositions using shader transitions:
1. No `transparent` in gradients (canvas interpolates through black)
2. No gradient backgrounds on elements < 4px
3. No CSS variables on captured elements
4. `data-no-capture` for uncapturable decoratives
5. No gradient opacity < 0.15
6. Every `.scene` must have explicit `background-color` matching `bgColor`

### Build output

- IIFE (~214KB with html2canvas bundled, ~65KB gzipped) — `window.HyperShader`
- ESM + CJS + TypeScript declarations
- tsup build following `@hyperframes/player` conventions

## Test plan
- [ ] `bun run build` succeeds (includes shader-transitions)
- [ ] `bunx oxlint packages/shader-transitions/src/` — 0 errors
- [ ] Create a composition using `HyperShader.init()` — verify transitions fire, DOM animations play, accent colors match
- [ ] Test graceful degradation: composition works without WebGL (no transitions, no crash)
- [ ] Verify pause/resume: scrub to transition boundary — no jump in progress

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-04-13 18:44:00 -07:00
Miguel Ángel 078ed7d5cd fix(deps): patch security vulnerabilities in pretext and vite (#237) 
## Summary

- Bump `@chenglou/pretext` ^0.0.3 → ^0.0.5 in `packages/core` — fixes **high-severity** algorithmic complexity DoS ([Dependabot #3](https://github.com/heygen-com/hyperframes/security/dependabot/3))
- Bump `vite` ^5.0.0 → ^6.4.2 in `packages/studio` — fixes **medium-severity** path traversal in optimized deps `.map` handling ([Dependabot #2](https://github.com/heygen-com/hyperframes/security/dependabot/2))

## Test plan

- [x] `pnpm --filter @hyperframes/core build` — passes
- [x] `pnpm --filter @hyperframes/studio build` — passes (vite 6.4.2, 4631 modules, 3.85s)
- [x] `@vitejs/plugin-react@^4.0.0` supports vite 6 (`peerDependencies: vite ^4.2.0 || ^5.0.0 || ^6.0.0 || ^7.0.0`)
 2026-04-10 03:09:17 +02:00
Miguel Ángel 5655dabff6 feat: allow clip animation + ship <hyperframes-player> web component (#209) 
## Summary

Two independent initiatives that improve agent DX and expand HyperFrames' reach.

### Initiative 1: Fix the Clip Animation Footgun

- `gsap_animates_clip_element` lint rule now uses smart detection — only errors when GSAP animates `visibility` or `display` on a clip element
- All other properties (opacity, transform, x, y, scale, etc.) are allowed silently
- This was the #1 agent failure in QA (10/10 agents hit it on v0.2.1)

### Initiative 2: `<hyperframes-player>` Web Component

- New `@hyperframes/player` package — zero dependencies, 3.3KB gzipped
- Iframe-based web component with Shadow DOM for perfect isolation
- Video-like API: `play()`, `pause()`, `seek()`, `currentTime`, `duration`, events
- Controls overlay with play/pause, scrubber (mouse + touch), time display, auto-hide
- Full docs page at `docs/packages/player.mdx`

## Before / After

### Clip animation lint

**Before (10/10 agents hit this):**

```
✗ gsap_animates_clip_element: GSAP animation targets a clip element.
  Selector "#title" resolves to element <div id="title" class="clip">.
  The framework manages clip visibility — animate an inner wrapper instead.
  Fix: Wrap content in a child <div> and target that with GSAP.
```

**After (only errors on actual conflicts):**

```
# This passes lint — no error:
tl.from("#title", { opacity: 0, y: -50, scale: 0.8 }, 0);

# This still errors — actual conflict with runtime:
tl.to("#title", { visibility: "hidden" }, 3);
✗ gsap_animates_clip_element: GSAP animation sets visibility on a clip element.
  Fix: Remove the visibility/display tween. Use opacity for fade effects.
```

### Embeddable player

**Before:** No way to embed a composition in a web page.
**After:**

```html
<script src="https://cdn.jsdelivr.net/npm/@hyperframes/player"></script>
<hyperframes-player src="./composition/index.html" controls></hyperframes-player>
```

```js
const player = document.querySelector('hyperframes-player');
player.play();
player.pause();
player.seek(2.5);
player.addEventListener('ready', (e) => console.log('Duration:', e.detail.duration));
```

## Test plan

- [x] 427 core tests pass (20 GSAP lint tests with smart detection)
- [x] 7 player tests pass (formatTime + element registration)
- [x] TypeScript compiles cleanly (core + player)
- [x] Lint: GSAP animating clip with safe props → 0 errors
- [x] Lint: GSAP animating clip with `visibility` → 1 error (correct)
- [x] Player builds to 3.3KB gzipped ESM
- [x] Lockfile updated for CI
- [x] Docs page added at `docs/packages/player.mdx`
 2026-04-06 19:59:39 +02:00
Miguel Ángel e2c8ed5d83 refactor(core): replace cheerio with linkedom to drop deprecated whatwg-encoding (#187) 
## Summary

- `cheerio` pulls `encoding-sniffer` → `whatwg-encoding@3.1.1` (deprecated), causing a warning on every `npm install -g hyperframes`
- `linkedom` was already bundled into the CLI via tsup `noExternal` and has zero deprecated transitive deps
- Rewrote `htmlBundler.ts` and `subComposition.ts` to use standard DOM APIs via `linkedom`
- Added a `parseHTMLContent` helper that wraps HTML fragments in a full document structure (required for `linkedom` to populate `document.body`)
- Removed `cheerio` from `cli` dependencies and tsup `external` list
- Replaced `cheerio` with `linkedom` in `core` `optionalDependencies`

## Test plan

- [x] All 411 tests pass (`bun run test` in `packages/core`)
- [x] Full monorepo build succeeds (`bun run build`)
- [x] TypeScript typecheck passes
 2026-04-03 16:06:30 +02:00
Vance IngallsandClaude Opus 4.6 e9c2e6f772 fix(cli): resolve SyntaxError in bundled ESM output (#203) 
Two issues prevented `npx hyperframes` from running:

1. The tsup banner declared `const __filename` which collided with
   esbuild's CJS-to-ESM `var __filename` shim. ESM strict mode
   rejects const+var redeclaration. Changed to `var` so both
   declarations coexist.

2. postcss (producer dependency) was not resolvable during bundling
   due to bun's isolated module layout. Added postcss as an external
   dependency of the CLI package.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 14:52:37 -07:00
Miguel ÁngelandClaude Opus 4.6 7294803fbc feat(fonts): add Playfair Display, Noto Sans JP, Roboto, and 4 more to deterministic font database (#196) 
* fix(engine): suppress font-loading 404 noise in render console output

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(engine): downgrade resource 404s to buffer-only instead of suppressing

Address review feedback: instead of silently dropping "Failed to load
resource" errors (which could hide real asset failures), keep them in
browserConsoleBuffer for diagnostics but don't print to stdout. Real
asset 404s are still caught by the file server's own logging.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(engine): narrow 404 filter to font CDN domains and woff2 files only

Address review: filter was too broad and could suppress real asset
failures. Now only suppresses 404s matching fonts.googleapis,
fonts.gstatic, or .woff2 file extensions. Missing images, scripts,
and videos will still surface as [Browser:ERROR] in render output.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(fonts): add Playfair Display, Noto Sans JP, Roboto, and 4 more to deterministic font database

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 14:32:08 -07:00
Miguel Ángel 7f4e4333e5 fix(cli): read actual composition dimensions in info and fix semver update check (#194) 
## Summary

- **`info`** **command** now reads actual `data-width`/`data-height` from the root composition element instead of hardcoding 1920x1080 or 1080x1920 based on a parser heuristic that defaults to "portrait"
- **Update check** now uses proper semver comparison instead of string inequality (`!== VERSION`). Previously reported "update available" when installed `0.2.1` and npm had `0.2.0`

**Part 2 of 5** in a stacked PR series fixing E2E test findings.

## Test plan

- [x] `npx hyperframes info` on a 1920x1080 project shows "1920x1080" (not "1080x1920")
- [x] `npx hyperframes info --json` returns correct width/height fields
- [x] Version check no longer reports downgrade as available update
 2026-04-02 14:12:48 -07:00
Miguel Ángel 1681350ac4 fix(preview): rewrite sub-composition asset urls in styles (#174) 
## Summary
- rewrite CSS `url(...)` asset paths from sub-compositions before styles are hoisted into bundled/master preview output
- rewrite standalone sub-composition preview HTML so `src`/`href` paths keep resolving correctly under the preview root `<base>`
- add regression tests for bundled CSS asset rewriting and standalone sub-composition preview rewriting

## Root cause
Standalone composition previews reused the project `<head>` with a preview-root `<base href="/api/projects/:id/preview/">`, but the sub-composition body still contained `../...` asset references. Those escaped the preview route and 404ed. Separately, bundled preview already rewrote `<img src="../...">` paths but left hoisted CSS asset references like `@font-face src: url("../font.woff2")` untouched.

## Validation
- `pnpm --filter @hyperframes/core exec vitest run src/compiler/htmlBundler.test.ts src/studio-api/helpers/subComposition.test.ts`
- `pnpm --filter @hyperframes/core exec tsc --noEmit`
- `pnpm --filter @hyperframes/producer exec tsc --noEmit` *(blocked by pre-existing `packages/producer/src/services/deterministicFonts.ts` importing missing generated file `./fontData.generated.js` in the worktree install)*
- browser verification with `agent-browser` against the local preview server using `/Users/miguel07code/dev/test-hyperframes/heygen-promo`

## Browser proof
Verified the fixed preview routes in-browser after seeking to visible frames:
- standalone composition preview
- bundled master preview
 2026-04-01 18:13:38 +02:00
James Russo b866a28545 feat(cli): add remote template fetching via giget (#162) 
* feat(cli): add remote template fetching via giget

* fix: update remote.ts to use templates/ instead of examples/

* fix(cli): validate template ID against manifest before downloading

Fails fast with available template list instead of downloading
an empty directory for nonexistent templates.

* refactor(cli): simplify to single --template flag with dynamic validation

- Remove --example flag (--template handles bundled + remote)
- Remove static ALL_TEMPLATE_IDS list (validates against GitHub manifest)
- No CLI release needed to add new templates — just add to templates/ and templates.json
- scaffoldProject auto-detects bundled vs remote

* chore: update lockfiles for giget dependency

* fix(cli): remove undefined isAudioOnly reference
 2026-03-31 13:41:01 -07:00
Vance IngallsJamesClaude Opus 4.6
 		ef6225da1d feat(core): add fitTextFontSize utility for pixel-accurate text measurement (#152) 
Add @chenglou/pretext dependency and fitTextFontSize() utility that uses
canvas measureText to compute the largest font size that fits text within
a given width. Replaces character-count heuristics with actual font-aware
measurement.

- New fitTextFontSize() in @hyperframes/core/text, exposed on window.__hyperframes
- Generalized for all text elements (captions, titles, etc.), not just captions
- Unit tests (mocked pretext) + browser integration test (real Chromium canvas)
- Updated captions skill docs with usage, exit guarantee, and self-lint patterns

Co-authored-by: James <james.russo@heygen.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 20:14:13 -07:00
Miguel Ángel bd175b64a6 refactor(core): extract shared studio API module (#113) 
## Summary
Extracts all studio API routes into a shared Hono-based module at `@hyperframes/core/studio-api`.

### Architecture
- **`StudioApiAdapter` interface** — consumers inject host-specific behavior (project resolution, bundling, rendering, thumbnails)
- **Shared route modules**: projects, files, preview, lint, render, thumbnail
- **Shared helpers**: `isSafePath`, `walkDir`, `getMimeType`, `buildSubCompositionHtml`

### What this PR does
- Creates the shared module with all API routes extracted from both `vite.config.ts` and `studioServer.ts`
- Both consumers will be refactored in follow-up commits to mount this module with their own adapter

### What stays in each consumer
- **Vite**: SSR module loading, Puppeteer thumbnails, file watcher + HMR, producer HTTP proxy, multi-project scanning
- **CLI**: in-process `executeRenderJob`, local runtime serving, browser management, SPA static file serving

### Follow-up needed
- [ ] Refactor `packages/studio/vite.config.ts` to use `createStudioApi(adapter)` via `@hono/node-server`'s `getRequestListener`
- [ ] Refactor `packages/cli/src/server/studioServer.ts` to use `createStudioApi(adapter)`
- [ ] Add `./studio-api` export path to `packages/core/package.json`
- [ ] Add `hono` as peer dependency of `@hyperframes/core`

## Test plan
- [ ] Verify shared module compiles without type errors
- [ ] After consumer refactoring: all studio features work identically via both vite dev and CLI embedded servers

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-03-28 22:28:12 +01:00
Miguel Ángel 40bd159103 feat(studio): render queue, layout restructure, home page, hover preview (#95) 
## Summary
- Add render queue panel with progress tracking, download, and delete actions
- Restructure App layout: home page with project picker, session-based routing
- Add ExpandOnHover component for preview-on-hover interactions (uses motion/react)
- CompositionsTab now supports hover preview with expanded iframe view
- Vite config: guard setInterval cleanup to dev-only (fixes CI build timeout)
- Add favicon and update studio package deps

## Test plan
- [x] Render queue shows progress, completes, and allows download
- [x] Home page lists projects and navigates to session view
- [x] ExpandOnHover shows expanded preview on mouse hover with spring animation
- [x] `vite build` exits cleanly (no hanging process from setInterval)

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-03-28 21:28:43 +01:00
Miguel Ángel 0769678f46 refactor(studio): improve Timeline, PlayerControls, and player hook (#63) 
## Summary
- **Timeline**: Refactored track rendering with zoom support, drag/resize interactions, and playhead scrubbing
- **PlayerControls**: Redesigned with seek bar, time display, playback rate selector
- **useTimelinePlayer**: Enhanced with iframe bridge communication, timeline message parsing, and deterministic seek
- Add Timeline unit tests (109 lines)

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-03-28 20:39:35 +01:00
JamesandClaude Opus 4.6 bac66c2152 fix(ci): keep @hyperframes/cli name, rewrite to hyperframes at publish time 
Reverting the package rename — Vance needs @hyperframes/cli for local
dev workflow. Instead, rewrite the name to "hyperframes" in the publish
workflow just before npm publish, so the monorepo name stays intact.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-26 22:36:06 +00:00
JamesandClaude Opus 4.6 1e2875ab56 chore: update lockfile for CLI package rename 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-26 22:27:03 +00:00
Vance IngallsandClaude Opus 4.6 71e00b4533 fix(cli): resolve npx hyperframes from inside monorepo (#47) 
* fix(cli): rename workspace package so npx resolves from registry

Rename packages/cli from "hyperframes" to "@hyperframes/cli" so that
npm/npx stops resolving it as a local workspace package when run from
inside the monorepo. The publish workflow sets the name back to
"hyperframes" before publishing so the npm package name is unchanged.

Root cause: npm sees workspaces in root package.json, finds packages/cli
named "hyperframes", assumes it's local, but bun manages node_modules
so there's no bin symlink — npx fails with "command not found" instead
of falling back to the registry.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): update lockfile for workspace package rename

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(ci): publish CLI from temp copy to avoid workspace mutation

Copy packages/cli to a temp directory before renaming to "hyperframes"
for publish. Avoids corrupting the workspace if the job fails mid-way.

Addresses review feedback on #47.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(ci): resolve leftover conflict markers in publish.yml

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-26 10:25:19 -07:00
Vance Ingalls 94e25443ae build: migrate from pnpm to bun as package manager (#28) 
## Summary
- Replace pnpm with bun for dependency installation, script running, and ad-hoc execution
- Keep pnpm for publish workflow only (`publishConfig` overrides + `--provenance`)
- `bun install` replaces `pnpm install` (~4-5x faster cold installs)
- `bun run` replaces `pnpm run` (~28x less startup overhead)
- `bunx` replaces `npx` in lefthook hooks
- CI workflows updated (`oven-sh/setup-bun@v2` + `actions/setup-node@v4`)
- `pnpm-lock.yaml` removed, `bun.lock` generated
- `pnpm-workspace.yaml` kept for publish compatibility
- CLI source code (`packages/cli/src/`) unchanged — shipped to end users who may not have bun

Part 5/5 of [VA-851](https://linear.app/heygen/issue/VA-851/pre-migration-configure-eslint-prettier-and-conventional-commits)

## Test plan
- [x] `bun run lint` — 0 errors
- [x] `bun run format:check` — all files pass
- [x] `bun run build` — all 5 packages build
- [x] 330 core tests pass
- [x] 18 engine tests pass
- [x] `publish.yml` unchanged (pnpm stays for npm publishing)
- [x] No `bunx`/`bun run` references in shipped source code (`packages/*/src/`)
 2026-03-23 19:50:57 -07:00