ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-05 10:14:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,518 Commits 885 Branches 406 Tags
03c47dbb37d7f8d046ae78fbf643d5f261e25b9d
Commit Graph
8 Commits
Author SHA1 Message Date
Vance Ingalls 341e65aea2 fix(slideshow): harden media controls in present decks (#1619) 2026-06-20 23:35:58 -07:00
Miguel Ángel 6f677292ae refactor(core): simplify packages/core — dead code, dedup, type safety (#1413) 
- Delete unused mediaPreloader module, 5 dead RuntimeState fields,
  emitPerformanceMetric, lintScriptUrls, 5 variable type guards
- Consolidate compiler utilities: unify CSS URL regex, relative URL
  predicate, MIME map, @import regex, bulk asset rewrite delegation
- Cache extractGsapWindows per script (eliminates 2 redundant recast
  parses per lint run), share stripJsComments and script extraction
- Deduplicate GSAP parser: share serializeValue/safeJsKey, centralize
  converted-id fallback (6 sites), keyframe codegen (3 sites),
  waypoint extraction, insert-after-anchor, script hoisting
- Replace 88 bare any annotations with typed AstNode/AstPath interfaces
- Derive RuntimeBridgeControlAction from HyperframeControlAction,
  alias RuntimePickerElementInfo, share macOS font profiler
- Gate generateHyperframesStyles on includeStyles, collapse 4 GSAP
  property mutation cases into 2
- Extract magic numbers into named constants, replace 5 double casts
  with type guards and typed accessors (runtime/globals.ts),
  reduce function complexity in htmlParser and files route
 2026-06-13 18:23:36 -04:00
Miguel Ángel e0cb8fcee3 fix(core): remove 1800s hard cap on timeline duration that silently truncated long compositions (#1114) 
The runtime had a maxTimelineDurationSeconds field defaulting to 1800
(30 minutes) that clamped the TransportClock duration. Any seek beyond
this cap was silently clamped, so GSAP tweens starting past ~1700s
never received their totalTime() call and stayed at their pre-tween
state (e.g. opacity:0).

The data-duration attribute is the authored source of truth. The loop-
inflation guard (timelineLooksLoopInflated) already handles the infinite
repeat:-1 case this cap was meant to protect against.

Closes #1107
 2026-05-28 16:26:40 -04:00
Miguel Ángel b009288df7 feat(runtime): single-clock transport — eliminate pause/play audio drift (#671) 
Replace the two-clock architecture (GSAP rAF ticker + HTMLMediaElement
pipeline reconciled by a 50ms polling loop) with a single TransportClock.
GSAP is always paused and seeked to clock.now() on each rAF tick.
Drift between visual timeline and audio is structurally impossible.

Architecture:

  TransportClock.now() ──rAF──▶ timeline.seek(t) + el.currentTime
       ▲
  AudioContext.currentTime (~21µs)  ← WebAudio active
       OR
  audio.currentTime (~33ms)         ← HTMLMediaElement fallback
       OR
  performance.now() (~1ms)          ← no audio

Key changes:
- TransportClock class with monotonic + audio-master clock sources
- WebAudioTransport: routes audio through AudioBufferSourceNode for
  sample-accurate scheduling, falls back gracefully to HTMLMediaElement
- rAF tick loop replaces 50ms setInterval poll; GSAP always paused
- Strict sync (40ms threshold, consecutive-sample gated) + forceSync
  on play/pause/seek transitions for sub-frame media accuracy
- Buffer-stall: visuals freeze when audio is buffering instead of
  running ahead
- Frame quantization preserved in seek/renderSeek (parity contract)

Browser-verified: 0.0ms drift after 40 pause/play cycles (was 400ms+).

Also fixes: CDN script HTML error responses in validate (pre-existing).

54 tests across clock, clock-drift, webAudioTransport, and media.

Closes #668
 2026-05-08 07:53:46 +02:00
Miguel ÁngelandClaude Opus 4.6 a7a6648852 feat(player): add volume/mute controls (#651) 
* feat(player): add volume/mute controls to the player

Adds a mute toggle button and volume slider to the controls bar,
positioned between the time display and speed selector. The slider
expands on hover for a compact default footprint.

- `volume` attribute/property (0–1, clamped) with `volumechange` event
- `muted` attribute now syncs to the controls UI (icon updates)
- Three volume icons: high, low, muted — updates reactively
- Volume forwarded to parent-frame audio proxies and iframe runtime
  via `set-volume` postMessage control
- 9 new tests covering volume clamping, events, controls rendering,
  mute toggle, and iframe message forwarding

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(player): wire set-volume through runtime bridge + address review feedback

Addresses the blocker from PR review: the iframe runtime now handles
`set-volume` messages end-to-end (types → bridge → init → media sync).

Runtime side:
- Add `set-volume` to RuntimeBridgeControlAction union
- Add `volume` field to RuntimeBridgeControlMessage
- Handle `set-volume` in bridge.ts with [0,1] clamping
- Store bridgeVolume in RuntimeState, apply to media elements
- syncRuntimeMedia composes userVolume × clip author volume

Player side:
- Muted toggle now dispatches `volumechange` (HTML5 spec compliance)
- Volume slider auto-unmutes when scrubbed above 0 while muted
- Touch support on volume slider (touchstart/move/end)

Tests: 5 new (3 bridge, 2 media)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(player): add ARIA keyboard controls, fix icon collision and clipVolume parity

- Volume slider: role="slider", aria-label, aria-valuemin/max/now,
  tabindex=0, arrow key support (5% steps, auto-unmutes)
- Volume=0 unmuted now shows low-volume icon instead of muted icon
- Fix clipVolume divergence: init.ts uses Number.isFinite() matching
  media.ts semantics (preserves data-volume="0")
- 3 new tests: ARIA attributes, volumechange on mute, icon collision

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-07 00:38:27 +02:00
Vance Ingalls f906797222 perf(player): coalesce _mirrorParentMediaTime writes (#396) 
## Summary

Coalesce writes to `el.currentTime` inside `_mirrorParentMediaTime` so a single jitter sample no longer triggers a parent-media seek. A drift correction now requires **two consecutive samples** above the threshold (~`MIRROR_DRIFT_THRESHOLD_SECONDS`) before the player writes back. One-shot alignment paths (`promoteToParentProxy`, `_onIframeMediaAdded`) opt out via `force: true` so initial alignment stays immediate.

## Why

Step `P1-4` of the player perf proposal. `_mirrorParentMediaTime` is called every animation frame on parent media proxies. Even without true drift, browser internals report tiny jitter on `currentTime` reads — typically below 30 ms but occasionally crossing the threshold for a frame. Writing to `currentTime` triggers a seek, which is expensive *and* invalidates pipeline buffers, which causes the next frame's reading to jitter further. The result was unnecessary seek thrash on otherwise-aligned media.

By requiring two consecutive over-threshold samples, transient jitter is filtered out while real drift (a sustained offset) still corrects within ~1 frame of latency. This eliminates the most common cause of dropped frames on the studio thumbnail grid.

## What changed

- Each `_parentMedia` entry gains a `driftSamples` counter that increments while the absolute drift is above `MIRROR_DRIFT_THRESHOLD_SECONDS` and resets to 0 on the first sample below.
- `_mirrorParentMediaTime(el, opts)` only writes back when `driftSamples >= 2`, except when `opts.force === true`.
- `promoteToParentProxy` and `_onIframeMediaAdded` pass `force: true` so the first alignment after registration is still immediate (these are user-visible state transitions, not steady-state telemetry).

## Test plan

- [x] 11 new unit/integration tests in `hyperframes-player.test.ts` covering:
  - Single-sample jitter does not trigger a write.
  - Two-sample sustained drift does trigger a write.
  - Trending drift correction (gradually increasing offset) is detected within 2 samples.
  - `force: true` override bypasses the sample requirement.
  - Out-of-range proxies (proxies whose source has been removed) do not panic.
  - Multiple proxies maintain independent counters — drift on one does not affect the other.
  - `_promoteToParentProxy` alignment is immediate.

## Stack

Step `P1-4` of the player perf proposal. Builds on `P1-1` (shared adopted stylesheets) and `P1-2` (scoped media observer). Together these three target the studio multi-player render path — `P0-1*` perf gate scenarios will pick up the wins automatically.
 2026-04-22 17:44:49 -07:00
Miguel Ángel 3256551a5e fix(player): single-owner audio to prevent double voice in preview (#298) 
## Summary

Fixes the double-voice issue in studio preview where narration plays twice with a drifting offset (measured 23ms → 80ms over a 28s clip).

## Root cause

Two audio pipelines were playing the same source in parallel:

1. The iframe runtime played `<audio data-start>` elements via `syncRuntimeMedia` — the intended path.
2. `<hyperframes-player>` also created parent-frame `<audio>` copies on iframe load and auto-played them in response to every runtime `state` message.

The existing `_muteIframeMedia` tried to silence the iframe copies via `el.volume = 0`, but `syncRuntimeMedia` re-asserts `el.volume` from `data-volume` every tick, so the mute never held. Studio seeks went through `__player.seek()`, which only updated the iframe timeline; parent copies kept their stale `currentTime` and drift compounded across seeks.

Confirmed via agent-browser instrumentation on `factory-series-c-video`:
- 6 `volumechange` events per play cycle (mute-fight signature)
- Both copies audible at `volume=1`, offset growing 23ms → 80ms
- Every seek widened the drift further

PR #295 (v0.4.2) actually **made it audible** — before that, parent copies 404'd on the wrong URL and played silently. Fixing the URL exposed the latent double-playback.

## Fix

Explicit single-owner audio ownership between `<hyperframes-player>` and the runtime.

- **Default ownership is `runtime`**: iframe drives audible playback; parent proxies stay paused and inert. Matches every desktop / studio code path. No parent `play()`, no `volumechange` thrash.
- **On `NotAllowedError`** from the runtime's `play()` attempt (autoplay-gated iframes), the runtime posts `media-autoplay-blocked` once. The player promotes to `parent` ownership: sends `set-media-output-muted: true` to the runtime, starts parent proxies, mirrors `currentTime` from state messages with a 150ms correction threshold.

Two orthogonal mute channels replace the volume fight:

| Channel | Purpose |
|---|---|
| `set-muted` | User's mute preference (existing, unchanged) |
| `set-media-output-muted` | Internal ownership handoff (new) |

`syncRuntimeMedia` now accepts `outputMuted` and asserts `el.muted = true` per active tick — sticky against sub-composition media that arrives mid-playback. Uses native `muted` (orthogonal to `volume`) so no other code path can clobber it.

## Why this shape

- **Single owner, explicit transition.** No races, no tug-of-war.
- **Probes reality, not device class.** We flip on an actual `NotAllowedError`, not on `matchMedia('(pointer: coarse)')` or user-agent sniffing.
- **Uses `muted` instead of abusing `volume`.** `muted` is orthogonal to `volume`; `syncRuntimeMedia` doesn't write to it; author / user settings stay intact.
- **Parent proxies become a thin mirror.** Under parent ownership, their `currentTime` is slaved to the iframe timeline via state messages — no independent drift.
- **Backwards compatible.** Old runtimes without the new bridge action ignore the message; old players without the new message just get the previous behavior.
- **Capture engine unaffected** — it bypasses both DOM pipelines and muxes audio from source files.

## Files changed

- `packages/core/src/runtime/types.ts` — `set-media-output-muted` action + `media-autoplay-blocked` outbound message types.
- `packages/core/src/runtime/state.ts` — `mediaOutputMuted` + `mediaAutoplayBlockedPosted` fields.
- `packages/core/src/runtime/bridge.ts` — route new action to `onSetMediaOutputMuted`.
- `packages/core/src/runtime/media.ts` — `outputMuted` param asserts `el.muted = true` per tick; `NotAllowedError` detection fires `onAutoplayBlocked`.
- `packages/core/src/runtime/init.ts` — wire new bridge handler; coordinate with `set-muted`; post `media-autoplay-blocked` once per session.
- `packages/player/src/hyperframes-player.ts` — `_audioOwner` state; delete `_muteIframeMedia`; `_promoteToParentProxy`; mirror parent `currentTime`; gate all parent play/pause/seek on ownership.

## Verified end-to-end with agent-browser on `factory-series-c-video`

**Runtime ownership (default — desktop studio):**

| | Before | After |
|---|---|---|
| `PARENT.play()` calls per play cycle | 1 | **0** |
| iframe `volumechange` events | 6 | **0** |
| Audible streams | 2 (drifting) | **1 (iframe)** |

**Parent ownership (simulated autoplay block — direct message):**

| | Value |
|---|---|
| iframe audio | `muted=true`, `volume=1` (untouched) |
| parent audio | `muted=false`, `volume=1`, audible |
| Parent ↔ iframe `currentTime` offset | ~6 ms steady state |
| Offset > 150 ms | corrected by mirror sync |

**Mobile path simulated with iPhone 14 emulation + injected `NotAllowedError` from iframe `<audio>.play()`:**

Event timeline captured via agent-browser instrumentation:

```
t=0.0 ms   IFRAME.play() called                       ← runtime attempts playback
t=0.4 ms   IFRAME.play() REJECTED: NotAllowedError    ← simulated mobile gate
t=0.4 ms   →IFRAME bridge set-media-output-muted=true ← player promotes
t=0.6 ms   PARENT.play() called                       ← parent proxy starts
t=0.8 ms   ←IFRAME msg media-autoplay-blocked         ← runtime signal
t=1.0 ms   PARENT.play() resolved                     ← audible
t=1.3 ms   IFRAME muted=true, volume=1                ← iframe silenced via native muted
```

Steady state at t=4 s under promoted parent ownership:

| Element | currentTime | paused | volume | muted |
|---|---|---|---|---|
| Parent audio | 4.060 s | false | 1.0 | **false** (audible) |
| Iframe audio | 4.068 s | false | 1.0 | **true** (silent) |

**Offset: 8 ms**, single audible stream, orthogonal mute channel respected.

## Test plan

- [x] `bunx vitest run` under `packages/core` — **467 / 467 pass** (incl. 4 new `media.test.ts` + 2 new `bridge.test.ts`)
- [x] `bunx vitest run` under `packages/player` — **23 / 23 pass** (3 rewrites for new contract, 2 new for promotion flow)
- [x] `bun run build` — all packages green
- [x] Fresh preview + browser repro on `factory-series-c-video`:
  - [x] Runtime ownership: single audio stream, no drift
  - [x] Parent ownership promotion via direct `media-autoplay-blocked` message: iframe muted, parent audible
  - [x] iPhone 14 emulation + injected `NotAllowedError`: full promotion chain verified in ~1 s, 8 ms steady-state offset
  - [x] No `volumechange` thrash in either ownership mode
- [x] One round of QA on a physical iOS / Android device before release — exercises real `NotAllowedError` path (expected behavior identical to simulation above)
 2026-04-17 04:46:22 +02:00
Vance IngallsandClaude Opus 4.6 9f8e5ba5a1 initial code (#2) 
* feat: initial code port from hyperframes-internal

Port all OSS-ready packages from the internal monorepo:
- @hyperframes/core — shared types, HTML generation, GSAP utilities, runtime
- @hyperframes/cli — CLI for creating, previewing, and rendering compositions
- @hyperframes/engine — framework-agnostic rendering engine (BeginFrame + FFmpeg)
- @hyperframes/producer — video rendering pipeline (Puppeteer + FFmpeg)
- @hyperframes/ui-player — browser-based video player component
- @hyperframes/studio — composition editor (React frontend + Hono backend)

Includes regression test suite with Docker-based test harness.

All HeyGen-internal references, deployment infrastructure, and
proprietary assets have been removed. Package names migrated
from @app/* to @hyperframes/*.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: scrub internal codenames and stale references from OSS port

- Replace static.heygen.ai runtime URLs in test fixtures
- Remove internal CDN publish script (publish-hyperframe-runtime.ts)
- Replace sandbox-studio, sandbox-interceptor, __magicEditRuntime
  with neutral names (studio, hyperframe-runtime, __hyperframeRuntime)
- Fix stale Vault API / localhost references in docs
- Remove broken deprecated_studio link

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove remaining internal codenames and stale references

- Delete stale producer README.md and PIPELINE.md (referenced nonexistent files)
- Replace "Cerberus" codename with "HyperFrames" in test design reviews
- Replace magic-edit postMessage identifiers with hf-preview/hf-parent
- Rename debug-magic-edit-timeline.ts to debug-timeline.ts
- Replace "Motion Cut" with "HyperFrames" in Timeline comments
- Fix studio/CLI references to nonexistent archive package
  (use local data/projects/ dir, stub render proxy)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-21 22:43:56 -07:00