mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
Vance IngallsandClaude Opus 5
7d38fed97f
* feat(studio): pure range ops for automation lane selections Add pointsIn() and replaceRange() functions for managing automation envelope edits within a time range. The key invariant: envelope values outside the selection never move. Implemented by anchoring the boundaries at t0 and t1 by sampling the original lane, so cutting middle sections cannot reshape the rest. Inner points from shape generators can suppress redundant anchors at merge distance. * fix(studio): budget replaceRange's inner points before capping, not after * feat(studio): automation selection slice * feat(studio): drag-select a time range on an automation lane Dragging on an automation lane's empty background now arms a range selection, snapped to the beat grid and clamped to the lane duration; a sub-3px drag counts as a click and clears instead. Point drags and Alt-drag segment bends still take priority, since the range arm only runs where the existing point/segment hit-test already returned null. useAutomationLanes binds the selection slice per element/lane so the rect renders from the store, matching the read pattern the writes already use. * feat(studio): delete an automation selection from the keyboard Escape clears the active automation-lane time selection; Delete/Backspace empties it via replaceRange(..., inner: []), which pins anchor points at both edges and leaves the envelope outside the selection untouched. Mounted in TimelineLanes.tsx next to the useAutomationLanes() call that already lives there. Also adds a stale-selection guard in TimelineAutomationLaneSlot that clears the selection if its lane's target stops existing on the bound element's automation (e.g. the automated effect was deleted). * test(studio): cover the automation selection stale-target guard * feat(studio): ramp, swell and dip generators for automation selections * fix(studio): let an automation range keep Delete from the clip useAppHotkeys listens on window/capture, so it runs before useAutomationSelectionKeyboard's document/capture handler. With a range selected, Delete fell straight through to the clip-delete branch and destroyed the whole audio clip the lane belongs to; Backspace hit the reset-keyframes branch on the way and wiped the clip's keyframes. Guard both by returning early when automationSelection is set, mirroring the selectedKeyframes precedent six lines above. No preventDefault: the downstream handler still needs the key. dispatchPlainKey is exported so the arbitration between keyframes, an automation range and the clip can be pinned without standing up the hook. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: suppress unused export for automationShapes (consumed upstack) * feat(studio): simplify dense automation runs Implements Ramer-Douglas-Peucker point-thinning for audio automation lane breakpoints, working in unit space for correct log-scaled parameter handling. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * chore: add automationSimplify to fallow complexity ignore The Ramer-Douglas-Peucker algorithm in automationSimplify.ts has inherent complexity (12 cyclomatic / 20 cognitive) that is by design and not refactorable. Added to health.ignore list and ignoreExports list since it's consumed by the UI layer one PR upstack. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(studio): internal clipboard for automation ranges * feat(studio): copy and paste automation ranges across lanes Extends the automation-selection keyboard hook with Cmd/Ctrl+C (copy the active range) and Cmd/Ctrl+V (paste onto the selected clip's lane, at the selection's start or the playhead, chaining the selection to the pasted span so a second paste lands right after the first). Paste falls through untouched when no target lane resolves, so clip-level paste keeps working. Also fixes a latent test-isolation bug: setup() never unmounted the previous test's Host, so document keydown listeners leaked across tests and could consume later events before the current test's own listener ran. * feat(studio): shape and simplify menu on an automation selection Right-click inside an active time-selection rectangle on an automation lane now opens a menu offering the four utility shapes (Ramp up, Ramp down, Swell, Dip) and Simplify, composing generateShape/simplifyPoints with pointsIn/replaceRange from the prior selection tasks. A point's own right-click still stops propagation and deletes it, unaffected. * feat(studio): retime an automation selection Add retimeRange pure operation that scales interior points proportionally into a new time span, then uses replaceRange to update the lane while preserving the envelope outside the union of old and new ranges. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(studio): repair the automation paste path and finish the key arbitration Paste was the least safe path in this feature: it resolved its target from the player store but committed through a different, asynchronously-lagging channel. Six review findings against this branch, plus the Cmd+C/Cmd+V half of the arbitration wa-15 started for Delete. - Write channel: resolvePasteTarget bails unless the binding's commitTargetKey equals the element it resolved. useAutomationLanes exposes that key, resolved through resolveTimelineIdForSelection — the same resolver applyDomSelection uses — and read in the same render as the commit handlers, so a handler and the key cannot describe different moments. Before this, clicking clip B then immediately pasting serialized B's automation onto A and left B untouched. - Chaining: the paste anchor comes from sel.t1, not sel.t0, so a second Cmd+V lands after the first instead of on top of it. The old comment claimed the new behaviour while the code did the opposite, and no test pressed Cmd+V twice. - Empty copy: copyRange returns false rather than arming a clipboard whose every paste is a destructive flatten, and samples the range's edges so copying a smooth stretch yields a real segment instead of no points. - Playhead: the playhead branch requires the playhead to be inside the clip rather than silently clamping an out-of-clip playhead to the clip's start. - Keys: one chord helper normalizes with toLowerCase() and gates on !shiftKey && !altKey, matching useAppHotkeys. CapsLock no longer kills the shortcut and Ctrl+Alt+V no longer pastes where the app declines. - Arbitration: useAppHotkeys consults automationOwnsKey before its c/v branch, so an active range keeps Cmd+C/Cmd+V from the clip clipboard the same way it keeps Delete. Without it Cmd+V duplicated the clip while the automation paste wrote the same file, and Cmd+C armed both clipboards. It returns without preventDefault — the downstream handler needs the key — and declines when the automation clipboard is empty so clip paste still works. dispatchModifierKey is exported to pin this, like dispatchPlainKey. - Double-action: the hook now returns early on e.defaultPrevented. useAppHotkeys is on window/capture and deliberately lets a keyframe selection outrank a range on Delete; without this that press deleted the keyframes there AND emptied the range here. - Project scoping: the clipboard scopes itself. Every entry point carries the project it speaks for and a mismatch empties the module, the shape keyframeSlice already uses to discard a request from a previous session. Scoping it inside the module rather than clearing it from the session seam is deliberate: the failure is silent and destructive — a range copied in project A pasted into B is remapped through A's captured sourceRange for an FX node B may not have, and the keystroke is consumed so clip paste never runs — so no future caller should be able to forget the guard. The mark isLastPasteSpan reads is scoped transitively, through the same check. - Session reset: createTimelineResetState clears automationSelection. It is as ephemeral as selectedKeyframes, and a range surviving a project switch can match a same-keyed clip in the new project and redirect a paste through sel.elementKey === paste.elementKey to a stale t0. Five of the six paste fixes above shipped without a test that fails without them, which is how the branch reached review with a comment describing chaining that the code did not do. Each now has one: a second Cmd+V landing after the first, a commit-target mismatch declining, an out-of-clip playhead declining, an empty-lane copy leaving an earlier clipboard intact, and Cmd+V with CapsLock on. All five fail against this branch's parent. * test(studio): probe retimeRange's actual guarantee, not sample-continuity past a moved edge The failing test probed t=5.1, which sits inside the reshaped transition segment between the new edge (t=5) and the existing point (t=6). When growing past an existing breakpoint, the transition TO that point legitimately reshapes — the edge moved (t=3→t=5) even though the far point (t=6) did not. The real guarantee: all BREAKPOINTS strictly outside the union keep exact (t, v) values. Corrected test to: 1. Verify sample continuity on unaffected side: t=[0,1,1.9] 2. Verify the breakpoint at t=6 keeps exact value: (t:6, v:0) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(studio): stretch an automation selection by its edges Add an edge-handle drag to a selection's rect: grabbing within 8px of either edge retimes the selection via the already-landed retimeRange, scaling interior points proportionally and clamping the dragged edge against its partner and the clip's duration. Priority is point-drag > curve-drag > edge-stretch > new-range-select, so a point sitting on an edge still wins the press. Cursor shows col-resize while hovering or dragging a handle. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(studio): retime edge-stretch from a fixed points snapshot moveEdge fed retimeRange the live draft on every pointermove while origin.t0/t1 stayed pinned to the drag's start. retimeRange is a relative transform that scales a lane's own current point positions, so repeated pointermoves compounded the scale factor (interior points drift toward the far edge) and could drop points that retimed past the selection's original bound out of the next move's `inner` set entirely. Snapshot lane.points at arm time (armBackgroundGesture) alongside the existing frozen origin, and always retime from that snapshot in moveEdge instead of the live draft. finishEdgeDrag is unchanged: it already just persists the last (now-correct) preview. Adds a regression test asserting a multi-pointermove edge-drag (both edges) lands on the exact same final points as a single-shot drag to the same target — the case that exposed the bug, since the existing suite only ever tested a single move. * fix(studio): clamp selection-start paste, sharpen clipboard test, cleanup - useAutomationSelectionKeyboard: clamp the selection-start paste branch to [0, element.duration - clip.span], same as the playhead branch already does. An unclamped paste near a clip's end could write points past element.duration and leave the resulting selection's edge ungrabbable off the visible lane. - automationClipboard.test.ts: swap the cross-parameter mapping test's target from fx.r.wet (numerically identical to VOLUME_RANGE) to the log-scaled fx.n1.frequency, so the test actually discriminates real unit-space mapping from a linear guess or a verbatim value copy. - automationLaneSelection.ts: drop the lone `!` non-null assertion in decimateEvenly's budget-of-1 branch for a guarded pattern, matching the loop right below it and the repo's no-`!` convention. - .fallowrc.jsonc: remove the two ignoreExports entries for AUTOMATION_SHAPES and simplifyPoints — both are now genuinely consumed (AutomationSelectionMenu.tsx, TimelineAutomationLane.tsx). - AutomationSelectionMenu.tsx: port TrackGapContextMenu's viewport-edge clamping so a right-click near the bottom/right of the timeline doesn't render the shape/simplify menu partially off-screen. * fix(studio): give edge-stretch the gesture contract the other four follow Seven review findings against this branch, five of which were one defect: edge-stretch was added as a fifth mutually-exclusive gesture on the lane without joining the threshold / live-preview / revert-on-cancel contract the point drag, curve bend, range drag and double-click all obey. Patching them one at a time would have been more code and less coherent, so this makes the stretch structurally parallel to its sibling range drag instead, and extracts it to useAutomationEdgeStretch on the way out — the gestures file had ~60 lines of headroom under the 600-line studio cap, and shaving comments to fit a refactor in is not a plan. - Threshold. A press within the 8px halo of either edge used to persist a no-op commit and push an undo entry that changed nothing (commitDataAttribute has no unchanged-value short-circuit). Worse, it made the pre-existing "click the background to clear the selection" escape unreachable anywhere near an edge. Below 3px of travel — the same threshold the range drag uses — the press now clears the selection and writes nothing at all. - Live preview. moveEdge never fired onRangeSelect and the hook discarded the drag's live position, so the highlight rect and both edge lines stayed pinned at the pre-drag bounds for the whole gesture and snapped into place on release: the user dragged an invisible handle. It now reports bounds on every move, exactly as the marquee drag does and for the same reason. - Revert on cancel. pointercancel means the browser abandoned the gesture; it was routed to the same handler as pointerup, which persisted whatever partial retime it had reached. It now restores the arm-time snapshot through the preview channel — there is nothing persisted to undo — and puts the selection back. A new cancelDrag handler owns that, so a release and an abandonment are no longer the same event. - Lost capture. capturePointer took the capture on e.target, i.e. whichever child the press landed on. A child that unmounts mid-drag takes the capture with it, silently, with no pointercancel — after which edgeDrag stayed non-null and every later button-less pointermove kept retiming and writing. Capture is now taken on the svg, which outlives every gesture on it, and a move reporting no buttons held ends the drag as a cancel. - Hit priority. A breakpoint sitting exactly on the selection's edge used to win the press. Since replaceRange pins an anchor at the union bound and finishEdgeDrag leaves the selection edge at that same time, EVERY range operation — stretch, delete, shape insert — leaves a point exactly on the edge it just created: the second stretch of the same edge resolved to a point-drag, at the one height (on the envelope) where a user naturally grabs it. The feature was not repeatable. An active selection's edge now outranks a point on it; clearing the selection reaches the point again, which is tested. - Clamp order. The dragged edge was bounded against its partner AFTER the 0-floor, so a selection thinner than the minimum width yielded a negative t0, which core's cleanPoint then collapses onto a duplicate t=0 on the serialize round-trip — silent envelope corruption. The floor is now applied last. The minimum width is its own MIN_SELECTION_SEC rather than a borrowed POINT_MERGE_SEC: when two breakpoints are the same breakpoint is a different question from how thin a time selection may get. One finding does not survive: edgeAt's `d0 <= d1` tiebreak was reported as making the t1 edge ungrabbable on a narrow selection, but that comparison IS nearest-wins, and a press right of the midpoint already resolved to t1. The midpoint split here is the same rule written so it is legible rather than inferred, and the test for it is labelled as characterizing behaviour, not fixing it. What was genuinely unreachable inside a narrow halo — starting a fresh range, or clearing the old one without Escape — the threshold above fixes. Also settles what retimeRange does with a breakpoint sitting ON a dragged edge, which was never decided: pointsIn is endpoint-inclusive, so it is interior and travels with the stretch. It has to be, because the commonest stretch of all is grabbing an edge to drag exactly that point outward, and anchoring it would delete it and flatten the span instead. The price is that the retimed point lands on the union's own boundary where a preservation anchor would go, and anchor() stands down within a merge radius — one time cannot hold two values — so the segment leaving the union reshapes. That is the one place replaceRange's outside-never-moves invariant bends, and both halves are now pinned: the exact points and the sampled slope for the on-edge case, and the full two-sided invariant for a selection whose edges are off any breakpoint. The earlier right-side probe at t=5.1 that caught this was deleted during development as inherent; it was reporting the real behaviour. * fix(core): make audio automation survive being rescheduled mid-playback Anything landing inside a running value curve is refused unless the parameter is cancelled first, and two paths were not cancelling: the chain observer wrote each knob straight onto its AudioParam before rescheduling, and a bent segment read as straight because only the curve exponent was checked, never the via point the timeline actually writes. The first threw NotSupportedError into the console and abandoned the rest of the envelope; the second played a dragged bend as a ramp. Measured against Chrome, in a live context and in an offline one suspended mid-curve: any cancel frees the span, and only a missing cancel is refused. clearParamLane takes the strongest form on purpose, because curve-over-curve refusals were reported with a cancel at the new schedule time already in place and have never reproduced; emit keeps a ramp fallback as the backstop for whatever that mechanism turns out to be. Dynamic carve is what exercises all of it, so it lands here too: - a `gain` primitive, so a carve can match levels as well as carve bands - carve settings collapse to one `strength`, with carveProfile deriving the six numbers that always moved together anyway - analyseCarveDynamics / analyseCarveDuck turn the analysis into envelopes, with a slow release so the bed does not snap back the instant a word ends - worklets are awaited inside attach, so adding a compressor to a carved bed no longer kills its envelopes and freezes every later edit - per-track failure detail in the render's audio stage, which was being discarded Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(studio): select automation points with a box, and stop them crossing Replaces the time-range selection with a rectangle. A lane selection is a set of breakpoints, not a span, so it now has value bounds as well as time bounds and a point is caught only if it falls inside both — which is what lets you take the peaks of an envelope and leave the dips between them. Delete, the group drag and the rings drawn on caught points all read the one rule, so what looks selected is exactly what those act on. Copy, paste, shape insert and simplify still work on the box's time span, because they act on the envelope over a stretch of time. Dragging is bounded by its neighbours in both the single and group cases. A point cannot cross another, and cannot land exactly on one either: the lane collapses points that share a `t`, keeping the later one, so arriving on top of a neighbour deleted it. It stops a millisecond short, which is under a pixel at any zoom the lane offers and keeps both points. Only stationary neighbours constrain a group, per member rather than per end, since a box can select a non-contiguous set. Edge-stretch is removed rather than fixed. Dragging a selection's edges to retime the points inside it was the feature this branch opened for, and it is not wanted: the hook, retimeRange, the edge handles, the col-resize cursor and the pointercancel revert path all go, along with the ~360 lines of tests that pinned them. Also: gesture-scoped coalescing keys, so one drag is one undo entry rather than a fragmented chain of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(studio): put #3207's edge-stretch back, folded into the unified hook The review blocks this PR for deleting a feature two PRs downstack: "#3209 deletes #3207 edge-stretch instead of folding it into the unified hook... merging the stack would ship #3207 and then silently remove its user-facing retime/edge-stretch feature." Verified, all four claims: `useAutomationEdgeStretch.ts` (248 lines) and its test were deleted, `retimeRange` was dropped from `automationLaneSelection.ts`, and the consolidated hook has no edge hit-test, arm/move/finish path, or resize cursor. Restored: the module, `retimeRange`, the selection regressions, and the lane wiring (`col-resize` cursor, `pointercancel` reverting a partial retime rather than persisting it). It is not a straight revert, because #3209 changed the selection from a time range to a box. Edge-stretch now takes `{t0,t1,v0,v1}` and moves only the time edges — the value extent rides through untouched, which keeps it the same gesture it was. **One arbitration call worth a second opinion.** #3207's rule was that a selection's edge outranks a point sitting on it, because every range operation leaves a breakpoint exactly on the edge it created — a point-first rule made the second stretch of an edge resolve to a point-drag. Under a box that rule now contradicts #3209's own test ("stops the group at a point it did not select"), which presses at t=0 v=1 — simultaneously the t0 edge and a selected point. I inverted it: selected content wins, the edge stretches everywhere it is not also selected content. The reasoning is that a box makes the point visibly part of the selection, and dragging selected content has to move it. That restores #3209's test and keeps the stretch usable along the rest of the edge — but it is a product decision between two deliberate designs, so flag it if #3207's original precedence was load-bearing. 954 player tests pass, including the 17 restored ones. * feat(studio): show every automated knob at the playhead, and carve as one module An automated parameter has two values: the number sitting in the chain, which is only the seed a lane replaced, and the number the envelope is on right now. The second is the true one, so the panel shows it — on the carve rack's readouts and on every effect's own fader and number field. A rack that showed the seed stood still while the carve was audibly working. Off the clip it keeps sampling rather than falling back to the stored number: a lane holds its first value backwards and its last forwards, so before the clip starts it already knows what it will open on, and the stored seed is a value nothing will ever play. Showing it made the fader jump the moment the clip came under the playhead. The playhead comes off the liveTime channel, throttled to 30 Hz — the RAF loop deliberately keeps frames out of the store, so a panel watching only the store would sit still for a whole take. PropertyPanel had that subscription inline; it is now one shared hook with two callers. Readouts reserve the width their parameter can need rather than what its current value takes, because an updating value one character narrower shunted everything after it sideways 30 times a second. The carve's effects are presented as one module: an author switched on a carve, and the peaking filters plus the level stage are how it is built, not six things to remove one at a time. Opening it lists every member's settings as readouts, since strength is what sets them. No carve control is offered on a track another track already carves against — that track is the voice, not the bed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(engine): render audio FX in an OfflineAudioContext Reads `data-fx-chain` off an audio element and runs the chain over the trimmed WAV before volume automation is baked in — effects should see the raw signal, and the envelope belongs on their output. The processing happens in an OfflineAudioContext inside the headless browser the engine already drives, running the same graph builders the studio previews with. That is the point of the approach: one implementation per effect, so the render agreeing with the preview is a property of the architecture rather than a tolerance to police. Reimplementing each effect as an FFmpeg filter would mean two implementations to keep in step, and for the dynamics processors and modulated delays there is no filter that behaves the same way. `build:audio-fx-runtime` bundles the graph builders into an injectable IIFE, following the same pattern as the existing runtime artifacts, so the browser runs exactly the code the studio does. The page loads from a file:// URL rather than about:blank because AudioWorklet is only exposed in a secure context — the compressor, limiter, gate and bitcrush processors would otherwise fail to register with an opaque error. file:// qualifies and needs no listening socket. The chain is serialised into the attribute the way colour grading carries its config, so there is no side-car file to resolve or lose. An FX failure is fatal for the whole mix rather than a per-track soft failure. Every other audio failure mode degrades gracefully — the track drops, siblings continue — but substituting the dry signal for a processed one ships a render that sounds plausible and is not what the author set up. Since the per-element work races under Promise.all, an internal AbortController chained off the caller's signal aborts in-flight siblings before workDir is removed. * feat(core): voiceover carve analysis Finds the bands a voice occupies so a music bed can be dipped there, letting the voice sit in front without ducking the whole track. Carve is a relationship between two tracks rather than an effect on one, so it stays out of the FX chain. What it emits is an ordinary chain of peaking filters, so a carve composes with whatever else is on the track and needs no separate rendering path. Selection is weighted toward intelligibility rather than raw voice energy. Ranking purely by power lands on the fundamental almost every time, because that is where a voice is loudest — but the masking that actually hurts a voiceover happens higher up, and dipping 160 Hz mostly just thins the bed. The bias is a control, not a constant: at 0 it follows raw energy, at 1 it weights toward 1-3 kHz. Ranking happens in dB, which matters more than it looks. Speech spreads 20-30 dB across these bands — it falls off roughly 6 dB per octave above the fundamental — so a weighting has to be on that scale to move anything at all. A multiplicative weight of `1 - bias + bias * shaped` is bounded below by `1 - bias`, capping its influence at 10*log10(1/(1 - bias)): 5.2 dB at the 0.7 default, 3 dB at 0.5. That is no influence against a real voice — every bias short of ~0.95 would rank exactly like bias 0 and carve the fundamental, the outcome the bias exists to prevent, while looking decisive against a fixture whose bands sit 2 dB apart. So the bias is a dB penalty, zero at 2 kHz and worth up to 30 dB at full strength, and relative cut depths come from a dB difference rather than a ratio of weighted linear powers. The bias reweights ranking without overriding the spectrum — a band the voice has no energy in is not worth carving, and scores -Infinity rather than competing — so a strongly low-pitched voice can still select low at full bias. What the tests hold is that biasing never selects lower than the unbiased ranking, that the DEFAULT bias reaches the presence region on a voice with a realistic tilt, and that bias 0 still follows raw power exactly. Includes a radix-2 FFT rather than a dependency; one Welch-style averaged spectrum over third-octave bands does not justify pulling in a DSP library. * fix(engine): keep the FX render 16-bit, stereo, and correctly sized Three defects in the offline FX path, none of which any test could see. **Float output silently disabled sample-accurate volume automation.** The writer emitted 32-bit IEEE float; the very next mixer step bakes the volume envelope into the samples and accepts only 16-bit PCM, returning null otherwise. So enabling any effect downgraded that track to the ffmpeg expression path — capped at 32 straight segments, quantising a curved envelope, and on a dense one falling back to base volume. It now writes 16-bit PCM, clamped rather than wrapped so a limiter at 0 dB or a resonant filter cannot turn overshoot into a click. A test asserts the baker accepts the writer's own output and actually fades it. **Everything was folded to mono.** `prepareAudioTrack` goes out of its way to emit stereo — its pan filter exists to dodge ffmpeg's 3 dB mono-to-stereo rematrix — and this folded it, then wrote one channel. So adding a single peaking EQ collapsed a bed's width and cost ~3 dB in the render, while preview stayed stereo. Channels now travel as one plane each, through an OfflineAudioContext of the same width, and come back interleaved. **Small results decoded the wrong length.** `new Float32Array(buf.buffer)` discards byteOffset and byteLength, and Node pools small allocations: a 400-byte payload sits at offset 8 inside an 8 KiB pool, so a clip under ~1024 samples decoded as 2048 samples of unrelated memory — and the empty-result guard could not see it. The reader has the mirror-image fix: a float data chunk on an odd boundary (ffmpeg's pcm_f32le writes fmt(18) + fact, landing `data` at 58) now copies instead of throwing RangeError on an unaligned view. The tail limitation is now stated rather than mis-stated: the context is exactly as long as the input, so a reverb or delay still ringing is cut there. The old comment claimed the opposite. How far a tail may run past a clip's end changes the clip's length in the mix, so it is a product decision, not one to make here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(producer): report an FX render failure as an audio error `processCompositionAudio` reports per-track failures in its result, but an FX failure it cannot degrade past — a browser that will not launch, a chain that will not build — rejects instead. `runAudioStage` had no try, so that rejection escaped to the orchestrator as an unclassified pipeline exception, losing the stage/owner/retryable classification this stage exists to attach, and skipping its abort check on the way out. It now lands in `audioError` alongside every other cause, while an abort still keeps its own shape rather than being reported as an audio problem. Not done here: committing the generated `audio-fx-runtime-inline.ts` so a fresh clone typechecks packages/engine without building first. The bundle is built from the stub, and the stub changes three times across this stack — so the artifact differs per branch and would conflict on every restack. Its model, position-edits-render-inline.ts, is committed only because it is stable. Building before testing is this monorepo's existing contract (studio's tests need core's dist too), so the gap is not specific to audio FX and is better closed by a build ordering gate than by committing a per-branch artifact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(engine): skip the browser FX render cases when there is no browser CI's `Test` job was red on this PR with four failures, all the same cause: Failed to launch the browser process: spawn /home/runner/.cache/hyperframes/chrome/chrome-headless-shell The job installs ffmpeg and no browser, deliberately — every other suite that needs an external binary already guards on it (`describe.skipIf(!HAS_FFMPEG)`). These cases were the only ones assuming a Chrome, so they failed on an absent dependency rather than on anything about the code. Guards on `resolveHeadlessShellPath()` — the same resolver `acquireBrowser` launches through, so the check cannot drift from the thing it guards the way a hard-coded cache path would. A configured path that does not exist throws; that is caught and read as "cannot run here". Checked both directions rather than just the green one: with a browser all 11 cases run and pass, and with `HYPERFRAMES_BROWSER_PATH` pointed at a missing binary exactly 3 skip and the other 8 still run. A guard that silently skipped everything would have looked identical in CI. They keep their value where it exists — every developer machine, and any job that has run `hyperframes browser ensure`. Not touched: the CodeQL failure on this PR is a run from 2026-08-07, five days and several force-pushes stale. None of the 17 open repo alerts are in files this PR changes; it re-runs on this push. * chore(engine): suppress the temp-file alert with the reason it is safe CodeQL flags `writeWav`'s `writeFileSync` as js/insecure-temporary-file (high) — the one new alert on #3021, and the reason its CodeQL check is red. It is a false positive, and the comment says why rather than just silencing it: `path` is always inside a directory made by `mkdtempSync`, never a name assembled directly under `tmpdir()`. Both callers are covered — the browser host page writes into `mkdtempSync(join(tmpdir(), "hf-fx-host-"))`, and the render output goes to the producer work dir, itself `mkdtempSync(join(tempRoot, "producer-project-"))`. mkdtemp picks the random suffix and creates the directory 0700 in one syscall, so the predictable filename inside it cannot be pre-created or symlinked by another user, which is the attack the rule is about. The analyzer sees the dataflow reach `tmpdir()` and not the mkdtemp in between. Suppressed inline rather than dismissed in the UI, so the justification lives next to the code and the rule stays live for anything added later in this file. Matches the repo's existing convention — `planV2.ts:222` carries an `lgtm[js/insecure-temporary-file]` for a different reason on the same rule. Correcting myself: I first reported this alert as not real, having intersected the PR's files against the default-branch alert list, which does not contain PR-ref alerts. Querying ?ref=refs/pull/3021/merge returns it straight away. * test(engine): probe ffmpeg and Chrome instead of assuming them Two failures on #3021's Test job, both about the environment rather than the code under test. **Bare `ffmpeg` is not on PATH in CI.** The 16-bit fixture shelled out to `execFileSync("ffmpeg", ...)` and died with ENOENT. The job does provide ffmpeg, through `prepare-ffmpeg-bin`, which is what `getFfmpegBinary()` resolves — every other ffmpeg-dependent suite in this package already goes through it. Now this one does too, and the case is `skipIf(!HAS_FFMPEG)` so a contributor without ffmpeg skips rather than fails. **The browser guard trusted the wrong thing.** It asked `resolveHeadlessShellPath()` and treated a returned path as "a browser is here". CI's cache holds a chrome-headless-shell that resolves and then fails to spawn — a partial download is indistinguishable from a working one by `existsSync`, which is all that resolver checks. So the three browser cases ran anyway and failed on the launch. It now runs `--version` and requires exit 0, which is the same probe the ffmpeg suites use: ask the binary, do not infer from the filesystem. Checked both directions rather than just the green one. With a working browser all 11 cases run and pass; with `HYPERFRAMES_BROWSER_PATH` pointed at a binary that exits non-zero — CI's exact situation — exactly 3 skip and the other 8 still run. A guard that quietly skipped everything would have looked identical on the CI summary. * feat(core): register the audio-fx-rack canary at 0% Lands the rollout switch dark, per the registry's own procedure: "Start at percentage: 0 and merge that — a canary at 0 is dead code you can land safely and ramp without a code review." Declared at the bottom of the stack so every branch above can read it. The gate itself goes in at wa-4-fx-panel, where the rack first appears. Scope is deliberate and stated in the description: it gates the AUTHORING surface only. A composition that already carries `data-fx-chain` still plays and renders it. A canary should stage who can REACH a feature, not make an attribute somebody already wrote silently inert — an agent that writes a chain through the skill would otherwise produce a file whose audio processing vanishes with no error. * feat(studio): audio FX panel generated from the registry Controls for the whole chain: add, remove, reorder, bypass, and every knob each effect declares. Nothing in the panel knows what a compressor is. The registry supplies each parameter's range, step, unit and scale and the panel renders what it finds, so adding an effect or a knob upstream needs no change here, and the panel cannot offer a value the renderer would reject — a typed-in figure is clamped into the declared range on the way through. Frequency and time controls span three or four decades, so those declare a log scale and the slider maps exponentially; a linear slider would spend most of its travel somewhere useless. Reorder is a first-class control because chain order changes the sound: a reverb before a compressor is not the same as after. Carve gets its own block rather than an entry in the add menu, with a picker for the voice track to listen to. It processes this track based on another one, which is how a sidechain control works — it lives on the track that changes, and names the source. * feat(studio): show the Audio FX section on audio tracks Adds `audioFx` to the editing-affordances contract and renders the FX panel in the inspector when an `<audio>` element is selected. The section is audio-only. A `<video>` carries its sound on a separate `<audio>` element, so an FX chain on the video would have nothing to process. Chain and carve settings are written straight back onto the element as serialised attributes, the way colour grading carries its config, so persistence is an ordinary attribute write and needs no new server route. A chain that cannot be parsed renders as empty rather than breaking the panel, and the attribute is left untouched until the user changes something. The collapsed group summarises what is on the track ("2 effects + carve") so the state is visible without expanding it. Wired into PropertyPanelFlat rather than PropertyPanel: STUDIO_FLAT_INSPECTOR_ENABLED defaults to true, so the flat inspector is what actually renders. * refactor(studio): lift audioFxSummary out of PropertyPanelFlat `PropertyPanelFlat.tsx` is 612 lines here against the repo's 600-line cap, so the required File size check is red — the sole reason this PR is blocked. The review says as much: "mechanical fix (~5 min), not a design problem. Code itself is LGTM." Moves `audioFxSummary` to `audioFxSummary.ts`, the same file a later branch creates for it. Deliberately the smallest cut that clears the cap rather than the whole `AudioFxGroup` extraction: every later commit in the stack edits AudioFxGroup, so moving it here would collide with each of them, while almost nothing touches this function. 595 lines. * feat(core,studio): hear the FX chain in preview, and run the carve analysis Splices an element's FX chain into the playback graph so preview stops being silent about effects, and wires the carve button that was previously inert. The chain goes between the decoded source and its gain stage: effects see the raw signal and volume automation rides on their output, matching the order the offline render uses. Since preview and render call the same graph builders, what is heard while scrubbing is what gets written. The splice lives in the transport rather than on the `<audio>` element. The transport plays each track from a decoded AudioBuffer and mutes the element to avoid doubling, so capturing the element with createMediaElementSource would have processed a stream nothing is listening to — it looked like it worked because the call succeeded, and the audio was unchanged. A chain that cannot be built plays dry rather than silencing the track, which is the right failure in preview: the author keeps working and hears the source. The render still refuses, because shipping the dry signal there would be wrong. Carve now analyses for real: it decodes the chosen voice track, ranks its bands and writes the resulting peaking filters onto this track. Generated nodes are tagged `fromCarve`, so re-running replaces the previous carve instead of stacking another set on top of hand-added effects. Known limitation: the graph is built when a source is scheduled, so a knob turned mid-playback takes effect on the next play or seek rather than immediately. Live re-parameterisation needs the transport to hold the handle and forward updates. * fix(studio,core): stop parameter drags from restarting playback Dragging a knob wrote the chain through the persisting attribute path on every input event. That path refreshes the preview, which reloads the composition and reschedules audio — so a single drag reloaded dozens of times and playback stuttered the whole way. Drags now go through `onSetAttributeLive`, the same path colour grading uses for scrubs: it coalesces undo entries and sets `skipRefresh`, so no reload happens. The persisting write fires once, when the gesture ends — pointer-up or blur for a slider, Enter or blur for a typed value. A select commits immediately since there is no drag to wait for. While dragging, the control is driven from local state. Waiting for the value to round-trip through the element attribute made the knob lag behind the pointer. For the change to be audible without a reload, the graph now follows the attribute: the chain installed by the transport observes the element and re-parameterises itself in place, so a value change lands on the next 128-sample quantum. A shape change (effect added, bypassed, pole count) cannot be patched into a running graph, so it still waits for the next schedule rather than cutting the audio mid-play. The regression test drags a slider through several values and asserts the persisting handler is untouched until release. * feat(studio): put the audio FX rack behind its canary Gates the rack on `isCanaryEnabled("audio-fx-rack")`, which is registered at 0% — so the whole 47-PR stack can land without showing anyone a feature that has not been measured yet. The gate sits on the AUTHORING surface and nowhere else. The runtime and the render still honour a `data-fx-chain` already on an element, so a composition written through the skill or by `carve.mjs` keeps its processing rather than going silently dry for anyone outside the cohort. A canary should stage who can REACH a feature, not make an attribute somebody already wrote stop working with no error. Gated at the panel rather than in `resolveEditingSections`: the affordance resolver is a pure function in core describing what an element CAN support, and rollout state is not a property of an `<audio>` tag. Pinned the 0% with a test, and checked it fails at 25 — a ramp should have to break something that says "this ships dark" out loud. One gap, stated rather than papered over: the gate itself has no unit test. I wrote one and deleted it, because `PropertyPanel.test.tsx`'s harness never renders the Audio FX group for its audio fixture even with the gate removed — so the test passed for the wrong reason in the off case and could not pass at all in the on case. A test that cannot fail for the right reason is worse than none. Verifying the gate needs the panel harness to mount that section first, which is its own change. * fix(core): register FX worklets before building nodes that need them An AudioWorkletNode cannot be constructed before its processor is registered — it throws, and the surrounding chain is lost with it. `attachElementFxChain` built the chain first and only then called `ensureAudioFxWorklets`, so every worklet-backed effect (compressor, limiter, gate, bitcrush) threw on construction and the track fell back to dry. Instrumenting the preview showed `hf-compressor: InvalidStateError` with addModule never called at all. When the module has not landed yet the track now plays dry and the graph is swapped in once registration resolves, so the effect arrives a moment late instead of never. Registration is also tracked per context rather than in one module-level promise. A processor registered on one AudioContext does not exist on another, so the shared promise made every context after the first believe it was ready when it was not — the studio's transport owns its own context, which is exactly that case. With the worklets actually running, the compressor's per-sample log10 and pow became real audio-thread work. Samples below the knee have a gain of exactly unity and need neither, so the envelope is now compared in the linear domain and the transcendentals only run for samples that are actually being compressed. * refactor(studio): split the FX node row out of FxSection Clears the health findings the FX stack left behind: the chain-node render callback was a 70-line closure over half of FxSection's state, and the two reorder arrows were the same button written twice. Also drops two exports with no consumers, and registers the audio FX runtime stub as an entry point — it is bundled by file path, so nothing imports it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(core): automation envelope model for audio tracks Adds the data model behind Ableton-style automation lanes: breakpoint envelopes over track volume or one knob of one effect in the track's FX chain, stored on the element as `data-automation`. Times are clip-local, so an envelope travels with the clip when it moves — the clip-envelope model rather than arrangement automation. `sampleAutomationLane` is the single interpolator. The lane drawing, the preview scheduler and the render bake all call it, so the picture and the sound cannot disagree about the curve. Log-scaled parameters interpolate in log space, matching what their own knob already promises. FX nodes gain a stable `id`, minted by count rather than randomly so the document is the same on every machine. Lanes address nodes by id, so reordering a chain never re-points a lane at a different effect, and a lane whose effect was deleted is dropped rather than left to reattach. Also warns when a track carries both a volume lane and a GSAP volume tween, since only the lane is heard and the tween silently does nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(studio): lift the audio FX group out of PropertyPanelFlat `PropertyPanelFlat.tsx` was 672 lines against the repo's 600-line cap, so the required File size check was red — the sole reason #3014 and #3022 are blocked. Both reviews say the same thing: "mechanical fix, not a design problem. Code itself is LGTM." Moves `AudioFxGroup` and `audioFxSummary` into `propertyPanelAudioFxGroup.tsx`, which is where a later branch puts them anyway — done here so the file is under the cap from the point it first crosses it, rather than ten branches later. 533 lines now. The four audio imports it no longer needs go with it. Not fixed here: three `FxSection carve` tests fail on this branch with "Cannot read properties of undefined (reading 'toFixed')". Confirmed pre-existing by stashing this change and re-running — that is the separate `Test` failure the review also flags. * feat(core): expose the AudioParams behind automatable FX knobs Marks the knobs an automation lane can drive and has each graph builder hand back the AudioParam behind them, so a scheduler can write to a running effect without knowing what the effect is. A knob is not always one AudioParam. A wet/dry mix is two gains moving in opposition, and a knob in milliseconds drives a delay time in seconds, so each target carries the mapping out of the knob's own declared unit. What stays unautomatable is stated where it is decided: a WaveShaper curve, a convolution impulse and a one-pole filter's coefficients are all rebuilt wholesale rather than scheduled, and the four worklet effects take values by postMessage rather than through AudioParams. The registry flag is written by hand, so a test builds every effect and checks the exposure both ways — nothing flagged is missing, nothing exposed is unflagged. A flag that lied would offer a lane that silently did nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(core): play automation envelopes in preview Schedules each lane onto the AudioParams behind its knob using native ramps and value curves. Nothing evaluates the envelope per frame: it is handed to the audio thread once, so it stays sample-accurate however busy the main thread is, and the offline render will schedule it the same way. Timing comes from the transport, so an envelope survives seeking into the middle of a clip, a clip that has not started yet, and a playback rate that compresses clip seconds into context seconds. A straight line is only scheduled as a ramp when nothing bends it — no curvature, a linear parameter scale, and no unit mapping. Log-scaled parameters and mapped ones are sampled instead, since a delay knob in milliseconds and a wet/dry pair moving in opposition are not linear in the parameter they drive. Lanes with nowhere to write are skipped rather than reported: a one-pole filter exposes no frequency param, and the worklet effects expose none at all. Editing an envelope mid-playback re-aims it at the live playhead rather than restarting the track. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(core): make the volume lane audible in preview The envelope was scheduled onto the transport's gain AudioParam, but the runtime rewrites that gain every tick from `data-volume` and the GSAP-seeked value — so it was erased within a frame. Volume automation was correct in the render and inaudible while previewing. The lane now feeds the per-tick path where the probed volume keyframes already sit, checked ahead of them so the two cannot fight, and the transport no longer schedules volume at all: one mechanism instead of two racing. The cost is honest — in preview the level steps per tick rather than per sample, exactly as the existing keyframe path does. The render still bakes it into the PCM sample-accurately, and FX parameters are still scheduled on their own AudioParams, since nothing rewrites those. Parsed lanes are cached by attribute text: the runtime asks once per tick per track, and parsing there would run the JSON parser 60 times a second for a value that only changes on an edit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(engine): bake automation envelopes into the render The offline render schedules FX lanes with the same scheduler preview uses, inside the OfflineAudioContext that already runs the same graph builders. The input WAV is the clip's own audio from its first sample, so clip-local time is offline time and the envelope needs no offset. Volume lanes take the existing PCM bake rather than a second mechanism: the lane is converted to keyframes, so a straight fade stays two of them and only a bent segment is sampled — the baker interpolates linearly and would otherwise quietly straighten the curve. A volume lane supersedes keyframes probed from the timeline, which `lint` already warns about. A browser test sweeps a lowpass from below a 2 kHz tone to well above it and measures both ends. Parsing the envelope is not the same as scheduling it, and only running the real thing tells the two apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(core): apply chain edits to the running graph A structural edit — an effect added, removed, bypassed, or a filter's pole count switched — was dropped. `buildFxChain`'s update reports false when the change is not merely new values, and the attribute observer ignored that, so the edit only took hold when the persisting write reloaded the composition. That reload restarted every playing track, which is what was heard as the audio chopping. The graph is now swapped in place: the old effects are detached, the new ones built and connected between the same source and gain, and any lanes re-scheduled onto the new nodes. The source node is never touched, so playback does not restart. A track with no chain is watched too, rather than wired through and forgotten, so adding its first effect is heard the same way. That means the function always returns a disposer instead of null for the empty case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(studio): drop the FX panel's dead __testables export Fallow audit flagged it — no test imports the module. * fix(core,studio): clear the remaining Fallow audit findings on the FX panel - Split FxSection's per-node row into FxNodeRow + FxNodeControls so the CRAP score (31.6, threshold 30) splits across two smaller units instead of moving wholesale with one extraction. - Dedupe the repeated "open the add menu, read its items" block in propertyPanelFxSection.test.tsx into openAddMenuItems(). - Merge build-audio-fx-runtime.ts and build-position-edits-render.ts into one build-inline-artifact.ts, config-selected by CLI arg — the two scripts were a byte-for-byte clone save for names. - Exempt canary.test.ts's rawFnv (a deliberate independent reimplementation used to cross-check canaryBucket, per its own docstring) and the property-panel test files' shared renderInto/mount scaffolding (pre-existing across 9 files, 2 outside this stack) in .fallowrc.jsonc, consistent with this file's existing exemptions for the same class of intentional/pre-existing duplication. * fix(ci): allowlist the build-script consolidation in the no-main-deletions guard build-audio-fx-runtime.ts and build-position-edits-render.ts were merged into build-inline-artifact.ts to kill a fallow duplication finding; the deletion guard flagged that as an accidental loss since main still has both originals. * fix(core): dedupe the wet/dry mix math between delayFeedback and chorusLfo Both effect builders set wet.gain to the mix and dry.gain to its complement in identical two-line blocks; fallow kept re-flagging it as a 10-line clone on every unrelated change. Extracted setWetDryMix. * fix(core): remove the build-audio-fx-runtime.ts stray resurrected by a main merge An earlier merge with main brought this deleted file back (git's merge/delete handling on an unchanged-on-one-side file); package.json already points at build-inline-artifact.ts, so it sat unreachable and duplicating that file's config, both of which fallow flagged. * fix(studio): pull TimelineLanes under the 600-line cap TimelineLanes.tsx hit 620 lines. Extracted the three per-clip pointer gestures (resize-start, pointer-down move-arm, click/razor-split) into createClipGestureHandlers — one factory call per rendered clip instead of ~120 lines of inline handler bodies in the render loop. 529 lines now. * fix(studio): split the extracted pointerdown handler under the CRAP threshold Moving the ~120-line gesture logic into timelineClipGestureHandlers.ts concentrated it into two functions fallow flagged (onPointerDown at CRAP 63.6, onResizeStart at 31.6). Split the decision logic (which gesture a pointerdown implies) into a pure resolvePointerDownAction, then split its own intent-blocking check into isIntentBlocked. onResizeStart's guard moved into canStartResize. Every function now scores under 30. * fix(studio): drop the unused DomEditSelection import in PropertyPanelFlat CI caught it on PR #3026 (wa-12-panel-params); a later refactor in the stack removed the last use of the type here without removing the import. * fix(studio): close the typecheck and fallow gaps wa-18b-reschedule opened useAutomationLanes.ts's write() assumed gesture-scoped coalescing and a preview-only commit that useDomEditAttributeCommits.ts never grew — backported that option support from its own later commit so the two sides of the API agree. The paste path and its tests were missing the box selection's v0/v1 bounds a sibling commit added to AutomationSelection. The FX panel's carve controls still edited the six mechanism numbers (maxCutDb, bands, intelligibilityBias) after carveProfile() collapsed authoring to one Strength knob, so those fields no longer existed on HfCarveSettings; UI now edits strength, and analyseCarveBands is called with carveProfile(strength). Also closes fallow's complexity, dead-code and duplication findings on this PR's diff: extracted automationLaneDragMath.ts (pure group/point-move math) and useAutomationRangeDrag.ts (the marquee-select gesture) out of useAutomationLaneGestures.ts, pulled a couple of render-loop ternaries and a resolver into named functions, dropped an export nothing outside its file used, and shared a step-simplifier between audioCarve's two envelope builders. The edge-stretch vs. box-select priority test in TimelineAutomationLane.test was still pinning the pre-box-select rule (edge wins over a point sitting on it) that a sibling commit deliberately reversed — a point inside the box is now selected content, so grabbing it drags the group instead. Updated the test to the shipped rule instead of the old one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(core): cap the via conic's weight so an edge-clamped via point can't NaN A via point pulled out past the segment (viaX: 5, viaY: -3) clamps to (0.999, 0.001) — exactly on the steady region's edge, where edge - viaX is 0. viaConic divided by that zero to get an infinite weight, and shapeVia turned Infinity into NaN a few steps later (Infinity - Infinity in the quadratic coefficient). NaN reaching setValueCurveAtTime silences the automated parameter for the rest of the render. Capped the weight at 1e6 instead of leaving it unbounded — past that point the arc already reads as touching the via point, so nothing visible is lost. Also hardened shapeVia's existing denominator guard (`<= 0`) to `!(> 0)`, since NaN fails the original comparison and fell through it. Review by Miga (PR #3208). * fix(studio): widen PropertyPanel's resetModules render timeout again The 20s margin (already once widened for the same reason) is timing out in CI's full-monorepo Test run — the resetModules()+fresh-import render this test needs is uncached and competes with every other package's test suite for the same worker pool, and the same test passes in well under 2s standalone. Went to 45s rather than re-tuning to whatever number happens to clear the current CI load, since that number moves every time CI gains a package. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Write HTML. Render video. Built for agents.
Quickstart | Showcase | Playground | Catalog | Docs | Discord
HyperFrames is an open-source framework for turning HTML, CSS, media, and seekable animations into deterministic MP4 videos. Use it locally with the CLI, from AI coding agents with skills, or as the rendering core behind hosted authoring workflows.
Quick Start
With an AI coding agent
Install the HyperFrames skills, then describe the video you want:
npx skills add heygen-com/hyperframes --full-depth
The picker opens with nothing pre-selected — the Core Skills group is all you need: the
/hyperframesrouter installs each creation workflow on demand. Agents and non-interactive runs should usenpx hyperframes skills updateinstead — it installs exactly the core set, whereas a non-interactiveskills addwithout--skillinstalls all 19.
--full-depthdoes a full clone of the repo's currentmain. Without it,skills addfetches the skills.sh registry blob, which lagsmainby hours — you'd get an older copy of a skill. (hyperframes skills updatealready installs full-depth.)
Try a prompt like:
Using
/hyperframes, create a 10-second product intro with a fade-in title, a background video, and subtle background music.
The skills teach agents the HyperFrames production loop: plan the video, write valid HTML, wire seekable animations, add media, lint, preview, and render. They work with Claude Code, Cursor, Gemini CLI, Codex, and other coding agents that support skills.
Skills
HyperFrames ships 19 skills agents load on demand. Read /hyperframes first — it's the router and capability map; it picks a workflow for any "make me a…" request — video, deck, or composition port — and points to the domain skills below.
Default to the core set — the router installs each creation workflow on demand. npx hyperframes skills update installs exactly that from anywhere; the interactive picker (npx skills add heygen-com/hyperframes --full-depth) lists it as the "Core Skills" group, nothing pre-selected. The picker is interactive-only — a non-interactive or agent run without --skill installs all 19. Use npx skills add heygen-com/hyperframes --all --full-depth to install all 19 deliberately (skips the picker), or npx skills add heygen-com/hyperframes --skill <name> --full-depth for just one (bare name, no leading /). Keep --full-depth — it installs the current main; without it skills add fetches the skills.sh blob, which lags by hours.
Installs stay lean after that: npx hyperframes init keeps the core set fresh (the router, the hyperframes-* domain skills, and media-use — plus whatever is already installed; /figma stays on demand) and never expands a partial install; the creation workflows install on demand — the router runs npx hyperframes skills update <workflow> before entering one. Nothing re-pulls the full set behind your back.
Upload to Codex
Build the upload-ready Codex plugin archive from the committed HEAD version of the manifest, brand assets, and skills:
bun run package:codex-plugin
This writes dist/hyperframes-plugin.zip with a hyperframes/ root folder and fails if the archive exceeds Codex's 100 MB upload limit.
Router
| Skill | Use when |
|---|---|
/hyperframes |
Read first for any request to make / create / edit / animate / render a video, animation, or motion graphic. Capability map for the domain skills, the intent layer that confirms every creation brief up front, and intent router for the creation workflows below. |
Creation workflows
| Skill | Use when |
|---|---|
/product-launch-video |
Any website — marketing / launching / promoting a product (from its URL, a brief, or a script), or a site tour / showcase / social clip featuring the site's own visuals. Up to ~3 min (sweet spot 30-90s). |
/faceless-explainer |
Explaining a topic / concept from arbitrary text — no product, no URL, no website capture; every visual is LLM-invented (typography / abstract / diagram / data-viz). |
/pr-to-video |
A GitHub pull request (PR URL, owner/repo#N ref, or "this PR") → changelog / feature-reveal / fix / refactor explainer, read via the gh CLI. |
/embedded-captions |
Adding captions / subtitles to an existing talking-head video (footage untouched) — verbatim rail, embedded climax behind the subject, or pure-cinematic embed. |
/talking-head-recut |
Packaging an existing talking-head / interview / podcast video with designed graphic overlays — lower-thirds, data callouts, kinetic titles, pull-quotes, side panels, PiP. |
/motion-graphics |
A short, unnarrated, design-led motion graphic (~under 10s) — kinetic type, stat / chart hit, logo sting, lower-third, animated tweet / headline. MP4 or transparent overlay. |
/music-to-video |
A music track (audio file, video to pull audio from, or one generated from a mood brief) → a beat-synced video — lyric, slideshow, or kinetic promo; music drives pacing. |
/slideshow |
A presentation / pitch deck / interactive deck — discrete slides, fragment reveals, branching, hotspot navigation, presenter mode. Output is a navigable deck, not a rendered video. |
/general-video |
Anything else — longer or multi-scene pieces, brand / sizzle reel, title card, static loop, freeform composition. Input- and length-agnostic fallback, and the home of companion mode (co-create with the full toolbox). |
/remotion-to-hyperframes |
Porting an existing Remotion (React) composition's source to HyperFrames HTML. One-way migration, not creation. |
Domain skills (loaded on demand)
Atomic capabilities the creation workflows compose against — pull one when you need that specific layer.
| Skill | Covers |
|---|---|
/hyperframes-core |
The composition contract — data-* timing attributes, class="clip", tracks, sub-compositions, variables, framework-owned media playback, determinism rules. |
/hyperframes-animation |
All animation knowledge — atomic motion rules, scene blueprints, transitions, runtime adapters (GSAP / Lottie / Three.js / Anime.js / CSS / WAAPI / TypeGPU). |
/hyperframes-keyframes |
Seek-safe keyframe authoring across runtimes — GSAP timelines, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, 3D depth — plus hyperframes keyframes diagnostics for rendered motion. |
/hyperframes-creative |
Non-animation creative direction — frame.md / design.md, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns. |
/media-use |
The media OS — resolve any media need (BGM, SFX, image, icon, logo, voice, color grade, LUT) into a frozen local file or paste-ready block + ledger record, generate via TTS/music/image models when the catalog misses, transcribe, caption, remove backgrounds, and reuse assets across projects. One shared audio engine + manifest tracking. |
/hyperframes-cli |
CLI dev loop — init, lint, check, snapshot, preview, render, publish, doctor, plus HeyGen-hosted cloud rendering (cloud render) and AWS Lambda rendering (lambda deploy / render / progress). |
/hyperframes-registry |
Install and wire registry blocks and components into compositions via hyperframes add. Authoring a new block or component to contribute upstream. |
/figma |
Import Figma assets, tokens, components, and storyboard sections → reconstructed motion (frames read as states, not slides) (REST/CLI) plus Motion animations (MCP) and shaders (MCP source / native export) into a composition. |
For visual design handoff workflows, see the Claude Design guide and Open Design guide.
Manually with the CLI
npx hyperframes init my-video
cd my-video
npx hyperframes preview # preview in browser with live reload
npx hyperframes render # render to MP4
Requirements: Node.js 22+, FFmpeg
What You Can Build
Need ideas? Browse the Showcase for finished videos you can watch, read, run, and remix.
- Product launch videos and feature announcements
- PR walkthroughs with animated code diffs, narration, and captions
- Data visualizations, chart races, and map animations
- Social videos with kinetic captions, overlays, and music
- Docs-to-video, PDF-to-video, and site-tour explainers
- Reusable motion graphics for automated content pipelines
Frame.md
frame.md — your design system, ready for video.
Every brand has a design.md. None of them were written for a camera. frame.md is the missing translation layer: it takes your web-context design spec and inverts it for the frame — the same tokens, the same rules, but rewritten so an AI agent can compose a promo video without guessing at scale or reaching for web chrome.
The output is a DESIGN.md superset your whole toolchain can read. Atoms stay sacred. Composition stays free. Numbers come from the script.
Biennale Yellow |
BlockFrame |
Blue Professional |
Bold Poster |
Broadside |
Capsule |
Cartesian |
Cobalt Grid |
Coral |
Creative Mode |
Browse and remix them all at hyperframes.dev/design.
How It Works
Define a video as HTML. Add data attributes for timing and tracks. Use GSAP, CSS, Lottie, Three.js, Anime.js, WAAPI, or your own frame adapter for seekable animation.
<div id="stage" data-composition-id="launch" data-start="0" data-width="1920" data-height="1080">
<video
class="clip"
data-start="0"
data-duration="6"
data-track-index="0"
src="intro.mp4"
muted
playsinline
></video>
<h1 id="title" class="clip" data-start="1" data-duration="4" data-track-index="1">Launch day</h1>
<audio
data-start="0"
data-duration="6"
data-track-index="2"
data-volume="0.5"
src="music.wav"
></audio>
<script src="https://cdn.jsdelivr.net/npm/gsap@3/dist/gsap.min.js"></script>
<script>
const tl = gsap.timeline({ paused: true });
tl.from("#title", { opacity: 0, y: 40, duration: 0.8 }, 1);
window.__timelines = window.__timelines || {};
window.__timelines.launch = tl;
</script>
</div>
Preview instantly in the browser. Render locally or in Docker. The renderer seeks each frame in headless Chrome and encodes the result with FFmpeg, so the same input produces the same video.
HyperFrames Stack
HyperFrames is the open-source rendering engine, plus a growing set of tools around HTML-native video creation.
| Piece | Status | What it does |
|---|---|---|
| CLI | Available | Scaffold, preview, lint, inspect, and render local video projects |
| Core / Engine / Producer | Available | Parse compositions, drive headless Chrome, encode video, and mix audio |
| Catalog | Available | Reusable blocks and components for transitions, overlays, captions, charts, maps, and effects |
| Agent skills | Available | Teach coding agents the video-production patterns that generic web docs miss |
| Studio | Available, evolving | Browser surface for previewing and editing compositions |
| AWS Lambda rendering | Available | Deploy a distributed render stack and drive renders from your laptop or CI |
| hyperframes.dev | Available | Community playground for previewing, iterating, sharing, and rendering HTML-native video projects |
| frame.md | Available | Invert your design system for the camera — a DESIGN.md superset an agent can compose video from |
Catalog
Install ready-to-use blocks and components:
npx hyperframes add flash-through-white # shader transition
npx hyperframes add instagram-follow # social overlay
npx hyperframes add data-chart # animated chart
Browse the catalog at hyperframes.heygen.com/catalog.
Why HyperFrames?
- HTML-native: compositions are HTML files with data attributes. No React requirement, no proprietary timeline format.
- Agent-friendly: agents already write HTML, and the CLI is non-interactive by default.
- Deterministic: same input, same frames, same output. Built for CI, regression tests, and automated rendering.
- No build step: an
index.htmlcomposition plays as-is and can be previewed directly in the browser. - Adapter-based animation: bring GSAP, CSS animations, Lottie, Three.js, Anime.js, WAAPI, or a custom runtime.
- Open source: Apache 2.0 license, with no per-render fees or commercial-use thresholds.
HyperFrames vs Remotion
HyperFrames is inspired by Remotion. Both tools render video with headless Chrome and FFmpeg. The main difference is the authoring model: Remotion's bet is React components; HyperFrames' bet is plain HTML that humans and agents can both write easily.
| HyperFrames | Remotion | |
|---|---|---|
| Authoring | HTML + CSS + seekable animation | React components |
| Build step | None; index.html plays as-is |
Bundler required |
| Agent handoff | Plain HTML files | JSX / React project |
| Library-clock animations | Seekable, frame-accurate via adapters | Wall-clock animation patterns need care |
| Distributed rendering | Local and AWS Lambda render paths | Remotion Lambda, mature cloud renderer |
| License | Apache 2.0 | Source-available Remotion License |
Read the full comparison in the HyperFrames vs Remotion guide.
Documentation
Full documentation: hyperframes.heygen.com/introduction
Packages
| Package | Description |
|---|---|
hyperframes |
CLI for creating, previewing, linting, and rendering compositions |
@hyperframes/core |
Types, parsers, generators, linter, runtime, and frame adapters |
@hyperframes/engine |
Seekable page-to-video capture engine using Puppeteer and FFmpeg |
@hyperframes/producer |
Full rendering pipeline for capture, encode, and audio mix |
@hyperframes/studio |
Browser-based composition editor UI |
@hyperframes/player |
Embeddable <hyperframes-player> web component |
@hyperframes/shader-transitions |
WebGL shader transitions for compositions |
@hyperframes/aws-lambda |
AWS Lambda SDK and deployment surface for distributed renders |
Community
HyperFrames is used in production at HeyGen, with community examples from teams like tldraw, TanStack, and others in ADOPTERS.md. Open a PR if your team is using HyperFrames.
- Questions and ideas: Discord
- Bugs and feature requests: GitHub Issues
- User research: Book a casual 30-minute conversation with the HyperFrames team — no preparation or sales pitch
- Security reports: SECURITY.md
- Contributions: CONTRIBUTING.md
Development Note
The repo uses Git LFS for golden regression-test baselines under packages/producer/tests/**/output.mp4 (about 240 MB of .mp4 files). If you're cloning the full repo for development, install Git LFS first:
# macOS
brew install git-lfs
# Ubuntu / Debian
sudo apt install git-lfs
# Windows
winget install GitHub.GitLFS
# Then, once per machine
git lfs install
If you only need source files, you can skip LFS content:
GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/heygen-com/hyperframes.git