mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-05 05:37:09 +00:00
feat: add captions, transitions, and media skills
Three new HyperFrames-specific skills: - captions: Build timed caption compositions from whisper.cpp transcripts. Covers word grouping, animation patterns, positioning, and timing rules. - transitions: Scene transition constraints and patterns (crossfade, wipe, scanline). Covers track overlap rules and relative timing. - media: Video, audio, and image handling constraints. Covers codecs, transparent video, trimming, PiP, and the framework playback contract. Also adds design spec for CLI transcription feature (future work). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
d9a20d72e1
commit
7cb5579d67
@@ -0,0 +1,232 @@
|
||||
---
|
||||
name: captions
|
||||
description: Build timed caption compositions from whisper.cpp transcripts in HyperFrames. Use when adding captions, subtitles, or timed text to a composition that has a transcript.json file.
|
||||
---
|
||||
|
||||
# Captions
|
||||
|
||||
Build HyperFrames caption compositions from whisper.cpp transcript data.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Project has a `transcript.json` (output from whisper.cpp `--output-json`)
|
||||
- User asks to "add captions", "add subtitles", or "show the words"
|
||||
- Editing an existing captions composition
|
||||
|
||||
## Whisper.cpp Transcript Format
|
||||
|
||||
The transcript file is whisper.cpp's native JSON output. Word-level timestamps are in the `tokens` array within each segment:
|
||||
|
||||
```json
|
||||
{
|
||||
"transcription": [
|
||||
{
|
||||
"timestamps": { "from": "00:00:00,000", "to": "00:00:05,000" },
|
||||
"offsets": { "from": 0, "to": 5000 },
|
||||
"text": " Hello world.",
|
||||
"tokens": [
|
||||
{
|
||||
"text": " Hello",
|
||||
"offsets": { "from": 0, "to": 1000 },
|
||||
"p": 0.98
|
||||
},
|
||||
{
|
||||
"text": " world",
|
||||
"offsets": { "from": 1000, "to": 2000 },
|
||||
"p": 0.95
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Key fields per token:
|
||||
|
||||
- `text` — word text (often has leading space for word-initial tokens)
|
||||
- `offsets.from` / `offsets.to` — milliseconds
|
||||
- `p` — confidence (0-1)
|
||||
|
||||
### Normalizing to a Word Array
|
||||
|
||||
Flatten the transcript into a simple word array for grouping:
|
||||
|
||||
```js
|
||||
const words = [];
|
||||
for (const segment of transcript.transcription) {
|
||||
for (const token of segment.tokens || []) {
|
||||
const text = token.text.trim();
|
||||
if (!text) continue;
|
||||
words.push({
|
||||
text: text,
|
||||
start: token.offsets.from / 1000,
|
||||
end: token.offsets.to / 1000,
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Word Grouping
|
||||
|
||||
Group words into caption lines. Never show more than 5 words at once. Break on:
|
||||
|
||||
1. **Sentence boundaries** — `.` `?` `!`
|
||||
2. **Pauses** — gap > 150ms between consecutive words
|
||||
3. **Max word count** — 3-5 words per group (3 for portrait, 5 for landscape)
|
||||
|
||||
```js
|
||||
function groupTranscript(words, maxWords) {
|
||||
const groups = [];
|
||||
let current = [];
|
||||
|
||||
words.forEach((word, i) => {
|
||||
current.push(word);
|
||||
|
||||
const next = words[i + 1];
|
||||
const isPause = next && next.start - word.end > 0.15;
|
||||
const isSentenceEnd = /[.?!]/.test(word.text);
|
||||
|
||||
if (current.length >= maxWords || isPause || isSentenceEnd || !next) {
|
||||
groups.push({
|
||||
text: current.map((w) => w.text).join(" "),
|
||||
start: current[0].start,
|
||||
end: current[current.length - 1].end,
|
||||
});
|
||||
current = [];
|
||||
}
|
||||
});
|
||||
return groups;
|
||||
}
|
||||
```
|
||||
|
||||
## Composition Structure
|
||||
|
||||
Caption compositions follow the standard HyperFrames composition format:
|
||||
|
||||
```html
|
||||
<template id="captions-template">
|
||||
<div data-composition-id="captions" data-width="1920" data-height="1080" data-duration="30">
|
||||
<div id="captions-container"></div>
|
||||
|
||||
<style>
|
||||
[data-composition-id="captions"] {
|
||||
width: 1920px;
|
||||
height: 1080px;
|
||||
pointer-events: none;
|
||||
}
|
||||
[data-composition-id="captions"] #captions-container {
|
||||
position: absolute;
|
||||
bottom: 100px;
|
||||
left: 50%;
|
||||
transform: translateX(-50%);
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
width: 100%;
|
||||
height: 150px;
|
||||
}
|
||||
.caption-group {
|
||||
position: absolute;
|
||||
opacity: 0;
|
||||
}
|
||||
</style>
|
||||
|
||||
<script src="https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js"></script>
|
||||
<script>
|
||||
(function () {
|
||||
// Word array from transcript — replace with actual data
|
||||
const WORDS = [
|
||||
/* ... */
|
||||
];
|
||||
|
||||
const container = document.getElementById("captions-container");
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
const groups = groupTranscript(WORDS, 5);
|
||||
|
||||
groups.forEach((group, i) => {
|
||||
const el = document.createElement("div");
|
||||
el.className = "caption-group";
|
||||
el.textContent = group.text;
|
||||
container.appendChild(el);
|
||||
|
||||
// Entrance
|
||||
tl.fromTo(
|
||||
el,
|
||||
{ opacity: 0, y: 20 },
|
||||
{ opacity: 1, y: 0, duration: 0.3, ease: "expo.out" },
|
||||
group.start,
|
||||
);
|
||||
|
||||
// Exit
|
||||
tl.to(el, { opacity: 0, y: -10, duration: 0.25, ease: "expo.out" }, group.end - 0.25);
|
||||
});
|
||||
|
||||
window.__timelines["captions"] = tl;
|
||||
})();
|
||||
</script>
|
||||
</div>
|
||||
</template>
|
||||
```
|
||||
|
||||
## Positioning
|
||||
|
||||
- **Landscape (1920x1080):** Bottom third — `bottom: 80-120px`, centered horizontally
|
||||
- **Portrait (1080x1920):** Lower middle — `bottom: 600-700px`, centered horizontally
|
||||
- Use `position: absolute` on caption groups, stacked in the same container
|
||||
- Set `pointer-events: none` on the composition so captions don't block clicks
|
||||
|
||||
## Animation Patterns
|
||||
|
||||
### Fade + slide (clean, modern)
|
||||
|
||||
```js
|
||||
tl.fromTo(el, { opacity: 0, y: 20 }, { opacity: 1, y: 0, duration: 0.4, ease: "expo.out" }, start);
|
||||
tl.to(el, { opacity: 0, y: -10, duration: 0.3, ease: "expo.out" }, end - 0.3);
|
||||
```
|
||||
|
||||
### Pop-in bounce (playful)
|
||||
|
||||
```js
|
||||
tl.to(el, { autoAlpha: 1, scale: 1.1, duration: 0.2, ease: "power2.out" }, start);
|
||||
tl.to(el, { scale: 1, duration: 0.1, ease: "power2.inOut" }, start + 0.2);
|
||||
tl.to(el, { autoAlpha: 0, scale: 0, duration: 0.15, ease: "power2.in" }, end);
|
||||
```
|
||||
|
||||
### Hard cut (no animation)
|
||||
|
||||
```js
|
||||
tl.set(el, { display: "flex" }, start);
|
||||
tl.set(el, { display: "none" }, end);
|
||||
```
|
||||
|
||||
## Timing Rules
|
||||
|
||||
- **Entrance animation must complete before exit starts.** If a group is shorter than entrance + exit duration, reduce animation durations.
|
||||
- **Exit before next group enters.** Use `Math.min(group.end, nextGroup.start)` as the hide time.
|
||||
- **Minimum display time:** 0.3s — skip groups shorter than this.
|
||||
- **No overlapping caption groups.** Only one group visible at a time.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Deterministic.** No `Math.random()`, no `Date.now()`.
|
||||
- **Register timeline.** `window.__timelines["captions"] = tl;`
|
||||
- **IIFE scope.** Wrap script in `(function() { ... })()` to avoid globals.
|
||||
- **Scoped CSS.** Prefix all selectors with `[data-composition-id="captions"]`.
|
||||
- **Set `data-duration`.** Must match or exceed the last caption's end time.
|
||||
- **Do not animate `visibility` or `display` on timed clips.** Use `opacity` or `autoAlpha`.
|
||||
- **Do not use `tl.set()` for property changes that need to survive timeline seeking.** Use `tl.to()` with a short duration instead.
|
||||
|
||||
## Loading in the Root Composition
|
||||
|
||||
```html
|
||||
<div
|
||||
id="el-captions"
|
||||
data-composition-id="captions"
|
||||
data-composition-src="compositions/captions.html"
|
||||
data-start="0"
|
||||
data-duration="30"
|
||||
data-track-index="5"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
></div>
|
||||
```
|
||||
Submodule
+1
Submodule .claude/skills/gsap added at 03d9f0c3db
@@ -0,0 +1,261 @@
|
||||
---
|
||||
name: media
|
||||
description: Video, audio, and image handling constraints for HyperFrames. Use when adding media elements to a composition or troubleshooting media playback.
|
||||
---
|
||||
|
||||
# Media
|
||||
|
||||
Constraints for working with video, audio, and images in HyperFrames compositions.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Adding video, audio, or image clips to a composition
|
||||
- Troubleshooting media that won't play or render
|
||||
- Working with transparent video, trimming, or volume control
|
||||
|
||||
## Video
|
||||
|
||||
### Required Attributes
|
||||
|
||||
Every `<video>` element must have:
|
||||
|
||||
```html
|
||||
<video
|
||||
id="el-1"
|
||||
data-start="0"
|
||||
data-duration="15"
|
||||
data-track-index="0"
|
||||
src="video.mp4"
|
||||
muted
|
||||
playsinline
|
||||
crossorigin="anonymous"
|
||||
></video>
|
||||
```
|
||||
|
||||
- **`muted`** — required. Video elements must not carry audio. Use a separate `<audio>` element.
|
||||
- **`playsinline`** — required. Prevents fullscreen on mobile.
|
||||
- **`crossorigin="anonymous"`** — required for CORS-safe rendering in headless Chrome.
|
||||
|
||||
### Supported Codecs
|
||||
|
||||
The renderer uses headless Chrome. Only web-compatible codecs work:
|
||||
|
||||
| Codec | Container | Works |
|
||||
| ---------- | --------- | --------------------- |
|
||||
| H.264 | MP4 | Yes |
|
||||
| VP8 | WebM | Yes |
|
||||
| VP9 | WebM | Yes (including alpha) |
|
||||
| AV1 | MP4/WebM | Yes |
|
||||
| HEVC/H.265 | MP4 | No |
|
||||
| ProRes | MOV | No |
|
||||
|
||||
If the source video uses an unsupported codec, transcode first:
|
||||
|
||||
```bash
|
||||
ffmpeg -i input.mov -c:v libx264 -preset fast -crf 18 -c:a aac -b:a 192k output.mp4
|
||||
```
|
||||
|
||||
### Transparent Video
|
||||
|
||||
Use WebM VP9 with alpha channel for overlays (talking heads, animated elements):
|
||||
|
||||
```html
|
||||
<video
|
||||
id="el-overlay"
|
||||
data-start="0"
|
||||
data-duration="10"
|
||||
data-track-index="2"
|
||||
src="overlay.webm"
|
||||
muted
|
||||
playsinline
|
||||
crossorigin="anonymous"
|
||||
style="z-index:50;"
|
||||
></video>
|
||||
```
|
||||
|
||||
Encode transparent video:
|
||||
|
||||
```bash
|
||||
ffmpeg -i input.mov -c:v libvpx-vp9 -pix_fmt yuva420p -b:v 2M output.webm
|
||||
```
|
||||
|
||||
### Trimming
|
||||
|
||||
Use `data-media-start` to set an in-point offset into the source file:
|
||||
|
||||
```html
|
||||
<!-- Play from 5s into the source, for 10s -->
|
||||
<video
|
||||
id="el-1"
|
||||
data-start="0"
|
||||
data-duration="10"
|
||||
data-media-start="5"
|
||||
data-track-index="0"
|
||||
src="video.mp4"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
```
|
||||
|
||||
The framework seeks the video to the `data-media-start` position and plays from there.
|
||||
|
||||
### Video Duration
|
||||
|
||||
If `data-duration` is omitted on a video element, the framework uses the source media's intrinsic duration. For predictable timing, always set `data-duration` explicitly.
|
||||
|
||||
## Audio
|
||||
|
||||
### Separate Audio Element
|
||||
|
||||
Audio must always be a separate `<audio>` element, even if the audio source is the same file as a video:
|
||||
|
||||
```html
|
||||
<!-- Video (muted) -->
|
||||
<video
|
||||
id="el-video"
|
||||
data-start="0"
|
||||
data-duration="30"
|
||||
data-track-index="0"
|
||||
src="interview.mp4"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
|
||||
<!-- Audio from same file -->
|
||||
<audio
|
||||
id="el-audio"
|
||||
data-start="0"
|
||||
data-duration="30"
|
||||
data-track-index="2"
|
||||
src="interview.mp4"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
```
|
||||
|
||||
### Volume Control
|
||||
|
||||
Use `data-volume` (0-1) to set playback volume:
|
||||
|
||||
```html
|
||||
<audio
|
||||
id="el-music"
|
||||
data-start="0"
|
||||
data-duration="60"
|
||||
data-track-index="3"
|
||||
src="bgm.mp3"
|
||||
data-volume="0.3"
|
||||
></audio>
|
||||
```
|
||||
|
||||
### Audio Trimming
|
||||
|
||||
Same as video — use `data-media-start`:
|
||||
|
||||
```html
|
||||
<audio
|
||||
id="el-sfx"
|
||||
data-start="5"
|
||||
data-duration="3"
|
||||
data-media-start="10"
|
||||
data-track-index="4"
|
||||
src="effects.mp3"
|
||||
></audio>
|
||||
```
|
||||
|
||||
## Images
|
||||
|
||||
```html
|
||||
<img
|
||||
id="el-bg"
|
||||
data-start="0"
|
||||
data-duration="30"
|
||||
data-track-index="1"
|
||||
src="background.jpg"
|
||||
style="width:1920px;height:1080px;object-fit:cover;"
|
||||
/>
|
||||
```
|
||||
|
||||
- `data-duration` is required on images (no intrinsic duration)
|
||||
- Use `object-fit: cover` for full-bleed backgrounds
|
||||
- Use `z-index` for layering with video
|
||||
|
||||
## Picture-in-Picture
|
||||
|
||||
To animate video position/size, wrap it in a non-timed div. Animate the wrapper, not the video:
|
||||
|
||||
```html
|
||||
<div
|
||||
id="pip-frame"
|
||||
style="
|
||||
position:absolute; top:0; left:0;
|
||||
width:1920px; height:1080px;
|
||||
z-index:50; overflow:hidden;
|
||||
"
|
||||
>
|
||||
<video
|
||||
id="el-video"
|
||||
data-start="0"
|
||||
data-duration="60"
|
||||
data-track-index="0"
|
||||
src="talking-head.mp4"
|
||||
muted
|
||||
playsinline
|
||||
style="width:100%;height:100%;object-fit:cover;"
|
||||
></video>
|
||||
</div>
|
||||
```
|
||||
|
||||
```js
|
||||
// Shrink to corner at 10s
|
||||
tl.to(
|
||||
"#pip-frame",
|
||||
{
|
||||
top: 700,
|
||||
left: 1360,
|
||||
width: 500,
|
||||
height: 280,
|
||||
borderRadius: 16,
|
||||
duration: 1,
|
||||
ease: "power2.inOut",
|
||||
},
|
||||
10,
|
||||
);
|
||||
```
|
||||
|
||||
**Key rules:**
|
||||
|
||||
- The wrapper div has **no** `data-start`, `data-duration`, or `data-track-index`
|
||||
- The video inside has all the data attributes
|
||||
- Animate the wrapper's CSS properties (top, left, width, height)
|
||||
- Never animate the video element's dimensions directly
|
||||
|
||||
## Framework Owns Playback
|
||||
|
||||
The framework manages all media playback. Never do any of these in scripts:
|
||||
|
||||
```js
|
||||
// ALL OF THESE ARE WRONG
|
||||
video.play();
|
||||
video.pause();
|
||||
video.currentTime = 5;
|
||||
audio.play();
|
||||
audio.pause();
|
||||
audio.volume = 0.5; // use data-volume attribute instead
|
||||
```
|
||||
|
||||
The framework:
|
||||
|
||||
- Plays/pauses media in sync with the master timeline
|
||||
- Seeks media when the timeline is scrubbed
|
||||
- Waits for media to load before resolving timing
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Video must be `muted`.** Audio comes from `<audio>` elements.
|
||||
- **Video must be `playsinline`.** Required for headless rendering.
|
||||
- **Do not nest video inside a timed div.** Video must be a direct child of the composition, or inside a non-timed wrapper.
|
||||
- **Do not animate video element dimensions.** Animate a wrapper div instead.
|
||||
- **Do not call play/pause/seek on media.** Framework owns playback.
|
||||
- **Set `data-duration` explicitly** on all clips for predictable timing.
|
||||
- **Use web-compatible codecs.** H.264, VP8, VP9, AV1 only.
|
||||
- **Separate audio from video.** Always use `<audio>` for sound.
|
||||
@@ -0,0 +1,206 @@
|
||||
---
|
||||
name: transitions
|
||||
description: Scene transition constraints and patterns for HyperFrames. Use when adding crossfades, wipes, or reveals between scenes in a composition.
|
||||
---
|
||||
|
||||
# Transitions
|
||||
|
||||
Constraints and patterns for scene-to-scene transitions in HyperFrames compositions.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Adding a crossfade, wipe, or reveal between video clips or scenes
|
||||
- User asks for "smooth transition", "crossfade", or "scene change"
|
||||
- Connecting two sequential clips visually
|
||||
|
||||
## Core Constraint: Same-Track Clips Cannot Overlap
|
||||
|
||||
Clips on the same `data-track-index` cannot overlap in time. This is enforced by the framework. If two clips need to overlap (for a crossfade), they must be on **different tracks**.
|
||||
|
||||
## Pattern: Crossfade
|
||||
|
||||
Two clips on separate tracks. The outgoing clip fades out while the incoming clip fades in during the overlap window.
|
||||
|
||||
```html
|
||||
<!-- Outgoing clip: track 0, ends at 10s -->
|
||||
<video
|
||||
id="el-1"
|
||||
data-start="0"
|
||||
data-duration="10"
|
||||
data-track-index="0"
|
||||
src="clip-a.mp4"
|
||||
muted
|
||||
playsinline
|
||||
style="z-index:1;"
|
||||
></video>
|
||||
|
||||
<!-- Incoming clip: track 1, starts at 9s (1s overlap) -->
|
||||
<video
|
||||
id="el-2"
|
||||
data-start="9"
|
||||
data-duration="15"
|
||||
data-track-index="1"
|
||||
src="clip-b.mp4"
|
||||
muted
|
||||
playsinline
|
||||
style="z-index:2;"
|
||||
></video>
|
||||
```
|
||||
|
||||
```js
|
||||
// Fade out clip A during overlap
|
||||
tl.to("#el-1", { opacity: 0, duration: 1, ease: "power2.inOut" }, 9);
|
||||
// Fade in clip B during overlap
|
||||
tl.fromTo("#el-2", { opacity: 0 }, { opacity: 1, duration: 1, ease: "power2.inOut" }, 9);
|
||||
```
|
||||
|
||||
**Key rules:**
|
||||
|
||||
- The overlap window is `incoming.start` to `outgoing.end`
|
||||
- Both clips must be on different tracks
|
||||
- Use `z-index` to control which clip renders on top during the transition
|
||||
- Animate `opacity` — do not animate `visibility` or `display`
|
||||
|
||||
## Pattern: Wipe / Reveal
|
||||
|
||||
An overlay composition that covers the frame, hides the cut, then reveals the new content.
|
||||
|
||||
```html
|
||||
<!-- Clip A and Clip B are sequential on track 0 -->
|
||||
<video
|
||||
id="el-1"
|
||||
data-start="0"
|
||||
data-duration="10"
|
||||
data-track-index="0"
|
||||
src="clip-a.mp4"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
<video
|
||||
id="el-2"
|
||||
data-start="10"
|
||||
data-duration="15"
|
||||
data-track-index="0"
|
||||
src="clip-b.mp4"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
|
||||
<!-- Wipe overlay on track 3, spans the cut point -->
|
||||
<div
|
||||
id="wipe-comp"
|
||||
data-composition-id="wipe"
|
||||
data-composition-src="compositions/wipe.html"
|
||||
data-start="9"
|
||||
data-duration="2"
|
||||
data-track-index="3"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
style="z-index:100;"
|
||||
></div>
|
||||
```
|
||||
|
||||
Wipe composition (`compositions/wipe.html`):
|
||||
|
||||
```html
|
||||
<template id="wipe-template">
|
||||
<div data-composition-id="wipe" data-width="1920" data-height="1080">
|
||||
<div
|
||||
id="wipe-bar"
|
||||
style="
|
||||
position:absolute; top:0; left:0;
|
||||
width:1920px; height:1080px;
|
||||
background:#111; transform:translateX(-100%);
|
||||
"
|
||||
></div>
|
||||
|
||||
<script src="https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js"></script>
|
||||
<script>
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
// Sweep in from left, covering the frame
|
||||
tl.to("#wipe-bar", { x: 0, duration: 0.5, ease: "power3.inOut" }, 0.5);
|
||||
// Hold for a beat while the cut happens underneath
|
||||
// Sweep out to the right, revealing new content
|
||||
tl.to("#wipe-bar", { x: 1920, duration: 0.5, ease: "power3.inOut" }, 1.2);
|
||||
window.__timelines["wipe"] = tl;
|
||||
</script>
|
||||
</div>
|
||||
</template>
|
||||
```
|
||||
|
||||
**Key rules:**
|
||||
|
||||
- The wipe composition spans the cut point (starts before the cut, ends after)
|
||||
- The wipe must have a high `z-index` to render above both clips
|
||||
- Clips can be on the same track (no overlap) since the wipe covers the hard cut
|
||||
- Time the wipe so the frame is fully covered at the exact cut point
|
||||
|
||||
## Pattern: Scanline Sweep
|
||||
|
||||
A partial-frame element sweeps across, creating a stylized transition.
|
||||
|
||||
```js
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
|
||||
// Scanline starts above frame, sweeps down
|
||||
tl.fromTo("#scanline", { y: -320 }, { y: 1920, duration: 1.5, ease: "power3.inOut" }, 0);
|
||||
```
|
||||
|
||||
Style the scanline with gradients, glow, or accent colors for visual interest.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Transitions are compositions.** Wrap transition elements in a composition with `data-composition-id`. Do not use bare divs without data attributes.
|
||||
- **Do not animate video element dimensions.** If you need to scale video during a transition, animate a wrapper div.
|
||||
- **Use GSAP, not CSS transitions.** CSS transitions are not deterministic in the renderer.
|
||||
- **Deterministic.** No `Math.random()`, no `Date.now()`.
|
||||
- **Transitions must not affect media playback.** Never call `play()`, `pause()`, or set `currentTime`. The framework owns media.
|
||||
- **Keep transitions short.** 0.3s–1.5s. Longer transitions feel sluggish in rendered video.
|
||||
|
||||
## Timing with Relative References
|
||||
|
||||
Use relative `data-start` to keep transitions aligned when clip durations change:
|
||||
|
||||
```html
|
||||
<video
|
||||
id="intro"
|
||||
data-start="0"
|
||||
data-duration="10"
|
||||
data-track-index="0"
|
||||
src="..."
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
|
||||
<!-- Transition starts 1s before intro ends -->
|
||||
<div
|
||||
id="wipe-comp"
|
||||
data-composition-id="wipe"
|
||||
data-composition-src="compositions/wipe.html"
|
||||
data-start="intro - 1"
|
||||
data-duration="2"
|
||||
data-track-index="3"
|
||||
data-width="1920"
|
||||
data-height="1080"
|
||||
style="z-index:100;"
|
||||
></div>
|
||||
|
||||
<video
|
||||
id="main"
|
||||
data-start="intro"
|
||||
data-duration="20"
|
||||
data-track-index="0"
|
||||
src="..."
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
```
|
||||
|
||||
If `intro`'s duration changes, the wipe and `main` shift automatically.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- **Overlapping clips on the same track.** Framework will reject this. Use different tracks for crossfades.
|
||||
- **CSS `transition` property.** Not deterministic in headless rendering. Use GSAP.
|
||||
- **Animating `display` property.** Use `opacity` instead.
|
||||
- **Transitions longer than 2s.** Feels slow in video.
|
||||
@@ -0,0 +1,118 @@
|
||||
# Skills & Transcription Design
|
||||
|
||||
## Overview
|
||||
|
||||
Three new HyperFrames skills (captions, transitions, media) and a CLI enhancement to add local transcription to `hyperframes init`.
|
||||
|
||||
---
|
||||
|
||||
## Part 1: CLI — Transcription in `hyperframes init`
|
||||
|
||||
### User Flow
|
||||
|
||||
```
|
||||
npx hyperframes init
|
||||
→ "Project name" → my-video
|
||||
→ "Got a video?" → Yes → /path/to/video.mp4
|
||||
→ (probe video: 1920x1080, 45.2s, has audio)
|
||||
→ "Generate captions from audio?" → Yes
|
||||
→ (extract audio → run whisper → save transcript)
|
||||
→ "Pick a template" → warm-grain
|
||||
→ (scaffold project with transcript.json + captions composition)
|
||||
→ "Open in studio" / "Render" / "Done"
|
||||
```
|
||||
|
||||
### Whisper Binary Management
|
||||
|
||||
Follow the same pattern as the Chrome browser manager (`packages/cli/src/browser/manager.ts`):
|
||||
|
||||
- **Cache location:** `~/.cache/hyperframes/whisper/`
|
||||
- **Binary:** whisper.cpp pre-built binary (download from GitHub releases)
|
||||
- **Model:** `ggml-base.en.bin` by default (~148MB), downloaded from Hugging Face on first use
|
||||
- **Resolution priority:** `--whisper-path` flag → `HYPERFRAMES_WHISPER_PATH` env var → cached binary → download
|
||||
|
||||
Structure:
|
||||
```
|
||||
~/.cache/hyperframes/whisper/
|
||||
├── bin/
|
||||
│ └── whisper # whisper.cpp binary
|
||||
└── models/
|
||||
└── ggml-base.en.bin # default model
|
||||
```
|
||||
|
||||
### Transcription Pipeline
|
||||
|
||||
1. **Extract audio:** `ffmpeg -i video.mp4 -vn -ar 16000 -ac 1 -f wav /tmp/audio.wav`
|
||||
2. **Run whisper:** `whisper --model ~/.cache/hyperframes/whisper/models/ggml-base.en.bin --output-json --dtw base.en /tmp/audio.wav`
|
||||
3. **Parse output:** whisper.cpp JSON output → normalize to `transcript.json`
|
||||
4. **Save:** `transcript.json` in project root
|
||||
|
||||
### transcript.json Format
|
||||
|
||||
Use whisper.cpp's native JSON output format directly. The captions skill documents whatever format whisper.cpp outputs — no normalization layer.
|
||||
|
||||
### Files to Change
|
||||
|
||||
- `packages/cli/src/whisper/manager.ts` — new module: download, cache, resolve whisper binary + model (mirrors browser/manager.ts)
|
||||
- `packages/cli/src/whisper/transcribe.ts` — new module: extract audio, run whisper, return parsed result
|
||||
- `packages/cli/src/commands/init.ts` — add "Generate captions?" prompt after video probe, call transcribe, save transcript.json, pass to scaffolding
|
||||
- `packages/cli/src/commands/doctor.ts` — add whisper check
|
||||
|
||||
---
|
||||
|
||||
## Part 2: Skills
|
||||
|
||||
### Skill: `captions`
|
||||
|
||||
**File:** `.claude/skills/captions/SKILL.md`
|
||||
|
||||
**Purpose:** Given a whisper transcript file in the project, build or modify HyperFrames caption compositions.
|
||||
|
||||
**Assumes:** `transcript.json` exists (output from `hyperframes init` or whisper.cpp directly).
|
||||
|
||||
**Content covers:**
|
||||
- Whisper.cpp JSON structure reference (segments, tokens, timestamps)
|
||||
- Word grouping strategies: by sentence breaks, pause gaps (>150ms), max word count (3-5)
|
||||
- Composition structure: `<template>` wrapper, `data-composition-id="captions"`, scoped CSS
|
||||
- GSAP patterns for caption entrance/exit (opacity, y offset, scale)
|
||||
- Positioning: bottom third centered, safe margins
|
||||
- Constraints: deterministic, no `Math.random()`, register in `window.__timelines`
|
||||
- Anti-patterns: don't animate `display`, don't overlap caption groups on same track
|
||||
|
||||
### Skill: `transitions`
|
||||
|
||||
**File:** `.claude/skills/transitions/SKILL.md`
|
||||
|
||||
**Purpose:** Constraints and patterns for scene-to-scene transitions in HyperFrames.
|
||||
|
||||
**Content covers:**
|
||||
- Track overlap rules: same track clips cannot overlap, use different tracks for crossfades
|
||||
- Crossfade pattern: two clips on separate tracks, opacity tween on outgoing clip
|
||||
- Wipe/reveal patterns: overlay div with GSAP-animated position
|
||||
- Timing: transition composition overlaps scene boundaries via `data-start` offset
|
||||
- Constraints: transitions are compositions (need `data-composition-id`), not bare divs
|
||||
- Anti-patterns: don't animate video element dimensions, don't use CSS transitions (use GSAP)
|
||||
|
||||
### Skill: `media`
|
||||
|
||||
**File:** `.claude/skills/media/SKILL.md`
|
||||
|
||||
**Purpose:** Constraints for working with video, audio, and images in HyperFrames.
|
||||
|
||||
**Content covers:**
|
||||
- Video: must be `muted` + `playsinline`, audio separate, `crossorigin="anonymous"` for CORS
|
||||
- Audio: separate `<audio>` element, `data-volume` for level control
|
||||
- Trimming: `data-media-start` for in-point offset
|
||||
- Codec requirements: H.264/VP8/VP9/AV1 for video, AAC/MP3/Opus for audio
|
||||
- Transparent video: WebM VP9 with alpha channel
|
||||
- Image: `<img>` with `data-start`/`data-duration`/`data-track-index`
|
||||
- Framework owns playback: never call `play()`, `pause()`, set `currentTime`
|
||||
- PiP pattern: animate wrapper div, not the video element
|
||||
- Anti-patterns: don't nest video in timed div, don't animate video dimensions
|
||||
|
||||
---
|
||||
|
||||
## Prioritization
|
||||
|
||||
1. **Skills first** — captions, transitions, media (can ship immediately, no code changes)
|
||||
2. **CLI transcription** — whisper binary manager + init flow (requires engineering)
|
||||
Reference in New Issue
Block a user