mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-10-30 09:16:55 +00:00
feat: keep a video's sound on the clip by default (1/8) (#4813)
* feat(cli): one owner for init media placeholders, audio stays on the video Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cli): init keeps a video's sound on the clip; silent files scaffold muted Supersedes the patchVideoSrc changes in #3977 and #3942 and carries their tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: miga-heygen <miga-heygen@users.noreply.github.com> Co-Authored-By: kapelame <kapelame@users.noreply.github.com> * feat(registry): example templates keep the a-roll's sound on the video Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lint): video audio rules recommend keeping sound on the video Update video_missing_muted and video_audio_double_source message and fixHint text to recommend the new default: keep audio on the video with data-has-audio="true" (not muted). Tests verify the new wording is in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(parsers): a video is either audible (data-has-audio) or muted, never both Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(studio): follow the compiler's audio rule in the audio-track toggle Confirm before background removal drops a video's sound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): hyperframes-core teaches sound on the video by default Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): workflows and scaffold guidance keep a video's sound on the clip Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: footage with sound keeps it on the video clip Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cli): init from-file a-roll is timed, muted strip is attribute-exact The from-file a-roll video gets data-start="0" so a scaffold lints clean, and the orphaned audio-only slot comment is dropped. setVideoAudio now walks the tag attribute by attribute, so a class value containing the word muted survives. The ffprobe-missing warning says the video is assumed to have sound and how to switch to muted, and DEFAULT_META notes the failure surfaces at render until lint support lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(studio): keep has-audio in step with the muted toggle, test cutout confirm Muting a video now clears has-audio and unmuting sets it to true, in both the Design and flat media panels, matching the Has audio track toggle. Audio elements are unchanged. Adds coverage for both panels plus the cutout window.confirm guard: declined leaves src alone, accepted swaps src and mutes, a silent video never prompts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(parsers): updateElementInHtml hasAudio only applies to videos, true or false An audio element or an undefined hasAudio no longer picks up a muted attribute; only an explicit false on a video writes muted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: video_missing_muted rule text, silent-footage notes, consistent audible examples lint.mdx and cli.mdx describe the current video_missing_muted rule instead of the retired unmuted-video wording. The editing recipes say how to adapt for silent footage and require the outgoing clip's sound to end at a J-cut. The Remotion media example shows the audible mapping consistently, data-attributes states that has-audio is required unless muted, and the hairline frame note no longer claims the video is muted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cli): encode init media filenames and probe --audio duration Function replacers stop dollar-sign pattern expansion in filenames; src is percent-encoded. --audio now passes the probed track length as the composition duration in both init paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lint): audio-state-aware media rules and time-aware double-source check video_missing_muted honors any explicit data-has-audio. video_audio_double_source only considers audible videos and only fires when the same-src clips overlap in time. duplicate_audio_track now covers audible videos. audio_file_without_element no longer claims silence when an audible video carries the sound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): audio-stream check for remotion ports, followable J-cut recipe Remotion Video maps to data-has-audio only when the file has an audio stream. The J/L-cut recipe now shows both shots with muted pictures and separate audio. embedded-captions removes the audio.mp3 scratch file it extracts. skills-manifest.json regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(lint): trim comments to satisfy the comment ratchet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(studio): atomic muted edits, cutout keeps sound on a linked audio Muted and Has audio track commit one batched write (one persist, one undo entry) and remove data-has-audio instead of writing an empty marker. Unmuting a video probes its src: audible -> true, silent -> false, probe failed -> absent. Remove background on an audible video inserts a linked sibling <audio> with the original sound in the same write, replacing the drop-the-sound confirm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(producer): skip hidden videos when auto-adding video audio A video hidden in Studio is dropped by the mixer, so its src never reached existingAudioSrcs and the render re-added it at full volume. The video parser now records the shared hidden predicate and the auto-detect step skips hidden videos. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(studio): keep useDomEditSession under the 600-line cap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: miga-heygen <miga-heygen@users.noreply.github.com> Co-authored-by: kapelame <kapelame@users.noreply.github.com>
This commit is contained in:
Vance Ingalls
co-authored by
Claude Opus 5.5
miga-heygen
kapelame
Claude Opus 5.5
miga-heygen
kapelame
parent
7a60b97bf8
commit
37f30b1d06
@@ -383,8 +383,9 @@ The skeleton handles most structural rules. These are the runtime rules the skel
|
||||
| Never | Use instead |
|
||||
| ------------------------------- | --------------------------- |
|
||||
| `video.play()`, `audio.play()` | Framework owns playback |
|
||||
| `<video>` without `muted` | Always `muted playsinline` |
|
||||
| Audio on `<video>` | Separate `<audio>` element |
|
||||
| `<video>` with sound | `data-has-audio="true"`, no `muted` |
|
||||
| Music, voiceover, or replacement audio | separate `<audio>` element |
|
||||
| Silent footage and b-roll | `muted` |
|
||||
| Base64 media | File reference or HTTPS URL |
|
||||
| Placeholder URLs (placehold.co) | Real assets |
|
||||
|
||||
|
||||
@@ -375,7 +375,9 @@ The cloud renderer seeks the timeline frame-by-frame. Non-deterministic or self-
|
||||
| `stagger: { from: "random" }` | `from: "start"`, `"center"`, or `"end"` |
|
||||
| async timeline construction | build synchronously at page load |
|
||||
| `video.play()` / `audio.play()` | the framework owns playback |
|
||||
| `<video>` without `muted playsinline` | always `muted playsinline`; audio on a separate `<audio>` |
|
||||
| `<video>` with sound | `data-has-audio="true"`, no `muted` |
|
||||
| Music, voiceover, or replacement audio | separate `<audio>` |
|
||||
| Silent footage and b-roll | `muted` |
|
||||
| exit tweens before a shader | the shader IS the exit — content stays visible |
|
||||
| `tl.set`/`tl.to` on an anchor container (beyond the required first-anchor `opacity:1` set) | HyperShader owns anchor opacity; only `tl.set(firstAnchor, { opacity: 1 }, start)` is allowed |
|
||||
| `autoAlpha` on an anchor scene | anchors use `opacity` (HyperShader-managed); `autoAlpha` is for non-anchor scenes |
|
||||
|
||||
@@ -641,7 +641,7 @@ npx hyperframes lint [dir] --json
|
||||
◆ Linting my-project/index.html
|
||||
|
||||
✗ missing_gsap_script: Composition uses GSAP but no GSAP script is loaded.
|
||||
⚠ unmuted-video [clip-1]: Video should have the 'muted' attribute for reliable autoplay.
|
||||
⚠ video_missing_muted [clip-1]: <video id="clip-1"> has data-start but declares neither muted nor data-has-audio. If the file has sound, add data-has-audio="true" (the sound stays on this clip). If it is silent, add muted.
|
||||
|
||||
◇ 1 error(s), 1 warning(s)
|
||||
```
|
||||
|
||||
@@ -103,7 +103,7 @@ The browser entry exposes `lintHyperframeHtml`, `lintMediaUrls`, and `shouldBloc
|
||||
Detected issues include:
|
||||
|
||||
- Missing timeline registration (`window.__timelines`)
|
||||
- Unmuted video elements (causes autoplay failures)
|
||||
- A timed `<video>` that declares neither `muted` (silent footage) nor `data-has-audio="true"` (has sound) — `video_missing_muted`
|
||||
- Missing `class="clip"` on timed visible elements
|
||||
- Deprecated attribute names
|
||||
- Missing composition dimensions (`data-width`, `data-height`)
|
||||
|
||||
@@ -95,7 +95,8 @@ The skeleton above is how you ask. This is what the agent writes, and it's worth
|
||||
| `class="clip"` | Marks a timed section. The framework owns its visibility — never animate a clip's own opacity or `display` to hide it. |
|
||||
| `data-start` / `data-duration` / `data-track-index` | When the clip begins, how long it runs, and which track it sits on. Two clips on the same track index must not overlap in time. |
|
||||
| `window.__timelines["<id>"]` | Every GSAP-driven composition registers exactly one paused timeline here, built synchronously at load. The renderer seeks that timeline frame by frame — which is why nothing may depend on wall-clock time or unseeded randomness. Other adapters, including Lottie, CSS animations, and WAAPI, register through their own seekable runtime contracts. |
|
||||
| `<video muted>` + a separate `<audio>` | Video elements render muted and their sound rides a sibling audio element, so the mixer can duck and balance it independently. |
|
||||
| Footage with sound | `<video playsinline data-has-audio="true">` — the sound stays on the clip |
|
||||
| Music / voiceover | separate `<audio>` |
|
||||
|
||||
You rarely write these by hand, but naming them in a prompt is legitimate and sometimes necessary — "put the captions on their own track index," "keep it one composition file, one timeline" — and it's the vocabulary every error message uses. The full contract lives in the `/hyperframes-core` skill; the [appendix](/prompting/rules-and-anti-patterns) carries the rules that most often bite.
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ Almost every rule below is taught in context somewhere earlier in this guide; a
|
||||
The skills enforce these automatically, but if you hand-edit compositions or debug issues, these are the rules that matter:
|
||||
|
||||
1. **Register all timelines** on `window.__timelines` — the renderer can't seek animations it doesn't know about.
|
||||
2. **Video elements must be `muted`** — audio goes in separate `<audio>` elements so the renderer can mix it.
|
||||
2. **A video with sound keeps it on the `<video>`** — `data-has-audio="true"`, no `muted`. Use a separate `<audio>` for music, voiceover, replacement audio, J/L cuts, or audio detached in Studio. Silent footage and b-roll: `muted`.
|
||||
3. **No `Math.random()`** — random values produce different frames on each render, breaking determinism. Use a seeded PRNG (e.g. mulberry32) if you need pseudo-random values. And when you want a stepped, stop-motion hold, quantize it on the **integer frame index**, not on elapsed seconds — seek times don't land on exact 1/fps doubles, so second-domain arithmetic drifts and the hold length stutters. See [Handmade, still deterministic](/prompting/motion#handmade-still-deterministic).
|
||||
4. **Synchronous timeline construction** — no `async`/`await` or `fetch()` during GSAP timeline setup.
|
||||
5. **Timed elements need `class="clip"`** — plus `data-start`, `data-duration`, and `data-track-index`.
|
||||
|
||||
@@ -133,8 +133,7 @@ Video and audio may omit `data-duration` when their intrinsic duration is known
|
||||
and the whole remaining source should play.
|
||||
|
||||
Do not imperatively call `play()`, `pause()`, or set `currentTime`.
|
||||
HyperFrames owns media playback and seeking. A video should be `muted` unless it
|
||||
is intentionally audible and declares `data-has-audio="true"`.
|
||||
HyperFrames owns media playback and seeking. A `<video>` with sound declares `data-has-audio="true"` and is not muted. Silent footage is `muted`. Every timed `<video>` must say one or the other (lint `video_missing_muted`).
|
||||
|
||||
Media can live inside nested composition markup. Keep element IDs unique across
|
||||
the assembled project because render-time frame injection targets those IDs.
|
||||
|
||||
Binary file not shown.
@@ -220,14 +220,63 @@ describe("hyperframes init flag rename", () => {
|
||||
const res = runInit([target, "--non-interactive", "--skip-transcribe", "--video", clip]);
|
||||
expect(res.status).toBe(0);
|
||||
const html = readFileSync(join(target, "index.html"), "utf-8");
|
||||
expect(html).toContain('id="a-roll"');
|
||||
expect(html).toContain('src="clip.mp4"');
|
||||
const video = html.match(/<video\b[^>]*>/)?.[0] ?? "";
|
||||
expect(video).toContain('id="a-roll"');
|
||||
expect(video).toContain('src="clip.mp4"');
|
||||
expect(video).toContain('data-has-audio="true"');
|
||||
expect(video).not.toMatch(/\bmuted\b/);
|
||||
expect(html).not.toMatch(/<audio\b/);
|
||||
expect(existsSync(join(target, "clip.mp4"))).toBe(true);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("scaffolds a silent --video as a muted clip with no audio element", () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-init-silent-"));
|
||||
const target = join(dir, "proj");
|
||||
const clip = join(dir, "silent.mp4");
|
||||
copyFileSync(resolve(fileURLToPath(import.meta.url), "../__fixtures__/silent.mp4"), clip);
|
||||
try {
|
||||
const res = runInit([target, "--non-interactive", "--skip-transcribe", "--video", clip]);
|
||||
expect(res.status).toBe(0);
|
||||
const html = readFileSync(join(target, "index.html"), "utf-8");
|
||||
const video = html.match(/<video\b[^>]*>/)?.[0] ?? "";
|
||||
expect(video).toMatch(/\bmuted\b/);
|
||||
expect(video).not.toContain("data-has-audio");
|
||||
expect(html).not.toMatch(/<audio\b/);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("wires --audio into the composition without creating a video clip", () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-init-audio-test-"));
|
||||
const target = join(dir, "proj");
|
||||
const audio = join(dir, "track.wav");
|
||||
copyFileSync(
|
||||
resolve(
|
||||
fileURLToPath(import.meta.url),
|
||||
"../../../../producer/tests/audio-mux-parity/src/assets/tone.wav",
|
||||
),
|
||||
audio,
|
||||
);
|
||||
try {
|
||||
const res = runInit([target, "--non-interactive", "--skip-transcribe", "--audio", audio]);
|
||||
expect(res.status).toBe(0);
|
||||
const html = readFileSync(join(target, "index.html"), "utf-8");
|
||||
expect(html).toMatch(/<audio\b[^>]*src="track\.wav"/);
|
||||
expect(html).not.toMatch(/<video\b/);
|
||||
// tone.wav is 3 s; the root and the <audio> both carry the probed length, not the 10 s default.
|
||||
expect(html).toMatch(/<audio\b[^>]*data-duration="3"/);
|
||||
expect(html).toMatch(/<div[^>]*id="root"[^>]*data-duration="3"/);
|
||||
expect(html).not.toContain("__VIDEO_SRC__");
|
||||
expect(html).not.toContain("__AUDIO_SRC__");
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("-v works as the short alias for --video", () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-init-test-"));
|
||||
const target = join(dir, "proj");
|
||||
|
||||
@@ -7,6 +7,7 @@ import { failCommand, finishCommand } from "../utils/commandResult.js";
|
||||
import { writeNewFileSync } from "../utils/writeNewFile.js";
|
||||
import { defineCommand, runCommand } from "citty";
|
||||
import type { Example } from "./_examples.js";
|
||||
import { patchMediaPlaceholders, type InitMediaOptions } from "./initMedia.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
["Create a project with the interactive wizard", "hyperframes init my-video"],
|
||||
@@ -82,7 +83,8 @@ const DEFAULT_META: VideoMeta = {
|
||||
width: 1920,
|
||||
height: 1080,
|
||||
fps: 30,
|
||||
hasAudio: false,
|
||||
// ffprobe unavailable: assume sound. A wrong guess fails loudly at RENDER (lint catches it only once #3638 lands); assuming silence would mute real audio.
|
||||
hasAudio: true,
|
||||
videoCodec: "h264",
|
||||
};
|
||||
|
||||
@@ -348,31 +350,12 @@ function writeTailwindSupport(destDir: string): void {
|
||||
}
|
||||
}
|
||||
|
||||
function patchVideoSrc(
|
||||
dir: string,
|
||||
videoFilename: string | undefined,
|
||||
durationSeconds?: number,
|
||||
): void {
|
||||
function patchVideoSrc(dir: string, media: InitMediaOptions): void {
|
||||
const htmlFiles = readdirSync(dir, { withFileTypes: true, recursive: true })
|
||||
.filter((e) => e.isFile() && e.name.endsWith(".html"))
|
||||
.map((e) => join(e.parentPath, e.name));
|
||||
|
||||
for (const file of htmlFiles) {
|
||||
let content = readFileSync(file, "utf-8");
|
||||
if (videoFilename) {
|
||||
content = content.replaceAll("__VIDEO_SRC__", videoFilename);
|
||||
} else {
|
||||
// Remove video elements with placeholder src
|
||||
content = content.replace(/<video[^>]*src="__VIDEO_SRC__"[^>]*>[\s\S]*?<\/video>/g, "");
|
||||
content = content.replace(/<video[^>]*src="__VIDEO_SRC__"[^>]*>/g, "");
|
||||
// Remove audio elements with placeholder src
|
||||
content = content.replace(/<audio[^>]*src="__VIDEO_SRC__"[^>]*>[\s\S]*?<\/audio>/g, "");
|
||||
content = content.replace(/<audio[^>]*src="__VIDEO_SRC__"[^>]*>/g, "");
|
||||
}
|
||||
// Patch duration — use probed duration or default
|
||||
const dur = durationSeconds ? String(Math.round(durationSeconds * 100) / 100) : "10";
|
||||
content = content.replaceAll("__VIDEO_DURATION__", dur);
|
||||
writeFileSync(file, content, "utf-8");
|
||||
writeFileSync(file, patchMediaPlaceholders(readFileSync(file, "utf-8"), media), "utf-8");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -404,7 +387,7 @@ async function handleVideoFile(
|
||||
);
|
||||
}
|
||||
} else {
|
||||
const msg = `ffprobe not found — using defaults (1920x1080, 5s, 30fps). Install: ${getFFmpegInstallHint()}`;
|
||||
const msg = `ffprobe not found — using defaults (1920x1080, 5s, 30fps) and assuming the video has sound (data-has-audio="true"); if it is silent, replace that with muted. Install: ${getFFmpegInstallHint()}`;
|
||||
if (interactive) {
|
||||
clack.log.warn(msg);
|
||||
} else {
|
||||
@@ -559,8 +542,7 @@ async function scaffoldProject(
|
||||
destDir: string,
|
||||
name: string,
|
||||
templateId: string,
|
||||
localVideoName: string | undefined,
|
||||
durationSeconds?: number,
|
||||
media: InitMediaOptions,
|
||||
tailwind = false,
|
||||
resolution?: CanvasResolution,
|
||||
authoringSkill?: string,
|
||||
@@ -576,7 +558,7 @@ async function scaffoldProject(
|
||||
} else {
|
||||
await fetchRemoteTemplate(templateId, destDir);
|
||||
}
|
||||
patchVideoSrc(destDir, localVideoName, durationSeconds);
|
||||
patchVideoSrc(destDir, media);
|
||||
if (tailwind) writeTailwindSupport(destDir);
|
||||
if (resolution) applyResolutionPreset(destDir, resolution);
|
||||
|
||||
@@ -863,6 +845,7 @@ export default defineCommand({
|
||||
mkdirSync(destDir, { recursive: true });
|
||||
|
||||
let localVideoName: string | undefined;
|
||||
let videoHasAudio = true;
|
||||
let videoDuration: number | undefined;
|
||||
let sourceFilePath: string | undefined;
|
||||
|
||||
@@ -871,6 +854,7 @@ export default defineCommand({
|
||||
sourceFilePath = videoPath;
|
||||
const result = await handleVideoFile(videoPath, destDir, false);
|
||||
localVideoName = result.localVideoName;
|
||||
videoHasAudio = result.meta.hasAudio;
|
||||
videoDuration = result.meta.durationSeconds;
|
||||
console.log(
|
||||
`Video: ${result.meta.width}x${result.meta.height}, ${result.meta.durationSeconds.toFixed(1)}s`,
|
||||
@@ -881,6 +865,8 @@ export default defineCommand({
|
||||
if (audioPath) {
|
||||
sourceFilePath = audioPath;
|
||||
copyFileSync(audioPath, resolve(destDir, basename(audioPath)));
|
||||
const { getMediaDurationSeconds } = await import("../whisper/transcribe.js");
|
||||
videoDuration = getMediaDurationSeconds(audioPath) ?? undefined;
|
||||
console.log(`Audio: ${basename(audioPath)}`);
|
||||
}
|
||||
|
||||
@@ -907,12 +893,16 @@ export default defineCommand({
|
||||
|
||||
// Scaffold
|
||||
try {
|
||||
const media: InitMediaOptions = {
|
||||
video: localVideoName ? { filename: localVideoName, hasAudio: videoHasAudio } : undefined,
|
||||
audio: audioPath ? { filename: basename(audioPath) } : undefined,
|
||||
durationSeconds: videoDuration,
|
||||
};
|
||||
await scaffoldProject(
|
||||
destDir,
|
||||
basename(destDir),
|
||||
templateId,
|
||||
localVideoName,
|
||||
videoDuration,
|
||||
media,
|
||||
tailwind,
|
||||
resolutionPreset,
|
||||
args.skill,
|
||||
@@ -1018,6 +1008,7 @@ export default defineCommand({
|
||||
|
||||
// 2. Video/audio file handling (only via --video/--audio flags, no interactive prompt)
|
||||
let localVideoName: string | undefined;
|
||||
let videoHasAudio = true;
|
||||
let sourceFilePath: string | undefined;
|
||||
let videoDuration: number | undefined;
|
||||
|
||||
@@ -1032,6 +1023,7 @@ export default defineCommand({
|
||||
sourceFilePath = videoPath;
|
||||
const result = await handleVideoFile(videoPath, destDir, true);
|
||||
localVideoName = result.localVideoName;
|
||||
videoHasAudio = result.meta.hasAudio;
|
||||
videoDuration = result.meta.durationSeconds;
|
||||
} else if (audioFlag) {
|
||||
const audioPath = resolve(audioFlag);
|
||||
@@ -1043,6 +1035,8 @@ export default defineCommand({
|
||||
mkdirSync(destDir, { recursive: true });
|
||||
sourceFilePath = audioPath;
|
||||
copyFileSync(audioPath, resolve(destDir, basename(audioPath)));
|
||||
const { getMediaDurationSeconds } = await import("../whisper/transcribe.js");
|
||||
videoDuration = getMediaDurationSeconds(audioPath) ?? undefined;
|
||||
clack.log.info(`Audio copied to ${c.accent(basename(audioPath))}`);
|
||||
}
|
||||
|
||||
@@ -1124,12 +1118,16 @@ export default defineCommand({
|
||||
spin.start(`Downloading example ${c.accent(templateId)}...`);
|
||||
}
|
||||
try {
|
||||
const media: InitMediaOptions = {
|
||||
video: localVideoName ? { filename: localVideoName, hasAudio: videoHasAudio } : undefined,
|
||||
audio: audioFlag ? { filename: basename(audioFlag) } : undefined,
|
||||
durationSeconds: videoDuration,
|
||||
};
|
||||
await scaffoldProject(
|
||||
destDir,
|
||||
name,
|
||||
templateId,
|
||||
localVideoName,
|
||||
videoDuration,
|
||||
media,
|
||||
tailwind,
|
||||
resolutionPreset,
|
||||
args.skill,
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { patchMediaPlaceholders } from "./initMedia.js";
|
||||
import { readdirSync, readFileSync, existsSync } from "node:fs";
|
||||
import { join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const NEW_TEMPLATE = `<div id="root">
|
||||
<video
|
||||
id="a-roll"
|
||||
class="clip"
|
||||
src="__VIDEO_SRC__"
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="0"
|
||||
></video>
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="__AUDIO_SRC__"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="2"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
</div>`;
|
||||
|
||||
const LEGACY_TEMPLATE = `<div id="root">
|
||||
<video id="a-roll" class="clip" src="__VIDEO_SRC__" muted playsinline data-duration="__VIDEO_DURATION__"></video>
|
||||
<audio id="a-roll-audio" src="__VIDEO_SRC__" data-start="0" data-duration="__VIDEO_DURATION__"></audio>
|
||||
</div>`;
|
||||
|
||||
const videoTag = (html: string) => html.match(/<video\b[^>]*>/)?.[0] ?? "";
|
||||
|
||||
describe("patchMediaPlaceholders — new template", () => {
|
||||
it("video with audio: one audible <video>, no <audio>", () => {
|
||||
const out = patchMediaPlaceholders(NEW_TEMPLATE, {
|
||||
video: { filename: "talk.mp4", hasAudio: true },
|
||||
durationSeconds: 6,
|
||||
});
|
||||
expect(videoTag(out)).toContain('data-has-audio="true"');
|
||||
expect(videoTag(out)).not.toMatch(/\bmuted\b/);
|
||||
expect(videoTag(out)).toContain('src="talk.mp4"');
|
||||
expect(out).not.toMatch(/<audio\b/);
|
||||
expect(out).toContain('data-duration="6"');
|
||||
expect(out).not.toContain("__");
|
||||
});
|
||||
|
||||
it("silent video: muted <video>, no data-has-audio, no <audio>", () => {
|
||||
const out = patchMediaPlaceholders(NEW_TEMPLATE, {
|
||||
video: { filename: "silent.mp4", hasAudio: false },
|
||||
durationSeconds: 3,
|
||||
});
|
||||
expect(videoTag(out)).toMatch(/\bmuted\b/);
|
||||
expect(videoTag(out)).not.toContain("data-has-audio");
|
||||
expect(out).not.toMatch(/<audio\b/);
|
||||
});
|
||||
|
||||
it("keeps every other attribute and the closing > of a multi-line tag", () => {
|
||||
const out = patchMediaPlaceholders(NEW_TEMPLATE, {
|
||||
video: { filename: "silent.mp4", hasAudio: false },
|
||||
});
|
||||
const tag = videoTag(out);
|
||||
for (const attr of ['id="a-roll"', 'class="clip"', "playsinline", 'data-track-index="0"']) {
|
||||
expect(tag).toContain(attr);
|
||||
}
|
||||
expect(out).toMatch(/<video\b[^>]*>\s*<\/video>/);
|
||||
});
|
||||
|
||||
it("audio only: one <audio> on the file, no <video>", () => {
|
||||
const out = patchMediaPlaceholders(NEW_TEMPLATE, { audio: { filename: "track.wav" } });
|
||||
expect(out).toMatch(/<audio\b[^>]*src="track\.wav"/);
|
||||
expect(out).not.toMatch(/<video\b/);
|
||||
expect(out).not.toContain("__");
|
||||
});
|
||||
|
||||
it("no media: strips both placeholders and defaults duration to 10", () => {
|
||||
const out = patchMediaPlaceholders(
|
||||
NEW_TEMPLATE.replace('<div id="root">', '<div id="root" data-duration="__VIDEO_DURATION__">'),
|
||||
{},
|
||||
);
|
||||
expect(out).not.toMatch(/<video\b|<audio\b/);
|
||||
expect(out).not.toContain("__");
|
||||
expect(out).toContain('<div id="root" data-duration="10">');
|
||||
});
|
||||
|
||||
it('normalises muted="" and a stray data-has-audio on the placeholder', () => {
|
||||
const html = `<video src="__VIDEO_SRC__" muted="" data-has-audio="false" data-x="1"></video>`;
|
||||
const out = patchMediaPlaceholders(html, { video: { filename: "a.mp4", hasAudio: true } });
|
||||
expect(out).toBe(`<video src="a.mp4" data-x="1" data-has-audio="true"></video>`);
|
||||
});
|
||||
});
|
||||
|
||||
describe("patchMediaPlaceholders — muted token matching", () => {
|
||||
it("does not strip the word muted from another attribute's value", () => {
|
||||
const html = `<video src="__VIDEO_SRC__" class="a muted b" muted></video>`;
|
||||
const out = patchMediaPlaceholders(html, { video: { filename: "a.mp4", hasAudio: true } });
|
||||
expect(out).toBe(`<video src="a.mp4" class="a muted b" data-has-audio="true"></video>`);
|
||||
});
|
||||
});
|
||||
|
||||
describe("patchMediaPlaceholders — legacy remote templates", () => {
|
||||
it("video with audio: keeps the legacy split untouched", () => {
|
||||
const out = patchMediaPlaceholders(LEGACY_TEMPLATE, {
|
||||
video: { filename: "talk.mp4", hasAudio: true },
|
||||
});
|
||||
expect(videoTag(out)).toMatch(/\bmuted\b/);
|
||||
expect(out).toMatch(/<audio\b[^>]*src="talk\.mp4"/);
|
||||
});
|
||||
|
||||
it("silent video: strips the legacy <audio> (a silent file would fail the render)", () => {
|
||||
const out = patchMediaPlaceholders(LEGACY_TEMPLATE, {
|
||||
video: { filename: "silent.mp4", hasAudio: false },
|
||||
});
|
||||
expect(out).not.toMatch(/<audio\b/);
|
||||
expect(videoTag(out)).toMatch(/\bmuted\b/);
|
||||
});
|
||||
|
||||
it("audio only: fills the legacy audio slot and drops the video", () => {
|
||||
const out = patchMediaPlaceholders(LEGACY_TEMPLATE, { audio: { filename: "track.wav" } });
|
||||
expect(out).toMatch(/<audio\b[^>]*src="track\.wav"/);
|
||||
expect(out).not.toMatch(/<video\b/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("registry examples use the audio-on-video form", () => {
|
||||
const examplesDir = resolve(fileURLToPath(import.meta.url), "../../../../../registry/examples");
|
||||
const withVideoSlot = readdirSync(examplesDir)
|
||||
.map((name) => join(examplesDir, name, "index.html"))
|
||||
.filter((file) => existsSync(file) && readFileSync(file, "utf-8").includes("__VIDEO_SRC__"));
|
||||
|
||||
it("finds the examples that carry a video slot", () => {
|
||||
expect(withVideoSlot.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it.each(withVideoSlot)("%s has no muted placeholder video and no __VIDEO_SRC__ audio", (file) => {
|
||||
const html = readFileSync(file, "utf-8");
|
||||
const video = html.match(/<video\b[^>]*src="__VIDEO_SRC__"[^>]*>/)?.[0] ?? "";
|
||||
expect(video).toContain('data-has-audio="true"');
|
||||
expect(video).not.toMatch(/\bmuted\b/);
|
||||
expect(html).not.toMatch(/<audio\b[^>]*src="__VIDEO_SRC__"/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("patchMediaPlaceholders — hostile filenames", () => {
|
||||
const cases: Array<[string, string]> = [
|
||||
["beat$$.wav", "beat$$.wav"],
|
||||
["cut$'s.mp4", "cut$'s.mp4"],
|
||||
["Track #1.wav", "Track%20%231.wav"],
|
||||
['a"b.mp4', "a%22b.mp4"],
|
||||
["q?x&y.mp4", "q%3Fx&y.mp4"],
|
||||
["p$&q.mp4", "p$&q.mp4"],
|
||||
];
|
||||
it.each(cases)("video %s is inserted literally as an encoded src", (name, encoded) => {
|
||||
const out = patchMediaPlaceholders(NEW_TEMPLATE, {
|
||||
video: { filename: name, hasAudio: true },
|
||||
durationSeconds: 6,
|
||||
});
|
||||
expect(out).toContain(`src="${encoded}"`);
|
||||
expect(out).toContain('data-duration="6"');
|
||||
expect(out).toContain("</div>");
|
||||
expect(out).not.toContain("__");
|
||||
});
|
||||
it.each(cases)("audio %s is inserted literally as an encoded src", (name, encoded) => {
|
||||
const out = patchMediaPlaceholders(NEW_TEMPLATE, {
|
||||
audio: { filename: name },
|
||||
durationSeconds: 6,
|
||||
});
|
||||
expect(out).toContain(`src="${encoded}"`);
|
||||
expect(out).not.toContain("__");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,72 @@
|
||||
// The one place `hyperframes init` (and the template-preview script) rewrites media placeholders.
|
||||
// New templates: an audible <video data-has-audio="true"> on __VIDEO_SRC__ plus an audio-only slot on __AUDIO_SRC__.
|
||||
// Legacy templates (muted <video> + <audio> both on __VIDEO_SRC__) still come from older remote registry examples.
|
||||
|
||||
export interface InitMediaOptions {
|
||||
video?: { filename: string; hasAudio: boolean };
|
||||
audio?: { filename: string };
|
||||
durationSeconds?: number;
|
||||
}
|
||||
|
||||
const VIDEO_SRC = "__VIDEO_SRC__";
|
||||
const AUDIO_SRC = "__AUDIO_SRC__";
|
||||
const DURATION = "__VIDEO_DURATION__";
|
||||
|
||||
// encodeURI leaves `#` and `?` alone, which would truncate the path at the URL fragment/query.
|
||||
function toSrcValue(filename: string): string {
|
||||
return encodeURI(filename).replace(/#/g, "%23").replace(/\?/g, "%3F");
|
||||
}
|
||||
|
||||
function openTag(tag: "video" | "audio", placeholder: string): string {
|
||||
return `<${tag}\\b[^>]*src="${placeholder}"[^>]*>`;
|
||||
}
|
||||
|
||||
// Element with a closing tag first, then a bare open tag — two passes so a bare tag never swallows a later element.
|
||||
function stripPlaceholder(html: string, tag: "video" | "audio", placeholder: string): string {
|
||||
const open = openTag(tag, placeholder);
|
||||
return html
|
||||
.replace(new RegExp(`${open}[\\s\\S]*?</${tag}>`, "g"), "")
|
||||
.replace(new RegExp(open, "g"), "");
|
||||
}
|
||||
|
||||
function setVideoAudio(html: string, hasAudio: boolean): string {
|
||||
return html.replace(new RegExp(openTag("video", VIDEO_SRC), "g"), (tag) => {
|
||||
// Attribute-by-attribute so a quoted value (class="a muted b") is consumed whole and never matched.
|
||||
const cleaned = tag.replace(
|
||||
/\s+([^\s=>"']+)(?:\s*=\s*(?:"[^"]*"|'[^']*'|[^\s>"']+))?/g,
|
||||
(attr, name: string) => (name === "muted" || name === "data-has-audio" ? "" : attr),
|
||||
);
|
||||
const attr = hasAudio ? ' data-has-audio="true"' : " muted";
|
||||
return cleaned.replace(/\s*>$/, `${attr}>`);
|
||||
});
|
||||
}
|
||||
|
||||
export function patchMediaPlaceholders(html: string, opts: InitMediaOptions): string {
|
||||
const legacyAudioSlot = new RegExp(openTag("audio", VIDEO_SRC)).test(html);
|
||||
let out = html;
|
||||
if (opts.video) {
|
||||
out = stripPlaceholder(out, "audio", AUDIO_SRC);
|
||||
if (legacyAudioSlot) {
|
||||
// Legacy split: keep it for a file with sound; a silent file's <audio> fails the render preflight.
|
||||
if (!opts.video.hasAudio) out = stripPlaceholder(out, "audio", VIDEO_SRC);
|
||||
} else {
|
||||
out = setVideoAudio(out, opts.video.hasAudio);
|
||||
}
|
||||
const videoSrc = toSrcValue(opts.video.filename);
|
||||
out = out.replaceAll(VIDEO_SRC, () => videoSrc);
|
||||
} else if (opts.audio) {
|
||||
out = stripPlaceholder(out, "video", VIDEO_SRC);
|
||||
const audioSrc = toSrcValue(opts.audio.filename);
|
||||
out = out.replaceAll(AUDIO_SRC, () => audioSrc);
|
||||
// Legacy templates carry the audio slot on __VIDEO_SRC__.
|
||||
out = out.replaceAll(VIDEO_SRC, () => audioSrc);
|
||||
} else {
|
||||
out = stripPlaceholder(out, "video", VIDEO_SRC);
|
||||
out = stripPlaceholder(out, "audio", VIDEO_SRC);
|
||||
out = stripPlaceholder(out, "audio", AUDIO_SRC);
|
||||
}
|
||||
const duration = opts.durationSeconds
|
||||
? String(Math.round(opts.durationSeconds * 100) / 100)
|
||||
: "10";
|
||||
return out.replaceAll(DURATION, duration);
|
||||
}
|
||||
@@ -13,7 +13,7 @@ Core attributes for controlling element timing and behavior.
|
||||
- `data-media-start="2"` — Media playback offset / trim point (seconds)
|
||||
- `data-volume="0.8"` — Audio/video gain. `1` is 0 dB, `0` is silence, and values above `1` boost up to `3.98` (+12 dB). For ducking and shaped envelopes use the `data-automation` volume lane (`hyperframes-core` skill, `creator-editing-recipes.md`)
|
||||
- `data-fade-in="0.5"` / `data-fade-out="1"` — Clip-edge fades (seconds): linear gain ramps anchored to the clip's start and end, multiplied on top of `data-volume` and any volume lane
|
||||
- `data-has-audio="true"` — Indicates video has an audio track
|
||||
- `data-has-audio="true"` — Keeps the file's sound on this clip (the default for footage with sound). Silent footage uses `muted` instead.
|
||||
|
||||
## Composition
|
||||
|
||||
|
||||
@@ -102,6 +102,6 @@ Fix all errors before presenting the result. Warnings should be reviewed before
|
||||
Scene timelines manually added to this root must not be paused. A paused
|
||||
child does not advance when the root is seeked. The runtime activates
|
||||
registered composition siblings, not arbitrary nested scene timelines.
|
||||
4. Videos use `muted` with a separate `<audio>` element for the audio track
|
||||
4. A video with sound keeps it on the `<video>` (`data-has-audio="true"`, no `muted`). Use a separate `<audio>` for music, voiceover, replacement audio, J/L cuts, or audio detached in Studio. Silent footage and b-roll: `muted`.
|
||||
5. Sub-compositions use `data-composition-src="compositions/file.html"` to reference other HTML files
|
||||
6. Only deterministic logic — no `Date.now()`, no `Math.random()`, no network fetches
|
||||
|
||||
@@ -102,6 +102,6 @@ Fix all errors before presenting the result. Warnings should be reviewed before
|
||||
Scene timelines manually added to this root must not be paused. A paused
|
||||
child does not advance when the root is seeked. The runtime activates
|
||||
registered composition siblings, not arbitrary nested scene timelines.
|
||||
4. Videos use `muted` with a separate `<audio>` element for the audio track
|
||||
4. A video with sound keeps it on the `<video>` (`data-has-audio="true"`, no `muted`). Use a separate `<audio>` for music, voiceover, replacement audio, J/L cuts, or audio detached in Studio. Silent footage and b-roll: `muted`.
|
||||
5. Sub-compositions use `data-composition-src="compositions/file.html"` to reference other HTML files
|
||||
6. Only deterministic logic — no `Date.now()`, no `Math.random()`, no network fetches
|
||||
|
||||
@@ -41,15 +41,16 @@
|
||||
id="a-roll"
|
||||
class="clip"
|
||||
src="__VIDEO_SRC__"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="0"
|
||||
style="position: absolute; inset: 0; width: 100%; height: 100%; object-fit: cover"
|
||||
></video>
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="__VIDEO_SRC__"
|
||||
src="__AUDIO_SRC__"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="2"
|
||||
|
||||
@@ -159,7 +159,7 @@ export function resolveAudioPreparationTimeoutMs(durationSeconds: number | null)
|
||||
);
|
||||
}
|
||||
|
||||
function getMediaDurationSeconds(filePath: string): number | null {
|
||||
export function getMediaDurationSeconds(filePath: string): number | null {
|
||||
try {
|
||||
const ffprobePath = findFFprobe();
|
||||
if (!ffprobePath) return null;
|
||||
|
||||
@@ -73,7 +73,7 @@
|
||||
(higher = in front, clips on same track cannot overlap)
|
||||
|
||||
REQUIRED on <video>:
|
||||
muted — framework manages audio via separate <audio> element
|
||||
data-has-audio="true" — keep the clip's own sound (no muted); silent footage: muted
|
||||
playsinline — prevents fullscreen on mobile
|
||||
|
||||
REQUIRED:
|
||||
@@ -82,7 +82,7 @@
|
||||
DO NOT:
|
||||
- Call video.play(), video.pause(), or set video.currentTime in scripts
|
||||
- Toggle visibility in scripts
|
||||
- Use the video element for audio (use a separate <audio> element)
|
||||
- Add muted to a video whose sound should play
|
||||
-->
|
||||
<video
|
||||
id="el-video"
|
||||
@@ -91,7 +91,7 @@
|
||||
data-duration="10"
|
||||
data-track-index="0"
|
||||
src="my-video.mp4"
|
||||
muted
|
||||
data-has-audio="true"
|
||||
playsinline
|
||||
></video>
|
||||
|
||||
@@ -117,7 +117,8 @@
|
||||
════════════════════════════════════════════════════
|
||||
|
||||
Audio is invisible — no class="clip" needed (no visual to hide).
|
||||
Use a separate <audio> element even if the source is the same .mp4 file.
|
||||
Use <audio> for music, voiceover, or replacement sound. A video with sound
|
||||
keeps it on the <video> (data-has-audio="true").
|
||||
The framework syncs audio playback to the timeline position.
|
||||
|
||||
Optional: data-volume="0.5" (0-1, default 1)
|
||||
@@ -128,7 +129,7 @@
|
||||
data-duration="15"
|
||||
data-track-index="1"
|
||||
data-volume="0.8"
|
||||
src="my-video.mp4"
|
||||
src="my-music.mp3"
|
||||
></audio>
|
||||
|
||||
<!--
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
* Processes and mixes audio tracks using FFmpeg.
|
||||
*/
|
||||
|
||||
import { isSelfOrAncestorHidden } from "./mediaHidden.js";
|
||||
import { closeSync, existsSync, mkdirSync, mkdtempSync, openSync, rmSync, writeFileSync } from "fs";
|
||||
import { join, dirname, isAbsolute, relative } from "path";
|
||||
import { parseHTML } from "linkedom";
|
||||
@@ -588,12 +589,7 @@ export function parseAudioElements(html: string): AudioElement[] {
|
||||
const parseEnd = (raw: string | null): number => {
|
||||
return parseStrictFiniteTimingNumber(raw) ?? 0;
|
||||
};
|
||||
const isHidden = (el: AudioMediaElement): boolean => {
|
||||
for (let current: AudioMediaElement | null = el; current; current = current.parentElement) {
|
||||
if (current.hasAttribute("data-hidden")) return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
const isHidden = isSelfOrAncestorHidden;
|
||||
|
||||
// Resolved once per parse. A group element carrying `data-hidden` drops
|
||||
// every member from the render (RULES: mute-by-drop, never
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
interface HiddenCheckEl {
|
||||
hasAttribute(name: string): boolean;
|
||||
parentElement: HiddenCheckEl | null;
|
||||
}
|
||||
|
||||
export function isSelfOrAncestorHidden(el: HiddenCheckEl): boolean {
|
||||
for (let current: HiddenCheckEl | null = el; current; current = current.parentElement) {
|
||||
if (current.hasAttribute("data-hidden")) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
@@ -3700,3 +3700,18 @@ describe("getFrameAtTime — IEEE 754 boundary precision", () => {
|
||||
expect(getFrameAtTime(extracted, 3, 0)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseVideoElements hidden flag", () => {
|
||||
it("marks a video hidden on itself or through a hidden ancestor", () => {
|
||||
const videos = parseVideoElements(
|
||||
`<div data-hidden><video id="a" src="a.mp4" data-has-audio="true"></video></div>` +
|
||||
`<video id="b" src="b.mp4" data-has-audio="true" data-hidden></video>` +
|
||||
`<video id="c" src="c.mp4" data-has-audio="true"></video>`,
|
||||
);
|
||||
expect(videos.map((v) => [v.id, v.hidden])).toEqual([
|
||||
["a", true],
|
||||
["b", true],
|
||||
["c", undefined],
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
* Videos are replaced with <img> elements during capture.
|
||||
*/
|
||||
|
||||
import { isSelfOrAncestorHidden } from "./mediaHidden.js";
|
||||
import { copyFileSync, existsSync, linkSync, mkdirSync, rmSync } from "fs";
|
||||
import { join } from "path";
|
||||
import { parseHTML } from "linkedom";
|
||||
@@ -81,6 +82,7 @@ export interface VideoElement {
|
||||
playbackRate?: RateSpec;
|
||||
loop: boolean;
|
||||
hasAudio: boolean;
|
||||
hidden?: boolean;
|
||||
}
|
||||
|
||||
export interface ExtractedFrames {
|
||||
@@ -720,6 +722,7 @@ export function parseVideoElements(html: string): VideoElement[] {
|
||||
playbackRate: readElementRateSpec(el),
|
||||
loop: el.hasAttribute("loop"),
|
||||
hasAudio: hasAudioAttr === "true",
|
||||
...(isSelfOrAncestorHidden(el) ? { hidden: true } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -834,3 +834,51 @@ describe("templating tokens are checked on the raw src, before cleanAssetUrl", (
|
||||
expect(c.has("missing_local_asset")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("audio-aware project rules", () => {
|
||||
const composition = (body: string) => validHtml().replace("></div>", `>${body}</div>`);
|
||||
const codesFor = async (html: string, files: string[] = []) => {
|
||||
const project = makeProject(html);
|
||||
for (const file of files) writeFileSync(join(project, file), "x");
|
||||
const { results } = await lintProject(project);
|
||||
return results.flatMap((result) => result.result.findings.map((f) => f.code));
|
||||
};
|
||||
const video = (attrs: string, start = 0) =>
|
||||
`<video id="v${start}${attrs.length}" src="a.mp4" data-start="${start}" data-duration="5" data-track-index="0" ${attrs}></video>`;
|
||||
|
||||
describe("duplicate_audio_track", () => {
|
||||
it("warns when audible videos overlap on one track", async () => {
|
||||
const codes = await codesFor(
|
||||
composition(video('data-has-audio="true"', 0) + video('data-has-audio="true"', 2)),
|
||||
);
|
||||
expect(codes).toContain("duplicate_audio_track");
|
||||
});
|
||||
|
||||
it("stays quiet when one of the overlapping videos is muted", async () => {
|
||||
const codes = await codesFor(
|
||||
composition(video('data-has-audio="true"', 0) + video("muted", 2)),
|
||||
);
|
||||
expect(codes).not.toContain("duplicate_audio_track");
|
||||
});
|
||||
|
||||
it("stays quiet when an overlapping video is explicitly non-audible", async () => {
|
||||
const codes = await codesFor(
|
||||
composition(video('data-has-audio="true"', 0) + video('data-has-audio="false"', 2)),
|
||||
);
|
||||
expect(codes).not.toContain("duplicate_audio_track");
|
||||
});
|
||||
});
|
||||
|
||||
describe("audio_file_without_element", () => {
|
||||
const audibleRoll = video('data-has-audio="true"');
|
||||
it("does not claim silence when an audible video carries the sound", async () => {
|
||||
const codes = await codesFor(composition(audibleRoll), ["audio.mp3"]);
|
||||
expect(codes).not.toContain("audio_file_without_element");
|
||||
});
|
||||
|
||||
it("still warns when the only video is muted", async () => {
|
||||
const codes = await codesFor(composition(video("muted")), ["audio.mp3"]);
|
||||
expect(codes).toContain("audio_file_without_element");
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -29,7 +29,7 @@ import type {
|
||||
HyperframeLinterOptions,
|
||||
} from "./types.js";
|
||||
import type { ParsableDocumentLike } from "@hyperframes/parsers/sub-composition-validity";
|
||||
import { mediaSrcTagRe } from "./utils";
|
||||
import { isAudibleVideoTag, mediaSrcTagRe } from "./utils";
|
||||
|
||||
/** Adapts linkedom's `parseHTML` to the `checkSubCompositionUsability` contract. */
|
||||
function parseSubCompHtml(html: string): ParsableDocumentLike {
|
||||
@@ -339,8 +339,11 @@ function lintProjectAudioFiles(
|
||||
if (audioFiles.length === 0) return findings;
|
||||
|
||||
const hasAudioElement = htmlSources.some(({ html }) => /<audio\b/i.test(html));
|
||||
const hasAudibleVideo = htmlSources.some(({ html }) =>
|
||||
(html.match(/<video\b[^>]*>/gi) ?? []).some(isAudibleVideoTag),
|
||||
);
|
||||
|
||||
if (!hasAudioElement) {
|
||||
if (!hasAudioElement && !hasAudibleVideo) {
|
||||
findings.push({
|
||||
code: "audio_file_without_element",
|
||||
severity: "warning",
|
||||
@@ -549,10 +552,11 @@ function lintDuplicateAudioTracks(htmlSources: HtmlSource[]): HyperframeLintFind
|
||||
const seen = new Set<string>();
|
||||
|
||||
for (const { html } of htmlSources) {
|
||||
const audioTagRe = /<audio\b[^>]*>/gi;
|
||||
const soundTagRe = /<(?:audio|video)\b[^>]*>/gi;
|
||||
let match: RegExpExecArray | null;
|
||||
while ((match = audioTagRe.exec(html)) !== null) {
|
||||
while ((match = soundTagRe.exec(html)) !== null) {
|
||||
const tag = match[0];
|
||||
if (/^<video/i.test(tag) && !isAudibleVideoTag(tag)) continue;
|
||||
const trackStr = extractAttr(tag, "data-track-index");
|
||||
const startStr = extractAttr(tag, "data-start");
|
||||
const durStr = extractAttr(tag, "data-duration");
|
||||
@@ -579,7 +583,7 @@ function lintDuplicateAudioTracks(htmlSources: HtmlSource[]): HyperframeLintFind
|
||||
findings.push({
|
||||
code: "duplicate_audio_track",
|
||||
severity: "warning",
|
||||
message: `Multiple <audio> elements on track ${a.trackIndex} overlap (${a.src} at ${a.start}-${Number.isFinite(a.end) ? a.end.toFixed(1) : "end"}s, ${b.src} at ${b.start}-${Number.isFinite(b.end) ? b.end.toFixed(1) : "end"}s). This causes layered audio playback.`,
|
||||
message: `Multiple audible <audio>/<video> elements on track ${a.trackIndex} overlap (${a.src} at ${a.start}-${Number.isFinite(a.end) ? a.end.toFixed(1) : "end"}s, ${b.src} at ${b.start}-${Number.isFinite(b.end) ? b.end.toFixed(1) : "end"}s). This causes layered audio playback.`,
|
||||
fixHint: "Use non-overlapping time windows or different track indices.",
|
||||
});
|
||||
}
|
||||
|
||||
@@ -293,6 +293,93 @@ describe("media rules", () => {
|
||||
expect(finding?.elementId).toBe("demo-video");
|
||||
});
|
||||
|
||||
it("video_missing_muted recommends data-has-audio first", async () => {
|
||||
const html = `
|
||||
<html><body>
|
||||
<div id="root" data-composition-id="c1" data-width="1920" data-height="1080">
|
||||
<video id="v" data-start="0" data-duration="5" src="clip.mp4" playsinline></video>
|
||||
</div>
|
||||
<script>window.__timelines = window.__timelines || {}; window.__timelines["c1"] = gsap.timeline({ paused: true });</script>
|
||||
</body></html>`;
|
||||
const result = await lintHyperframeHtml(html);
|
||||
const finding = result.findings.find((f) => f.code === "video_missing_muted");
|
||||
expect(finding?.severity).toBe("error");
|
||||
expect(finding?.message).toContain("declares neither muted nor data-has-audio");
|
||||
expect(finding?.message).not.toContain("separate <audio>");
|
||||
expect(finding?.fixHint?.indexOf("data-has-audio")).toBeLessThan(
|
||||
finding?.fixHint?.indexOf("muted") ?? -1,
|
||||
);
|
||||
});
|
||||
|
||||
it("video_audio_double_source suggests removing the <audio> first", async () => {
|
||||
const html = `
|
||||
<html><body>
|
||||
<div id="root" data-composition-id="c1" data-width="1920" data-height="1080">
|
||||
<video id="v" data-start="0" data-duration="5" data-has-audio="true" src="clip.mp4" playsinline></video>
|
||||
<audio id="a" data-start="0" data-duration="5" src="clip.mp4"></audio>
|
||||
</div>
|
||||
<script>window.__timelines = window.__timelines || {}; window.__timelines["c1"] = gsap.timeline({ paused: true });</script>
|
||||
</body></html>`;
|
||||
const result = await lintHyperframeHtml(html);
|
||||
const finding = result.findings.find((f) => f.code === "video_audio_double_source");
|
||||
expect(finding?.fixHint?.startsWith("Remove the <audio>")).toBe(true);
|
||||
});
|
||||
|
||||
const wrap = (body: string) => `
|
||||
<html><body>
|
||||
<div id="root" data-composition-id="c1" data-width="1920" data-height="1080">
|
||||
${body}
|
||||
</div>
|
||||
<script>window.__timelines = window.__timelines || {}; window.__timelines["c1"] = gsap.timeline({ paused: true });</script>
|
||||
</body></html>`;
|
||||
const codes = async (html: string) =>
|
||||
(await lintHyperframeHtml(html)).findings.map((f) => f.code);
|
||||
|
||||
it.each(['data-has-audio="false"', 'data-has-audio=""', "data-has-audio"])(
|
||||
"video_missing_muted stays quiet when the video declares its audio state (%s)",
|
||||
async (attr) => {
|
||||
const found = await codes(
|
||||
wrap(`<video id="v" data-start="0" data-duration="5" ${attr} src="clip.mp4"></video>`),
|
||||
);
|
||||
expect(found).not.toContain("video_missing_muted");
|
||||
},
|
||||
);
|
||||
|
||||
it.each(['data-has-audio="false"', 'data-has-audio=""', "muted"])(
|
||||
"video_audio_double_source ignores an explicitly silent video (%s)",
|
||||
async (attr) => {
|
||||
const found = await codes(
|
||||
wrap(`<video id="v" data-start="0" data-duration="5" ${attr} src="clip.mp4"></video>
|
||||
<audio id="a" data-start="0" data-duration="5" src="clip.mp4"></audio>`),
|
||||
);
|
||||
expect(found).not.toContain("video_audio_double_source");
|
||||
},
|
||||
);
|
||||
|
||||
it("video_audio_double_source ignores same-src clips that do not overlap in time", async () => {
|
||||
const found = await codes(
|
||||
wrap(`<video id="v" data-start="0" data-duration="4" data-has-audio="true" src="clip.mp4"></video>
|
||||
<audio id="a" data-start="4" data-duration="4" src="clip.mp4"></audio>`),
|
||||
);
|
||||
expect(found).not.toContain("video_audio_double_source");
|
||||
});
|
||||
|
||||
it("video_audio_double_source flags same-src clips that overlap in time", async () => {
|
||||
const found = await codes(
|
||||
wrap(`<video id="v" data-start="0" data-duration="4" data-has-audio="true" src="clip.mp4"></video>
|
||||
<audio id="a" data-start="3" data-duration="4" src="clip.mp4"></audio>`),
|
||||
);
|
||||
expect(found).toContain("video_audio_double_source");
|
||||
});
|
||||
|
||||
it("video_audio_double_source keeps flagging when a duration is not numeric", async () => {
|
||||
const found = await codes(
|
||||
wrap(`<video id="v" data-start="0" data-has-audio="true" src="clip.mp4"></video>
|
||||
<audio id="a" data-start="9" data-duration="4" src="clip.mp4"></audio>`),
|
||||
);
|
||||
expect(found).toContain("video_audio_double_source");
|
||||
});
|
||||
|
||||
it("does NOT flag <video> as nested in a void element with data-start (regression)", async () => {
|
||||
// Regression: void elements like <img> have no closing tag, so the previous
|
||||
// implementation kept them on the parent stack indefinitely and flagged any
|
||||
|
||||
@@ -1,5 +1,15 @@
|
||||
import type { LintContext, HyperframeLintFinding, OpenTag } from "../context";
|
||||
import { readAttr, readDecodedAttr, stripJsComments, truncateSnippet, isMediaTag } from "../utils";
|
||||
import {
|
||||
readAttr,
|
||||
readDecodedAttr,
|
||||
stripJsComments,
|
||||
truncateSnippet,
|
||||
isMediaTag,
|
||||
hasAttrName,
|
||||
isAudibleVideoTag,
|
||||
mediaTimeWindow,
|
||||
mediaWindowsOverlap,
|
||||
} from "../utils";
|
||||
import { validateColorGradingContract } from "@hyperframes/parsers/color-grading-contract";
|
||||
import { extractMediaSrcMutations } from "@hyperframes/parsers/composition";
|
||||
import { parseHTML } from "linkedom";
|
||||
@@ -37,12 +47,6 @@ function escapeRegExp(value: string): string {
|
||||
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
}
|
||||
|
||||
function hasAttrName(tagSource: string, attr: string): boolean {
|
||||
const escaped = escapeRegExp(attr);
|
||||
const attrs = tagSource.replace(/^<\s*[a-z][\w:-]*/i, "");
|
||||
return new RegExp(`(?:^|\\s)${escaped}(?:\\s*=|\\s|/?>)`, "i").test(attrs);
|
||||
}
|
||||
|
||||
const IMAGE_SRC_EXT = new Set([
|
||||
"jpg",
|
||||
"jpeg",
|
||||
@@ -513,15 +517,16 @@ export const mediaRules: Array<(ctx: LintContext) => HyperframeLintFinding[]> =
|
||||
if (tag.name !== "video") continue;
|
||||
const hasMuted = hasAttrName(tag.raw, "muted");
|
||||
const hasDeclaredAudio = readAttr(tag.raw, "data-has-audio") === "true";
|
||||
if (!hasMuted && !hasDeclaredAudio && readAttr(tag.raw, "data-start")) {
|
||||
const declaresAudioState = hasAttrName(tag.raw, "data-has-audio");
|
||||
if (!hasMuted && !declaresAudioState && readAttr(tag.raw, "data-start")) {
|
||||
const elementId = readAttr(tag.raw, "id") || undefined;
|
||||
findings.push({
|
||||
code: "video_missing_muted",
|
||||
severity: "error",
|
||||
message: `<video${elementId ? ` id="${elementId}"` : ""}> has data-start but is not muted. Mark audible videos with data-has-audio="true"; otherwise keep video muted and use a separate <audio> element for sound.`,
|
||||
message: `<video${elementId ? ` id="${elementId}"` : ""}> has data-start but declares neither muted nor data-has-audio. If the file has sound, add data-has-audio="true" (the sound stays on this clip). If it is silent, add muted.`,
|
||||
elementId,
|
||||
fixHint:
|
||||
'Add the `muted` attribute for silent video, or add data-has-audio="true" when the video track should contribute audio.',
|
||||
'Add data-has-audio="true" when the file has sound (recommended — keeps picture and sound on one clip), or add `muted` for silent footage.',
|
||||
snippet: truncateSnippet(tag.raw),
|
||||
});
|
||||
}
|
||||
@@ -769,34 +774,37 @@ export const mediaRules: Array<(ctx: LintContext) => HyperframeLintFinding[]> =
|
||||
// <audio> pointing to the same file, which causes double playback at runtime
|
||||
({ tags }) => {
|
||||
const findings: HyperframeLintFinding[] = [];
|
||||
const videoSources = new Map<string, { id?: string; raw: string }>();
|
||||
const audioSources = new Map<string, { id?: string; raw: string }>();
|
||||
type Source = { id?: string; raw: string };
|
||||
const videos: Array<[string, Source]> = [];
|
||||
const audios: Array<[string, Source]> = [];
|
||||
|
||||
for (const tag of tags) {
|
||||
if (!readAttr(tag.raw, "data-start")) continue;
|
||||
const src = readAttr(tag.raw, "src");
|
||||
if (!src) continue;
|
||||
const elementId = readAttr(tag.raw, "id") || undefined;
|
||||
const source = { id: readAttr(tag.raw, "id") || undefined, raw: tag.raw };
|
||||
if (tag.name === "video") {
|
||||
const isMuted = hasAttrName(tag.raw, "muted");
|
||||
if (!isMuted) {
|
||||
videoSources.set(src, { id: elementId, raw: tag.raw });
|
||||
}
|
||||
if (isAudibleVideoTag(tag.raw)) videos.push([src, source]);
|
||||
} else if (tag.name === "audio") {
|
||||
audioSources.set(src, { id: elementId, raw: tag.raw });
|
||||
audios.push([src, source]);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [src, audioInfo] of audioSources) {
|
||||
const videoInfo = videoSources.get(src);
|
||||
if (!videoInfo) continue;
|
||||
for (const [src, audioInfo] of audios) {
|
||||
const match = videos.find(
|
||||
([videoSrc, video]) =>
|
||||
videoSrc === src &&
|
||||
mediaWindowsOverlap(mediaTimeWindow(video.raw), mediaTimeWindow(audioInfo.raw)),
|
||||
);
|
||||
if (!match) continue;
|
||||
const videoInfo = match[1];
|
||||
findings.push({
|
||||
code: "video_audio_double_source",
|
||||
severity: "error",
|
||||
message: `<audio${audioInfo.id ? ` id="${audioInfo.id}"` : ""}> and <video${videoInfo.id ? ` id="${videoInfo.id}"` : ""}> both point to the same source. The unmuted video already provides audio — the duplicate <audio> will cause double playback and echo.`,
|
||||
message: `<audio${audioInfo.id ? ` id="${audioInfo.id}"` : ""}> and <video${videoInfo.id ? ` id="${videoInfo.id}"` : ""}> both point to the same source at the same time. The unmuted video already provides audio — the duplicate <audio> will cause double playback and echo.`,
|
||||
elementId: audioInfo.id,
|
||||
fixHint:
|
||||
"Either mute the video (add `muted` attribute) and keep the separate <audio>, or remove the <audio> element and let the video provide its own audio track.",
|
||||
"Remove the <audio> element and let the video carry its own sound (recommended), or mute the video (add `muted`) and keep the separate <audio> when picture and sound must be cut independently.",
|
||||
snippet: truncateSnippet(audioInfo.raw),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -233,6 +233,36 @@ export function readAttr(tagSource: string, attr: string): string | null {
|
||||
return match?.[1] || null;
|
||||
}
|
||||
|
||||
export function hasAttrName(tagSource: string, attr: string): boolean {
|
||||
const escaped = attr.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
const attrs = tagSource.replace(/^<\s*[a-z][\w:-]*/i, "");
|
||||
return new RegExp(`(?:^|\\s)${escaped}(?:\\s*=|\\s|/?>)`, "i").test(attrs);
|
||||
}
|
||||
|
||||
// An explicit data-has-audio is authoritative for the compiler; "false" and empty mean silent.
|
||||
export function isAudibleVideoTag(tagSource: string): boolean {
|
||||
if (hasAttrName(tagSource, "muted")) return false;
|
||||
if (!hasAttrName(tagSource, "data-has-audio")) return true;
|
||||
const declared = readAttr(tagSource, "data-has-audio");
|
||||
return declared !== null && declared.trim().toLowerCase() !== "false";
|
||||
}
|
||||
|
||||
export function mediaTimeWindow(tagSource: string): { start: number; end: number } | null {
|
||||
const start = Number(readAttr(tagSource, "data-start"));
|
||||
const duration = Number(readAttr(tagSource, "data-duration"));
|
||||
if (!readAttr(tagSource, "data-start") || !readAttr(tagSource, "data-duration")) return null;
|
||||
if (!Number.isFinite(start) || !Number.isFinite(duration)) return null;
|
||||
return { start, end: start + duration };
|
||||
}
|
||||
|
||||
export function mediaWindowsOverlap(
|
||||
a: { start: number; end: number } | null,
|
||||
b: { start: number; end: number } | null,
|
||||
): boolean {
|
||||
if (!a || !b) return true;
|
||||
return a.start < b.end && b.start < a.end;
|
||||
}
|
||||
|
||||
/** Read an HTML attribute using browser-equivalent character-reference decoding. */
|
||||
export function readDecodedAttr(tagSource: string, attr: string): string | null {
|
||||
if (!tagSource) return null;
|
||||
|
||||
@@ -908,3 +908,65 @@ describe("extractCompositionMetadata", () => {
|
||||
expect(meta.variables.find((v) => v.id === "ok")).toBeDefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("video audio attributes", () => {
|
||||
const DOC = `<!DOCTYPE html>
|
||||
<html><body>
|
||||
<div id="stage">
|
||||
<div id="stage-zoom-container"><video id="v" src="a.mp4" muted data-start="0" data-duration="2"></video></div>
|
||||
</div>
|
||||
</body></html>`;
|
||||
|
||||
const videoTag = (html: string, id: string): string =>
|
||||
html.match(new RegExp(`<video[^>]*id="${id}"[^>]*>`))?.[0] ?? "";
|
||||
|
||||
it("addElementToHtml: an audible video gets data-has-audio and no muted", () => {
|
||||
const { html, id } = addElementToHtml(DOC, {
|
||||
type: "video",
|
||||
name: "Audible",
|
||||
src: "b.mp4",
|
||||
startTime: 0,
|
||||
duration: 2,
|
||||
zIndex: 0,
|
||||
hasAudio: true,
|
||||
});
|
||||
const tag = videoTag(html, id);
|
||||
expect(tag).toContain('data-has-audio="true"');
|
||||
expect(tag).not.toMatch(/\bmuted\b/);
|
||||
});
|
||||
|
||||
it("addElementToHtml: a silent video is muted with no data-has-audio", () => {
|
||||
const { html, id } = addElementToHtml(DOC, {
|
||||
type: "video",
|
||||
name: "Silent",
|
||||
src: "b.mp4",
|
||||
startTime: 0,
|
||||
duration: 2,
|
||||
zIndex: 0,
|
||||
});
|
||||
const tag = videoTag(html, id);
|
||||
expect(tag).toMatch(/\bmuted\b/);
|
||||
expect(tag).not.toContain("data-has-audio");
|
||||
});
|
||||
|
||||
it("updateElementInHtml hasAudio:true clears muted; false restores it", () => {
|
||||
const on = updateElementInHtml(DOC, "v", { hasAudio: true });
|
||||
expect(videoTag(on, "v")).not.toMatch(/\bmuted\b/);
|
||||
expect(videoTag(on, "v")).toContain('data-has-audio="true"');
|
||||
const off = updateElementInHtml(on, "v", { hasAudio: false });
|
||||
const tag = videoTag(off, "v");
|
||||
expect(tag).toMatch(/\bmuted\b/);
|
||||
expect(tag).not.toContain("data-has-audio");
|
||||
});
|
||||
|
||||
it("updateElementInHtml hasAudio only touches videos, and only true/false", () => {
|
||||
const html =
|
||||
'<html><body><video id="v" src="a.mp4" data-has-audio="true"></video><audio id="a" src="m.wav"></audio></body></html>';
|
||||
const audio = updateElementInHtml(html, "a", { hasAudio: false });
|
||||
expect(audio).not.toMatch(/<audio\b[^>]*\bmuted\b/);
|
||||
expect(audio).not.toMatch(/<audio\b[^>]*data-has-audio/);
|
||||
const untouched = updateElementInHtml(html, "v", { hasAudio: undefined });
|
||||
expect(videoTag(untouched, "v")).toContain('data-has-audio="true"');
|
||||
expect(videoTag(untouched, "v")).not.toMatch(/\bmuted\b/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -617,11 +617,13 @@ export function updateElementInHtml(
|
||||
}
|
||||
|
||||
// Handle hasAudio property for videos
|
||||
if ("hasAudio" in updates) {
|
||||
if (el.tagName.toLowerCase() === "video" && "hasAudio" in updates) {
|
||||
if (updates.hasAudio === true) {
|
||||
el.setAttribute("data-has-audio", "true");
|
||||
} else {
|
||||
el.removeAttribute("muted");
|
||||
} else if (updates.hasAudio === false) {
|
||||
el.removeAttribute("data-has-audio");
|
||||
el.setAttribute("muted", "");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -672,12 +674,10 @@ export function addElementToHtml(
|
||||
case "video": {
|
||||
const mediaEl = element as TimelineMediaElement;
|
||||
newEl = doc.createElement("video");
|
||||
newEl.setAttribute("muted", "");
|
||||
newEl.setAttribute("playsinline", "");
|
||||
applyMediaAttrs(newEl, mediaEl);
|
||||
if (mediaEl.hasAudio) {
|
||||
newEl.setAttribute("data-has-audio", "true");
|
||||
}
|
||||
if (mediaEl.hasAudio) newEl.setAttribute("data-has-audio", "true");
|
||||
else newEl.setAttribute("muted", "");
|
||||
break;
|
||||
}
|
||||
case "image": {
|
||||
|
||||
@@ -114,6 +114,15 @@ describe("appendAutoDetectedVideoAudio", () => {
|
||||
expect(composition.audios).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("skips a hidden video so a track hidden in Studio stays silent in the render", () => {
|
||||
const composition = {
|
||||
videos: [makeVideo({ hidden: true })],
|
||||
audios: [] as never[],
|
||||
};
|
||||
appendAutoDetectedVideoAudio(composition, [makeExtracted("v1", true)]);
|
||||
expect(composition.audios).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("skips when the source file has no audio track", () => {
|
||||
const composition = { videos: [makeVideo()], audios: [] as never[] };
|
||||
appendAutoDetectedVideoAudio(composition, [makeExtracted("v1", false)]);
|
||||
|
||||
@@ -562,7 +562,7 @@ export function appendAutoDetectedVideoAudio(
|
||||
for (const ext of extracted) {
|
||||
if (!ext.metadata.hasAudio) continue;
|
||||
const video = composition.videos.find((v) => v.id === ext.videoId);
|
||||
if (!video || !video.hasAudio || existingAudioSrcs.has(video.src)) continue;
|
||||
if (!video || !video.hasAudio || video.hidden || existingAudioSrcs.has(video.src)) continue;
|
||||
composition.audios.push({
|
||||
id: `${video.id}-audio`,
|
||||
src: video.src,
|
||||
|
||||
@@ -62,6 +62,7 @@ export function StudioRightPanels({
|
||||
handleDomAttributeQuietCommit,
|
||||
handleDomHtmlAttributeCommit,
|
||||
handleDomAttributesCommit,
|
||||
handleDomAttributeBatchCommit,
|
||||
handleDomPathOffsetCommit,
|
||||
handleDomBoxSizeCommit,
|
||||
handleDomRotationCommit,
|
||||
@@ -210,6 +211,7 @@ export function StudioRightPanels({
|
||||
onSetStyle={handleDomStyleCommit}
|
||||
onSetAttribute={handleDomAttributeCommit}
|
||||
onSetAttributes={handleDomAttributesCommit}
|
||||
onSetAttributeBatch={handleDomAttributeBatchCommit}
|
||||
onSetAttributeLive={setAttributeWhileDragging}
|
||||
onSetAttributeQuiet={handleDomAttributeQuietCommit}
|
||||
onApplyColorGradingScope={handleApplyColorGradingScope}
|
||||
|
||||
@@ -58,7 +58,6 @@ export {
|
||||
export const PropertyPanel = memo(function PropertyPanel(props: PropertyPanelProps) {
|
||||
const {
|
||||
projectId,
|
||||
projectDir,
|
||||
assets,
|
||||
element,
|
||||
multiSelectCount = 0,
|
||||
@@ -72,8 +71,6 @@ export const PropertyPanel = memo(function PropertyPanel(props: PropertyPanelPro
|
||||
onSetAttribute,
|
||||
onSetAttributeLive,
|
||||
onApplyColorGradingScope,
|
||||
onSetHtmlAttribute,
|
||||
onRemoveBackground,
|
||||
onSetManualOffset,
|
||||
onSetManualSize,
|
||||
onSetManualRotation,
|
||||
@@ -374,17 +371,7 @@ export const PropertyPanel = memo(function PropertyPanel(props: PropertyPanelPro
|
||||
/>
|
||||
)}
|
||||
|
||||
{sections.media && (
|
||||
<MediaSection
|
||||
projectDir={projectDir}
|
||||
element={element}
|
||||
styles={styles}
|
||||
onSetStyle={onSetStyle}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={onSetHtmlAttribute}
|
||||
onRemoveBackground={onRemoveBackground}
|
||||
/>
|
||||
)}
|
||||
{sections.media && <MediaSection {...props} element={element} styles={styles} />}
|
||||
|
||||
{sections.layout && (
|
||||
<Section title="Layout" icon={<Move size={15} />}>
|
||||
|
||||
@@ -61,7 +61,6 @@ export function PropertyPanelFlat({
|
||||
clipboardCopied,
|
||||
onCopyElementInfo,
|
||||
projectId,
|
||||
projectDir,
|
||||
assets,
|
||||
previewIframeRef,
|
||||
onClearSelection,
|
||||
@@ -73,8 +72,6 @@ export function PropertyPanelFlat({
|
||||
onSetAttributeLive,
|
||||
onSetAttributeQuiet,
|
||||
onApplyColorGradingScope,
|
||||
onSetHtmlAttribute,
|
||||
onRemoveBackground,
|
||||
onSetText,
|
||||
onSetTextFieldStyle,
|
||||
onPreviewTextFieldStyle,
|
||||
@@ -130,6 +127,7 @@ export function PropertyPanelFlat({
|
||||
onUpdateKeyframeEase,
|
||||
onUpdateSegmentEase,
|
||||
onSetAllKeyframeEases,
|
||||
...forwardedProps
|
||||
}: PropertyPanelFlatProps) {
|
||||
// PropertyPanel keys this component by selection, so the default is per element.
|
||||
const [openGroupId, setOpenGroupId] = useState<string>(() =>
|
||||
@@ -502,13 +500,12 @@ export function PropertyPanelFlat({
|
||||
summary: element.tagName,
|
||||
content: (
|
||||
<FlatMediaSection
|
||||
projectDir={projectDir}
|
||||
{...forwardedProps}
|
||||
projectId={projectId}
|
||||
element={element}
|
||||
styles={styles}
|
||||
onSetStyle={onSetStyle}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={onSetHtmlAttribute}
|
||||
onRemoveBackground={onRemoveBackground}
|
||||
{...volumeAutomation}
|
||||
/>
|
||||
),
|
||||
|
||||
@@ -0,0 +1,212 @@
|
||||
// @vitest-environment happy-dom
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { applyPatchByTarget } from "../../utils/sourcePatcher";
|
||||
import {
|
||||
buildKeepSoundCutoutEdit,
|
||||
cutoutOps,
|
||||
hasAudioToggleOps,
|
||||
mintLinkId,
|
||||
mutedToggleOps,
|
||||
} from "./mediaAudioEdits";
|
||||
|
||||
const attr = (property: string, value: string | null) => ({
|
||||
type: "attribute" as const,
|
||||
property,
|
||||
value,
|
||||
});
|
||||
const html = (property: string, value: string | null) => ({
|
||||
type: "html-attribute" as const,
|
||||
property,
|
||||
value,
|
||||
});
|
||||
|
||||
describe("mutedToggleOps", () => {
|
||||
it("mutes a video and removes has-audio instead of writing an empty marker", () => {
|
||||
expect(mutedToggleOps({ isVideo: true, nextMuted: true, probedHasAudio: null })).toEqual([
|
||||
html("muted", "true"),
|
||||
attr("has-audio", null),
|
||||
]);
|
||||
});
|
||||
|
||||
it("unmutes a video the probe hears as audible", () => {
|
||||
expect(mutedToggleOps({ isVideo: true, nextMuted: false, probedHasAudio: true })).toEqual([
|
||||
html("muted", null),
|
||||
attr("has-audio", "true"),
|
||||
]);
|
||||
});
|
||||
|
||||
it("unmutes a silent-probe video with an explicit has-audio=false", () => {
|
||||
expect(mutedToggleOps({ isVideo: true, nextMuted: false, probedHasAudio: false })).toEqual([
|
||||
html("muted", null),
|
||||
attr("has-audio", "false"),
|
||||
]);
|
||||
});
|
||||
|
||||
it("leaves has-audio absent when the probe fails", () => {
|
||||
expect(mutedToggleOps({ isVideo: true, nextMuted: false, probedHasAudio: null })).toEqual([
|
||||
html("muted", null),
|
||||
attr("has-audio", null),
|
||||
]);
|
||||
});
|
||||
|
||||
it("toggles only muted on an audio element", () => {
|
||||
expect(mutedToggleOps({ isVideo: false, nextMuted: true, probedHasAudio: null })).toEqual([
|
||||
html("muted", "true"),
|
||||
]);
|
||||
expect(mutedToggleOps({ isVideo: false, nextMuted: false, probedHasAudio: null })).toEqual([
|
||||
html("muted", null),
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("hasAudioToggleOps", () => {
|
||||
it("turns sound on by stamping has-audio and clearing muted", () => {
|
||||
expect(hasAudioToggleOps(true)).toEqual([attr("has-audio", "true"), html("muted", null)]);
|
||||
});
|
||||
|
||||
it("turns sound off by removing has-audio rather than writing an empty value", () => {
|
||||
expect(hasAudioToggleOps(false)).toEqual([attr("has-audio", null), html("muted", "true")]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("cutoutOps", () => {
|
||||
it("swaps the src of an image and nothing else", () => {
|
||||
expect(cutoutOps({ isVideo: false, cutoutSrc: "assets/cut.png" })).toEqual([
|
||||
html("src", "assets/cut.png"),
|
||||
]);
|
||||
});
|
||||
|
||||
it("swaps a silent video's src and keeps it muted without a has-audio marker", () => {
|
||||
expect(cutoutOps({ isVideo: true, cutoutSrc: "assets/cut.webm" })).toEqual([
|
||||
html("src", "assets/cut.webm"),
|
||||
html("muted", "true"),
|
||||
attr("has-audio", null),
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("mintLinkId", () => {
|
||||
it("dedupes against element ids and existing links", () => {
|
||||
const doc = document.implementation.createHTMLDocument("t");
|
||||
expect(mintLinkId(doc)).toBe("link");
|
||||
doc.body.innerHTML = '<div id="link"></div><video data-link="link-2"></video>';
|
||||
expect(mintLinkId(doc)).toBe("link-3");
|
||||
});
|
||||
});
|
||||
|
||||
function makeVideo(attrs: Record<string, string>): HTMLVideoElement {
|
||||
const doc = document.implementation.createHTMLDocument("t");
|
||||
const video = doc.createElement("video");
|
||||
for (const [name, value] of Object.entries(attrs)) video.setAttribute(name, value);
|
||||
doc.body.append(video);
|
||||
return video;
|
||||
}
|
||||
|
||||
function applyEdit(source: string, edit: ReturnType<typeof buildKeepSoundCutoutEdit>): string {
|
||||
const target = { id: "clip" };
|
||||
const patched = edit.ops.reduce((acc, op) => applyPatchByTarget(acc, target, op), source);
|
||||
return edit.prepareContent(patched);
|
||||
}
|
||||
|
||||
function parse(source: string): Document {
|
||||
return new DOMParser().parseFromString(source, "text/html");
|
||||
}
|
||||
|
||||
describe("buildKeepSoundCutoutEdit", () => {
|
||||
const automation = JSON.stringify({
|
||||
version: 1,
|
||||
lanes: [
|
||||
{ target: "volume", points: [{ t: 0, v: 1 }] },
|
||||
{ target: "rate", points: [{ t: 0, v: 2 }] },
|
||||
],
|
||||
});
|
||||
|
||||
const source = [
|
||||
'<div data-composition-id="main" data-duration="10">',
|
||||
' <video id="clip" class="clip" src="assets/talk.mp4" data-start="1" data-duration="4" data-media-start="2" data-track-index="0" data-has-audio="true" data-volume="0.5" data-fade-in="0.3" data-audio-group="voiceover"></video>',
|
||||
' <audio id="music" class="clip" src="assets/bgm.mp3" data-start="0" data-duration="10" data-track-index="1"></audio>',
|
||||
"</div>",
|
||||
].join("\n");
|
||||
|
||||
const liveVideo = () =>
|
||||
makeVideo({
|
||||
id: "clip",
|
||||
src: "assets/talk.mp4",
|
||||
"data-has-audio": "true",
|
||||
"data-volume": "0.5",
|
||||
"data-fade-in": "0.3",
|
||||
"data-audio-group": "voiceover",
|
||||
"data-automation": automation,
|
||||
});
|
||||
|
||||
it("mutes the video onto the cutout and moves its sound to a linked sibling audio", () => {
|
||||
const edit = buildKeepSoundCutoutEdit({
|
||||
video: liveVideo(),
|
||||
videoId: "clip",
|
||||
target: { id: "clip" },
|
||||
cutoutSrc: "assets/talk-cutout.webm",
|
||||
});
|
||||
const doc = parse(applyEdit(source, edit));
|
||||
const video = doc.getElementById("clip");
|
||||
const audio = doc.getElementById("clip-audio");
|
||||
if (!video || !audio) throw new Error("expected both clips");
|
||||
|
||||
expect(video.getAttribute("src")).toBe("assets/talk-cutout.webm");
|
||||
expect(video.hasAttribute("muted")).toBe(true);
|
||||
expect(video.hasAttribute("data-has-audio")).toBe(false);
|
||||
expect(video.hasAttribute("data-volume")).toBe(false);
|
||||
expect(video.hasAttribute("data-fade-in")).toBe(false);
|
||||
expect(video.hasAttribute("data-audio-group")).toBe(false);
|
||||
expect(JSON.parse(video.getAttribute("data-automation") ?? "").lanes).toEqual([
|
||||
{ target: "rate", points: [{ t: 0, v: 2 }] },
|
||||
]);
|
||||
|
||||
expect(audio.tagName).toBe("AUDIO");
|
||||
expect(audio.getAttribute("src")).toBe("assets/talk.mp4");
|
||||
expect(audio.getAttribute("data-start")).toBe("1");
|
||||
expect(audio.getAttribute("data-duration")).toBe("4");
|
||||
expect(audio.getAttribute("data-media-start")).toBe("2");
|
||||
expect(audio.hasAttribute("data-playback-rate")).toBe(false);
|
||||
expect(audio.getAttribute("data-track-index")).toBe("2");
|
||||
expect(audio.getAttribute("data-volume")).toBe("0.5");
|
||||
expect(audio.getAttribute("data-fade-in")).toBe("0.3");
|
||||
expect(audio.getAttribute("data-audio-group")).toBe("voiceover");
|
||||
expect(JSON.parse(audio.getAttribute("data-automation") ?? "").lanes).toEqual([
|
||||
{ target: "volume", points: [{ t: 0, v: 1 }] },
|
||||
{ target: "rate", points: [{ t: 0, v: 2 }] },
|
||||
]);
|
||||
expect(audio.getAttribute("data-link")).toBe(video.getAttribute("data-link"));
|
||||
expect(video.getAttribute("data-link")).toBe("link");
|
||||
expect(audio.parentElement).toBe(video.parentElement);
|
||||
});
|
||||
|
||||
it("drops the video's automation attribute when it held no rate lane", () => {
|
||||
const video = makeVideo({
|
||||
id: "clip",
|
||||
src: "assets/talk.mp4",
|
||||
"data-automation": JSON.stringify({
|
||||
version: 1,
|
||||
lanes: [{ target: "volume", points: [{ t: 0, v: 1 }] }],
|
||||
}),
|
||||
});
|
||||
const edit = buildKeepSoundCutoutEdit({
|
||||
video,
|
||||
videoId: "clip",
|
||||
target: { id: "clip" },
|
||||
cutoutSrc: "assets/cut.webm",
|
||||
});
|
||||
expect(edit.ops).toContainEqual(attr("automation", null));
|
||||
});
|
||||
|
||||
it("suffixes the audio id when <video-id>-audio is taken", () => {
|
||||
const edit = buildKeepSoundCutoutEdit({
|
||||
video: liveVideo(),
|
||||
videoId: "clip",
|
||||
target: { id: "clip" },
|
||||
cutoutSrc: "assets/cut.webm",
|
||||
});
|
||||
const taken = source.replace('id="music"', 'id="clip-audio"');
|
||||
const doc = parse(applyEdit(taken, edit));
|
||||
expect(doc.getElementById("clip-audio-2")?.getAttribute("src")).toBe("assets/talk.mp4");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,253 @@
|
||||
import {
|
||||
HF_AUDIO_AUTOMATION_ATTR,
|
||||
RATE_TARGET,
|
||||
parseAutomation,
|
||||
serializeAutomation,
|
||||
type HfAutomationLane,
|
||||
} from "@hyperframes/core/audio-automation";
|
||||
import { HF_AUDIO_CARVE_ATTR } from "@hyperframes/core/audio-carve";
|
||||
import { HF_AUDIO_FADE_IN_ATTR, HF_AUDIO_FADE_OUT_ATTR } from "@hyperframes/core/audio-fade";
|
||||
import { HF_AUDIO_FX_ATTR } from "@hyperframes/core/audio-fx";
|
||||
import { HF_AUDIO_GROUP_ATTR } from "@hyperframes/core/audio-groups";
|
||||
import {
|
||||
escapeHtmlAttribute,
|
||||
findTagByTarget,
|
||||
readAttributeByTarget,
|
||||
type PatchOperation,
|
||||
type PatchTarget,
|
||||
} from "../../utils/sourcePatcher";
|
||||
import { collectHtmlIds, resolveAssetHasAudio } from "../../utils/studioHelpers";
|
||||
import type { CommitDomAttributeBatch } from "../../hooks/domEditCommitTypes";
|
||||
import { buildDomEditPatchTarget, type DomEditSelection } from "./domEditing";
|
||||
import { generateId } from "../../utils/generateId";
|
||||
|
||||
const MEDIA_LINK_ATTR = "data-link";
|
||||
const SOUND_KEPT_ON_LINKED_AUDIO_STAGE = "Background removed. Sound kept on a linked audio track.";
|
||||
|
||||
const MOVED_SOUND_ATTRS = [
|
||||
"data-volume",
|
||||
HF_AUDIO_FADE_IN_ATTR,
|
||||
HF_AUDIO_FADE_OUT_ATTR,
|
||||
HF_AUDIO_FX_ATTR,
|
||||
HF_AUDIO_CARVE_ATTR,
|
||||
HF_AUDIO_GROUP_ATTR,
|
||||
];
|
||||
|
||||
const COPIED_TIMING_ATTRS = [
|
||||
"start",
|
||||
"end",
|
||||
"duration",
|
||||
"media-start",
|
||||
"playback-start",
|
||||
"playback-rate",
|
||||
"source-duration",
|
||||
];
|
||||
|
||||
const htmlAttr = (property: string, value: string | null): PatchOperation => ({
|
||||
type: "html-attribute",
|
||||
property,
|
||||
value,
|
||||
});
|
||||
|
||||
const dataAttr = (property: string, value: string | null): PatchOperation => ({
|
||||
type: "attribute",
|
||||
property,
|
||||
value,
|
||||
});
|
||||
|
||||
function stripDataPrefix(name: string): string {
|
||||
return name.startsWith("data-") ? name.slice("data-".length) : name;
|
||||
}
|
||||
|
||||
function probedHasAudioValue(probedHasAudio: boolean | null): string | null {
|
||||
if (probedHasAudio === null) return null;
|
||||
return probedHasAudio ? "true" : "false";
|
||||
}
|
||||
|
||||
export function mutedToggleOps(input: {
|
||||
isVideo: boolean;
|
||||
nextMuted: boolean;
|
||||
probedHasAudio: boolean | null;
|
||||
}): PatchOperation[] {
|
||||
const muted = htmlAttr("muted", input.nextMuted ? "true" : null);
|
||||
if (!input.isVideo) return [muted];
|
||||
const hasAudio = input.nextMuted ? null : probedHasAudioValue(input.probedHasAudio);
|
||||
return [muted, dataAttr("has-audio", hasAudio)];
|
||||
}
|
||||
|
||||
export function hasAudioToggleOps(next: boolean): PatchOperation[] {
|
||||
return next
|
||||
? [dataAttr("has-audio", "true"), htmlAttr("muted", null)]
|
||||
: [dataAttr("has-audio", null), htmlAttr("muted", "true")];
|
||||
}
|
||||
|
||||
export function cutoutOps(input: { isVideo: boolean; cutoutSrc: string }): PatchOperation[] {
|
||||
const src = htmlAttr("src", input.cutoutSrc);
|
||||
return input.isVideo ? [src, htmlAttr("muted", "true"), dataAttr("has-audio", null)] : [src];
|
||||
}
|
||||
|
||||
function firstFreeName(base: string, taken: Set<string>): string {
|
||||
if (!taken.has(base)) return base;
|
||||
let n = 2;
|
||||
while (taken.has(`${base}-${n}`)) n += 1;
|
||||
return `${base}-${n}`;
|
||||
}
|
||||
|
||||
export function mintLinkId(doc: Document): string {
|
||||
const taken = new Set<string>();
|
||||
for (const el of Array.from(doc.querySelectorAll(`[id], [${MEDIA_LINK_ATTR}]`))) {
|
||||
if (el.id) taken.add(el.id);
|
||||
const link = el.getAttribute(MEDIA_LINK_ATTR);
|
||||
if (link) taken.add(link);
|
||||
}
|
||||
return firstFreeName("link", taken);
|
||||
}
|
||||
|
||||
function firstFreeTrackIndex(source: string): number {
|
||||
const used = new Set(
|
||||
Array.from(source.matchAll(/\bdata-track-index=["'](-?\d+)["']/g), (m) => Number(m[1])),
|
||||
);
|
||||
let track = 0;
|
||||
while (used.has(track)) track += 1;
|
||||
return track;
|
||||
}
|
||||
|
||||
function splitAutomation(raw: string): { videoKeeps: string | null; audioTakes: string } {
|
||||
let lanes: HfAutomationLane[];
|
||||
try {
|
||||
lanes = parseAutomation(raw).lanes;
|
||||
} catch {
|
||||
return { videoKeeps: null, audioTakes: raw };
|
||||
}
|
||||
const rateLanes = lanes.filter((lane) => lane.target === RATE_TARGET);
|
||||
const soundLanes = lanes.filter((lane) => lane.target !== RATE_TARGET);
|
||||
return {
|
||||
videoKeeps: rateLanes.length > 0 ? serializeAutomation({ version: 1, lanes: rateLanes }) : null,
|
||||
audioTakes: serializeAutomation({ version: 1, lanes: [...soundLanes, ...rateLanes] }),
|
||||
};
|
||||
}
|
||||
|
||||
function formatAttrs(attrs: Array<[string, string]>): string {
|
||||
return attrs.map(([name, value]) => `${name}="${escapeHtmlAttribute(value)}"`).join(" ");
|
||||
}
|
||||
|
||||
function insertBeforeTarget(source: string, target: PatchTarget, markup: string): string {
|
||||
const match = findTagByTarget(source, target);
|
||||
if (!match) return source;
|
||||
const lineStart = source.lastIndexOf("\n", match.start) + 1;
|
||||
const indent = /^[ \t]*$/.test(source.slice(lineStart, match.start))
|
||||
? source.slice(lineStart, match.start)
|
||||
: "";
|
||||
const separator = indent ? `\n${indent}` : "";
|
||||
return `${source.slice(0, match.start)}${markup}${separator}${source.slice(match.start)}`;
|
||||
}
|
||||
|
||||
export interface KeepSoundCutoutEdit {
|
||||
ops: PatchOperation[];
|
||||
prepareContent: (source: string) => string;
|
||||
}
|
||||
|
||||
export function buildKeepSoundCutoutEdit(input: {
|
||||
video: Element;
|
||||
videoId: string | null;
|
||||
target: PatchTarget;
|
||||
cutoutSrc: string;
|
||||
}): KeepSoundCutoutEdit {
|
||||
const { video, target } = input;
|
||||
const linkId = mintLinkId(video.ownerDocument);
|
||||
const originalSrc = video.getAttribute("src") ?? "";
|
||||
|
||||
const movedSound: Array<[string, string]> = [];
|
||||
const ops: PatchOperation[] = [...cutoutOps({ isVideo: true, cutoutSrc: input.cutoutSrc })];
|
||||
for (const name of MOVED_SOUND_ATTRS) {
|
||||
const value = video.getAttribute(name);
|
||||
if (value === null) continue;
|
||||
movedSound.push([name, value]);
|
||||
ops.push(dataAttr(stripDataPrefix(name), null));
|
||||
}
|
||||
const automation = video.getAttribute(HF_AUDIO_AUTOMATION_ATTR);
|
||||
if (automation !== null) {
|
||||
const { videoKeeps, audioTakes } = splitAutomation(automation);
|
||||
movedSound.push([HF_AUDIO_AUTOMATION_ATTR, audioTakes]);
|
||||
ops.push(dataAttr(stripDataPrefix(HF_AUDIO_AUTOMATION_ATTR), videoKeeps));
|
||||
}
|
||||
ops.push(dataAttr(stripDataPrefix(MEDIA_LINK_ATTR), linkId));
|
||||
|
||||
const prepareContent = (source: string): string => {
|
||||
const audioId = firstFreeName(
|
||||
`${input.videoId || "video"}-audio`,
|
||||
new Set(collectHtmlIds(source)),
|
||||
);
|
||||
const timing: Array<[string, string]> = [];
|
||||
for (const name of COPIED_TIMING_ATTRS) {
|
||||
const value = readAttributeByTarget(source, target, name);
|
||||
if (value !== undefined) timing.push([`data-${name}`, value]);
|
||||
}
|
||||
const attrs: Array<[string, string]> = [
|
||||
["id", audioId],
|
||||
["data-hf-id", `hf-${generateId()}`],
|
||||
["class", "clip"],
|
||||
["src", originalSrc],
|
||||
...timing,
|
||||
["data-track-index", String(firstFreeTrackIndex(source))],
|
||||
[MEDIA_LINK_ATTR, linkId],
|
||||
...movedSound,
|
||||
];
|
||||
return insertBeforeTarget(source, target, `<audio ${formatAttrs(attrs)}></audio>`);
|
||||
};
|
||||
|
||||
return { ops, prepareContent };
|
||||
}
|
||||
|
||||
export interface MediaEditContext {
|
||||
element: DomEditSelection;
|
||||
projectId: string | null;
|
||||
projectSrc: string;
|
||||
commit: CommitDomAttributeBatch;
|
||||
}
|
||||
|
||||
const isVideoSelection = (element: DomEditSelection) => element.tagName === "video";
|
||||
|
||||
export async function commitMutedToggle(ctx: MediaEditContext, nextMuted: boolean): Promise<void> {
|
||||
const isVideo = isVideoSelection(ctx.element);
|
||||
const probedHasAudio =
|
||||
isVideo && !nextMuted && ctx.projectId && ctx.projectSrc
|
||||
? await resolveAssetHasAudio(ctx.projectId, ctx.projectSrc)
|
||||
: null;
|
||||
await ctx.commit(ctx.element, mutedToggleOps({ isVideo, nextMuted, probedHasAudio }), {
|
||||
label: "Edit muted",
|
||||
});
|
||||
}
|
||||
|
||||
export async function commitHasAudioToggle(ctx: MediaEditContext, next: boolean): Promise<void> {
|
||||
await ctx.commit(ctx.element, hasAudioToggleOps(next), { label: "Edit has audio" });
|
||||
}
|
||||
|
||||
export async function commitCutout(
|
||||
ctx: MediaEditContext,
|
||||
cutoutSrc: string,
|
||||
keepSound: boolean,
|
||||
): Promise<string> {
|
||||
const isVideo = isVideoSelection(ctx.element);
|
||||
const label = "Remove background";
|
||||
const keep = isVideo && keepSound;
|
||||
const landed = keep
|
||||
? await commitKeepSoundCutout(ctx, cutoutSrc, label)
|
||||
: await ctx.commit(ctx.element, cutoutOps({ isVideo, cutoutSrc }), { label });
|
||||
if (!landed) throw new Error("Couldn't apply the cutout");
|
||||
return keep ? SOUND_KEPT_ON_LINKED_AUDIO_STAGE : "Applied cutout";
|
||||
}
|
||||
|
||||
function commitKeepSoundCutout(
|
||||
ctx: MediaEditContext,
|
||||
cutoutSrc: string,
|
||||
label: string,
|
||||
): Promise<boolean> {
|
||||
const edit = buildKeepSoundCutoutEdit({
|
||||
video: ctx.element.element,
|
||||
videoId: ctx.element.id ?? null,
|
||||
target: buildDomEditPatchTarget(ctx.element),
|
||||
cutoutSrc,
|
||||
});
|
||||
return ctx.commit(ctx.element, edit.ops, { label, prepareContent: edit.prepareContent });
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { FlatMediaSection } from "./propertyPanelFlatMediaSection";
|
||||
import { MediaSection } from "./propertyPanelMediaSection";
|
||||
import type { DomEditSelection } from "./domEditing";
|
||||
import type { PatchOperation } from "../../utils/sourcePatcher";
|
||||
import { formatTimingValue } from "./propertyPanelHelpers";
|
||||
import { readMediaOffsetSeconds } from "@hyperframes/parsers/media-duration";
|
||||
|
||||
@@ -62,6 +63,7 @@ function renderSection(overrides: Partial<DomEditSelection> = {}) {
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={vi.fn()}
|
||||
onSetHtmlAttribute={vi.fn()}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
@@ -102,6 +104,7 @@ describe("FlatMediaSection — cutout", () => {
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={onSetHtmlAttribute}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
onRemoveBackground={onRemoveBackground}
|
||||
/>,
|
||||
);
|
||||
@@ -149,6 +152,7 @@ describe("FlatMediaSection — volume/rate/media-start", () => {
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={vi.fn()}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
@@ -174,6 +178,7 @@ describe("FlatMediaSection — volume/rate/media-start", () => {
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={vi.fn()}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
@@ -208,6 +213,7 @@ describe("FlatMediaSection — volume/rate/media-start", () => {
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={vi.fn()}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
volumeAutomated
|
||||
onCommitVolumeAt={onCommitVolumeAt}
|
||||
/>,
|
||||
@@ -242,6 +248,7 @@ describe("FlatMediaSection — volume/rate/media-start", () => {
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={vi.fn()}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
@@ -273,6 +280,7 @@ describe("FlatMediaSection — volume/rate/media-start", () => {
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={vi.fn()}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
@@ -308,6 +316,7 @@ describe("FlatMediaSection — loop/muted/has-audio", () => {
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={onSetHtmlAttribute}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
@@ -323,92 +332,6 @@ describe("FlatMediaSection — loop/muted/has-audio", () => {
|
||||
expect(hasAudioToggle?.getAttribute("aria-checked")).toBe("true");
|
||||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it("toggles muted via onSetHtmlAttribute", () => {
|
||||
const onSetHtmlAttribute = vi.fn();
|
||||
const onSetAttribute = vi.fn();
|
||||
const element = makeVideoElement();
|
||||
const host = document.createElement("div");
|
||||
document.body.append(host);
|
||||
const root = createRoot(host);
|
||||
act(() => {
|
||||
root.render(
|
||||
<FlatMediaSection
|
||||
projectDir={null}
|
||||
element={element}
|
||||
styles={{}}
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={onSetHtmlAttribute}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
const mutedToggle = host.querySelector<HTMLButtonElement>(
|
||||
'[data-flat-toggle="true"][aria-label="Muted"]',
|
||||
);
|
||||
expect(mutedToggle?.getAttribute("aria-checked")).toBe("false");
|
||||
act(() => mutedToggle?.dispatchEvent(new MouseEvent("click", { bubbles: true })));
|
||||
expect(onSetHtmlAttribute).toHaveBeenCalledWith("muted", "true");
|
||||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it("enables has-audio-track and clears muted on click", () => {
|
||||
const onSetHtmlAttribute = vi.fn();
|
||||
const onSetAttribute = vi.fn();
|
||||
const element = makeVideoElement();
|
||||
const host = document.createElement("div");
|
||||
document.body.append(host);
|
||||
const root = createRoot(host);
|
||||
act(() => {
|
||||
root.render(
|
||||
<FlatMediaSection
|
||||
projectDir={null}
|
||||
element={element}
|
||||
styles={{}}
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={onSetHtmlAttribute}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
const hasAudioToggle = host.querySelector<HTMLButtonElement>(
|
||||
'[data-flat-toggle="true"][aria-label="Has audio track"]',
|
||||
);
|
||||
expect(hasAudioToggle?.getAttribute("aria-checked")).toBe("false");
|
||||
act(() => hasAudioToggle?.dispatchEvent(new MouseEvent("click", { bubbles: true })));
|
||||
expect(onSetAttribute).toHaveBeenCalledWith("has-audio", "true");
|
||||
expect(onSetHtmlAttribute).toHaveBeenCalledWith("muted", null);
|
||||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it("disables has-audio-track and sets muted on click", () => {
|
||||
const onSetHtmlAttribute = vi.fn();
|
||||
const onSetAttribute = vi.fn();
|
||||
const element = makeVideoElement({ dataAttributes: { "has-audio": "true" } });
|
||||
const host = document.createElement("div");
|
||||
document.body.append(host);
|
||||
const root = createRoot(host);
|
||||
act(() => {
|
||||
root.render(
|
||||
<FlatMediaSection
|
||||
projectDir={null}
|
||||
element={element}
|
||||
styles={{}}
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={onSetHtmlAttribute}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
const hasAudioToggle = host.querySelector<HTMLButtonElement>(
|
||||
'[data-flat-toggle="true"][aria-label="Has audio track"]',
|
||||
);
|
||||
expect(hasAudioToggle?.getAttribute("aria-checked")).toBe("true");
|
||||
act(() => hasAudioToggle?.dispatchEvent(new MouseEvent("click", { bubbles: true })));
|
||||
expect(onSetAttribute).toHaveBeenCalledWith("has-audio", "");
|
||||
expect(onSetHtmlAttribute).toHaveBeenCalledWith("muted", "true");
|
||||
act(() => root.unmount());
|
||||
});
|
||||
});
|
||||
|
||||
describe("FlatMediaSection — fit/position", () => {
|
||||
@@ -428,6 +351,7 @@ describe("FlatMediaSection — fit/position", () => {
|
||||
onSetStyle={onSetStyle}
|
||||
onSetAttribute={vi.fn()}
|
||||
onSetHtmlAttribute={vi.fn()}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
@@ -461,6 +385,7 @@ describe("FlatMediaSection — fit/position", () => {
|
||||
onSetStyle={onSetStyle}
|
||||
onSetAttribute={vi.fn()}
|
||||
onSetHtmlAttribute={vi.fn()}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
@@ -504,6 +429,7 @@ function renderWithRate(element: DomEditSelection, onSetAttribute = vi.fn()) {
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={vi.fn()}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
rate={{
|
||||
automated: false,
|
||||
automatedValue: undefined,
|
||||
@@ -543,7 +469,9 @@ describe("FlatMediaSection — audio clips", () => {
|
||||
);
|
||||
act(() => audio.root.unmount());
|
||||
|
||||
const silentVideo = renderWithRate(makeVideoElement({ dataAttributes: { duration: "10" } }));
|
||||
const mutedSelection = makeVideoElement({ dataAttributes: { duration: "10" } });
|
||||
mutedSelection.element.setAttribute("muted", "");
|
||||
const silentVideo = renderWithRate(mutedSelection);
|
||||
expect(labelsOf(silentVideo.host)).not.toContain("Fade in");
|
||||
act(() => silentVideo.root.unmount());
|
||||
|
||||
@@ -659,6 +587,7 @@ describe("FlatMediaSection — audio clips", () => {
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={vi.fn()}
|
||||
onSetAttributeBatch={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
@@ -708,3 +637,273 @@ function typeInto(input: HTMLInputElement, value: string) {
|
||||
setter?.call(input, value);
|
||||
input.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
}
|
||||
|
||||
type PanelKind = "flat" | "design";
|
||||
|
||||
type BatchOptions = { label: string; prepareContent?: (html: string) => string };
|
||||
|
||||
function makeHandlers() {
|
||||
return {
|
||||
onSetAttribute: vi.fn(),
|
||||
onSetHtmlAttribute: vi.fn(),
|
||||
onSetAttributeBatch: vi.fn<
|
||||
(
|
||||
selection: DomEditSelection,
|
||||
ops: PatchOperation[],
|
||||
options: BatchOptions,
|
||||
) => Promise<boolean>
|
||||
>(async () => true),
|
||||
};
|
||||
}
|
||||
|
||||
function mountPanel(
|
||||
kind: PanelKind,
|
||||
element: DomEditSelection,
|
||||
handlers: ReturnType<typeof makeHandlers> & {
|
||||
onRemoveBackground?: () => Promise<{ outputPath: string }>;
|
||||
},
|
||||
) {
|
||||
const host = document.createElement("div");
|
||||
document.body.append(host);
|
||||
const root = createRoot(host);
|
||||
const Panel = kind === "flat" ? FlatMediaSection : MediaSection;
|
||||
act(() => {
|
||||
root.render(
|
||||
<Panel
|
||||
projectId="p1"
|
||||
projectDir={null}
|
||||
element={element}
|
||||
styles={{}}
|
||||
onSetStyle={vi.fn()}
|
||||
{...handlers}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
return { host, root };
|
||||
}
|
||||
|
||||
async function flush() {
|
||||
for (let i = 0; i < 8; i++) await Promise.resolve();
|
||||
}
|
||||
|
||||
async function clickToggle(kind: PanelKind, host: HTMLElement, label: string, option: string) {
|
||||
if (kind === "flat") {
|
||||
const toggle = host.querySelector<HTMLButtonElement>(
|
||||
`[data-flat-toggle="true"][aria-label="${label}"]`,
|
||||
);
|
||||
if (!toggle) throw new Error(`expected ${label} toggle`);
|
||||
await act(async () => {
|
||||
toggle.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
||||
await flush();
|
||||
});
|
||||
return;
|
||||
}
|
||||
const row = Array.from(host.querySelectorAll("span")).find(
|
||||
(n) => n.textContent === label,
|
||||
)?.parentElement;
|
||||
const button = Array.from(row?.querySelectorAll("button") ?? []).find(
|
||||
(b) => b.textContent === option,
|
||||
);
|
||||
if (!button) throw new Error(`expected ${label} segmented button`);
|
||||
await act(async () => {
|
||||
button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
||||
await flush();
|
||||
});
|
||||
}
|
||||
|
||||
function stubHasAudioProbe(hasAudio: boolean | "fail") {
|
||||
const fetchMock = vi.fn(async (_input: Parameters<typeof fetch>[0]) =>
|
||||
hasAudio === "fail"
|
||||
? new Response("nope", { status: 500 })
|
||||
: new Response(JSON.stringify({ metadata: { hasAudio } })),
|
||||
);
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
return fetchMock;
|
||||
}
|
||||
|
||||
const htmlOp = (property: string, value: string | null) => ({
|
||||
type: "html-attribute",
|
||||
property,
|
||||
value,
|
||||
});
|
||||
const dataOp = (property: string, value: string | null) => ({
|
||||
type: "attribute",
|
||||
property,
|
||||
value,
|
||||
});
|
||||
|
||||
describe.each<PanelKind>(["flat", "design"])("%s panel — Muted is one atomic edit", (kind) => {
|
||||
afterEach(() => vi.unstubAllGlobals());
|
||||
|
||||
it("muting an audible video sets muted and removes has-audio in ONE write", async () => {
|
||||
const handlers = makeHandlers();
|
||||
const element = makeVideoElement({ dataAttributes: { "has-audio": "true" } });
|
||||
const { host, root } = mountPanel(kind, element, handlers);
|
||||
await clickToggle(kind, host, "Muted", "On");
|
||||
expect(handlers.onSetAttributeBatch).toHaveBeenCalledTimes(1);
|
||||
expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
|
||||
htmlOp("muted", "true"),
|
||||
dataOp("has-audio", null),
|
||||
]);
|
||||
expect(handlers.onSetAttribute).not.toHaveBeenCalled();
|
||||
expect(handlers.onSetHtmlAttribute).not.toHaveBeenCalled();
|
||||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it("unmuting a video the probe hears stamps has-audio=true in the same write", async () => {
|
||||
const probe = stubHasAudioProbe(true);
|
||||
const handlers = makeHandlers();
|
||||
const element = makeVideoElement();
|
||||
element.element.setAttribute("muted", "");
|
||||
const { host, root } = mountPanel(kind, element, handlers);
|
||||
await clickToggle(kind, host, "Muted", "Off");
|
||||
expect(String(probe.mock.calls[0]?.[0])).toContain("assets%2Fintro-loop.mp4");
|
||||
expect(handlers.onSetAttributeBatch).toHaveBeenCalledTimes(1);
|
||||
expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
|
||||
htmlOp("muted", null),
|
||||
dataOp("has-audio", "true"),
|
||||
]);
|
||||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it("unmuting a silent-probe video writes has-audio=false", async () => {
|
||||
stubHasAudioProbe(false);
|
||||
const handlers = makeHandlers();
|
||||
const element = makeVideoElement();
|
||||
element.element.setAttribute("muted", "");
|
||||
const { host, root } = mountPanel(kind, element, handlers);
|
||||
await clickToggle(kind, host, "Muted", "Off");
|
||||
expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
|
||||
htmlOp("muted", null),
|
||||
dataOp("has-audio", "false"),
|
||||
]);
|
||||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it("unmuting when the probe fails leaves has-audio absent", async () => {
|
||||
stubHasAudioProbe("fail");
|
||||
const handlers = makeHandlers();
|
||||
const element = makeVideoElement();
|
||||
element.element.setAttribute("muted", "");
|
||||
const { host, root } = mountPanel(kind, element, handlers);
|
||||
await clickToggle(kind, host, "Muted", "Off");
|
||||
expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
|
||||
htmlOp("muted", null),
|
||||
dataOp("has-audio", null),
|
||||
]);
|
||||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it("audio Muted toggles only the muted attribute", async () => {
|
||||
const handlers = makeHandlers();
|
||||
const { host, root } = mountPanel(kind, makeAudioElement({}), handlers);
|
||||
await clickToggle(kind, host, "Muted", "On");
|
||||
expect(handlers.onSetAttributeBatch).toHaveBeenCalledTimes(1);
|
||||
expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([htmlOp("muted", "true")]);
|
||||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it("Has audio track No removes has-audio and mutes in ONE write", async () => {
|
||||
const handlers = makeHandlers();
|
||||
const element = makeVideoElement({ dataAttributes: { "has-audio": "true" } });
|
||||
const { host, root } = mountPanel(kind, element, handlers);
|
||||
await clickToggle(kind, host, "Has audio track", "No");
|
||||
expect(handlers.onSetAttributeBatch).toHaveBeenCalledTimes(1);
|
||||
expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
|
||||
dataOp("has-audio", null),
|
||||
htmlOp("muted", "true"),
|
||||
]);
|
||||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it("Has audio track Yes stamps has-audio and clears muted in ONE write", async () => {
|
||||
const handlers = makeHandlers();
|
||||
const element = makeVideoElement();
|
||||
element.element.setAttribute("muted", "");
|
||||
const { host, root } = mountPanel(kind, element, handlers);
|
||||
await clickToggle(kind, host, "Has audio track", "Yes");
|
||||
expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
|
||||
dataOp("has-audio", "true"),
|
||||
htmlOp("muted", null),
|
||||
]);
|
||||
act(() => root.unmount());
|
||||
});
|
||||
});
|
||||
|
||||
describe.each<PanelKind>(["flat", "design"])("%s panel — cutout keeps the sound", (kind) => {
|
||||
async function runCutout(element: DomEditSelection) {
|
||||
const handlers = makeHandlers();
|
||||
const onRemoveBackground = vi.fn().mockResolvedValue({ outputPath: "assets/cut.webm" });
|
||||
const confirm = vi.spyOn(window, "confirm");
|
||||
const { host, root } = mountPanel(kind, element, { ...handlers, onRemoveBackground });
|
||||
const button =
|
||||
kind === "flat"
|
||||
? host.querySelector<HTMLButtonElement>('[data-flat-media-remove-bg="true"]')
|
||||
: Array.from(host.querySelectorAll("button")).find((b) =>
|
||||
/remove bg/i.test(b.textContent ?? ""),
|
||||
);
|
||||
if (!button) throw new Error("expected remove background button");
|
||||
await act(async () => {
|
||||
button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
||||
await flush();
|
||||
});
|
||||
const text = host.textContent ?? "";
|
||||
act(() => root.unmount());
|
||||
return { ...handlers, confirm, text };
|
||||
}
|
||||
|
||||
afterEach(() => vi.restoreAllMocks());
|
||||
|
||||
it("audible video: one write mutes onto the cutout and inserts a linked sibling audio", async () => {
|
||||
const element = makeVideoElement({
|
||||
dataAttributes: { "has-audio": "true", volume: "0.5" },
|
||||
});
|
||||
element.element.setAttribute("id", "s1-bg");
|
||||
element.element.setAttribute("data-has-audio", "true");
|
||||
element.element.setAttribute("data-volume", "0.5");
|
||||
const { onSetAttributeBatch, onSetAttribute, onSetHtmlAttribute, confirm, text } =
|
||||
await runCutout(element);
|
||||
|
||||
expect(confirm).not.toHaveBeenCalled();
|
||||
expect(onSetAttribute).not.toHaveBeenCalled();
|
||||
expect(onSetHtmlAttribute).not.toHaveBeenCalled();
|
||||
expect(onSetAttributeBatch).toHaveBeenCalledTimes(1);
|
||||
const [selection, ops, options] = onSetAttributeBatch.mock.calls[0] ?? [];
|
||||
expect(selection).toBe(element);
|
||||
expect(ops).toEqual([
|
||||
htmlOp("src", "assets/cut.webm"),
|
||||
htmlOp("muted", "true"),
|
||||
dataOp("has-audio", null),
|
||||
dataOp("volume", null),
|
||||
dataOp("link", "link"),
|
||||
]);
|
||||
const source =
|
||||
'<div data-composition-id="main"><video id="s1-bg" src="assets/cut.webm" muted data-start="2" data-duration="3" data-track-index="0" data-link="link"></video></div>';
|
||||
const prepared = options?.prepareContent?.(source) ?? "";
|
||||
const audio = new DOMParser()
|
||||
.parseFromString(prepared, "text/html")
|
||||
.getElementById("s1-bg-audio");
|
||||
expect(audio?.getAttribute("src")).toBe("assets/intro-loop.mp4");
|
||||
expect(audio?.getAttribute("data-start")).toBe("2");
|
||||
expect(audio?.getAttribute("data-duration")).toBe("3");
|
||||
expect(audio?.getAttribute("data-link")).toBe("link");
|
||||
expect(audio?.getAttribute("data-volume")).toBe("0.5");
|
||||
expect(text).toContain("Background removed. Sound kept on a linked audio track.");
|
||||
});
|
||||
|
||||
it("silent video: one write swaps the src and inserts no audio", async () => {
|
||||
const silent = makeVideoElement();
|
||||
silent.element.setAttribute("muted", "");
|
||||
const { onSetAttributeBatch, confirm, text } = await runCutout(silent);
|
||||
expect(confirm).not.toHaveBeenCalled();
|
||||
expect(onSetAttributeBatch).toHaveBeenCalledTimes(1);
|
||||
const [, ops, options] = onSetAttributeBatch.mock.calls[0] ?? [];
|
||||
expect(ops).toEqual([
|
||||
htmlOp("src", "assets/cut.webm"),
|
||||
htmlOp("muted", "true"),
|
||||
dataOp("has-audio", null),
|
||||
]);
|
||||
expect(options?.prepareContent).toBeUndefined();
|
||||
expect(text).toContain("Applied cutout");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { useEffect, useState } from "react";
|
||||
import { isVideoAudible } from "../../player/lib/timelineElementHelpers";
|
||||
import { useTrackDesignInput } from "../../contexts/DesignPanelInputContext";
|
||||
import { Check, ClipboardList } from "../../icons/SystemIcons";
|
||||
import type { DomEditSelection } from "./domEditing";
|
||||
@@ -34,15 +35,19 @@ import {
|
||||
readFadeSeconds,
|
||||
} from "@hyperframes/core/audio-fade";
|
||||
import { parseGainInput, parseRateInput, parseSecondsInput } from "./audioInspectorInput";
|
||||
import type { CommitDomAttributeBatch } from "../../hooks/domEditCommitTypes";
|
||||
import { commitCutout, commitHasAudioToggle, commitMutedToggle } from "./mediaAudioEdits";
|
||||
|
||||
// fallow-ignore-next-line complexity
|
||||
export function FlatMediaSection({
|
||||
projectId = null,
|
||||
projectDir,
|
||||
element,
|
||||
styles,
|
||||
onSetStyle,
|
||||
onSetAttribute,
|
||||
onSetHtmlAttribute,
|
||||
onSetAttributeBatch,
|
||||
onRemoveBackground,
|
||||
volumeAutomated,
|
||||
onAutomateVolume,
|
||||
@@ -51,12 +56,14 @@ export function FlatMediaSection({
|
||||
automatedVolumeValue,
|
||||
rate,
|
||||
}: {
|
||||
projectId?: string | null;
|
||||
projectDir: string | null;
|
||||
element: DomEditSelection;
|
||||
styles: Record<string, string>;
|
||||
onSetStyle: (prop: string, value: string) => void | Promise<unknown>;
|
||||
onSetAttribute: (attr: string, value: string) => void | Promise<void>;
|
||||
onSetHtmlAttribute: (attr: string, value: string | null) => void | Promise<void>;
|
||||
onSetAttributeBatch: CommitDomAttributeBatch;
|
||||
/** A volume lane in the timeline drives the level; the slider writes a keyframe instead. */
|
||||
volumeAutomated?: boolean;
|
||||
onAutomateVolume?: () => void;
|
||||
@@ -104,7 +111,11 @@ export function FlatMediaSection({
|
||||
const fadeMax = clipDuration > 0 ? clipDuration : 10;
|
||||
const hasLoop = el.hasAttribute("loop");
|
||||
const hasMuted = el.hasAttribute("muted");
|
||||
const hasAudio = element.dataAttributes["has-audio"] === "true";
|
||||
const hasAudio = isVideoAudible({
|
||||
tag: el.tagName,
|
||||
hasAudioAttr: element.dataAttributes["has-audio"],
|
||||
muted: el.hasAttribute("muted"),
|
||||
});
|
||||
const objectFit = styles["object-fit"] || "contain";
|
||||
const objectPosition = styles["object-position"] || "center";
|
||||
|
||||
@@ -128,13 +139,7 @@ export function FlatMediaSection({
|
||||
setCreatePlate(false);
|
||||
}, [srcAttr]);
|
||||
|
||||
const applyCutoutResult = async (result: BackgroundRemovalResult) => {
|
||||
await onSetHtmlAttribute("src", result.outputPath);
|
||||
if (isVideo) {
|
||||
await onSetAttribute("has-audio", "");
|
||||
await onSetHtmlAttribute("muted", "true");
|
||||
}
|
||||
};
|
||||
const mediaEdit = { element, projectId, projectSrc, commit: onSetAttributeBatch };
|
||||
|
||||
const runBackgroundRemoval = async () => {
|
||||
if (!onRemoveBackground || !projectSrc || removeBusy) return;
|
||||
@@ -147,8 +152,8 @@ export function FlatMediaSection({
|
||||
quality,
|
||||
onProgress: setRemoveProgress,
|
||||
});
|
||||
await applyCutoutResult(result);
|
||||
setRemoveProgress({ status: "complete", progress: 100, stage: "Applied cutout", ...result });
|
||||
const stage = await commitCutout(mediaEdit, result.outputPath, hasAudio);
|
||||
setRemoveProgress({ status: "complete", progress: 100, stage, ...result });
|
||||
} catch (error) {
|
||||
setRemoveProgress({
|
||||
status: "failed",
|
||||
@@ -362,21 +367,13 @@ export function FlatMediaSection({
|
||||
<FlatToggle
|
||||
label="Muted"
|
||||
checked={hasMuted}
|
||||
onChange={(next) => void onSetHtmlAttribute("muted", next ? "true" : null)}
|
||||
onChange={(next) => void commitMutedToggle(mediaEdit, next)}
|
||||
/>
|
||||
{isVideo && (
|
||||
<FlatToggle
|
||||
label="Has audio track"
|
||||
checked={hasAudio}
|
||||
onChange={(next) => {
|
||||
if (next) {
|
||||
void onSetAttribute("has-audio", "true");
|
||||
void onSetHtmlAttribute("muted", null);
|
||||
} else {
|
||||
void onSetAttribute("has-audio", "");
|
||||
void onSetHtmlAttribute("muted", "true");
|
||||
}
|
||||
}}
|
||||
onChange={(next) => void commitHasAudioToggle(mediaEdit, next)}
|
||||
/>
|
||||
)}
|
||||
</>
|
||||
|
||||
@@ -19,6 +19,7 @@ export type PropertyPanelFlatProps = Pick<
|
||||
| "onSetAttributeQuiet"
|
||||
| "onApplyColorGradingScope"
|
||||
| "onSetHtmlAttribute"
|
||||
| "onSetAttributeBatch"
|
||||
| "onRemoveBackground"
|
||||
| "onSetText"
|
||||
| "onSetTextFieldStyle"
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { useEffect, useState } from "react";
|
||||
import { isVideoAudible } from "../../player/lib/timelineElementHelpers";
|
||||
import { Check, ClipboardList, Film, Music, Scissors } from "../../icons/SystemIcons";
|
||||
import type { DomEditSelection } from "./domEditing";
|
||||
import {
|
||||
@@ -22,23 +23,29 @@ import {
|
||||
audioGainToFaderPosition,
|
||||
audioGainToText,
|
||||
} from "@hyperframes/core/audio-gain";
|
||||
import type { CommitDomAttributeBatch } from "../../hooks/domEditCommitTypes";
|
||||
import { commitCutout, commitHasAudioToggle, commitMutedToggle } from "./mediaAudioEdits";
|
||||
|
||||
// fallow-ignore-next-line complexity
|
||||
export function MediaSection({
|
||||
projectId = null,
|
||||
projectDir,
|
||||
element,
|
||||
styles,
|
||||
onSetStyle,
|
||||
onSetAttribute,
|
||||
onSetHtmlAttribute,
|
||||
onSetAttributeBatch,
|
||||
onRemoveBackground,
|
||||
}: {
|
||||
projectId?: string | null;
|
||||
projectDir: string | null;
|
||||
element: DomEditSelection;
|
||||
styles: Record<string, string>;
|
||||
onSetStyle: (prop: string, value: string) => void | Promise<unknown>;
|
||||
onSetAttribute: (attr: string, value: string) => void | Promise<void>;
|
||||
onSetHtmlAttribute: (attr: string, value: string | null) => void | Promise<void>;
|
||||
onSetAttributeBatch: CommitDomAttributeBatch;
|
||||
onRemoveBackground?: (
|
||||
inputPath: string,
|
||||
options: {
|
||||
@@ -62,7 +69,11 @@ export function MediaSection({
|
||||
|
||||
const hasLoop = el.hasAttribute("loop");
|
||||
const hasMuted = el.hasAttribute("muted");
|
||||
const hasAudio = element.dataAttributes["has-audio"] === "true";
|
||||
const hasAudio = isVideoAudible({
|
||||
tag: el.tagName,
|
||||
hasAudioAttr: element.dataAttributes["has-audio"],
|
||||
muted: el.hasAttribute("muted"),
|
||||
});
|
||||
|
||||
const playbackRate = Number.parseFloat(element.dataAttributes["playback-rate"] ?? "1") || 1;
|
||||
|
||||
@@ -96,13 +107,7 @@ export function MediaSection({
|
||||
setCreatePlate(false);
|
||||
}, [srcAttr]);
|
||||
|
||||
const applyCutoutResult = async (result: BackgroundRemovalResult) => {
|
||||
await onSetHtmlAttribute("src", result.outputPath);
|
||||
if (isVideo) {
|
||||
await onSetAttribute("has-audio", "");
|
||||
await onSetHtmlAttribute("muted", "true");
|
||||
}
|
||||
};
|
||||
const mediaEdit = { element, projectId, projectSrc, commit: onSetAttributeBatch };
|
||||
|
||||
const runBackgroundRemoval = async () => {
|
||||
if (!onRemoveBackground || !projectSrc || removeBusy) return;
|
||||
@@ -115,13 +120,8 @@ export function MediaSection({
|
||||
quality,
|
||||
onProgress: setRemoveProgress,
|
||||
});
|
||||
await applyCutoutResult(result);
|
||||
setRemoveProgress({
|
||||
status: "complete",
|
||||
progress: 100,
|
||||
stage: "Applied cutout",
|
||||
...result,
|
||||
});
|
||||
const stage = await commitCutout(mediaEdit, result.outputPath, hasAudio);
|
||||
setRemoveProgress({ status: "complete", progress: 100, stage, ...result });
|
||||
} catch (error) {
|
||||
setRemoveProgress({
|
||||
status: "failed",
|
||||
@@ -320,9 +320,7 @@ export function MediaSection({
|
||||
<SegmentedControl
|
||||
trackName="Muted"
|
||||
value={hasMuted ? "on" : "off"}
|
||||
onChange={(next) => {
|
||||
void onSetHtmlAttribute("muted", next === "on" ? "true" : null);
|
||||
}}
|
||||
onChange={(next) => void commitMutedToggle(mediaEdit, next === "on")}
|
||||
options={[
|
||||
{ label: "On", value: "on" },
|
||||
{ label: "Off", value: "off" },
|
||||
@@ -337,15 +335,7 @@ export function MediaSection({
|
||||
<SegmentedControl
|
||||
trackName="Has audio track"
|
||||
value={hasAudio ? "yes" : "no"}
|
||||
onChange={(next) => {
|
||||
if (next === "yes") {
|
||||
void onSetAttribute("has-audio", "true");
|
||||
void onSetHtmlAttribute("muted", null);
|
||||
} else {
|
||||
void onSetAttribute("has-audio", "");
|
||||
void onSetHtmlAttribute("muted", "true");
|
||||
}
|
||||
}}
|
||||
onChange={(next) => void commitHasAudioToggle(mediaEdit, next === "yes")}
|
||||
options={[
|
||||
{ label: "Yes", value: "yes" },
|
||||
{ label: "No", value: "no" },
|
||||
|
||||
@@ -3,6 +3,7 @@ import type { ArcPathSegment, GsapAnimation } from "@hyperframes/parsers/gsap-pa
|
||||
import type { DomEditSelection } from "./domEditing";
|
||||
import type { ImportedFontAsset } from "./fontAssets";
|
||||
import type { GsapAnimationEditCallbacks } from "./gsapAnimationCallbacks";
|
||||
import type { CommitDomAttributeBatch } from "../../hooks/domEditCommitTypes";
|
||||
|
||||
export interface BackgroundRemovalProgress {
|
||||
status: "processing" | "complete" | "failed";
|
||||
@@ -68,6 +69,7 @@ export interface PropertyPanelProps {
|
||||
value: string | null,
|
||||
) => Promise<{ changedFiles: number; changedElements: number }>;
|
||||
onSetHtmlAttribute: (attr: string, value: string | null) => void | Promise<void>;
|
||||
onSetAttributeBatch: CommitDomAttributeBatch;
|
||||
onRemoveBackground?: (
|
||||
inputPath: string,
|
||||
options: {
|
||||
|
||||
@@ -21,6 +21,7 @@ const SAVES_AN_EDIT = {
|
||||
handleDomAttributeQuietCommit: true,
|
||||
handleDomHtmlAttributeCommit: true,
|
||||
handleDomAttributesCommit: true,
|
||||
handleDomAttributeBatchCommit: true,
|
||||
handleDomPathOffsetCommit: true,
|
||||
handleDomGroupPathOffsetCommit: true,
|
||||
handleDomZIndexReorderCommit: true,
|
||||
@@ -175,6 +176,7 @@ export function DomEditProvider({
|
||||
handleDomAttributeQuietCommit,
|
||||
handleDomHtmlAttributeCommit,
|
||||
handleDomAttributesCommit,
|
||||
handleDomAttributeBatchCommit,
|
||||
handleDomPathOffsetCommit,
|
||||
handleDomGroupPathOffsetCommit,
|
||||
handleDomZIndexReorderCommit,
|
||||
@@ -269,6 +271,7 @@ export function DomEditProvider({
|
||||
handleDomAttributeQuietCommit,
|
||||
handleDomHtmlAttributeCommit,
|
||||
handleDomAttributesCommit,
|
||||
handleDomAttributeBatchCommit,
|
||||
handleDomPathOffsetCommit,
|
||||
handleDomGroupPathOffsetCommit,
|
||||
handleDomZIndexReorderCommit,
|
||||
@@ -344,6 +347,7 @@ export function DomEditProvider({
|
||||
handleDomAttributeQuietCommit,
|
||||
handleDomHtmlAttributeCommit,
|
||||
handleDomAttributesCommit,
|
||||
handleDomAttributeBatchCommit,
|
||||
handleDomPathOffsetCommit,
|
||||
handleDomGroupPathOffsetCommit,
|
||||
handleDomZIndexReorderCommit,
|
||||
|
||||
@@ -57,3 +57,10 @@ export type PersistDomEditOperations = (
|
||||
shouldSave?: () => boolean;
|
||||
},
|
||||
) => Promise<DomEditPersistOutcome | undefined>;
|
||||
|
||||
/** Several data-* and HTML attribute ops on one element as ONE persist and ONE undo entry. */
|
||||
export type CommitDomAttributeBatch = (
|
||||
selection: DomEditSelection,
|
||||
operations: PatchOperation[],
|
||||
options: { label: string; prepareContent?: (html: string) => string },
|
||||
) => Promise<boolean>;
|
||||
|
||||
@@ -5,7 +5,7 @@ import {
|
||||
getDomEditTargetKey,
|
||||
type DomEditSelection,
|
||||
} from "../components/editor/domEditing";
|
||||
import type { PersistDomEditOperations } from "./domEditCommitTypes";
|
||||
import type { CommitDomAttributeBatch, PersistDomEditOperations } from "./domEditCommitTypes";
|
||||
import { reportDomEditPersistFailure } from "./domEditPersistFailure";
|
||||
import { bumpDomEditCommitMapVersion, runDomEditCommit } from "./domEditCommitRunner";
|
||||
import { syncStoredAutomationFromPreview } from "../player/lib/automationStoreSync";
|
||||
@@ -54,6 +54,15 @@ export function resolveOptimisticAttributeValue(attr: string, value: string | nu
|
||||
return value === "false" && HTML_BOOLEAN_ATTRIBUTES.has(attr) ? null : value;
|
||||
}
|
||||
|
||||
function previewAttributeName(op: PatchOperation): string {
|
||||
return op.type === "attribute" ? resolveFullAttrName(op.property, true) : op.property;
|
||||
}
|
||||
|
||||
function previewAttributeValue(op: PatchOperation): string | null {
|
||||
if (op.type === "html-attribute") return resolveOptimisticAttributeValue(op.property, op.value);
|
||||
return op.value === "" ? null : op.value;
|
||||
}
|
||||
|
||||
function setOrRemovePreviewAttribute(
|
||||
el: HTMLElement,
|
||||
fullAttr: string,
|
||||
@@ -209,7 +218,7 @@ export function useDomEditAttributeCommits({
|
||||
],
|
||||
);
|
||||
|
||||
// Commits several data-* attributes on the SAME element in ONE persist call
|
||||
// Commits several data-* and HTML attributes on the SAME element in ONE persist call
|
||||
// — needed when two attributes together describe a single logical value
|
||||
// (e.g. a pinned timing range's start+duration): committing them through two
|
||||
// separate sequential `commitDataAttribute` calls leaves a window where the
|
||||
@@ -221,32 +230,30 @@ export function useDomEditAttributeCommits({
|
||||
// explicit, caller-supplied `selection` (not the "current" one) closes both
|
||||
// gaps — matching `onCommitAnimatedProperties`'s same-shaped fix for GSAP
|
||||
// property batches.
|
||||
const commitDataAttributes = useCallback(
|
||||
const commitAttributeOps = useCallback(
|
||||
async (
|
||||
selection: DomEditSelection,
|
||||
attrs: Record<string, string | null>,
|
||||
options: DataAttributeCommitOptions,
|
||||
) => {
|
||||
ops: PatchOperation[],
|
||||
options: DataAttributeCommitOptions & {
|
||||
prepareContent?: (html: string) => string;
|
||||
syncAutomation?: boolean;
|
||||
},
|
||||
): Promise<boolean> => {
|
||||
const iframe = previewIframeRef.current;
|
||||
const entries = Object.entries(attrs).map(([attr, value]) => ({
|
||||
attr,
|
||||
fullAttr: resolveFullAttrName(attr, true),
|
||||
value,
|
||||
const entries = ops.map((op) => ({
|
||||
fullAttr: previewAttributeName(op),
|
||||
value: previewAttributeValue(op),
|
||||
}));
|
||||
const commitKey = `${options.coalescePrefix}:${entries
|
||||
.map((entry) => entry.attr)
|
||||
const commitKey = `${options.coalescePrefix}:${ops
|
||||
.map((op) => op.property)
|
||||
.sort()
|
||||
.join(",")}:${getDomEditTargetKey(selection)}`;
|
||||
const isLatestCommit = bumpDomEditCommitMapVersion(
|
||||
domAttributeCommitVersionRef.current,
|
||||
commitKey,
|
||||
);
|
||||
const ops: PatchOperation[] = entries.map((entry) => ({
|
||||
type: "attribute",
|
||||
property: entry.attr,
|
||||
value: entry.value,
|
||||
}));
|
||||
let captured: CapturedMultiAttributeElement | null = null;
|
||||
let landed = false;
|
||||
|
||||
await runDomEditCommit({
|
||||
capture: () => {
|
||||
@@ -260,8 +267,7 @@ export function useDomEditAttributeCommits({
|
||||
apply: () => {
|
||||
if (!captured) return;
|
||||
for (const entry of entries) {
|
||||
const nextValue = entry.value === null || entry.value === "" ? null : entry.value;
|
||||
setOrRemovePreviewAttribute(captured.element, entry.fullAttr, nextValue);
|
||||
setOrRemovePreviewAttribute(captured.element, entry.fullAttr, entry.value);
|
||||
}
|
||||
},
|
||||
persist: () =>
|
||||
@@ -269,6 +275,7 @@ export function useDomEditAttributeCommits({
|
||||
label: options.label,
|
||||
coalesceKey: commitKey,
|
||||
skipRefresh: options.skipRefresh,
|
||||
prepareContent: options.prepareContent,
|
||||
}),
|
||||
shouldRevert: () => isLatestCommit(),
|
||||
revert: () => {
|
||||
@@ -283,10 +290,19 @@ export function useDomEditAttributeCommits({
|
||||
},
|
||||
onError: (error) => reportDomEditPersistFailure(selection, ops, error, showToast),
|
||||
shouldResync: () => isLatestCommit() && !!options.refreshAfter,
|
||||
resync: () => refreshDomEditSelectionFromPreview(selection),
|
||||
onSettled: options.onSettled,
|
||||
resync: () => {
|
||||
refreshDomEditSelectionFromPreview(selection);
|
||||
if (options.syncAutomation) {
|
||||
syncStoredAutomationFromPreview(previewIframeRef.current?.contentDocument ?? null);
|
||||
}
|
||||
},
|
||||
onSettled: (ok) => {
|
||||
landed = ok;
|
||||
options.onSettled?.(ok);
|
||||
},
|
||||
onFinally: isLatestCommit.release,
|
||||
});
|
||||
return landed;
|
||||
},
|
||||
[
|
||||
activeCompPath,
|
||||
@@ -299,14 +315,35 @@ export function useDomEditAttributeCommits({
|
||||
|
||||
const handleDomAttributesCommit = useCallback(
|
||||
async (selection: DomEditSelection, attrs: Record<string, string>) => {
|
||||
await commitDataAttributes(selection, attrs, {
|
||||
label: "Edit timing",
|
||||
coalescePrefix: "attrs",
|
||||
await commitAttributeOps(
|
||||
selection,
|
||||
Object.entries(attrs).map(([property, value]) => ({
|
||||
type: "attribute",
|
||||
property,
|
||||
value,
|
||||
})),
|
||||
{
|
||||
label: "Edit timing",
|
||||
coalescePrefix: "attrs",
|
||||
skipRefresh: false,
|
||||
refreshAfter: true,
|
||||
},
|
||||
);
|
||||
},
|
||||
[commitAttributeOps],
|
||||
);
|
||||
|
||||
const handleDomAttributeBatchCommit: CommitDomAttributeBatch = useCallback(
|
||||
(selection, ops, options) =>
|
||||
commitAttributeOps(selection, ops, {
|
||||
label: options.label,
|
||||
coalescePrefix: "attr-batch",
|
||||
skipRefresh: false,
|
||||
refreshAfter: true,
|
||||
});
|
||||
},
|
||||
[commitDataAttributes],
|
||||
syncAutomation: true,
|
||||
prepareContent: options.prepareContent,
|
||||
}),
|
||||
[commitAttributeOps],
|
||||
);
|
||||
|
||||
const handleDomAttributeCommit = useCallback(
|
||||
@@ -438,5 +475,6 @@ export function useDomEditAttributeCommits({
|
||||
handleDomAttributeQuietCommit,
|
||||
handleDomHtmlAttributeCommit,
|
||||
handleDomAttributesCommit,
|
||||
handleDomAttributeBatchCommit,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1736,3 +1736,128 @@ describe("useDomEditCommits attribute persist handling", () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("useDomEditCommits attribute batch commit", () => {
|
||||
beforeEach(() => {
|
||||
ensureCssEscape();
|
||||
vi.clearAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
document.body.replaceChildren();
|
||||
});
|
||||
|
||||
const original = '<video data-hf-id="hf-card" data-has-audio="true"></video>';
|
||||
const patched = '<video data-hf-id="hf-card" muted></video>';
|
||||
const mute = [
|
||||
{ type: "html-attribute" as const, property: "muted", value: "true" },
|
||||
{ type: "attribute" as const, property: "has-audio", value: null },
|
||||
];
|
||||
|
||||
it("sends mixed html and data attribute ops in ONE patch and ONE undo entry", async () => {
|
||||
const fetchMock = stubPatchFetch(
|
||||
{
|
||||
ok: true,
|
||||
changed: true,
|
||||
matched: true,
|
||||
content: patched,
|
||||
path: "index.html",
|
||||
version: "v2",
|
||||
},
|
||||
original,
|
||||
);
|
||||
const { iframe, element } = createPreviewElement(original);
|
||||
const rendered = renderDomEditCommits(createSelection(element, { tagName: "video" }), iframe);
|
||||
|
||||
try {
|
||||
let ok = false;
|
||||
await act(async () => {
|
||||
ok = await rendered.hook.handleDomAttributeBatchCommit(
|
||||
createSelection(element, { tagName: "video" }),
|
||||
mute,
|
||||
{ label: "Edit muted" },
|
||||
);
|
||||
});
|
||||
|
||||
const patches = fetchMock.mock.calls.filter(([input]) =>
|
||||
requestUrl(input).includes("/file-mutations/patch-element/"),
|
||||
);
|
||||
expect(ok).toBe(true);
|
||||
expect(patches).toHaveLength(1);
|
||||
const body = JSON.parse(String(patches[0]?.[1]?.body));
|
||||
expect(body.operations).toEqual(mute);
|
||||
expect(rendered.recordEdit).toHaveBeenCalledTimes(1);
|
||||
expect(rendered.recordEdit).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
files: { "index.html": { before: original, after: patched } },
|
||||
}),
|
||||
);
|
||||
expect(rendered.reloadPreview).toHaveBeenCalledTimes(1);
|
||||
expect(element.hasAttribute("muted")).toBe(true);
|
||||
expect(element.hasAttribute("data-has-audio")).toBe(false);
|
||||
} finally {
|
||||
rendered.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it("folds a document-level prepareContent into the same single undo entry", async () => {
|
||||
stubPatchFetch(
|
||||
{
|
||||
ok: true,
|
||||
changed: true,
|
||||
matched: true,
|
||||
content: patched,
|
||||
path: "index.html",
|
||||
version: "v2",
|
||||
},
|
||||
original,
|
||||
);
|
||||
const writes: string[] = [];
|
||||
const { iframe, element } = createPreviewElement(original);
|
||||
const rendered = renderDomEditCommits(createSelection(element), iframe, {
|
||||
writeProjectFile: async (_path, content) => {
|
||||
writes.push(content);
|
||||
},
|
||||
});
|
||||
const withSibling = `<audio id="a"></audio>${patched}`;
|
||||
|
||||
try {
|
||||
await act(async () => {
|
||||
await rendered.hook.handleDomAttributeBatchCommit(createSelection(element), mute, {
|
||||
label: "Remove background",
|
||||
prepareContent: () => withSibling,
|
||||
});
|
||||
});
|
||||
|
||||
expect(writes).toEqual([withSibling]);
|
||||
expect(rendered.recordEdit).toHaveBeenCalledTimes(1);
|
||||
expect(rendered.recordEdit).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
label: "Remove background",
|
||||
files: { "index.html": { before: original, after: withSibling } },
|
||||
}),
|
||||
);
|
||||
} finally {
|
||||
rendered.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it("reverts every op in the batch and reports failure when the patch rejects", async () => {
|
||||
stubPatchFetch(new Error("network down"), original);
|
||||
const { iframe, element } = createPreviewElement(original);
|
||||
const rendered = renderDomEditCommits(createSelection(element), iframe);
|
||||
|
||||
try {
|
||||
let ok = true;
|
||||
await act(async () => {
|
||||
ok = await rendered.hook.handleDomAttributeBatchCommit(createSelection(element), mute, {
|
||||
label: "Edit muted",
|
||||
});
|
||||
});
|
||||
|
||||
expect(ok).toBe(false);
|
||||
expect(element.hasAttribute("muted")).toBe(false);
|
||||
expect(element.getAttribute("data-has-audio")).toBe("true");
|
||||
} finally {
|
||||
rendered.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -239,6 +239,7 @@ export function useDomEditCommits({
|
||||
handleDomAttributeQuietCommit,
|
||||
handleDomHtmlAttributeCommit,
|
||||
handleDomAttributesCommit,
|
||||
handleDomAttributeBatchCommit,
|
||||
handleDomTextCommit,
|
||||
handleDomTextCommitForSelection,
|
||||
handleDomRichTextCommit,
|
||||
@@ -305,6 +306,7 @@ export function useDomEditCommits({
|
||||
handleDomAttributeQuietCommit,
|
||||
handleDomHtmlAttributeCommit,
|
||||
handleDomAttributesCommit,
|
||||
handleDomAttributeBatchCommit,
|
||||
handleDomTextCommit,
|
||||
handleDomTextCommitForSelection,
|
||||
handleDomRichTextCommit,
|
||||
|
||||
@@ -170,7 +170,6 @@ export function useDomEditSession({
|
||||
refreshDomEditSelectionFromPreview,
|
||||
});
|
||||
// ── GSAP cache (hoisted so both useGsapScriptCommits and useDomEditWiring share the same instance) ──
|
||||
|
||||
const { version: gsapCacheVersion, bump: bumpGsapCache } = useGsapCacheVersion();
|
||||
|
||||
const {
|
||||
@@ -202,6 +201,7 @@ export function useDomEditSession({
|
||||
handleDomAttributeQuietCommit,
|
||||
handleDomHtmlAttributeCommit,
|
||||
handleDomAttributesCommit,
|
||||
handleDomAttributeBatchCommit,
|
||||
handleDomTextCommit,
|
||||
handleDomTextCommitForSelection,
|
||||
handleDomRichTextCommit,
|
||||
@@ -288,7 +288,6 @@ export function useDomEditSession({
|
||||
});
|
||||
|
||||
// ── Element groups (wrap selected elements in a data-hf-group div) ──
|
||||
|
||||
const { groupSelection, ungroupSelection } = useGroupCommits({
|
||||
activeCompPath,
|
||||
showToast,
|
||||
@@ -363,7 +362,6 @@ export function useDomEditSession({
|
||||
}, [domEditSelectionRef, ungroupSelection, setActiveGroupElement, showToast]);
|
||||
|
||||
// ── Wiring: selection sync, GSAP cache, preview sync, selection handlers ──
|
||||
|
||||
const {
|
||||
onClickToSource,
|
||||
selectedGsapAnimations,
|
||||
@@ -496,6 +494,7 @@ export function useDomEditSession({
|
||||
handleDomAttributeQuietCommit,
|
||||
handleDomHtmlAttributeCommit,
|
||||
handleDomAttributesCommit,
|
||||
handleDomAttributeBatchCommit,
|
||||
handleDomPathOffsetCommit: handleGsapAwarePathOffsetCommit,
|
||||
handleDomGroupPathOffsetCommit: handleGsapAwareGroupPathOffsetCommit,
|
||||
handleDomZIndexReorderCommit,
|
||||
|
||||
@@ -99,6 +99,7 @@ export function useDomEditTextCommits({
|
||||
handleDomAttributeQuietCommit,
|
||||
handleDomHtmlAttributeCommit,
|
||||
handleDomAttributesCommit,
|
||||
handleDomAttributeBatchCommit,
|
||||
} = useDomEditAttributeCommits({
|
||||
activeCompPath,
|
||||
previewIframeRef,
|
||||
@@ -485,6 +486,7 @@ export function useDomEditTextCommits({
|
||||
handleDomAttributeQuietCommit,
|
||||
handleDomHtmlAttributeCommit,
|
||||
handleDomAttributesCommit,
|
||||
handleDomAttributeBatchCommit,
|
||||
handleDomTextCommit,
|
||||
handleDomTextCommitForSelection,
|
||||
handleDomRichTextCommit,
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { isVideoAudible } from "./timelineElementHelpers";
|
||||
|
||||
describe("isVideoAudible — the compiler's data-has-audio rule", () => {
|
||||
it("explicit data-has-audio wins", () => {
|
||||
expect(isVideoAudible({ tag: "video", hasAudioAttr: "true", muted: false })).toBe(true);
|
||||
expect(isVideoAudible({ tag: "video", hasAudioAttr: "false", muted: false })).toBe(false);
|
||||
});
|
||||
it("no attribute: an unmuted video is audible, a muted one is not", () => {
|
||||
expect(isVideoAudible({ tag: "video", hasAudioAttr: null, muted: false })).toBe(true);
|
||||
expect(isVideoAudible({ tag: "video", hasAudioAttr: undefined, muted: true })).toBe(false);
|
||||
});
|
||||
it("never true for non-video without the attribute", () => {
|
||||
expect(isVideoAudible({ tag: "img", hasAudioAttr: null, muted: false })).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -220,12 +220,26 @@ function readVolume(el: Element, media: Element): number | undefined {
|
||||
return Number.isFinite(volume) ? volume : undefined;
|
||||
}
|
||||
|
||||
/** The compiler's rule (timingCompiler): explicit data-has-audio wins; otherwise an unmuted <video> is audible. */
|
||||
export function isVideoAudible(opts: {
|
||||
tag: string;
|
||||
hasAudioAttr: string | null | undefined;
|
||||
muted: boolean;
|
||||
}): boolean {
|
||||
if (opts.hasAudioAttr === "true") return true;
|
||||
if (opts.hasAudioAttr === "false" || opts.hasAudioAttr === "") return false;
|
||||
return opts.tag.toLowerCase() === "video" && !opts.muted;
|
||||
}
|
||||
|
||||
/** What the mixer gets: the compiler's `data-has-audio` rule, muted and volume. */
|
||||
function applyAudioMetadataFromElement(entry: TimelineElement, el: Element): void {
|
||||
const media = resolveMediaElement(el) ?? el;
|
||||
const muted = el.hasAttribute("muted") || media.hasAttribute("muted");
|
||||
const hasAudio = el.getAttribute("data-has-audio");
|
||||
const sound = hasAudio === null ? el.tagName === "VIDEO" && !muted : hasAudio === "true";
|
||||
const sound = isVideoAudible({
|
||||
tag: el.tagName,
|
||||
hasAudioAttr: el.getAttribute("data-has-audio"),
|
||||
muted,
|
||||
});
|
||||
setOptional(entry, "hasAudio", sound ? true : undefined);
|
||||
setOptional(entry, "muted", muted ? true : undefined);
|
||||
setOptional(entry, "volume", readVolume(el, media));
|
||||
|
||||
@@ -12,7 +12,7 @@ function escapeStyleAttributeValue(value: string, quote: string): string {
|
||||
}
|
||||
|
||||
/** Escape a string for safe use inside a double-quoted HTML attribute. */
|
||||
function escapeHtmlAttribute(value: string): string {
|
||||
export function escapeHtmlAttribute(value: string): string {
|
||||
return value
|
||||
.replace(/&/g, "&")
|
||||
.replace(/"/g, """)
|
||||
|
||||
@@ -5,6 +5,7 @@ import {
|
||||
historyTooltipLabel,
|
||||
findMatchingTimelineElementId,
|
||||
findTimelineIdByAncestor,
|
||||
resolveAssetHasAudio,
|
||||
resolveDroppedAssetDimensions,
|
||||
resolveElementTrack,
|
||||
resolveTimelineIdForSelection,
|
||||
@@ -235,3 +236,32 @@ describe("historyTooltipLabel", () => {
|
||||
expect(historyTooltipLabel("redo", null)).toMatch(/^Redo \(.+\)$/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("resolveAssetHasAudio", () => {
|
||||
afterEach(() => vi.unstubAllGlobals());
|
||||
|
||||
const stubMetadata = (response: Response | Error) =>
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () => {
|
||||
if (response instanceof Error) throw response;
|
||||
return response;
|
||||
}),
|
||||
);
|
||||
|
||||
it("answers the metadata endpoint's audio flag", async () => {
|
||||
stubMetadata(new Response(JSON.stringify({ metadata: { hasAudio: true } })));
|
||||
expect(await resolveAssetHasAudio("p1", "assets/talk.mp4")).toBe(true);
|
||||
stubMetadata(new Response(JSON.stringify({ metadata: { hasAudio: false } })));
|
||||
expect(await resolveAssetHasAudio("p1", "assets/cutout.webm")).toBe(false);
|
||||
});
|
||||
|
||||
it("answers null when the probe cannot tell", async () => {
|
||||
stubMetadata(new Response("nope", { status: 500 }));
|
||||
expect(await resolveAssetHasAudio("p1", "assets/talk.mp4")).toBeNull();
|
||||
stubMetadata(new Error("offline"));
|
||||
expect(await resolveAssetHasAudio("p1", "assets/talk.mp4")).toBeNull();
|
||||
stubMetadata(new Response(JSON.stringify({ metadata: {} })));
|
||||
expect(await resolveAssetHasAudio("p1", "assets/talk.mp4")).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -388,13 +388,27 @@ export async function resolveDroppedAssetHasAudio(
|
||||
kind: TimelineAssetKind,
|
||||
): Promise<boolean> {
|
||||
if (kind !== "video") return false;
|
||||
return (await resolveAssetHasAudio(projectId, assetPath)) === true;
|
||||
}
|
||||
|
||||
function isPlainRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null;
|
||||
}
|
||||
|
||||
/** An asset's audio stream from the metadata endpoint: null when the probe can't tell. */
|
||||
export async function resolveAssetHasAudio(
|
||||
projectId: string,
|
||||
assetPath: string,
|
||||
): Promise<boolean | null> {
|
||||
try {
|
||||
const response = await fetch(mediaMetadataUrl(projectId, assetPath));
|
||||
if (!response.ok) return false;
|
||||
const data = (await response.json()) as { metadata?: { hasAudio?: boolean } } | null;
|
||||
return data?.metadata?.hasAudio === true;
|
||||
if (!response.ok) return null;
|
||||
const data: unknown = await response.json();
|
||||
const metadata = isPlainRecord(data) ? data.metadata : undefined;
|
||||
const hasAudio = isPlainRecord(metadata) ? metadata.hasAudio : undefined;
|
||||
return typeof hasAudio === "boolean" ? hasAudio : null;
|
||||
} catch {
|
||||
return false;
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+1
-9
@@ -142,22 +142,14 @@
|
||||
<video
|
||||
id="short_mag_cut"
|
||||
src="__VIDEO_SRC__"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="1"
|
||||
style="width: 100%; height: auto"
|
||||
></video>
|
||||
</div>
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="__VIDEO_SRC__"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="5"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
<style>
|
||||
#aroll-container {
|
||||
position: absolute;
|
||||
|
||||
+1
-9
@@ -202,20 +202,12 @@
|
||||
<video
|
||||
id="short_mag_cut"
|
||||
src="__VIDEO_SRC__"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="1"
|
||||
></video>
|
||||
</div>
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="__VIDEO_SRC__"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="5"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
+1
-9
@@ -176,20 +176,12 @@
|
||||
<video
|
||||
id="a-roll"
|
||||
src="__VIDEO_SRC__"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="0"
|
||||
></video>
|
||||
</div>
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="__VIDEO_SRC__"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="5"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
+2
-10
@@ -125,24 +125,16 @@
|
||||
</script>
|
||||
</div>
|
||||
|
||||
<!-- A-Roll Video (muted — audio is separate) -->
|
||||
<!-- A-roll keeps its own sound -->
|
||||
<video
|
||||
id="a-roll"
|
||||
src="__VIDEO_SRC__"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="0"
|
||||
></video>
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="__VIDEO_SRC__"
|
||||
data-start="0"
|
||||
data-duration="__VIDEO_DURATION__"
|
||||
data-track-index="5"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
|
||||
<!-- Compositions -->
|
||||
<div
|
||||
|
||||
@@ -180,7 +180,7 @@ test("core and general-video author temporal edits as duplicated source-range cl
|
||||
/data-media-start[\s\S]{0,120}data-duration/i,
|
||||
/data-start/,
|
||||
/hard cut[\s\S]{0,100}trim[\s\S]{0,100}splice[\s\S]{0,100}reorder/i,
|
||||
/separate(?:ly)? authored audio[\s\S]{0,180}(identical|same)[\s\S]{0,100}(range|timing)/i,
|
||||
/sound stays on the clip[\s\S]{0,160}data-has-audio/i,
|
||||
],
|
||||
surface,
|
||||
);
|
||||
@@ -239,7 +239,7 @@ test("creator editing recipes are copyable, owned, mathematical, and limitation-
|
||||
/data-playback-rate/,
|
||||
/consumed source\s*=\s*timeline duration\s*[×*]\s*rate/i,
|
||||
/natural timeline duration\s*=\s*remaining source\s*\/\s*rate/i,
|
||||
/separate audio track/i,
|
||||
/separate `?<audio>`?[\s\S]{0,160}(J\/L cut|replacement audio|voiceover|music)/i,
|
||||
/final-source[\s\S]{0,100}subcomp[\s\S]{0,100}visual pose/i,
|
||||
/arbitrary mid-source[\s\S]{0,120}preprocess/i,
|
||||
/distinct tracks[\s\S]{0,120}overlap[\s\S]{0,120}opposing/i,
|
||||
@@ -288,7 +288,7 @@ test("crossfade and volume recipes contain executable opposing envelopes", async
|
||||
/gsap\.timeline\(\{\s*paused:\s*true\s*\}\)/,
|
||||
/window\.__timelines/,
|
||||
/autoAlpha|opacity/,
|
||||
/<audio[\s\S]*<audio/,
|
||||
/data-has-audio="true"[\s\S]*data-has-audio="true"/,
|
||||
/data-automation='/,
|
||||
],
|
||||
"crossfade recipe",
|
||||
@@ -318,8 +318,7 @@ test("natural media duration is rate-scaled across every preview surface", async
|
||||
}
|
||||
});
|
||||
|
||||
// fallow-ignore-next-line complexity
|
||||
test("every temporal source-range recipe includes matching audio markup", async () => {
|
||||
test("every temporal source-range recipe keeps sound on the video", async () => {
|
||||
const recipes = await read(files.editingRecipes);
|
||||
for (const name of [
|
||||
"Hard cut",
|
||||
@@ -331,21 +330,17 @@ test("every temporal source-range recipe includes matching audio markup", async
|
||||
const start = recipes.indexOf(`## ${name}`);
|
||||
const next = recipes.indexOf("## ", start + 3);
|
||||
const section = recipes.slice(start, next < 0 ? recipes.length : next);
|
||||
const videos = [...section.matchAll(/<video[\s\S]*?<\/video>/g)].map((m) => m[0]);
|
||||
const audios = [...section.matchAll(/<audio[\s\S]*?<\/audio>/g)].map((m) => m[0]);
|
||||
assert.ok(
|
||||
videos.length > 0 && audios.length >= videos.length,
|
||||
`${name}: matching audio missing`,
|
||||
);
|
||||
const videos = [...section.matchAll(/<video\b[^>]*>/g)].map((m) => m[0]);
|
||||
assert.ok(videos.length > 0, `${name}: no video markup`);
|
||||
for (const video of videos) {
|
||||
for (const attr of ["src", "data-start", "data-duration", "data-media-start"]) {
|
||||
const value = video.match(new RegExp(`${attr}="([^"]+)"`))?.[1];
|
||||
assert.ok(
|
||||
value && audios.some((audio) => audio.includes(`${attr}="${value}"`)),
|
||||
`${name}: audio mismatch ${attr}`,
|
||||
);
|
||||
}
|
||||
assert.match(video, /data-has-audio="true"/, `${name}: video must keep its sound`);
|
||||
assert.doesNotMatch(video, /\bmuted\b/, `${name}: video must not be muted`);
|
||||
}
|
||||
assert.doesNotMatch(
|
||||
section,
|
||||
/<audio\b/,
|
||||
`${name}: no duplicate <audio> for the video's own sound`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
@@ -37,6 +37,7 @@ import {
|
||||
executeRenderJob,
|
||||
} from "../packages/producer/src/index.js";
|
||||
import { openOpaqueCapture } from "./preview-capture.js";
|
||||
import { patchMediaPlaceholders } from "../packages/cli/src/commands/initMedia.ts";
|
||||
|
||||
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
||||
const repoRoot = resolve(scriptDir, "..");
|
||||
@@ -64,13 +65,7 @@ function patchTemplateHtml(dir: string, durationSeconds: number): void {
|
||||
|
||||
for (const entry of htmlEntries) {
|
||||
const file = join(entry.parentPath, entry.name);
|
||||
let content = readFileSync(file, "utf-8");
|
||||
content = content.replace(/<video[^>]*src="__VIDEO_SRC__"[^>]*>[\s\S]*?<\/video>/g, "");
|
||||
content = content.replace(/<video[^>]*src="__VIDEO_SRC__"[^>]*>/g, "");
|
||||
content = content.replace(/<audio[^>]*src="__VIDEO_SRC__"[^>]*>[\s\S]*?<\/audio>/g, "");
|
||||
content = content.replace(/<audio[^>]*src="__VIDEO_SRC__"[^>]*>/g, "");
|
||||
const dur = String(Math.round(durationSeconds * 100) / 100);
|
||||
content = content.replaceAll("__VIDEO_DURATION__", dur);
|
||||
const content = patchMediaPlaceholders(readFileSync(file, "utf-8"), { durationSeconds });
|
||||
writeFileSync(file, content, "utf-8");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"source": "heygen-com/hyperframes",
|
||||
"skills": {
|
||||
"embedded-captions": {
|
||||
"hash": "7c6f147f722cc4c4",
|
||||
"hash": "9ae9e4e06bf4b61e",
|
||||
"files": 144
|
||||
},
|
||||
"faceless-explainer": {
|
||||
@@ -14,7 +14,7 @@
|
||||
"files": 2
|
||||
},
|
||||
"general-video": {
|
||||
"hash": "77938376e7cc61fc",
|
||||
"hash": "a4f4532304ff7ec5",
|
||||
"files": 5
|
||||
},
|
||||
"hyperframes": {
|
||||
@@ -22,7 +22,7 @@
|
||||
"files": 29
|
||||
},
|
||||
"hyperframes-animation": {
|
||||
"hash": "3055d5038b7fe96b",
|
||||
"hash": "2a567656b9291f5a",
|
||||
"files": 122
|
||||
},
|
||||
"hyperframes-audio": {
|
||||
@@ -34,11 +34,11 @@
|
||||
"files": 11
|
||||
},
|
||||
"hyperframes-core": {
|
||||
"hash": "b632d12d11eb54e8",
|
||||
"hash": "db99256dd85d0460",
|
||||
"files": 11
|
||||
},
|
||||
"hyperframes-creative": {
|
||||
"hash": "922bce76a69b97d9",
|
||||
"hash": "7835cfaad29f1fcb",
|
||||
"files": 79
|
||||
},
|
||||
"hyperframes-keyframes": {
|
||||
@@ -62,7 +62,7 @@
|
||||
"files": 23
|
||||
},
|
||||
"music-to-video": {
|
||||
"hash": "3c982c2ddeac43f2",
|
||||
"hash": "887a71e8575e0342",
|
||||
"files": 169
|
||||
},
|
||||
"pr-to-video": {
|
||||
@@ -74,7 +74,7 @@
|
||||
"files": 34
|
||||
},
|
||||
"remotion-to-hyperframes": {
|
||||
"hash": "4fcef874801e58f7",
|
||||
"hash": "e13e3681d9b0dd75",
|
||||
"files": 77
|
||||
},
|
||||
"slideshow": {
|
||||
@@ -82,7 +82,7 @@
|
||||
"files": 2
|
||||
},
|
||||
"talking-head-recut": {
|
||||
"hash": "eac2082ea7096802",
|
||||
"hash": "865d98665f0011da",
|
||||
"files": 28
|
||||
}
|
||||
}
|
||||
|
||||
@@ -153,23 +153,14 @@
|
||||
<video
|
||||
id="a-roll"
|
||||
src="source.mp4"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
data-duration="{{DURATION}}"
|
||||
data-track-index="0"
|
||||
style="z-index: 1"
|
||||
></video>
|
||||
|
||||
<div id="stage">{{GROUPS_HTML}}</div>
|
||||
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="source.mp4"
|
||||
data-start="0"
|
||||
data-duration="{{DURATION}}"
|
||||
data-track-index="3"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
typography. Compiled by make-composition.cjs.
|
||||
|
||||
Contract (unchanged from cinematic-cream — gates depend on it):
|
||||
#root[data-composition-id="main"] · #a-roll · #stage · #a-roll-audio
|
||||
#root[data-composition-id="main"] · #a-roll · #stage
|
||||
.plane > .cap > .w spans · paused seek-safe GSAP on window.__timelines["main"]
|
||||
container fade via tl.set at g.in−0.01 · exit fade + display:none at g.out
|
||||
no Math.random / Date.now / CSS keyframes / repeat:-1
|
||||
@@ -102,23 +102,14 @@
|
||||
<video
|
||||
id="a-roll"
|
||||
src="source.mp4"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
data-duration="{{DURATION}}"
|
||||
data-track-index="0"
|
||||
style="z-index: 1"
|
||||
></video>
|
||||
|
||||
<div id="stage">{{GROUPS_HTML}}</div>
|
||||
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="source.mp4"
|
||||
data-start="0"
|
||||
data-duration="{{DURATION}}"
|
||||
data-track-index="3"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
|
||||
@@ -580,13 +580,11 @@ ${css}
|
||||
<body>
|
||||
<div id="root" data-composition-id="main" data-start="0" data-duration="${DUR}"
|
||||
data-width="${W}" data-height="${H}">
|
||||
<video id="a-roll" src="source.mp4" muted playsinline
|
||||
<video id="a-roll" src="source.mp4" playsinline data-has-audio="true"
|
||||
data-duration="${DUR}" data-track-index="0"></video>
|
||||
<div id="stage">
|
||||
${stageHtml}
|
||||
</div>
|
||||
<audio id="a-roll-audio" src="source.mp4" data-start="0" data-duration="${DUR}"
|
||||
data-track-index="3" data-volume="1"></audio>
|
||||
</div>
|
||||
<script>
|
||||
window.__timelines = window.__timelines || {};
|
||||
|
||||
@@ -143,7 +143,9 @@ function main() {
|
||||
process.exit(2);
|
||||
}
|
||||
const audio = path.join(project, "audio.mp3");
|
||||
if (!fs.existsSync(audio))
|
||||
// A stray audio.mp3 in the project makes lint call the render silent; only remove one this run made.
|
||||
const audioIsScratch = !fs.existsSync(audio);
|
||||
if (audioIsScratch)
|
||||
cp.execFileSync(
|
||||
"ffmpeg",
|
||||
["-y", "-i", src, "-vn", "-acodec", "libmp3lame", "-q:a", "2", audio],
|
||||
@@ -281,6 +283,8 @@ function main() {
|
||||
}
|
||||
}
|
||||
|
||||
if (audioIsScratch) fs.rmSync(audio, { force: true });
|
||||
|
||||
const text = words
|
||||
.map((w) => w.text)
|
||||
.join(" ")
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import test from "node:test";
|
||||
@@ -39,3 +39,31 @@ test("the CLI fallback labels the transcript with the engine the CLI ran", () =>
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("the extracted audio scratch file does not outlive the run", () => {
|
||||
const root = mkdtempSync(join(tmpdir(), "embedded-captions-transcribe-"));
|
||||
try {
|
||||
const cliDir = join(root, "hf", "packages", "cli", "dist");
|
||||
mkdirSync(cliDir, { recursive: true });
|
||||
writeFileSync(join(cliDir, "cli.js"), STUB_CLI);
|
||||
const project = join(root, "project");
|
||||
mkdirSync(project);
|
||||
execFileSync(
|
||||
"ffmpeg",
|
||||
["-y", "-f", "lavfi", "-i", "sine=frequency=440:duration=1", join(project, "source.mp4")],
|
||||
{ stdio: "ignore" },
|
||||
);
|
||||
execFileSync(
|
||||
process.execPath,
|
||||
[fileURLToPath(new URL("./transcribe.cjs", import.meta.url)), project],
|
||||
{
|
||||
stdio: "ignore",
|
||||
env: { ...process.env, HYPERFRAMES_ROOT: join(root, "hf"), TRANSCRIBE_ENGINE: "whisper" },
|
||||
},
|
||||
);
|
||||
assert.equal(existsSync(join(project, "audio.mp3")), false);
|
||||
assert.equal(existsSync(join(project, "transcript.json")), true);
|
||||
} finally {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
@@ -64,8 +64,7 @@ Do not invent synonyms for these states. An ongoing “just build it” signal i
|
||||
For a hard cut, trim, splice, or reorder of existing footage, duplicate the same
|
||||
video source into multiple clip elements. On each copy, set the source range
|
||||
with `data-media-start` plus `data-duration`, then set authored placement/order
|
||||
with `data-start`. Separately authored audio follows the identical clip ranges
|
||||
and timing on matching `<audio>` elements. `/hyperframes-core` owns this temporal
|
||||
with `data-start`. Each video segment keeps its sound: the sound stays on the clip (`data-has-audio="true"`), so cutting the video cuts its sound. `/hyperframes-core` owns this temporal
|
||||
edit; use `/hyperframes-keyframes` only for visual-property animation such as
|
||||
zoom, punch, pan, crop, mask, or `clip-path` on an inner wrapper.
|
||||
Copy the full contracts from `../hyperframes-core/references/creator-editing-recipes.md`.
|
||||
|
||||
@@ -393,7 +393,7 @@
|
||||
<!--
|
||||
Real asset path (replace for production).
|
||||
If missing, the CSS mockup below shows through the transparent
|
||||
poster region. The video stays muted per HyperFrames rules.
|
||||
poster region. Muted: decorative footage.
|
||||
-->
|
||||
<!-- Decorative CSS-mockup of an editor — paused screenshot feel. -->
|
||||
<div class="demo-mockup">
|
||||
|
||||
@@ -203,7 +203,7 @@ Vector animations that play inside a composition. Use for logos, character anima
|
||||
|
||||
## 6. Video Compositing
|
||||
|
||||
Embed real video footage inside compositions. Videos must be `muted` with `playsinline`.
|
||||
Embed real video footage inside compositions. Videos are `playsinline`; add `muted` for silent footage or b-roll, or `data-has-audio="true"` when the clip's own sound should play.
|
||||
|
||||
```html
|
||||
<div class="video-frame" style="width:680px;height:840px;border-radius:16px;overflow:hidden;">
|
||||
|
||||
@@ -4,7 +4,7 @@ Use these copyable contracts after `tracks-and-clips.md`. Global math: **consume
|
||||
|
||||
Before any edit, run `npx hyperframes timeline` (add `--json` for a machine-readable list) to see the project's tracks and clips instead of reading the HTML.
|
||||
|
||||
These recipes keep sound on a separate `<audio>` element with the `<video>` muted, which is the pattern to reach for when picture and sound are cut independently. An unmuted `<video>` that declares `data-has-audio="true"` is also mixed, so a separate track is a choice, not a requirement.
|
||||
These recipes keep a video's sound on the `<video>` (`data-has-audio="true"`, no `muted`), so cutting the video cuts its sound. Use a separate `<audio>` only when picture and sound must be cut independently (J/L cuts, replacement audio) or for other sound (music, voiceover) — the recipes that do say so. Silent footage or b-roll: replace `data-has-audio="true"` with `muted` in these blocks.
|
||||
|
||||
**Every `<video>` and `<audio>` below carries an `id`, and that is not cosmetic**: `lint` errors with `media_missing_id` on timed media without one, and an id-less `<audio>` is never picked up by the mixer, so the render comes out silent. Keep the ids when you copy a recipe.
|
||||
|
||||
@@ -18,8 +18,8 @@ These recipes keep sound on a separate `<audio>` element with the `<video>` mute
|
||||
data-duration="2"
|
||||
data-media-start="4"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
<video
|
||||
id="b"
|
||||
@@ -28,28 +28,12 @@ These recipes keep sound on a separate `<audio>` element with the `<video>` mute
|
||||
data-duration="3"
|
||||
data-media-start="10"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
<audio
|
||||
id="a-audio"
|
||||
src="take.mp4"
|
||||
data-start="0"
|
||||
data-duration="2"
|
||||
data-media-start="4"
|
||||
data-track-index="10"
|
||||
></audio>
|
||||
<audio
|
||||
id="b-audio"
|
||||
src="take.mp4"
|
||||
data-start="2"
|
||||
data-duration="3"
|
||||
data-media-start="10"
|
||||
data-track-index="10"
|
||||
></audio>
|
||||
```
|
||||
|
||||
Timeline math: B starts at A start + duration. Source math: each range starts at `data-media-start`; consumed source = timeline duration × rate. Audio follows: duplicate matching `<audio>` ranges/timing. Owner: `/hyperframes-core`. Limit: adjacent windows only; author the two windows edge to edge. Same-track overlap is valid; both clips paint in CSS order.
|
||||
Timeline math: B starts at A start + duration. Source math: each range starts at `data-media-start`; consumed source = timeline duration × rate. Audio follows: the sound moves with each video clip. Owner: `/hyperframes-core`. Limit: adjacent windows only; author the two windows edge to edge. Same-track overlap is valid; both clips paint in CSS order.
|
||||
|
||||
## Trim in/out
|
||||
|
||||
@@ -61,20 +45,12 @@ Timeline math: B starts at A start + duration. Source math: each range starts at
|
||||
data-duration="3"
|
||||
data-media-start="6"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
<audio
|
||||
id="shot-1-audio"
|
||||
src="take.mp4"
|
||||
data-start="1"
|
||||
data-duration="3"
|
||||
data-media-start="6"
|
||||
data-track-index="10"
|
||||
></audio>
|
||||
```
|
||||
|
||||
Timeline math: visible window is `[1,4]`. Source math: in=6, out=6+3 at 1x; never invent source-end syntax. Audio follows: matching separate audio track uses the same three attributes. Owner: `/hyperframes-core`. Limit: use another clip for another range.
|
||||
Timeline math: visible window is `[1,4]`. Source math: in=6, out=6+3 at 1x; never invent source-end syntax. Audio follows: the sound moves with the video clip, using the same three attributes. Owner: `/hyperframes-core`. Limit: use another clip for another range.
|
||||
|
||||
## Split / splice
|
||||
|
||||
@@ -86,8 +62,8 @@ Timeline math: visible window is `[1,4]`. Source math: in=6, out=6+3 at 1x; neve
|
||||
data-duration="2"
|
||||
data-media-start="0"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
<video
|
||||
id="shot-2"
|
||||
@@ -96,28 +72,12 @@ Timeline math: visible window is `[1,4]`. Source math: in=6, out=6+3 at 1x; neve
|
||||
data-duration="2"
|
||||
data-media-start="8"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
<audio
|
||||
id="shot-1-audio"
|
||||
src="take.mp4"
|
||||
data-start="0"
|
||||
data-duration="2"
|
||||
data-media-start="0"
|
||||
data-track-index="10"
|
||||
></audio>
|
||||
<audio
|
||||
id="shot-2-audio"
|
||||
src="take.mp4"
|
||||
data-start="2"
|
||||
data-duration="2"
|
||||
data-media-start="8"
|
||||
data-track-index="10"
|
||||
></audio>
|
||||
```
|
||||
|
||||
Timeline math: splice at t=2. Source math: independent source offsets select kept pieces. Audio follows: split matching audio identically. Owner: `/hyperframes-core`. Limit: source cuts are core, never keyframes.
|
||||
Timeline math: splice at t=2. Source math: independent source offsets select kept pieces. Audio follows: the sound moves with each video clip, so it splits identically. Owner: `/hyperframes-core`. Limit: source cuts are core, never keyframes.
|
||||
|
||||
## Duplicate / reuse same source
|
||||
|
||||
@@ -129,8 +89,8 @@ Timeline math: splice at t=2. Source math: independent source offsets select kep
|
||||
data-duration="1"
|
||||
data-media-start="2"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
<video
|
||||
id="shot-2"
|
||||
@@ -139,28 +99,12 @@ Timeline math: splice at t=2. Source math: independent source offsets select kep
|
||||
data-duration="1"
|
||||
data-media-start="2"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
<audio
|
||||
id="shot-1-audio"
|
||||
src="take.mp4"
|
||||
data-start="0"
|
||||
data-duration="1"
|
||||
data-media-start="2"
|
||||
data-track-index="10"
|
||||
></audio>
|
||||
<audio
|
||||
id="shot-2-audio"
|
||||
src="take.mp4"
|
||||
data-start="4"
|
||||
data-duration="1"
|
||||
data-media-start="2"
|
||||
data-track-index="10"
|
||||
></audio>
|
||||
```
|
||||
|
||||
Timeline math: copies may occupy different starts. Source math: identical offsets reuse identical source. Audio follows: duplicate the separate audio track too. Owner: `/hyperframes-core`. Limit: every element needs a unique id when ids are present.
|
||||
Timeline math: copies may occupy different starts. Source math: identical offsets reuse identical source. Audio follows: the sound moves with each video clip, so each copy carries its own. Owner: `/hyperframes-core`. Limit: every element needs a unique id when ids are present.
|
||||
|
||||
## Reorder
|
||||
|
||||
@@ -172,8 +116,8 @@ Timeline math: copies may occupy different starts. Source math: identical offset
|
||||
data-duration="2"
|
||||
data-media-start="10"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
<video
|
||||
id="shot-2"
|
||||
@@ -182,28 +126,12 @@ Timeline math: copies may occupy different starts. Source math: identical offset
|
||||
data-duration="2"
|
||||
data-media-start="2"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
<audio
|
||||
id="shot-1-audio"
|
||||
src="take.mp4"
|
||||
data-start="0"
|
||||
data-duration="2"
|
||||
data-media-start="10"
|
||||
data-track-index="10"
|
||||
></audio>
|
||||
<audio
|
||||
id="shot-2-audio"
|
||||
src="take.mp4"
|
||||
data-start="2"
|
||||
data-duration="2"
|
||||
data-media-start="2"
|
||||
data-track-index="10"
|
||||
></audio>
|
||||
```
|
||||
|
||||
Timeline math: `data-start` defines authored order. Source math: source offsets need not be chronological. Audio follows: reorder identical matching audio windows. Owner: `/hyperframes-core`. Limit: reordering changes placement only, not source ranges.
|
||||
Timeline math: `data-start` defines authored order. Source math: source offsets need not be chronological. Audio follows: the sound moves with the video clip, so reordering clips reorders their sound. Owner: `/hyperframes-core`. Limit: reordering changes placement only, not source ranges.
|
||||
|
||||
## Freeze / hold
|
||||
|
||||
@@ -224,12 +152,12 @@ Timeline math: the still owns its hold duration. Source math: final-source frame
|
||||
data-media-start="4"
|
||||
data-playback-rate="0.5"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
```
|
||||
|
||||
Timeline math: duration is authored timeline time. Source math: consumed source = timeline duration × rate; natural timeline duration = remaining source / rate. Audio follows: matching separate audio track uses the same constant rate. Owner: `/hyperframes-core`. Limit: normalized 0.1..10. For a speed ramp put a `rate` lane in `data-automation`, e.g. `{"version":1,"lanes":[{"target":"rate","points":[{"t":0,"v":1},{"t":2,"v":4}]}]}`; it wins over the constant.
|
||||
Timeline math: duration is authored timeline time. Source math: consumed source = timeline duration × rate; natural timeline duration = remaining source / rate. Audio follows: the sound moves with the video clip and plays at the same constant rate. Owner: `/hyperframes-core`. Limit: normalized 0.1..10. For a speed ramp put a `rate` lane in `data-automation`, e.g. `{"version":1,"lanes":[{"target":"rate","points":[{"t":0,"v":1},{"t":2,"v":4}]}]}`; it wins over the constant.
|
||||
|
||||
## Zoom / punch
|
||||
|
||||
@@ -250,7 +178,7 @@ tl.fromTo(
|
||||
);
|
||||
```
|
||||
|
||||
Timeline math: move spans four authored seconds. Source math: unchanged. Audio follows: matching clip timing remains separate. Owner: `/hyperframes-keyframes`. Limit: authored geometry, not automatic face tracking.
|
||||
Timeline math: move spans four authored seconds. Source math: unchanged. Audio follows: the sound stays on the video clip; the tween does not touch it. Owner: `/hyperframes-keyframes`. Limit: authored geometry, not automatic face tracking.
|
||||
|
||||
## Crop / reframe
|
||||
|
||||
@@ -271,7 +199,7 @@ tl.fromTo(
|
||||
);
|
||||
```
|
||||
|
||||
Timeline math: overlap placed clips for the 0.5s handoff. Source math: each clip keeps its own range. Audio follows: place matching audio on its own tracks. Owner: `/hyperframes-keyframes` + `/hyperframes-animation`. Limit: visual mask/polygon/split-screen only; source cuts stay `/hyperframes-core`.
|
||||
Timeline math: overlap placed clips for the 0.5s handoff. Source math: each clip keeps its own range. Audio follows: the sound stays on each video clip. Owner: `/hyperframes-keyframes` + `/hyperframes-animation`. Limit: visual mask/polygon/split-screen only; source cuts stay `/hyperframes-core`.
|
||||
|
||||
## Crossfade
|
||||
|
||||
@@ -283,8 +211,9 @@ Timeline math: overlap placed clips for the 0.5s handoff. Source math: each clip
|
||||
data-duration="3"
|
||||
data-track-index="0"
|
||||
src="a.mp4"
|
||||
muted
|
||||
data-automation='{"version":1,"lanes":[{"target":"volume","points":[{"t":0,"v":1},{"t":2.5,"v":1},{"t":3,"v":0}]}]}'
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
</div>
|
||||
<div id="b-visual" class="inner">
|
||||
@@ -294,26 +223,11 @@ Timeline math: overlap placed clips for the 0.5s handoff. Source math: each clip
|
||||
data-duration="3"
|
||||
data-track-index="1"
|
||||
src="b.mp4"
|
||||
muted
|
||||
data-automation='{"version":1,"lanes":[{"target":"volume","points":[{"t":0,"v":0},{"t":0.5,"v":1},{"t":3,"v":1}]}]}'
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
</div>
|
||||
<audio
|
||||
id="a-audio"
|
||||
src="a.mp4"
|
||||
data-start="0"
|
||||
data-duration="3"
|
||||
data-track-index="10"
|
||||
data-automation='{"version":1,"lanes":[{"target":"volume","points":[{"t":0,"v":1},{"t":2.5,"v":1},{"t":3,"v":0}]}]}'
|
||||
></audio>
|
||||
<audio
|
||||
id="b-audio"
|
||||
src="b.mp4"
|
||||
data-start="2.5"
|
||||
data-duration="3"
|
||||
data-track-index="11"
|
||||
data-automation='{"version":1,"lanes":[{"target":"volume","points":[{"t":0,"v":0},{"t":0.5,"v":1},{"t":3,"v":1}]}]}'
|
||||
></audio>
|
||||
<script>
|
||||
const tl = gsap.timeline({ paused: true });
|
||||
tl.set("#b-visual", { opacity: 0 }, 0)
|
||||
@@ -323,7 +237,7 @@ Timeline math: overlap placed clips for the 0.5s handoff. Source math: each clip
|
||||
</script>
|
||||
```
|
||||
|
||||
Timeline math: distinct tracks overlap by 0.5s with opposing opacity envelopes. Source math: each source range remains independent. Audio follows: opposing volume envelopes on distinct audio tracks. Owner: `/hyperframes-core` + `/hyperframes-keyframes` + `/hyperframes-audio`. Limit: the crossfade is the opacity/volume envelopes, not a source-level dissolve.
|
||||
Timeline math: distinct tracks overlap by 0.5s with opposing opacity envelopes. Source math: each source range remains independent. Audio follows: opposing volume envelopes on each video's own `data-automation`, because the sound stays on the video. Owner: `/hyperframes-core` + `/hyperframes-keyframes` + `/hyperframes-audio`. Limit: the crossfade is the opacity/volume envelopes, not a source-level dissolve.
|
||||
|
||||
## Volume fades / ducking
|
||||
|
||||
@@ -338,7 +252,7 @@ Timeline math: distinct tracks overlap by 0.5s with opposing opacity envelopes.
|
||||
></audio>
|
||||
```
|
||||
|
||||
Timeline math: lane `t` is clip-local authored time: fade-in 0–1, duck down 2–2.2, hold 2.2–3, duck up 3–3.2, fade-out 4–5. Source math: source selection still uses core attributes. Audio follows: the explicit down-hold-up envelope affects this separate audio track. Owner: `/hyperframes-audio`. Limit: automation is not source retiming.
|
||||
Timeline math: lane `t` is clip-local authored time: fade-in 0–1, duck down 2–2.2, hold 2.2–3, duck up 3–3.2, fade-out 4–5. Source math: source selection still uses core attributes. Audio follows: the explicit down-hold-up envelope affects this `<audio>` (music is separate sound). The same lane works on a `<video data-has-audio="true">`. Owner: `/hyperframes-audio`. Limit: automation is not source retiming.
|
||||
|
||||
**One rule for volume over time: use the lane.** `lint` accepts a timeline tween on `volume` too, but when a track has both, the lane wins and the tween is ignored (`audio_volume_double_automation`). Never add a lane to a track that already has a `volume` tween, and never add a tween to a track that has a lane; edit the one that exists. To ramp 0.1 to 0.5 over ten seconds, write `{"t":0,"v":0.1},{"t":10,"v":0.5}`. `t` is seconds from the clip's own start, so a ramp past `data-duration` never finishes: check the clip's length before choosing the times. `data-volume` stays as the static level of the clip and combines with nothing else you author here.
|
||||
|
||||
@@ -353,21 +267,55 @@ Timeline math: lane `t` is clip-local authored time: fade-in 0–1, duck down 2
|
||||
data-media-start="8"
|
||||
data-playback-rate="2"
|
||||
data-track-index="0"
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
```
|
||||
|
||||
Timeline math: picture and sound share start/duration because the sound stays on the clip (`data-has-audio="true"`). Source math: both consume four source seconds. Audio follows: identical timing, range, and rate, with nothing to keep in sync. Owner: `/hyperframes-core` + `/hyperframes-audio`. Limit: no waveform auto-sync or drift correction.
|
||||
|
||||
A J cut or L cut is the case that needs a separate `<audio>`: picture and sound are cut independently, so the sound gets its own element (the same goes for replacement audio, a voiceover, or music). Mute the video whose sound you are replacing.
|
||||
|
||||
```html
|
||||
<!-- Outgoing shot: picture runs 0-5, its own sound is a separate clip that ends at the audio cut (4). -->
|
||||
<video
|
||||
id="shot-1"
|
||||
src="intro.mp4"
|
||||
data-start="0"
|
||||
data-duration="5"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
<audio
|
||||
id="shot-1-audio"
|
||||
src="intro.mp4"
|
||||
data-start="0"
|
||||
data-duration="4"
|
||||
data-track-index="10"
|
||||
></audio>
|
||||
<!-- Incoming shot: picture starts at 5, its sound leads it by one second. -->
|
||||
<video
|
||||
id="shot-2"
|
||||
src="take.mp4"
|
||||
data-start="3"
|
||||
data-duration="2"
|
||||
data-media-start="8"
|
||||
data-playback-rate="2"
|
||||
data-start="5"
|
||||
data-duration="3"
|
||||
data-media-start="12"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
<audio
|
||||
id="shot-2-audio"
|
||||
src="take.mp4"
|
||||
data-start="4"
|
||||
data-duration="4"
|
||||
data-media-start="11"
|
||||
data-track-index="10"
|
||||
></audio>
|
||||
```
|
||||
|
||||
Timeline math: picture and sound share start/duration. Source math: both consume four source seconds. Audio follows: identical timing, range, and rate on the separate audio track. Owner: `/hyperframes-core` + `/hyperframes-audio`. Limit: no waveform auto-sync or drift correction.
|
||||
The sound leads the picture by one second (a J cut): `shot-2-audio` starts at 4 and reads from source 11, while the picture starts at 5 and reads from 12. Both stay on the same source clock. Every video in a J or L cut is `muted` and its sound is its own `<audio>`: the outgoing shot's `<audio>` ends at the audio cut (4) while its picture carries on to 5, so two sounds never overlap on the same source. An audible `<video>` and an `<audio>` on the same file are only flagged when their time windows overlap.
|
||||
|
||||
## Align a sound to an on-screen event
|
||||
|
||||
@@ -426,7 +374,7 @@ Write what Studio writes when a person drops a file on the timeline, so an agent
|
||||
- **Image: `data-duration` is optional and defaults to 3 seconds**, the same as a dropped image, because a still has no length of its own. Write it only for another length. A test keeps the 3 equal to the default in code.
|
||||
- **Video and audio: `data-start` is enough.** The length comes from the media itself. An authored `data-duration` shorter than the file is a trim, never a requirement; leave it out unless the request asks for a shorter clip.
|
||||
- **Start: the playhead or the requested time, never a silent `0`.** Studio's asset-panel Add uses the playhead time on track `0`; a drop uses the drop point.
|
||||
- Give every clip `id`, `class="clip"`, `data-start` and `data-track-index`. Video is `muted playsinline`; audio carries `data-volume="1"`.
|
||||
- Give every clip `id`, `class="clip"`, `data-start` and `data-track-index`. A video with sound is `playsinline data-has-audio="true"`; silent footage and b-roll is `muted playsinline`. Audio carries `data-volume="1"`.
|
||||
- Then make sure the root composition's `data-duration` is at least the clip's end (`data-start` plus its length: 3 for an image unless you set another, the media's length for video and audio): Studio raises a declared root duration to cover the new clip, so an agent must too, or the clip lies past the end and never plays.
|
||||
- **Images and video fill the whole frame**: absolutely positioned at `left: 0; top: 0`, `width` and `height` equal to the composition's `data-width` and `data-height`, `object-fit: contain`. Studio does not know a dropped file's natural size, so it does not centre a smaller one.
|
||||
- `z-index` is the number of top-level clips already in that file plus one (at least `1`); later clips stack above earlier ones.
|
||||
@@ -479,13 +427,13 @@ Inside a sub-composition file, `data-start` is scene-local (see `## Align a soun
|
||||
data-duration="4"
|
||||
data-media-start="0"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
></video>
|
||||
```
|
||||
|
||||
Timeline math: change only `src`. Source math: reset `data-media-start` to the offset you want in the NEW file, and set `data-duration` no longer than the new file's remaining length (probe it with `ffprobe`). Audio follows: a separate `<audio>` that pointed at the old file needs the same `src` swap. Keep `id`, `data-start`, `data-track-index` and any `data-automation` so nothing else moves. Run `lint`: `audio_src_not_found` and `media_src_kind_mismatch` catch a wrong path or kind. Owner: `/hyperframes-core`. Limit: a still swapped for a video (or the reverse) is a tag change, not a swap.
|
||||
Timeline math: change only `src`. Source math: reset `data-media-start` to the offset you want in the NEW file, and set `data-duration` no longer than the new file's remaining length (probe it with `ffprobe`). Audio follows: the sound moves with the video clip; a separate `<audio>` that pointed at the old file (music, voiceover) needs its own `src` swap. Keep `id`, `data-start`, `data-track-index` and any `data-automation` so nothing else moves. Run `lint`: `audio_src_not_found` and `media_src_kind_mismatch` catch a wrong path or kind. Owner: `/hyperframes-core`. Limit: a still swapped for a video (or the reverse) is a tag change, not a swap.
|
||||
|
||||
## Split a section and change its speed
|
||||
|
||||
Timeline math: a section that is a sub-composition or a group of clips has no `data-playback-rate` of its own to set; split it by giving each half its own host or clips and shift everything after the cut by the length change. New length of a part = old length / rate. Every later `data-start` (clips, audio, root-timeline tweens) moves by the same delta. Source math: `<video>` and `<audio>` parts use `data-playback-rate` (0.1 to 10, constant) per the constant-speed recipe above, with matching audio. A speed ramp (a rate that changes within one clip) is a `rate` lane in `data-automation` on the `<video>`/`<audio>`; see `docs/reference/speed-ramps`. Say which you did.
|
||||
Timeline math: a section that is a sub-composition or a group of clips has no `data-playback-rate` of its own to set; split it by giving each half its own host or clips and shift everything after the cut by the length change. New length of a part = old length / rate. Every later `data-start` (clips, audio, root-timeline tweens) moves by the same delta. Source math: `<video>` and `<audio>` parts use `data-playback-rate` (0.1 to 10, constant) per the constant-speed recipe above, the sound moves with the video. A speed ramp (a rate that changes within one clip) is a `rate` lane in `data-automation` on the `<video>`/`<audio>`; see `docs/reference/speed-ramps`. Say which you did.
|
||||
|
||||
@@ -26,15 +26,15 @@ The root should be `position: relative`, have explicit pixel dimensions, and hid
|
||||
|
||||
**Nesting is allowed.** A timed element inside a wrapper is still timed, and a timed ancestor clamps its descendants: a child cannot be visible while its timed ancestor is hidden. Direct children of the root get automatic layout (see "Root-level clips get automatic layout" below); nested ones do not, so give them their own positioning.
|
||||
|
||||
| Attribute | Required | Meaning |
|
||||
| ------------------ | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `id` | Yes on `<video>`/`<audio>`, else recommended | `lint` errors with `media_missing_id` on media without one, and an id-less `<audio>` is never mixed, so the render is **silent**. Elsewhere it is a warning (`studio_missing_editable_id`): Studio needs a stable edit target, and timeline targets reference it. |
|
||||
| `data-start` | Yes | Start time in seconds, or a supported clip-time reference. This attribute is what marks the element as timed. |
|
||||
| `data-duration` | Required for `div` and sub-compositions | Duration in seconds. An `img` with `data-start` defaults to 3 s; video/audio default to the source length (less the playback offset, over the rate) once known; an authored value trims. Without any resolvable duration the element has no end and stays visible for the rest of the composition. |
|
||||
| `data-track-index` | No | Studio timeline lane, display only. The render never reads it, and clips on one track may overlap in time. Absent, the parser defaults it and Studio lays out one lane per clip. Two `<audio>` elements on the same index that overlap in time raise a `lint` warning. |
|
||||
| `data-media-start` | No | Offset into the media source, in seconds. |
|
||||
| `data-volume` | No | Static audio gain, default `1` (0 dB). `0` is silence and values above `1` boost, up to `3.98` (+12 dB) — Studio's fader writes this. For fades and ducking, use the `data-automation` volume lane (see `creator-editing-recipes.md`). |
|
||||
| `data-has-audio` | No (`<video>` only) | `"true"` to declare the video carries an audio track when auto-detection would miss it. |
|
||||
| Attribute | Required | Meaning |
|
||||
| ------------------ | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `id` | Yes on `<video>`/`<audio>`, else recommended | `lint` errors with `media_missing_id` on media without one, and an id-less `<audio>` is never mixed, so the render is **silent**. Elsewhere it is a warning (`studio_missing_editable_id`): Studio needs a stable edit target, and timeline targets reference it. |
|
||||
| `data-start` | Yes | Start time in seconds, or a supported clip-time reference. This attribute is what marks the element as timed. |
|
||||
| `data-duration` | Required for `div` and sub-compositions | Duration in seconds. An `img` with `data-start` defaults to 3 s; video/audio default to the source length (less the playback offset, over the rate) once known; an authored value trims. Without any resolvable duration the element has no end and stays visible for the rest of the composition. |
|
||||
| `data-track-index` | No | Studio timeline lane, display only. The render never reads it, and clips on one track may overlap in time. Absent, the parser defaults it and Studio lays out one lane per clip. Two `<audio>` elements on the same index that overlap in time raise a `lint` warning. |
|
||||
| `data-media-start` | No | Offset into the media source, in seconds. |
|
||||
| `data-volume` | No | Static audio gain, default `1` (0 dB). `0` is silence and values above `1` boost, up to `3.98` (+12 dB) — Studio's fader writes this. For fades and ducking, use the `data-automation` volume lane (see `creator-editing-recipes.md`). |
|
||||
| `data-has-audio` | Required on a timed `<video>` unless it is `muted` | `"true"` keeps the file's sound on this clip (the default for footage with sound). Silent footage uses `muted` instead; a video with neither fails lint (`video_missing_muted`). |
|
||||
|
||||
**The visibility window is half-open: `[start, start + duration)`.** A clip shows while `start ≤ t < start + duration` and is hidden at exactly `t = start + duration`. Land an animation's resolved end state slightly **before** `data-duration`, not on it, or its last frame is never rendered. Two clips can therefore be authored back to back (`b.start === a.start + a.duration`) with no overlapping frame.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ A clip is any DOM element with `data-start` and, where required, `data-duration`
|
||||
|
||||
- **Visual `<div>` clips** — scenes, cards, overlays. Always require `data-duration`.
|
||||
- **Sub-composition hosts** — `<div>` with `data-composition-src`. Always require `data-duration`.
|
||||
- **Video clips** — `<video>` with `muted` and `playsinline`. Duration can default to media length.
|
||||
- **Video clips** — `<video playsinline>`. With sound: `data-has-audio="true"` (the sound stays on the clip). Silent: `muted`. Duration can default to media length.
|
||||
- **Audio clips** — `<audio>`. Duration can default to media length.
|
||||
- **Image clips** — `<img>`. `data-duration` is optional and defaults to 3 seconds; write it only for another length.
|
||||
|
||||
@@ -46,9 +46,7 @@ multiple clip elements. Each copy selects its source range with
|
||||
timeline with `data-start`. Change the source offsets and placement order; do
|
||||
not try to keyframe source cutting.
|
||||
|
||||
Separately authored audio gives each audio copy the identical source range and
|
||||
timing as its matching video clip (`data-media-start`, `data-duration`, and
|
||||
`data-start`). Video stays muted; the separate audio elements carry sound.
|
||||
Each video segment keeps its sound: the sound stays on the clip (`data-has-audio="true"`), so cutting the video cuts its sound. A separate `<audio>` is for other sound (music, voiceover, replacement audio, J/L cuts).
|
||||
|
||||
## Relative Timing
|
||||
|
||||
|
||||
@@ -66,27 +66,29 @@ document.getElementById("title").textContent = title;
|
||||
|
||||
The one real constraint is about **timelines, not media placement**: a sub-composition timeline **cannot reach or animate host elements** — neither `document.querySelector("#host-id")` nor a gsap selector string (`tl.to("#host-id", …)`) resolves across the boundary; a sub-comp timeline only drives its own subtree. So if a media element lives at the host root, **its per-scene motion (scale/opacity/morph/tilt/breathing) must be authored on the MAIN timeline in `index.html`, at GLOBAL time** (scene-local time + the scene slot's `data-start`). Keeping the media inside the scene sub-comp instead lets that sub-comp's own timeline animate it with scene-local time. For 3D tilt without a perspective parent, use gsap `transformPerspective` on the element. See `composition-patterns.md` archetype B.
|
||||
|
||||
Video elements must be muted and inline. Audio must be a separate `<audio>` element, even when it uses the same source file.
|
||||
A video with sound keeps it on the `<video>` (`data-has-audio="true"`, no `muted`). Use a separate `<audio>` for music, voiceover, replacement audio, J/L cuts, or audio detached in Studio. Silent footage and b-roll: `muted`.
|
||||
|
||||
```html
|
||||
<video
|
||||
id="a-roll"
|
||||
class="clip"
|
||||
src="assets/demo.mp4"
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
data-start="0"
|
||||
data-duration="12"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
data-volume="1"
|
||||
></video>
|
||||
|
||||
<!-- Separate <audio> only for other sound: music, voiceover, replacement audio. -->
|
||||
<audio
|
||||
id="a-roll-audio"
|
||||
src="assets/demo.mp4"
|
||||
id="music"
|
||||
src="assets/bed.mp3"
|
||||
data-start="0"
|
||||
data-duration="12"
|
||||
data-track-index="10"
|
||||
data-volume="1"
|
||||
data-track-index="2"
|
||||
data-volume="0.4"
|
||||
></audio>
|
||||
```
|
||||
|
||||
@@ -99,7 +101,7 @@ Video elements must be muted and inline. Audio must be a separate `<audio>` elem
|
||||
- **Sub-compositions are exempt and work.** A `<video>`/`<audio>` inside a sub-composition renders identically to one at the host root, because a composition host propagates its offset. Only plain timed wrappers (a `<section data-start>` around a `<video data-start>`) break.
|
||||
- **Never** add `crossorigin` to `<video>`/`<audio>`. `lint` rejects it unconditionally (`media_crossorigin_breaks_preview`, error) because a media host without `Access-Control-Allow-Origin` then fails silently in preview while renders still work, hiding the bug. There is no suppression, so this holds even for the canvas/WebGL/WebAudio readback case.
|
||||
- **Every `<audio>` needs an `id`.** The mixer selects `audio[id][src]`, so an id-less `<audio>` is never mixed and the render is **silent**. `lint` catches it as `media_missing_id`.
|
||||
- Audio always lives on a separate `<audio>` element — even if its source file is the same as a `<video>`. The `<video>` is muted; the `<audio>` carries sound.
|
||||
- A video's own sound stays on the `<video>` (`data-has-audio="true"`, no `muted`). Add a separate `<audio>` only for other sound (music, voiceover, replacement audio, J/L cuts) and mute the video it replaces.
|
||||
- For volume fades and ducking, use the `data-automation` volume lane; the exact form is in `creator-editing-recipes.md`. `data-volume` is the static baseline. A timeline `volume` tween is ignored when a lane is present.
|
||||
|
||||
For media duration: `<video>` and `<audio>` can omit `data-duration` if the media's intrinsic length is known and you want the full clip. Otherwise provide `data-duration` explicitly.
|
||||
|
||||
@@ -23,7 +23,7 @@ Animate a wrapper div for position/size. The video fills the wrapper. The wrappe
|
||||
data-duration="60"
|
||||
data-track-index="0"
|
||||
src="talking-head.mp4"
|
||||
muted
|
||||
data-has-audio="true"
|
||||
playsinline
|
||||
></video>
|
||||
</div>
|
||||
@@ -53,7 +53,7 @@ Three layers, plus one critical rule:
|
||||
data-media-start="0"
|
||||
data-track-index="0"
|
||||
src="presenter.mp4"
|
||||
muted
|
||||
data-has-audio="true"
|
||||
playsinline
|
||||
></video>
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ realizes it inside the frame file (HOW). **Obey the frame's `pacing`.**
|
||||
The asset-driven analogue of a per-onset typographic group: cut to a new clip on each anchor
|
||||
(the frame's beats/onsets from the audiomap). Each clip is a `class="clip"` element
|
||||
(`<img>` for a photo, **muted** `<video>` for a motion clip) placed at its anchor with
|
||||
`data-start`/`data-duration`/`data-track-index` per the core clip contract. Between clips,
|
||||
`data-start`/`data-duration`/`data-track-index` per the core clip contract. (Muted on purpose: the music track drives the sound.) Between clips,
|
||||
crossfade the outgoing content to `opacity:0` ending **at** the next anchor.
|
||||
Cut on the **strong** anchors; land a hero clip on a `key_moment`/downbeat.
|
||||
|
||||
|
||||
@@ -53,16 +53,16 @@ See [timing.md](timing.md) — this is the highest-leverage section.
|
||||
|
||||
See [media.md](media.md) for trim, volume ramps, and decoder notes.
|
||||
|
||||
| Remotion | HyperFrames |
|
||||
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `<Audio src volume>` | `<audio data-start data-duration data-track-index data-volume src>` |
|
||||
| `<Audio playbackRate startFrom endAt>` | `data-playback-rate`, `data-trim-start`, `data-trim-end` |
|
||||
| `<Video src>` | `<video muted playsinline data-start data-duration data-track-index src>` |
|
||||
| `<OffthreadVideo>` | `<video>` — HF doesn't need the off-thread variant (uses headless Chrome) |
|
||||
| `<Img src>` | `<img>` |
|
||||
| `<IFrame src>` | `<iframe>` — HF auto-falls back to screenshot mode for nested iframes |
|
||||
| `staticFile("x.png")` | `"assets/x.png"` in the root, `"../assets/x.png"` inside a `compositions/` scene file — copy the file into `hf-src/assets/` next to `index.html` |
|
||||
| `delayRender()` / `continueRender()` | drop — HF waits on asset readiness via the Frame Adapter pattern |
|
||||
| Remotion | HyperFrames |
|
||||
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `<Audio src volume>` | `<audio data-start data-duration data-track-index data-volume src>` |
|
||||
| `<Audio playbackRate startFrom endAt>` | `data-playback-rate`, `data-trim-start`, `data-trim-end` |
|
||||
| `<Video src>` | `<video playsinline data-has-audio="true" data-start data-duration data-track-index src>` only when the file has an audio stream (check with `ffprobe -v error -select_streams a -show_entries stream=codec_type -of csv=p=0 <file>`: non-empty means it has audio) and the source is not `muted` or `volume={0}`; otherwise `muted` instead |
|
||||
| `<OffthreadVideo>` | `<video>` — HF doesn't need the off-thread variant (uses headless Chrome) |
|
||||
| `<Img src>` | `<img>` |
|
||||
| `<IFrame src>` | `<iframe>` — HF auto-falls back to screenshot mode for nested iframes |
|
||||
| `staticFile("x.png")` | `"assets/x.png"` in the root, `"../assets/x.png"` inside a `compositions/` scene file — copy the file into `hf-src/assets/` next to `index.html` |
|
||||
| `delayRender()` / `continueRender()` | drop — HF waits on asset readiness via the Frame Adapter pattern |
|
||||
|
||||
## Transitions
|
||||
|
||||
|
||||
@@ -70,14 +70,25 @@ is the static baseline.
|
||||
## `<Video>` and `<OffthreadVideo>`
|
||||
|
||||
```tsx
|
||||
<Video src={staticFile("intro.mp4")} muted playsInline />
|
||||
<OffthreadVideo src={staticFile("intro.mp4")} muted />
|
||||
<Video src={staticFile("intro.mp4")} playsInline />
|
||||
<OffthreadVideo src={staticFile("intro.mp4")} />
|
||||
```
|
||||
|
||||
```html
|
||||
<!-- intro.mp4 has an audio stream -->
|
||||
<video
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
data-start="0"
|
||||
data-duration="5"
|
||||
data-track-index="0"
|
||||
src="assets/intro.mp4"
|
||||
></video>
|
||||
|
||||
<!-- intro.mp4 is silent (or the Remotion element is muted) -->
|
||||
<video
|
||||
playsinline
|
||||
muted
|
||||
data-start="0"
|
||||
data-duration="5"
|
||||
data-track-index="0"
|
||||
@@ -89,8 +100,14 @@ is the static baseline.
|
||||
rendering. HF runs in headless Chrome already, so the off-thread variant
|
||||
collapses to a regular `<video>`.
|
||||
|
||||
`muted` and `playsinline` are required for the runtime to autoplay
|
||||
(browser policy). Always emit them.
|
||||
`playsinline` is required for the runtime to autoplay (browser policy); always
|
||||
emit it. Check the source file for an audio stream first:
|
||||
`ffprobe -v error -select_streams a -show_entries stream=codec_type -of csv=p=0 <file>`
|
||||
(non-empty output means it has audio). Remotion `<Video>` maps to
|
||||
`<video playsinline>` + `data-has-audio="true"` only when the file has an audio
|
||||
stream and the Remotion source does not set `muted` or `volume={0}`; otherwise
|
||||
emit `muted`. Silent b-roll with `data-has-audio="true"` aborts the render
|
||||
(`ASSET_MEDIA_TYPE_MISMATCH`).
|
||||
|
||||
## `<Img>`
|
||||
|
||||
|
||||
@@ -939,22 +939,13 @@ ffmpeg -y -i "$VIDEO_PATH" -c:v libx264 -crf 18 -g 30 -keyint_min 30 \
|
||||
<video
|
||||
id="bg-video"
|
||||
src="input-video.mp4"
|
||||
muted
|
||||
playsinline
|
||||
data-has-audio="true"
|
||||
data-start="0"
|
||||
data-duration="121.2"
|
||||
data-track-index="1"
|
||||
></video>
|
||||
</div>
|
||||
<!-- Preserve the source program audio while the visual video stays muted. -->
|
||||
<audio
|
||||
id="source-audio"
|
||||
src="input-video.mp4"
|
||||
data-start="0"
|
||||
data-duration="121.2"
|
||||
data-track-index="10"
|
||||
data-volume="1"
|
||||
></audio>
|
||||
|
||||
<!-- Layer 2: each card-host sits at the bounds dictated by its layout. -->
|
||||
<!-- IMPORTANT: every card-host MUST carry BOTH "card-host" and "clip" classes. -->
|
||||
@@ -1182,11 +1173,8 @@ PRODUCER_BROWSER_GPU_MODE=hardware npx hyperframes render public \
|
||||
```
|
||||
|
||||
`hyperframes render <dir>` reads `<dir>/index.html` and produces the MP4.
|
||||
The canonical composition keeps the visual `<video>` muted and mounts the same
|
||||
source as the root `#source-audio` track, so the rendered MP4 preserves the
|
||||
talking-head audio without a manual remux. This uses a separate audio track
|
||||
rather than `data-has-audio="true"` so its volume and ducking remain independently
|
||||
controllable on the timeline.
|
||||
The source program audio stays on `#bg-video`, so the rendered MP4 preserves the
|
||||
talking-head audio without a manual remux.
|
||||
The flag `PRODUCER_BROWSER_GPU_MODE=hardware` (or `--browser-gpu`) is
|
||||
strongly recommended on macOS — software-only Chrome rendering times out
|
||||
on most laptops.
|
||||
|
||||
@@ -16,20 +16,9 @@ function readAttribute(tag, attribute) {
|
||||
return match[1];
|
||||
}
|
||||
|
||||
test("canonical composition preserves source audio as a root media track", () => {
|
||||
test("canonical composition keeps the source program audio on the video", () => {
|
||||
const video = findTag("video", "bg-video");
|
||||
const audio = findTag("audio", "source-audio");
|
||||
|
||||
assert.match(video, /\bmuted\b/);
|
||||
assert.match(
|
||||
skill,
|
||||
/<\/div>\s*<!-- Preserve the source program audio[\s\S]*?<audio\b[^>]*\bid="source-audio"[^>]*>[\s\S]*?<\/audio>/,
|
||||
);
|
||||
assert.equal(readAttribute(audio, "src"), readAttribute(video, "src"));
|
||||
assert.equal(readAttribute(audio, "data-start"), readAttribute(video, "data-start"));
|
||||
assert.equal(readAttribute(audio, "data-duration"), readAttribute(video, "data-duration"));
|
||||
assert.notEqual(
|
||||
readAttribute(audio, "data-track-index"),
|
||||
readAttribute(video, "data-track-index"),
|
||||
);
|
||||
assert.doesNotMatch(video, /\bmuted\b/);
|
||||
assert.equal(readAttribute(video, "data-has-audio"), "true");
|
||||
assert.doesNotMatch(skill, /<audio\b[^>]*\bid="source-audio"/);
|
||||
});
|
||||
|
||||
@@ -9,8 +9,8 @@
|
||||
|
||||
How to apply in the composition
|
||||
Add ONE decorative div as a sibling of #video-wrap (not a child — the
|
||||
video element is muted/playsinline and the decorative div should sit
|
||||
above it). The accent color is configurable per scene.
|
||||
video element keeps its sound (data-has-audio="true") and plays inline,
|
||||
and the decorative div should sit above it). The accent color is configurable per scene.
|
||||
|
||||
<div class="video-wrapper" id="video-wrap">
|
||||
<video src="input-video.mp4" ...></video>
|
||||
|
||||
Reference in New Issue
Block a user