feat(studio,lint): carve targets voiceover groups — always, when plural

Plural voiceover carve now targets a group instead of naming each clip:
`resolveCarveSourceIds` (core `audioGroups.ts`) expands a group id to its
current members at analysis time, so a clip added to the group later is
covered without touching `sources`. The picker (`useFxCarve.ts`) offers a
grouped voice as one option instead of one row per member, tests overlap
as a union of member spans (a group overlaps the bed if ANY member does),
and prefers a qualifying group over its individual members in
`autoSourceIds`.

Picking two or more ungrouped voice clips in the carve flow now mints a
group behind them (`mintGroupId`, de-duped against every id in the
document) and writes `data-audio-group` on each picked clip atomically,
one undo entry — `createAudioGroupAndAssignMembers` in
`timelineTrackVisibility.ts` copies `setElementsHidden`'s multi-target
write shape. The DSP is untouched: `mixCarveSources` already sums
multiple sources correctly (verified in the design doc's own
investigation) — this only fixes the picker.

New lint rule `audio_carve_ungrouped_sources` (`packages/lint/src/rules/
media.ts`, alongside `audio_volume_double_automation`) warns when a
`data-fx-carve`'s `sources` names two or more plain clip ids instead of a
group — the shape that silently rots when a clip is added. `/hyperframes-
audio` states the same rule as an invariant, not a tip, with the grouped-
narration HTML example from the design doc.

The group-matching and auto-group logic (`withAutoGroupedSources`,
`collectCarveCandidates`) is split into `useFxCarveGrouping.ts` —
`useFxCarve.ts` was pushing past the 600-line cap. `resolveNextCarveSettings`
is deliberately NOT an `async function`: wrapping it in one would force a
microtask on every call, including the synchronous branch — the exact bug
`withAutoGroupedSources`'s own sync-when-possible contract exists to avoid,
and one caught via `propertyPanelAudioFxGroup.test.tsx` (10 failures)
before fixing it back to a plain function the caller conditionally awaits.

Also extracted `useEffectiveTimelineDuration` out of `App.tsx` and
`useRemoveBackground` out of `StudioRightPanel.tsx` (both pushed past 600
lines from an added prop wire), and decomposed `useFxCarve.ts`'s picker
IIFE to clear fallow's complexity gate.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
2026-08-20 02:11:37 -07:00
co-authored by Claude Sonnet 5
parent 8d48a6f52f
commit 1abad17650
22 changed files with 914 additions and 68 deletions
+47 -1
View File
@@ -1,5 +1,10 @@
import { beforeEach, describe, expect, it } from "vitest";
import { audioGroupOf, HF_AUDIO_GROUP_ATTR, resolveAudioGroups } from "./audioGroups.js";
import {
audioGroupOf,
HF_AUDIO_GROUP_ATTR,
resolveAudioGroups,
resolveCarveSourceIds,
} from "./audioGroups.js";
beforeEach(() => {
document.body.innerHTML = "";
@@ -64,6 +69,47 @@ describe("audioGroupOf", () => {
});
});
describe("resolveCarveSourceIds", () => {
it("expands a group id to its current members", () => {
document.body.innerHTML = `
<audio id="vo-1" data-audio-group="voiceover"></audio>
<audio id="vo-2" data-audio-group="voiceover"></audio>
`;
expect(resolveCarveSourceIds(document, ["voiceover"])).toEqual(["vo-1", "vo-2"]);
});
it("picks up a member added to the group after the carve was set (analysis-time, not frozen)", () => {
document.body.innerHTML = `
<audio id="vo-1" data-audio-group="voiceover"></audio>
<audio id="vo-2" data-audio-group="voiceover"></audio>
`;
expect(resolveCarveSourceIds(document, ["voiceover"])).toEqual(["vo-1", "vo-2"]);
document.body.insertAdjacentHTML(
"beforeend",
`<audio id="vo-3" data-audio-group="voiceover"></audio>`,
);
expect(resolveCarveSourceIds(document, ["voiceover"])).toEqual(["vo-1", "vo-2", "vo-3"]);
});
it("passes through a plain clip id that still exists", () => {
document.body.innerHTML = `<audio id="vo-1"></audio>`;
expect(resolveCarveSourceIds(document, ["vo-1"])).toEqual(["vo-1"]);
});
it("drops an id that resolves to nothing — a deleted clip, an empty or vanished group", () => {
document.body.innerHTML = `<audio id="vo-1"></audio>`;
expect(resolveCarveSourceIds(document, ["vo-1", "deleted", "no-such-group"])).toEqual(["vo-1"]);
});
it("dedupes and preserves first-seen order across a mix of group and plain ids", () => {
document.body.innerHTML = `
<audio id="vo-1" data-audio-group="voiceover"></audio>
<audio id="vo-2" data-audio-group="voiceover"></audio>
`;
expect(resolveCarveSourceIds(document, ["voiceover", "vo-1"])).toEqual(["vo-1", "vo-2"]);
});
});
describe(HF_AUDIO_GROUP_ATTR, () => {
it("is the attribute name membership is keyed on", () => {
expect(HF_AUDIO_GROUP_ATTR).toBe("data-audio-group");
+31
View File
@@ -51,6 +51,37 @@ export function resolveAudioGroups(root: ParentNode): HfAudioGroup[] {
return groups;
}
/**
* Expand a list of source ids for a carve: a plain id passes through if it
* still exists, a group id expands to its CURRENT members. Resolved fresh
* every time — group membership is never frozen into the carve's own
* attribute, so adding a fourth voice to a group already named in a carve's
* `sources` picks it up on the next analysis without editing that carve.
*
* Dedupes and preserves first-seen order; an id that resolves to nothing
* (a deleted clip, an empty or vanished group) is dropped rather than kept
* as a dangling reference the analysis would only fail to find anyway.
*/
export function resolveCarveSourceIds(doc: Document, ids: readonly string[]): string[] {
const groupsById = new Map(resolveAudioGroups(doc).map((group) => [group.id, group] as const));
const seen = new Set<string>();
const out: string[] = [];
const add = (id: string): void => {
if (seen.has(id)) return;
seen.add(id);
out.push(id);
};
for (const id of ids) {
const group = groupsById.get(id);
if (group) {
group.memberIds.forEach(add);
} else if (doc.getElementById(id)) {
add(id);
}
}
return out;
}
/** The group a member belongs to, or null. Groups do not nest — this ignores
* `data-audio-group` on an `<hf-audio-group>` element itself.
*
+48
View File
@@ -555,3 +555,51 @@ describe("audio_volume_double_automation", () => {
expect(res.findings.some((f) => f.code === "audio_volume_double_automation")).toBe(false);
});
});
describe("audio_carve_ungrouped_sources", () => {
const withCarve = (carveJson: string, extra = "") => `<!DOCTYPE html><html><body>
<div id="root" data-composition-id="main" data-start="0" data-width="1920" data-height="1080" data-duration="10">
<audio id="bed" src="bed.wav" data-start="0" data-duration="10" data-fx-carve='${carveJson}'></audio>
${extra}
</div>
</body></html>`;
it("warns when sources names two or more plain clip ids", async () => {
const res = await lintHyperframeHtml(
withCarve(`{"enabled":true,"sources":["vo-1","vo-2"],"strength":0.35}`),
);
const finding = res.findings.find((f) => f.code === "audio_carve_ungrouped_sources");
expect(finding?.severity).toBe("warning");
expect(finding?.elementId).toBe("bed");
});
it("stays quiet when sources names a group", async () => {
const res = await lintHyperframeHtml(
withCarve(
`{"enabled":true,"sources":["voiceover"],"strength":0.35}`,
`<hf-audio-group id="voiceover" data-label="Voiceover"></hf-audio-group>`,
),
);
expect(res.findings.some((f) => f.code === "audio_carve_ungrouped_sources")).toBe(false);
});
it("stays quiet for a single-clip sources list", async () => {
const res = await lintHyperframeHtml(
withCarve(`{"enabled":true,"sources":["narration"],"strength":0.35}`),
);
expect(res.findings.some((f) => f.code === "audio_carve_ungrouped_sources")).toBe(false);
});
it("still warns when one entry is a group and the rest are plain clip ids", async () => {
// Mixing a group with two more bare clip ids is still an ungrouped-source
// rot risk for those two clips — only fully-grouped sources are silent.
const res = await lintHyperframeHtml(
withCarve(
`{"enabled":true,"sources":["voiceover","vo-3","vo-4"],"strength":0.35}`,
`<hf-audio-group id="voiceover" data-label="Voiceover"></hf-audio-group>`,
),
);
const finding = res.findings.find((f) => f.code === "audio_carve_ungrouped_sources");
expect(finding?.severity).toBe("warning");
});
});
+47
View File
@@ -632,6 +632,8 @@ export const mediaRules: Array<(ctx: LintContext) => HyperframeLintFinding[]> =
// audio_volume_tween_overrides_gain
findVolumeTweenOverridesGainFindings,
// audio_carve_ungrouped_sources
findCarveUngroupedSourcesFindings,
];
/**
@@ -722,3 +724,48 @@ function findVolumeDoubleAutomationFindings(ctx: LintContext): HyperframeLintFin
}
return findings;
}
/**
* A carve's `sources` naming two or more plain clip ids is the normative
* mistake groups exist to prevent (groups doc §1.6): the list silently rots
* when a voice clip is added or removed, since nothing re-derives it. Naming
* a group instead means membership resolves at analysis time. Silent when
* `sources` already names a group, or names at most one clip.
*/
function findCarveUngroupedSourcesFindings(ctx: LintContext): HyperframeLintFinding[] {
const groupIds = new Set(
ctx.tags.filter((tag) => tag.name === "hf-audio-group").map((tag) => readAttr(tag.raw, "id")),
);
const findings: HyperframeLintFinding[] = [];
for (const tag of ctx.tags) {
const raw = readDecodedAttr(tag.raw, "data-fx-carve");
if (raw === null) continue;
const trimmed = raw.trim();
if (!trimmed.startsWith("{")) continue;
let parsed: unknown;
try {
parsed = JSON.parse(trimmed);
} catch {
continue;
}
const sources = (parsed as { sources?: unknown }).sources;
if (!Array.isArray(sources)) continue;
const clipIds = sources.filter(
(id): id is string => typeof id === "string" && !groupIds.has(id),
);
if (clipIds.length < 2) continue;
const elementId = readAttr(tag.raw, "id") || undefined;
findings.push({
code: "audio_carve_ungrouped_sources",
severity: "warning",
message: `${elementId ? `#${elementId}'s` : "This"} carve names ${clipIds.length} voice clips directly (${clipIds.join(", ")}) instead of a group.`,
elementId,
fixHint:
"Group the voice clips and carve against the group — a hand-rolled clip list silently rots when a clip is added.",
snippet: truncateSnippet(tag.raw),
});
}
return findings;
}
+6 -7
View File
@@ -37,6 +37,7 @@ import { useCompositionDimensions } from "./hooks/useCompositionDimensions";
import { useToast } from "./hooks/useToast";
import { useCompositionContentLoader } from "./hooks/useCompositionContentLoader";
import { useStudioUrlState } from "./hooks/useStudioUrlState";
import { useEffectiveTimelineDuration } from "./hooks/useEffectiveTimelineDuration";
import {
buildStudioContextValue,
useGlobalFileDrop,
@@ -95,13 +96,10 @@ export function StudioApp() {
const setTimelineSelectionSet = usePlayerStore((s) => s.setSelectedElementIds);
const timelineDuration = usePlayerStore((s) => s.duration);
const isPlaying = usePlayerStore((s) => s.isPlaying);
const effectiveTimelineDuration = useMemo(() => {
const maxEnd =
timelineElements.length > 0
? Math.max(...timelineElements.map((el) => el.start + el.duration))
: 0;
return Math.max(timelineDuration, maxEnd);
}, [timelineDuration, timelineElements]);
const effectiveTimelineDuration = useEffectiveTimelineDuration(
timelineDuration,
timelineElements,
);
const { toasts, showToast, dismissToast } = useToast();
const panelLayout = usePanelLayout({
rightCollapsed: initialUrlStateRef.current.rightCollapsed,
@@ -532,6 +530,7 @@ export function StudioApp() {
domEditSaveTimestampRef={domEditSaveTimestampRef}
recordEdit={editHistory.recordEdit}
onToggleElementHidden={timelineEditing.handleToggleElementHidden}
onAutoGroupCarveSources={timelineEditing.handleAutoGroupCarveSources}
onAddMediaOverlay={handleAddMediaOverlay}
/>
)
@@ -1,4 +1,4 @@
import { useCallback, useEffect, useRef } from "react";
import { useCallback } from "react";
import type { StudioRightPanelProps } from "./StudioRightPanel.types";
export type { StudioRightPanelProps };
@@ -20,15 +20,14 @@ import { usePanelLayoutContext } from "../contexts/PanelLayoutContext";
import { useFileManagerContext } from "../contexts/FileManagerContext";
import { useDomEditContext } from "../contexts/DomEditContext";
import { usePlayerStore } from "../player";
import { waitForMediaJob } from "./studioMediaJobs";
import {
applyColorGradingScopeUpdate,
EMPTY_COLOR_GRADING_SCOPE_RESULT,
type ColorGradingScope,
} from "./studioColorGradingScope";
import type { BackgroundRemovalProgress } from "./editor/propertyPanelTypes";
import { timelineKeysForSelections } from "../utils/studioHelpers";
import { useInspectorSplitResize } from "../hooks/useInspectorSplitResize";
import { useRemoveBackground } from "../hooks/useRemoveBackground";
// fallow-ignore-next-line complexity
export function StudioRightPanel({
@@ -45,6 +44,7 @@ export function StudioRightPanel({
domEditSaveTimestampRef,
recordEdit,
onToggleElementHidden,
onAutoGroupCarveSources,
onAddMediaOverlay,
}: StudioRightPanelProps) {
const {
@@ -163,14 +163,6 @@ export function StudioRightPanel({
handleInspectorSplitResizeMove,
handleInspectorSplitResizeEnd,
} = useInspectorSplitResize();
const backgroundRemovalAbortRef = useRef<AbortController | null>(null);
useEffect(
() => () => {
backgroundRemovalAbortRef.current?.abort();
},
[],
);
const renderJobs = renderQueue.jobs as RenderJob[];
const inspectorTabActive = rightPanelTab === "design" || rightPanelTab === "layers";
@@ -237,52 +229,7 @@ export function StudioRightPanel({
],
);
const handleRemoveBackground = useCallback(
// fallow-ignore-next-line complexity
async (
inputPath: string,
options: {
createBackgroundPlate?: boolean;
quality?: "fast" | "balanced" | "best";
onProgress?: (progress: BackgroundRemovalProgress) => void;
},
) => {
const response = await fetch(
`/api/projects/${encodeURIComponent(projectId)}/media/remove-background`,
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
inputPath,
createBackgroundPlate: options.createBackgroundPlate === true,
quality: options.quality ?? "balanced",
}),
},
);
const data = (await response.json().catch(() => ({}))) as {
jobId?: string;
error?: string;
};
if (!response.ok || !data.jobId) {
throw new Error(data.error || `Background removal failed (${response.status})`);
}
showToast("Removing background...", "info");
backgroundRemovalAbortRef.current?.abort();
const controller = new AbortController();
backgroundRemovalAbortRef.current = controller;
try {
const result = await waitForMediaJob(data.jobId, options.onProgress, controller.signal);
await refreshFileTree();
showToast(`Created transparent asset: ${result.outputPath.split("/").pop()}`, "info");
return result;
} finally {
if (backgroundRemovalAbortRef.current === controller) {
backgroundRemovalAbortRef.current = null;
}
}
},
[projectId, refreshFileTree, showToast],
);
const handleRemoveBackground = useRemoveBackground(projectId, refreshFileTree, showToast);
/**
* A dial being dragged writes to the preview and stops there.
@@ -328,6 +275,7 @@ export function StudioRightPanel({
copiedAgentPrompt={copiedAgentPrompt}
onClearSelection={clearDomSelection}
onToggleElementHidden={onToggleElementHidden}
onAutoGroupCarveSources={onAutoGroupCarveSources}
onUngroup={handleUngroupSelection}
onSetStyle={handleDomStyleCommit}
onSetAttribute={handleDomAttributeCommit}
@@ -51,5 +51,6 @@ export interface StudioRightPanelProps extends StudioEditPersistenceProps {
files: Record<string, { before: string; after: string }>;
}) => Promise<void>;
onToggleElementHidden?: ToggleHiddenHandler;
onAutoGroupCarveSources?: (clipIds: readonly string[], groupId: string) => Promise<void>;
onAddMediaOverlay?: AddMediaOverlayHandler;
}
@@ -79,6 +79,7 @@ export function PropertyPanelFlat({
onRemoveTextField,
onAskAgent,
onToggleElementHidden,
onAutoGroupCarveSources,
onImportAssets,
onAddMediaOverlay,
onImportFonts,
@@ -440,6 +441,7 @@ export function PropertyPanelFlat({
element={element}
onSetAttributeQuiet={onSetAttributeQuiet ?? onSetAttributeLive}
onSetAttributeLive={onSetAttributeLive}
onAutoGroupCarveSources={onAutoGroupCarveSources}
/>
),
});
@@ -1507,6 +1507,142 @@ describe("AudioFxGroup carve source list", () => {
});
});
describe("AudioFxGroup carve targets groups (B6)", () => {
// These tests are the only ones in this file whose auto-carve effect makes a
// real cross-element write call (`onAutoGroupCarveSources`), which can be a
// genuine Promise. None of this file's other `mount*` helpers ever unmount
// their React root — harmless everywhere else because their effects only
// ever touch a plain `onSetAttributeQuiet` mock, so an orphaned root left
// over from an earlier test does nothing observable if it ever re-renders.
// Here it can re-fire the auto-group effect against WHATEVER a later test's
// fixture put in the (file-global) `document`, calling a long-dead test's
// mock — unmounting is what prevents that.
const roots: ReturnType<typeof createRoot>[] = [];
afterEach(() => {
for (const root of roots.splice(0)) act(() => root.unmount());
});
/** A bed, plus tracks that may carry `data-audio-group`, plus an optional auto-group handler. */
const mountWith = (
tracks: { id: string; group?: string; start?: string; duration?: string }[],
onAutoGroupCarveSources?: (clipIds: readonly string[], groupId: string) => Promise<void>,
) => {
const bed = document.createElement("audio");
bed.id = "bed";
document.body.append(bed);
for (const t of tracks) {
const el = document.createElement("audio");
el.id = t.id;
if (t.group) el.setAttribute("data-audio-group", t.group);
if (t.start !== undefined) el.setAttribute("data-start", t.start);
if (t.duration !== undefined) el.setAttribute("data-duration", t.duration);
document.body.append(el);
}
const onSetAttributeQuiet = vi.fn();
const host = document.createElement("div");
document.body.append(host);
const root = createRoot(host);
roots.push(root);
act(() => {
root.render(
<AudioFxGroup
element={{ dataAttributes: {}, id: "bed", element: bed } as unknown as DomEditSelection}
onSetAttributeQuiet={onSetAttributeQuiet}
onSetAttributeLive={vi.fn()}
onAutoGroupCarveSources={onAutoGroupCarveSources}
/>,
);
});
const offered = Array.from(host.querySelectorAll<HTMLElement>("[data-carve-source]"));
const options = offered.map((el) => el.dataset["carveSource"] ?? "");
const boxes = offered.filter((el): el is HTMLInputElement => el instanceof HTMLInputElement);
return { host, options, boxes, onSetAttributeQuiet };
};
it("offers one entry for a group and hides its members individually", () => {
const { options } = mountWith([
{ id: "vo-1", group: "voiceover" },
{ id: "vo-2", group: "voiceover" },
]);
expect(options).toEqual(["voiceover"]);
});
it("offers the group when only ONE member overlaps the bed (union, not per-clip)", () => {
// The bed spans the whole default window (no start/duration on the
// selection itself in this harness resolves to [0, Infinity)), so give the
// members explicit, non-overlapping-with-each-other spans and confirm the
// group still appears as long as at least one of them is in range.
const { options } = mountWith([
{ id: "vo-1", group: "voiceover", start: "0", duration: "5" },
{ id: "vo-2", group: "voiceover", start: "1000", duration: "5" },
]);
expect(options).toEqual(["voiceover"]);
});
it("auto-selects the group over its individual members", () => {
const { onSetAttributeQuiet } = mountWith([
{ id: "vo-1", group: "voiceover" },
{ id: "vo-2", group: "voiceover" },
]);
const write = writeTo(onSetAttributeQuiet.mock.calls, "data-fx-carve");
expect(JSON.parse(String(write![1])).sources).toEqual(["voiceover"]);
});
it("auto-groups a multi-voice carve into one named group, atomically", async () => {
// Two ungrouped, both voice-classified: the existing "every candidate
// carves itself" mount effect names both by id, which is exactly the
// plural-ungrouped case B6 intercepts — the carve should land on the
// minted group, not on the two ids directly.
const onAutoGroupCarveSources = vi.fn().mockResolvedValue(undefined);
const { onSetAttributeQuiet } = mountWith(
[
{ id: "narration", start: "0", duration: "5" },
{ id: "interview-guest", start: "10", duration: "5" },
],
onAutoGroupCarveSources,
);
expect(onAutoGroupCarveSources).toHaveBeenCalledWith(
["narration", "interview-guest"],
"voiceover",
);
// Explicitly await the exact promise `assignGroup` returned — a bare
// `await Promise.resolve()`/`setTimeout` flush is guessing how deep the
// chain behind it goes (its `.then(...)`, the write, and the re-analysis
// `setCarve` awaits afterward). Left genuinely unresolved when this test
// returns, that chain settles during a LATER test instead, after this
// one's mocks and DOM are gone.
await act(async () => {
await onAutoGroupCarveSources.mock.results[0]?.value;
// Two more turns of the microtask queue: one for the `.then(...)` that
// builds the grouped settings, one for `setCarve`'s own trailing
// `await analyse(next)` (a no-op here — the fixture's tracks have no
// `src`, so `resolveCarveVoices` returns empty and `analyse` exits
// immediately, but it still has to actually run to completion).
await Promise.resolve();
await Promise.resolve();
});
const write = writeTo(onSetAttributeQuiet.mock.calls, "data-fx-carve");
expect(write).toBeTruthy();
expect(JSON.parse(String(write![1])).sources).toEqual(["voiceover"]);
});
it("leaves sources alone when one of them already names a group", () => {
const onAutoGroupCarveSources = vi.fn();
const { onSetAttributeQuiet } = mountWith(
[
{ id: "vo-1", group: "voiceover" },
{ id: "vo-2", group: "voiceover" },
],
onAutoGroupCarveSources,
);
// The default-carve effect already named the group (previous test above),
// so no auto-group call should ever fire for an all-group source list.
expect(onAutoGroupCarveSources).not.toHaveBeenCalled();
const write = writeTo(onSetAttributeQuiet.mock.calls, "data-fx-carve");
expect(JSON.parse(String(write![1])).sources).toEqual(["voiceover"]);
});
});
describe("AudioFxGroup carve source range", () => {
const spanned = (tracks: { id: string; start?: string; duration?: string }[]): string[] => {
const bed = document.createElement("audio");
@@ -55,6 +55,7 @@ export function AudioFxGroup({
element,
onSetAttributeQuiet: onSetAttributeQuietRaw,
onSetAttributeLive,
onAutoGroupCarveSources,
}: {
element: DomEditSelection;
/**
@@ -70,6 +71,8 @@ export function AudioFxGroup({
onSetAttributeQuiet: (attr: string, value: string | null) => void | Promise<void>;
/** Continuous, non-persisting write for a dial being dragged. */
onSetAttributeLive: (attr: string, value: string | null) => void | Promise<void>;
/** Write `data-audio-group` on every named clip, atomically, one undo entry. */
onAutoGroupCarveSources?: (clipIds: readonly string[], groupId: string) => Promise<void>;
}) {
const chain = ((): HfAudioFxChain => {
const raw = element.dataAttributes?.[HF_AUDIO_FX_DATA_KEY];
@@ -231,6 +234,7 @@ export function AudioFxGroup({
onSetAttributeQuiet,
writeAutomation,
setAnalysing,
onAutoGroupCarveSources,
);
const { runLeveller, auditionTransport, auditioningLevel, auditionLevel, removeLeveller } =
@@ -27,6 +27,7 @@ export type PropertyPanelFlatProps = Pick<
| "onRemoveTextField"
| "onAskAgent"
| "onToggleElementHidden"
| "onAutoGroupCarveSources"
| "onImportAssets"
| "onAddMediaOverlay"
| "onImportFonts"
@@ -92,6 +92,8 @@ export interface PropertyPanelProps {
onRemoveTextField: (fieldKey: string) => void;
onAskAgent: () => void;
onToggleElementHidden?: (elementKey: string, hidden: boolean) => void | Promise<void>;
/** B6: group two or more picked voice clips, atomically, one undo entry. */
onAutoGroupCarveSources?: (clipIds: readonly string[], groupId: string) => Promise<void>;
onImportAssets?: (files: FileList, dir?: string) => Promise<string[]>;
onAddMediaOverlay?: AddMediaOverlayHandler;
fontAssets?: ImportedFontAsset[];
Binary file not shown.
@@ -0,0 +1,132 @@
/**
* B6's normative rule: plural voiceover carve always targets a group.
*
* Split out of `useFxCarve.ts`, which owned all of this before the file grew
* past a size where "carve targets a group" was still one thing to read
* same reason `useFxCarve.ts` itself was split out of
* `propertyPanelAudioFxGroup.tsx`.
*/
import { classifyAudioName, type HfCarveSettings } from "@hyperframes/core/audio-carve";
import { resolveAudioGroups } from "@hyperframes/core/audio-groups";
/**
* An id for a new voiceover group, de-duped against every id already in the
* document group ids and plain element ids share one namespace, so both
* have to be checked.
*/
function mintGroupId(doc: Document): string {
const taken = new Set([
...resolveAudioGroups(doc).map((g) => g.id),
...Array.from(doc.querySelectorAll("[id]")).map((el) => el.id),
]);
if (!taken.has("voiceover")) return "voiceover";
let n = 2;
while (taken.has(`voiceover-${n}`)) n += 1;
return `voiceover-${n}`;
}
export function isPromiseLike<T>(value: T | Promise<T>): value is Promise<T> {
return typeof (value as { then?: unknown })?.then === "function";
}
/**
* Plural voiceover carve, always against a group normative, not a
* suggestion (groups doc §1.6). Picking a second ungrouped voice clip mints a
* group behind the two of them and points the carve at it instead, the same
* way a hand-authored composition is expected to work; a source list already
* naming a group is left alone; this only fires on a run of plain clip ids.
*
* Returns the settings unchanged, synchronously, when there is nothing to do
* NOT wrapped in a promise even though the caller may `await` the result. An
* `async` function always yields a microtask, which would push every
* `setCarve` call (grouped or not) one tick later than before this existed;
* several tests assert on `onSetAttributeQuiet` synchronously after mount and
* would miss that write.
*/
function withAutoGroupedSources(
doc: Document,
next: HfCarveSettings,
assignGroup: ((clipIds: readonly string[], groupId: string) => Promise<void>) | undefined,
): HfCarveSettings | Promise<HfCarveSettings> {
if (!assignGroup || next.sources.length < 2) return next;
const groupIds = new Set(resolveAudioGroups(doc).map((g) => g.id));
if (next.sources.some((id) => groupIds.has(id))) return next;
const groupId = mintGroupId(doc);
return assignGroup(next.sources, groupId).then(() => ({ ...next, sources: [groupId] }));
}
/**
* `setCarve`'s first step: apply the auto-group rule, if a document and a
* write-back are both available.
*
* Deliberately NOT an `async function`: wrapping this in one would make every
* call return a promise-wrapped value, forcing the caller's `await` to yield
* a microtask even on the synchronous branch the exact bug
* `withAutoGroupedSources`'s own sync-when-possible contract exists to avoid.
* The caller does the `isPromiseLike` check (re-exported from here) and only
* awaits the branch that is genuinely async.
*/
export function resolveNextCarveSettings(
nextRaw: HfCarveSettings | null,
doc: Document | undefined,
assignGroup: ((clipIds: readonly string[], groupId: string) => Promise<void>) | undefined,
): HfCarveSettings | Promise<HfCarveSettings> | null {
return nextRaw && doc ? withAutoGroupedSources(doc, nextRaw, assignGroup) : nextRaw;
}
export interface CarveCandidate {
id: string;
label: string;
kind: ReturnType<typeof classifyAudioName>;
}
/**
* One row per ungrouped clip that overlaps the bed, one row per group that has
* ANY overlapping member (union, not per-clip a narration group spanning
* the whole timeline is relevant even if each of its segments only overlaps
* part of the bed). Grouped members never appear individually.
*/
export function collectCarveCandidates(
doc: Document,
others: readonly HTMLAudioElement[],
overlapsBed: (a: Element) => boolean,
): CarveCandidate[] {
const groupByMember = new Map(
resolveAudioGroups(doc).flatMap((group) => group.memberIds.map((id) => [id, group] as const)),
);
const offeredGroupIds = new Set<string>();
const described: CarveCandidate[] = [];
for (const a of others) {
const group = groupByMember.get(a.id);
if (!group) {
if (overlapsBed(a)) {
described.push({
id: a.id,
label: a.id,
kind: classifyAudioName(a.id, a.getAttribute("src")),
});
}
continue;
}
if (offeredGroupIds.has(group.id)) continue;
const members = group.memberIds
.map((id) => doc.getElementById(id))
// Not `instanceof HTMLAudioElement`: these belong to the composition's
// iframe document, so the constructor is a different realm's and the
// instanceof is false for every one (mirrors resolveCarveVoices in
// useFxCarve.ts).
.filter((el): el is HTMLElement => el?.tagName === "AUDIO");
if (!members.some(overlapsBed)) continue;
offeredGroupIds.add(group.id);
described.push({
id: group.id,
label: `${group.label} (${members.length})`,
kind: classifyAudioName(
group.label,
...members.flatMap((m) => [m.id, m.getAttribute("src")]),
),
});
}
return described;
}
@@ -2,7 +2,11 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { usePlayerStore, type TimelineElement } from "../player";
import { toggleTimelineElementHidden, toggleTimelineTrackHidden } from "./timelineTrackVisibility";
import {
createAudioGroupAndAssignMembers,
toggleTimelineElementHidden,
toggleTimelineTrackHidden,
} from "./timelineTrackVisibility";
afterEach(() => {
document.body.innerHTML = "";
@@ -395,3 +399,139 @@ describe("toggleTimelineElementHidden", () => {
expect(recordEdit.mock.calls[0]?.[0]?.label).toBe("Hide 2 elements");
});
});
describe("createAudioGroupAndAssignMembers", () => {
it("writes data-audio-group on every member in ONE atomic edit and updates the player store", async () => {
const iframe = document.createElement("iframe");
document.body.append(iframe);
if (iframe.contentDocument) {
iframe.contentDocument.body.innerHTML = `
<audio id="narration"></audio>
<audio id="interview-guest"></audio>
`;
}
const files = new Map([
[
"index.html",
`<audio id="narration" data-start="0" data-duration="5"></audio>
<audio id="interview-guest" data-start="10" data-duration="5"></audio>
<audio id="sfx-boom" data-start="0" data-duration="1"></audio>`,
],
]);
stubProjectFiles(files);
const narration = element({
id: "narration",
key: "index.html:#narration",
domId: "narration",
track: 0,
});
const guest = element({
id: "interview-guest",
key: "index.html:#interview-guest",
domId: "interview-guest",
track: 1,
});
usePlayerStore.getState().setElements([narration, guest]);
const writes = new Map<string, string>();
const recordEdit = vi.fn();
const changedPaths = await createAudioGroupAndAssignMembers({
projectId: "project-1",
activeCompPath: "index.html",
elements: [narration, guest],
groupId: "voiceover",
previewIframe: iframe,
writeProjectFile: async (path, content) => {
writes.set(path, content);
},
recordEdit,
domEditSaveTimestampRef: { current: 0 },
pendingTimelineEditPathRef: { current: new Set() },
});
expect(changedPaths).toEqual(["index.html"]);
expect(
iframe.contentDocument?.getElementById("narration")?.getAttribute("data-audio-group"),
).toBe("voiceover");
expect(
iframe.contentDocument?.getElementById("interview-guest")?.getAttribute("data-audio-group"),
).toBe("voiceover");
// One write carrying BOTH members — per-element writes would clobber each
// other (each starts from the original file content).
expect(writes.get("index.html")).toContain(
'id="narration" data-start="0" data-duration="5" data-audio-group="voiceover"',
);
expect(writes.get("index.html")).toContain(
'id="interview-guest" data-start="10" data-duration="5" data-audio-group="voiceover"',
);
expect(writes.get("index.html")).toContain(
'id="sfx-boom" data-start="0" data-duration="1"></audio>',
);
expect(recordEdit).toHaveBeenCalledTimes(1);
expect(recordEdit.mock.calls[0]?.[0]?.label).toBe("Group 2 voice clips");
expect(
usePlayerStore.getState().elements.find((el) => el.key === "index.html:#narration")
?.audioGroup,
).toBe("voiceover");
expect(
usePlayerStore.getState().elements.find((el) => el.key === "index.html:#interview-guest")
?.audioGroup,
).toBe("voiceover");
});
it("does nothing for fewer than two elements — grouping is a plural concept", async () => {
const recordEdit = vi.fn();
const changedPaths = await createAudioGroupAndAssignMembers({
projectId: "project-1",
activeCompPath: "index.html",
elements: [element({ id: "narration", domId: "narration" })],
groupId: "voiceover",
previewIframe: null,
writeProjectFile: async () => {},
recordEdit,
domEditSaveTimestampRef: { current: 0 },
pendingTimelineEditPathRef: { current: new Set() },
});
expect(changedPaths).toEqual([]);
expect(recordEdit).not.toHaveBeenCalled();
});
it("reverts the optimistic live patch when the save fails", async () => {
const iframe = document.createElement("iframe");
document.body.append(iframe);
if (iframe.contentDocument) {
iframe.contentDocument.body.innerHTML = `
<audio id="narration"></audio>
<audio id="interview-guest"></audio>
`;
}
// No stubbed fetch: readFileContent's request will fail, forcing the
// catch path.
const narration = element({ id: "narration", domId: "narration" });
const guest = element({ id: "interview-guest", domId: "interview-guest" });
await expect(
createAudioGroupAndAssignMembers({
projectId: "project-1",
activeCompPath: "index.html",
elements: [narration, guest],
groupId: "voiceover",
previewIframe: iframe,
writeProjectFile: async () => {},
recordEdit: vi.fn(),
domEditSaveTimestampRef: { current: 0 },
pendingTimelineEditPathRef: { current: new Set() },
}),
).rejects.toThrow();
expect(
iframe.contentDocument?.getElementById("narration")?.hasAttribute("data-audio-group"),
).toBe(false);
expect(
iframe.contentDocument?.getElementById("interview-guest")?.hasAttribute("data-audio-group"),
).toBe(false);
});
});
@@ -8,6 +8,7 @@ import {
} from "../player/components/timelineTrackDisplay";
import { saveProjectFilesWithHistory } from "../utils/studioFileHistory";
import { isAudioTimelineElement } from "../utils/timelineInspector";
import { HF_AUDIO_GROUP_ATTR } from "@hyperframes/core/audio-groups";
import { readTagSnippetByTarget, type PatchOperation } from "../utils/sourcePatcher";
import {
applyPatchByTarget,
@@ -277,6 +278,114 @@ export async function toggleTimelineElementHidden({
});
}
function patchLiveAudioGroupState(
iframe: HTMLIFrameElement | null,
elements: readonly TimelineElement[],
groupId: string | null,
activeCompPath: string | null,
): void {
for (const element of elements) {
const target = findTimelineElementInIframe(iframe, element, activeCompPath);
if (!target) continue;
if (groupId) target.setAttribute(HF_AUDIO_GROUP_ATTR, groupId);
else target.removeAttribute(HF_AUDIO_GROUP_ATTR);
}
}
interface CreateAudioGroupAndAssignMembersInput {
projectId: string;
activeCompPath: string | null;
elements: readonly TimelineElement[];
groupId: string;
previewIframe: HTMLIFrameElement | null;
writeProjectFile: (path: string, content: string) => Promise<void>;
recordEdit: (input: RecordEditInput) => Promise<void>;
domEditSaveTimestampRef: MutableRef<number>;
pendingTimelineEditPathRef: MutableRef<Set<string>>;
}
/**
* Group two or more voice clips: write `data-audio-group="<groupId>"` on
* every one of them, atomically, one undo entry the same multi-target shape
* `setElementsHidden` uses for mute. The group needs no `<hf-audio-group>`
* element of its own to exist: `resolveAudioGroups` already degrades
* gracefully to label = id when one is absent, and a naming dialog is out of
* scope here the id itself is the default name.
*/
// fallow-ignore-next-line complexity
export async function createAudioGroupAndAssignMembers({
projectId,
activeCompPath,
elements,
groupId,
previewIframe,
writeProjectFile,
recordEdit,
domEditSaveTimestampRef,
pendingTimelineEditPathRef,
}: CreateAudioGroupAndAssignMembersInput): Promise<string[]> {
if (elements.length < 2) return [];
patchLiveAudioGroupState(previewIframe, elements, groupId, activeCompPath);
reseekPreviewRuntime(previewIframe);
const groupOperation: PatchOperation = {
type: "attribute",
property: HF_AUDIO_GROUP_ATTR,
value: groupId,
};
const originalByPath = new Map<string, string>();
const files: Record<string, string> = {};
try {
for (const [targetPath, fileElements] of groupElementsByTargetPath(elements, activeCompPath)) {
let patchedContent = await readFileContent(projectId, targetPath);
originalByPath.set(targetPath, patchedContent);
for (const element of fileElements) {
const patchTarget = buildPatchTarget(element);
if (!patchTarget) {
throw new Error(`Timeline element ${element.id} is missing a patchable target`);
}
if (readTagSnippetByTarget(patchedContent, patchTarget) === undefined) {
throw new Error(`Unable to patch timeline element ${element.id} in ${targetPath}`);
}
patchedContent = applyPatchByTarget(patchedContent, patchTarget, groupOperation);
}
files[targetPath] = patchedContent;
pendingTimelineEditPathRef.current.add(targetPath);
}
domEditSaveTimestampRef.current = Date.now();
const changedPaths = await saveProjectFilesWithHistory({
projectId,
label: `Group ${elements.length} voice clips`,
kind: "timeline",
files,
readFile: async (path) => {
const original = originalByPath.get(path);
if (original !== undefined) return original;
return readFileContent(projectId, path);
},
writeFile: writeProjectFile,
recordEdit,
});
domEditSaveTimestampRef.current = Date.now();
for (const element of elements) {
usePlayerStore.getState().updateElement(element.key ?? element.id, { audioGroup: groupId });
}
return changedPaths;
} catch (error) {
// Mirrors setElementsHidden's failure path: the optimistic live patch
// already ran, so a save failure has to be unwound or the preview shows a
// grouping that never made it to disk.
patchLiveAudioGroupState(previewIframe, elements, null, activeCompPath);
reseekPreviewRuntime(previewIframe);
throw error;
}
}
export function useTimelineTrackVisibilityEditing({
projectIdRef,
activeCompPath,
@@ -407,3 +516,67 @@ export function useTimelineElementVisibilityEditing({
],
);
}
/**
* The write behind B6's auto-group: pick two or more voice clips in the carve
* picker and they land in a group instead of naming each other by id. Same
* expanded-rows resolution as element-visibility, for the same reason a
* nested sub-composition child has no entry in the raw store list.
*/
export function useAudioGroupCarveAssignment({
projectIdRef,
activeCompPath,
showToast,
writeProjectFile,
recordEdit,
domEditSaveTimestampRef,
previewIframeRef,
pendingTimelineEditPathRef,
isRecordingRef,
}: UseTimelineElementVisibilityEditingInput): (
clipIds: readonly string[],
groupId: string,
) => Promise<void> {
const expandedElements = useExpandedTimelineElements();
return useCallback(
async (clipIds: readonly string[], groupId: string) => {
if (isRecordingRef?.current) {
showToast("Cannot edit timeline while recording", "error");
return;
}
const pid = projectIdRef.current;
if (!pid) return;
const keys = new Set(clipIds);
const elements = expandedElements.filter((item) => keys.has(item.key ?? item.id));
try {
await createAudioGroupAndAssignMembers({
projectId: pid,
activeCompPath,
elements,
groupId,
previewIframe: previewIframeRef.current,
writeProjectFile,
recordEdit,
domEditSaveTimestampRef,
pendingTimelineEditPathRef,
});
} catch (error) {
console.error("[Timeline] Failed to group voice clips", error);
const message = error instanceof Error ? error.message : "Failed to group voice clips";
showToast(message);
}
},
[
activeCompPath,
expandedElements,
previewIframeRef,
writeProjectFile,
recordEdit,
domEditSaveTimestampRef,
pendingTimelineEditPathRef,
isRecordingRef,
showToast,
projectIdRef,
],
);
}
@@ -0,0 +1,20 @@
import { useMemo } from "react";
import type { TimelineElement } from "../player/store/timelineElement";
/**
* The stored `duration` lags a moment behind an edit that pushes an element
* past it (drag, trim, paste) this is the actual end of the timeline, the
* later of the stored duration and the furthest element's end.
*/
export function useEffectiveTimelineDuration(
timelineDuration: number,
timelineElements: readonly TimelineElement[],
): number {
return useMemo(() => {
const maxEnd =
timelineElements.length > 0
? Math.max(...timelineElements.map((el) => el.start + el.duration))
: 0;
return Math.max(timelineDuration, maxEnd);
}, [timelineDuration, timelineElements]);
}
@@ -0,0 +1,70 @@
import { useCallback, useEffect, useRef } from "react";
import { waitForMediaJob } from "../components/studioMediaJobs";
import type { BackgroundRemovalProgress } from "../components/editor/propertyPanelTypes";
interface RemoveBackgroundOptions {
createBackgroundPlate?: boolean;
quality?: "fast" | "balanced" | "best";
onProgress?: (progress: BackgroundRemovalProgress) => void;
}
/**
* One removal in flight at a time: starting a second one aborts whichever job
* is still running, so a stale progress callback can't overwrite a newer
* result. Unmounting aborts too, or the job would keep running against a
* panel that is no longer there to show its progress.
*/
export function useRemoveBackground(
projectId: string,
refreshFileTree: () => Promise<void>,
showToast: (message: string, kind?: "info" | "error") => void,
) {
const abortRef = useRef<AbortController | null>(null);
useEffect(
() => () => {
abortRef.current?.abort();
},
[],
);
return useCallback(
// fallow-ignore-next-line complexity
async (inputPath: string, options: RemoveBackgroundOptions) => {
const response = await fetch(
`/api/projects/${encodeURIComponent(projectId)}/media/remove-background`,
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
inputPath,
createBackgroundPlate: options.createBackgroundPlate === true,
quality: options.quality ?? "balanced",
}),
},
);
const data = (await response.json().catch(() => ({}))) as {
jobId?: string;
error?: string;
};
if (!response.ok || !data.jobId) {
throw new Error(data.error || `Background removal failed (${response.status})`);
}
showToast("Removing background...", "info");
abortRef.current?.abort();
const controller = new AbortController();
abortRef.current = controller;
try {
const result = await waitForMediaJob(data.jobId, options.onProgress, controller.signal);
await refreshFileTree();
showToast(`Created transparent asset: ${result.outputPath.split("/").pop()}`, "info");
return result;
} finally {
if (abortRef.current === controller) {
abortRef.current = null;
}
}
},
[projectId, refreshFileTree, showToast],
);
}
@@ -28,6 +28,7 @@ import {
import type { PersistTimelineEditInput } from "./timelineEditingHelpers";
import type { TimelineStackingReorderIntent } from "../player/components/timelineEditing";
import {
useAudioGroupCarveAssignment,
useTimelineElementVisibilityEditing,
useTimelineTrackVisibilityEditing,
} from "./timelineTrackVisibility";
@@ -388,6 +389,18 @@ export function useTimelineEditing({
forceReloadSdkSession,
});
const handleAutoGroupCarveSources = useAudioGroupCarveAssignment({
projectIdRef,
activeCompPath,
showToast,
writeProjectFile,
recordEdit,
domEditSaveTimestampRef,
previewIframeRef,
pendingTimelineEditPathRef,
isRecordingRef,
});
// fallow-ignore-next-line complexity
const handleTimelineElementsDelete = useCallback(
// fallow-ignore-next-line complexity
@@ -558,6 +571,7 @@ export function useTimelineEditing({
handleTimelineElementResize,
handleToggleTrackHidden,
handleToggleElementHidden,
handleAutoGroupCarveSources,
handleTimelineElementDelete,
handleTimelineElementsDelete,
handleTimelineElementSplit: handleRazorSplit,
@@ -146,7 +146,14 @@ interface PlayerState extends KeyframeSlice, AutomationSelectionSlice, Thumbnail
updates: Partial<
Pick<
TimelineElement,
"start" | "duration" | "track" | "zIndex" | "hasExplicitZIndex" | "playbackStart" | "hidden"
| "start"
| "duration"
| "track"
| "zIndex"
| "hasExplicitZIndex"
| "playbackStart"
| "hidden"
| "audioGroup"
>
>,
) => void;
+1 -1
View File
@@ -26,7 +26,7 @@
"files": 121
},
"hyperframes-audio": {
"hash": "94aba963d262d71d",
"hash": "f224ea9481998c08",
"files": 6
},
"hyperframes-cli": {
+25
View File
@@ -236,6 +236,31 @@ so one analysis covers all of them: the bands come from all the speech there is,
the envelopes rise wherever any of it is happening. Voices that never play while the
bed does are left out; they cannot mask it.
**A carve against more than one clip id is wrong. Group the clips and carve
against the group.** This is an invariant, not a tip. Naming clips one by one has
to be exhaustively right and stays right only until the next edit — a fourth
narration clip added later plays outside the carve's awareness, and the bed
fails to duck under it silently. Naming the group instead resolves membership at
analysis time, so a clip added to the group later is covered without touching
`sources` at all:
```html
<!-- group the narration, then carve the bed against the group -->
<audio id="vo-intro" data-audio-group="voiceover" …></audio>
<audio id="vo-middle" data-audio-group="voiceover" …></audio>
<audio id="vo-outro" data-audio-group="voiceover" …></audio>
<audio
id="music"
data-fx-carve='{"enabled":true,"sources":["voiceover"],"strength":0.25}'
></audio>
```
A `sources` list naming two or more plain clip ids instead of a group is caught
by the `audio_carve_ungrouped_sources` lint rule — it still works, but it is the
version that silently rots when a clip is added.
**One knob.** `strength` is 0..1 and derives everything: how deep to cut, how
many bands, how wide, how far to favour intelligibility over raw voice energy,
how far the level may drop, how far under the voice to aim. Those six move