From 5e2781b4595f0e2396015e30e089d3322233c187 Mon Sep 17 00:00:00 2001 From: Vance Ingalls Date: Thu, 2 Apr 2026 11:38:35 -0700 Subject: [PATCH] fix(studio): address caption designer PR feedback (#200) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(studio): address caption designer PR feedback Fixes from review comments on feature/caption-designer (#180): - fix(generator): guard named colors in hexToRgba — "red", "transparent" no longer produce NaN rgba values - fix(sync): log auto-save failures instead of silently swallowing them - fix(sync): check res.ok before parsing caption-overrides response - refactor(components): extract Section, Row, inputCls into shared.tsx to eliminate duplication between CaptionPropertyPanel and CaptionAnimationPanel - fix(store): replace non-deterministic Date.now()+Math.random() ID with counter-based group IDs - fix(store): read selectedGroupId from state param instead of get() to avoid stale reads in batched set() calls - fix(overlay): remove cssScale multiplier from getBoundingClientRect coords — the browser already accounts for CSS transforms - docs(parser): add comment explaining the lazy ]; regex assumption Co-Authored-By: Claude Opus 4.6 (1M context) * fix(studio): address remaining caption designer feedback Overlay: handle both per-word spans (generator output) and grouped text nodes (existing templates). Wraps text nodes into individual spans on demand so the overlay can target words in any caption format. Property panel: add Typography (font, size, weight, spacing) and Color (color, active, dim, opacity) sections alongside existing Position and Transform controls. Timeline: move caption timeline into a dedicated flex-shrink-0 section below the main timeline tracks instead of inside the scrollable area. Gives it fixed 60px height that's always visible. Caption overrides: classify color tweens by comparing target color to the dim baseline instead of relying on timeline position order. This handles compositions with custom color tweens correctly. App.tsx: remove polling interval, rely on runtime postMessage events for caption detection. Add clarifying comment on why useEffect is appropriate (external event subscription). Co-Authored-By: Claude Opus 4.6 (1M context) * fix(studio): restore cssScale in overlay coordinate conversion getBoundingClientRect() on iframe-internal elements returns coordinates in the iframe's native resolution (1920x1080), not the CSS-scaled display size. The cssScale multiplier is needed to convert to parent window coordinates. The earlier removal was incorrect — it only worked at 1:1 scale. Co-Authored-By: Claude Opus 4.6 (1M context) * fix(studio): fix reversed scaling on left-side corner handles Scale interaction used horizontal dx from start position, which goes negative when dragging left handles outward. Now uses distance from box center — dragging away from center increases scale regardless of which corner handle is used. Co-Authored-By: Claude Opus 4.6 (1M context) * fix(studio): make rotation respond to horizontal drag only Rotation handle sits directly above the word, so atan2-based rotation barely responds to left/right movement. Replace with linear horizontal mapping: drag right = clockwise, drag left = counter-clockwise, 200px = 90 degrees. Vertical movement is ignored. Co-Authored-By: Claude Opus 4.6 (1M context) * fix(studio): remove animation tab and typography/color from property panel Keep only Position (X, Y) and Transform (Scale, Rotation) controls. Remove tab switcher UI since there's only one view now. Co-Authored-By: Claude Opus 4.6 (1M context) * style: fix oxfmt formatting in CLAUDE.md and captions skill docs Co-Authored-By: Claude Opus 4.6 (1M context) --------- Co-authored-by: Claude Opus 4.6 (1M context) --- .claude/worktrees/agent-a692178b | 1 - CLAUDE.md | 1 - packages/core/src/runtime/captionOverrides.ts | 31 +- packages/studio/src/App.tsx | 36 +- .../components/CaptionAnimationPanel.tsx | 33 +- .../captions/components/CaptionOverlay.tsx | 378 +++++++++++++----- .../components/CaptionPropertyPanel.tsx | 203 ++++------ .../studio/src/captions/components/shared.tsx | 26 ++ packages/studio/src/captions/generator.ts | 4 + .../src/captions/hooks/useCaptionSync.ts | 14 +- packages/studio/src/captions/parser.ts | 2 + packages/studio/src/captions/store.ts | 8 +- .../studio/src/components/nle/NLELayout.tsx | 2 +- skills/hyperframes-captions/SKILL.md | 1 - .../hyperframes-captions/transcript-guide.md | 14 +- 15 files changed, 433 insertions(+), 321 deletions(-) delete mode 160000 .claude/worktrees/agent-a692178b create mode 100644 packages/studio/src/captions/components/shared.tsx diff --git a/.claude/worktrees/agent-a692178b b/.claude/worktrees/agent-a692178b deleted file mode 160000 index 1e4c101fb..000000000 --- a/.claude/worktrees/agent-a692178b +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 1e4c101fb4412e554d740d4c662ef4779a63eba6 diff --git a/CLAUDE.md b/CLAUDE.md index 6ddf7412c..b64e4b87b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -45,7 +45,6 @@ npx skills add greensock/gsap-skills # GSAP skills Uses [vercel-labs/skills](https://github.com/vercel-labs/skills). Installs to Claude Code, Gemini CLI, and Codex CLI by default. Pass `-a ` for other targets. - ## Project Overview Open-source video rendering framework: write HTML, render video. diff --git a/packages/core/src/runtime/captionOverrides.ts b/packages/core/src/runtime/captionOverrides.ts index 47699474e..8e5b34360 100644 --- a/packages/core/src/runtime/captionOverrides.ts +++ b/packages/core/src/runtime/captionOverrides.ts @@ -83,25 +83,30 @@ export function applyCaptionOverrides(): void { if (override.fontWeight !== undefined) styleProps.fontWeight = override.fontWeight; if (override.fontFamily !== undefined) styleProps.fontFamily = override.fontFamily; - // Replace color values in existing GSAP tweens by timeline order. - // For any word, color tweens follow: dim (setup) → active (spoken) → after. - // Sort by startTime and assign by position, not by content heuristics. + // Replace color values in existing GSAP tweens. + // Instead of relying on timeline position order (fragile if custom + // color tweens exist), we classify each tween by comparing its + // target color to the current computed color of the element. + // Tweens that match the current color are "dim" tweens; tweens + // with a different color are "active" tweens. if (override.activeColor || override.dimColor) { const allTweens = gsap.getTweensOf(el); const colorTweens = allTweens .filter((tw) => tw.vars.color !== undefined) .sort((a, b) => a.startTime() - b.startTime()); - for (let i = 0; i < colorTweens.length; i++) { - if (i === 0 && override.dimColor) { - // First color tween = dim setup - colorTweens[i].vars.color = override.dimColor; - } else if (i === 1 && override.activeColor) { - // Second color tween = active/spoken - colorTweens[i].vars.color = override.activeColor; - } else if (i >= 2 && override.dimColor) { - // Third+ = after/deactivate (use dim color) - colorTweens[i].vars.color = override.dimColor; + // Use the first tween's color as the dim baseline — if no tweens, + // fall back to computed style. + const dimBaseline = colorTweens.length > 0 ? String(colorTweens[0].vars.color) : ""; + + for (const tw of colorTweens) { + const tweenColor = String(tw.vars.color); + if (tweenColor === dimBaseline) { + // This tween targets the dim/inactive color + if (override.dimColor) tw.vars.color = override.dimColor; + } else { + // This tween targets the active/spoken color + if (override.activeColor) tw.vars.color = override.activeColor; } } diff --git a/packages/studio/src/App.tsx b/packages/studio/src/App.tsx index 3595613e7..9c42a7969 100644 --- a/packages/studio/src/App.tsx +++ b/packages/studio/src/App.tsx @@ -65,20 +65,18 @@ export function StudioApp() { const [rightWidth, setRightWidth] = useState(400); const [leftCollapsed, setLeftCollapsed] = useState(false); const [rightCollapsed, setRightCollapsed] = useState(true); - // Auto-enter caption edit mode when viewing a captions composition // Auto-enter caption edit mode when the iframe contains .caption-group elements. - // Listens for the runtime's postMessage events (state/timeline) which fire after - // all compositions are loaded, then checks for caption groups. + // This is a subscription to external events (postMessage from runtime) — useEffect + // is appropriate here. The runtime fires "state"/"timeline" messages after all + // compositions load, which triggers caption detection. // eslint-disable-next-line no-restricted-syntax useEffect(() => { if (!projectId) return; - let pollId: ReturnType | null = null; let activating = false; const tryActivateCaptions = () => { if (useCaptionStore.getState().isEditMode || activating) { - if (pollId) { clearInterval(pollId); pollId = null; } return; } @@ -88,7 +86,9 @@ export function StudioApp() { try { doc = iframe?.contentDocument ?? null; win = iframe?.contentWindow ?? null; - } catch { return; } + } catch { + return; + } if (!doc || !win) return; const groups = doc.querySelectorAll(".caption-group"); @@ -102,7 +102,8 @@ export function StudioApp() { // Strategy 1: data-composition-src or data-composition-file attributes const compHosts = doc.querySelectorAll("[data-composition-src], [data-composition-file]"); for (const host of compHosts) { - const src = host.getAttribute("data-composition-src") || host.getAttribute("data-composition-file"); + const src = + host.getAttribute("data-composition-src") || host.getAttribute("data-composition-file"); if (src && src.includes("captions")) { captionSrcPath = src; break; @@ -154,7 +155,9 @@ export function StudioApp() { captionSync.loadOverrides(); }) .catch(() => {}) - .finally(() => { activating = false; }); + .finally(() => { + activating = false; + }); }; // Listen for runtime messages that signal composition loading is complete @@ -168,14 +171,11 @@ export function StudioApp() { window.addEventListener("message", handleMessage); // Try immediately in case compositions are already loaded tryActivateCaptions(); - // Poll until captions are detected — sub-composition scripts run async - pollId = setInterval(tryActivateCaptions, 200); return () => { window.removeEventListener("message", handleMessage); - if (pollId) clearInterval(pollId); }; - }, [activeCompPath, projectId, compIdToSrc]); + }, [activeCompPath, projectId, compIdToSrc, captionSync]); // Auto-expand right panel when a caption word is selected // eslint-disable-next-line no-restricted-syntax @@ -296,7 +296,6 @@ export function StudioApp() { const previewIframeRef = useRef(null); const consoleErrorsRef = useRef([]); - // Listen for external file changes (user editing HTML outside the editor). // In dev: use Vite HMR. In embedded/production: use SSE from /api/events. useMountEffect(() => { @@ -850,14 +849,15 @@ export function StudioApp() { }); }} previewOverlay={ - captionEditMode ? ( - - ) : undefined + captionEditMode ? : undefined } timelineFooter={ captionEditMode ? ( -
-
+
+
Captions diff --git a/packages/studio/src/captions/components/CaptionAnimationPanel.tsx b/packages/studio/src/captions/components/CaptionAnimationPanel.tsx index 62ebe1349..c120876b5 100644 --- a/packages/studio/src/captions/components/CaptionAnimationPanel.tsx +++ b/packages/studio/src/captions/components/CaptionAnimationPanel.tsx @@ -61,38 +61,7 @@ const EASE_PRESETS = [ "bounce.out", ]; -// --------------------------------------------------------------------------- -// Shared input class (matches CaptionPropertyPanel) -// --------------------------------------------------------------------------- - -const inputCls = - "w-full bg-neutral-900 border border-neutral-800 rounded px-1.5 py-0.5 text-2xs text-neutral-200 font-mono outline-none focus:border-neutral-600"; - -// --------------------------------------------------------------------------- -// Helper Components -// --------------------------------------------------------------------------- - -function Section({ label, children }: { label: string; children: React.ReactNode }) { - return ( -
-
- - {label} - -
-
{children}
-
- ); -} - -function Row({ label, children }: { label: string; children: React.ReactNode }) { - return ( -
- {label} -
{children}
-
- ); -} +import { Section, Row, inputCls } from "./shared"; // --------------------------------------------------------------------------- // Animation phase controls diff --git a/packages/studio/src/captions/components/CaptionOverlay.tsx b/packages/studio/src/captions/components/CaptionOverlay.tsx index ec557c5f1..a30f788f7 100644 --- a/packages/studio/src/captions/components/CaptionOverlay.tsx +++ b/packages/studio/src/captions/components/CaptionOverlay.tsx @@ -37,6 +37,10 @@ function readWordBoxes( const iframeDisplayRect = iframe.getBoundingClientRect(); const overlayRect = overlayEl.getBoundingClientRect(); + // The iframe renders at native resolution (e.g. 1920x1080) but is + // CSS-scaled to fit the viewport. getBoundingClientRect() on elements + // inside the iframe returns coordinates in the iframe's native space. + // Multiply by cssScale to convert to parent window coordinates. const nativeW = parseFloat(iframe.style.width) || iframeDisplayRect.width; const cssScale = iframeDisplayRect.width / nativeW; const offsetX = iframeDisplayRect.left - overlayRect.left; @@ -53,7 +57,9 @@ function readWordBoxes( if (!groupEl) continue; const computed = win.getComputedStyle(groupEl); if (parseFloat(computed.opacity) <= 0.01 || computed.visibility === "hidden") continue; - // Find word spans — may be direct children or inside wrappers + // Find word elements — handles both per-word spans (generator output) + // and grouped text nodes (existing caption templates that use + // el.textContent = line.text instead of individual word spans). const resolvedWordEls: HTMLElement[] = []; for (const child of groupEl.children) { const c = child as HTMLElement; @@ -64,13 +70,49 @@ function readWordBoxes( resolvedWordEls.push(c); } } + // Fallback: if no word spans found but group has text content, + // the template uses grouped text. Wrap each word in a span so + // the overlay can target them individually. + if (resolvedWordEls.length === 0 && groupEl.textContent?.trim()) { + const textNode = groupEl.childNodes[0]; + if (textNode && textNode.nodeType === Node.TEXT_NODE) { + const words = (textNode.textContent || "").split(/\s+/).filter(Boolean); + const frag = doc.createDocumentFragment(); + for (const word of words) { + const span = doc.createElement("span"); + span.textContent = word + " "; + span.style.display = "inline"; + frag.appendChild(span); + resolvedWordEls.push(span); + } + groupEl.replaceChild(frag, textNode); + } else { + // Single span child with all text (e.g. vignelli template) + const singleSpan = groupEl.querySelector(":scope > span"); + if (singleSpan && singleSpan.textContent?.trim()) { + const words = singleSpan.textContent.split(/\s+/).filter(Boolean); + const frag = doc.createDocumentFragment(); + for (const word of words) { + const span = doc.createElement("span"); + span.textContent = word + " "; + span.style.display = "inline"; + frag.appendChild(span); + resolvedWordEls.push(span); + } + singleSpan.replaceWith(frag); + } + } + } for (let wi = 0; wi < group.segmentIds.length; wi++) { const segId = group.segmentIds[wi]; const wordEl = resolvedWordEls[wi] as HTMLElement | undefined; if (!wordEl) continue; const rect = wordEl.getBoundingClientRect(); boxes.push({ - segmentId: segId, groupId, groupIndex: gi, wordIndex: wi, + segmentId: segId, + groupId, + groupIndex: gi, + wordIndex: wi, x: rect.left * cssScale + offsetX, y: rect.top * cssScale + offsetY, width: rect.width * cssScale, @@ -81,9 +123,17 @@ function readWordBoxes( return boxes; } -function getWordEl(iframe: HTMLIFrameElement, groupIndex: number, wordIndex: number): HTMLElement | null { +function getWordEl( + iframe: HTMLIFrameElement, + groupIndex: number, + wordIndex: number, +): HTMLElement | null { let doc: Document | null = null; - try { doc = iframe.contentDocument; } catch { return null; } + try { + doc = iframe.contentDocument; + } catch { + return null; + } if (!doc) return null; const groupEl = doc.querySelectorAll(".caption-group")[groupIndex]; if (!groupEl) return null; @@ -108,8 +158,13 @@ function getWordEl(iframe: HTMLIFrameElement, groupIndex: number, wordIndex: num * Read GSAP's internal transform state for an element. * GSAP stores transforms in its own cache, not in el.style.transform. */ -function readGsapTransform(el: HTMLElement, iframeWin: Window): { x: number; y: number; scale: number; rotation: number } { - const gsap = (iframeWin as unknown as { gsap?: { getProperty?: (el: HTMLElement, prop: string) => number } }).gsap; +function readGsapTransform( + el: HTMLElement, + iframeWin: Window, +): { x: number; y: number; scale: number; rotation: number } { + const gsap = ( + iframeWin as unknown as { gsap?: { getProperty?: (el: HTMLElement, prop: string) => number } } + ).gsap; if (gsap && gsap.getProperty) { return { x: gsap.getProperty(el, "x") || 0, @@ -155,9 +210,20 @@ function getOrCreateWrapper(el: HTMLElement): HTMLElement { * Write transform values to a wrapper span around the word element. * The word keeps its GSAP animations; the wrapper handles editor transforms. */ -function writeTransform(el: HTMLElement, iframeWin: Window, x: number, y: number, scale: number, rotation: number) { +function writeTransform( + el: HTMLElement, + iframeWin: Window, + x: number, + y: number, + scale: number, + rotation: number, +) { const wrapper = getOrCreateWrapper(el); - const gsap = (iframeWin as unknown as { gsap?: { set?: (el: HTMLElement, props: Record) => void } }).gsap; + const gsap = ( + iframeWin as unknown as { + gsap?: { set?: (el: HTMLElement, props: Record) => void }; + } + ).gsap; if (gsap && gsap.set) { gsap.set(wrapper, { x, y, scale, rotation }); } else { @@ -173,7 +239,10 @@ function syncToStore(segmentId: string, el: HTMLElement, iframeWin: Window) { const style: Record = {}; if (Math.abs(x) > 0.5) style.x = x; if (Math.abs(y) > 0.5) style.y = y; - if (Math.abs(scale - 1) > 0.001) { style.scaleX = scale; style.scaleY = scale; } + if (Math.abs(scale - 1) > 0.001) { + style.scaleX = scale; + style.scaleY = scale; + } if (Math.abs(rotation) > 0.1) style.rotation = rotation; if (Object.keys(style).length > 0) { useCaptionStore.getState().updateSegmentStyle(segmentId, style); @@ -183,9 +252,7 @@ function syncToStore(segmentId: string, el: HTMLElement, iframeWin: Window) { const HANDLE = 8; const ROTATION_OFFSET = 20; // px above the selection box -export const CaptionOverlay = memo(function CaptionOverlay({ - iframeRef, -}: CaptionOverlayProps) { +export const CaptionOverlay = memo(function CaptionOverlay({ iframeRef }: CaptionOverlayProps) { const isEditMode = useCaptionStore((s) => s.isEditMode); const model = useCaptionStore((s) => s.model); const selectedSegmentIds = useCaptionStore((s) => s.selectedSegmentIds); @@ -199,9 +266,38 @@ export const CaptionOverlay = memo(function CaptionOverlay({ // Interaction mode — only one active at a time const interactionRef = useRef< - | { type: "move"; wordEl: HTMLElement; segmentId: string; startMX: number; startMY: number; origTX: number; origTY: number; origScale: number; origRotation: number } - | { type: "scale"; wordEl: HTMLElement; segmentId: string; startMX: number; startWidth: number; origTX: number; origTY: number; origScale: number; origRotation: number } - | { type: "rotate"; wordEl: HTMLElement; segmentId: string; centerX: number; centerY: number; startAngle: number; origTX: number; origTY: number; origRotation: number; origScale: number } + | { + type: "move"; + wordEl: HTMLElement; + segmentId: string; + startMX: number; + startMY: number; + origTX: number; + origTY: number; + origScale: number; + origRotation: number; + } + | { + type: "scale"; + wordEl: HTMLElement; + segmentId: string; + startMX: number; + startDxFromCenter: number; + origTX: number; + origTY: number; + origScale: number; + origRotation: number; + } + | { + type: "rotate"; + wordEl: HTMLElement; + segmentId: string; + startMX: number; + origTX: number; + origTY: number; + origRotation: number; + origScale: number; + } | null >(null); @@ -215,8 +311,13 @@ export const CaptionOverlay = memo(function CaptionOverlay({ if (!iframe || !m || !overlay) return; const next = readWordBoxes(iframe, m, overlay); // Skip state update if nothing changed (avoids re-render every 66ms) - if (next.length === prevBoxes.length && - next.every((b, i) => Math.abs(b.x - prevBoxes[i].x) < 0.5 && Math.abs(b.y - prevBoxes[i].y) < 0.5)) return; + if ( + next.length === prevBoxes.length && + next.every( + (b, i) => Math.abs(b.x - prevBoxes[i].x) < 0.5 && Math.abs(b.y - prevBoxes[i].y) < 0.5, + ) + ) + return; prevBoxes = next; setWordBoxes(next); }; @@ -273,93 +374,131 @@ export const CaptionOverlay = memo(function CaptionOverlay({ }, [iframeRef]); // --- Move --- - const startMove = useCallback((groupIndex: number, wordIndex: number, segmentId: string, e: React.PointerEvent) => { - e.stopPropagation(); - (e.currentTarget as HTMLElement).setPointerCapture(e.pointerId); - const iframe = iframeRef.current; - if (!iframe) return; - const wordEl = getWordEl(iframe, groupIndex, wordIndex); - const win = iframe.contentWindow; - if (!wordEl || !win) return; - const state = readGsapTransform(getOrCreateWrapper(wordEl), win); - interactionRef.current = { - type: "move", wordEl, segmentId, - startMX: e.clientX, startMY: e.clientY, - origTX: state.x, origTY: state.y, - origScale: state.scale, origRotation: state.rotation, - }; - }, [iframeRef]); + const startMove = useCallback( + (groupIndex: number, wordIndex: number, segmentId: string, e: React.PointerEvent) => { + e.stopPropagation(); + (e.currentTarget as HTMLElement).setPointerCapture(e.pointerId); + const iframe = iframeRef.current; + if (!iframe) return; + const wordEl = getWordEl(iframe, groupIndex, wordIndex); + const win = iframe.contentWindow; + if (!wordEl || !win) return; + const state = readGsapTransform(getOrCreateWrapper(wordEl), win); + interactionRef.current = { + type: "move", + wordEl, + segmentId, + startMX: e.clientX, + startMY: e.clientY, + origTX: state.x, + origTY: state.y, + origScale: state.scale, + origRotation: state.rotation, + }; + }, + [iframeRef], + ); // --- Scale --- - const startScale = useCallback((groupIndex: number, wordIndex: number, segmentId: string, e: React.PointerEvent) => { - e.stopPropagation(); - e.preventDefault(); - (e.currentTarget as HTMLElement).setPointerCapture(e.pointerId); - const iframe = iframeRef.current; - if (!iframe) return; - const wordEl = getWordEl(iframe, groupIndex, wordIndex); - const win = iframe.contentWindow; - if (!wordEl || !win) return; - const rect = wordEl.getBoundingClientRect(); - const state = readGsapTransform(getOrCreateWrapper(wordEl), win); - interactionRef.current = { - type: "scale", wordEl, segmentId, - startMX: e.clientX, startWidth: rect.width, - origTX: state.x, origTY: state.y, - origScale: state.scale, origRotation: state.rotation, - }; - }, [iframeRef]); + const startScale = useCallback( + (groupIndex: number, wordIndex: number, segmentId: string, e: React.PointerEvent) => { + e.stopPropagation(); + e.preventDefault(); + (e.currentTarget as HTMLElement).setPointerCapture(e.pointerId); + const iframe = iframeRef.current; + if (!iframe) return; + const wordEl = getWordEl(iframe, groupIndex, wordIndex); + const win = iframe.contentWindow; + if (!wordEl || !win) return; + const rect = wordEl.getBoundingClientRect(); + const cssScale = getCssScale(); + const boxCenterX = + rect.left * cssScale + + (iframeRef.current?.getBoundingClientRect().left ?? 0) + + (rect.width * cssScale) / 2; + const state = readGsapTransform(getOrCreateWrapper(wordEl), win); + interactionRef.current = { + type: "scale", + wordEl, + segmentId, + startMX: e.clientX, + startDxFromCenter: e.clientX - boxCenterX, + origTX: state.x, + origTY: state.y, + origScale: state.scale, + origRotation: state.rotation, + }; + }, + [iframeRef, getCssScale], + ); // --- Rotate --- - const startRotate = useCallback((box: WordBox, e: React.PointerEvent) => { - e.stopPropagation(); - e.preventDefault(); - (e.currentTarget as HTMLElement).setPointerCapture(e.pointerId); - const iframe = iframeRef.current; - if (!iframe) return; - const wordEl = getWordEl(iframe, box.groupIndex, box.wordIndex); - const win = iframe.contentWindow; - if (!wordEl || !win) return; - const cx = box.x + box.width / 2; - const cy = box.y + box.height / 2; - const startAngle = Math.atan2(e.clientY - cy, e.clientX - cx) * (180 / Math.PI); - const state = readGsapTransform(getOrCreateWrapper(wordEl), win); - interactionRef.current = { - type: "rotate", wordEl, segmentId: box.segmentId, - centerX: cx, centerY: cy, - startAngle, origTX: state.x, origTY: state.y, - origRotation: state.rotation, origScale: state.scale, - }; - }, [iframeRef]); + const startRotate = useCallback( + (box: WordBox, e: React.PointerEvent) => { + e.stopPropagation(); + e.preventDefault(); + (e.currentTarget as HTMLElement).setPointerCapture(e.pointerId); + const iframe = iframeRef.current; + if (!iframe) return; + const wordEl = getWordEl(iframe, box.groupIndex, box.wordIndex); + const win = iframe.contentWindow; + if (!wordEl || !win) return; + const state = readGsapTransform(getOrCreateWrapper(wordEl), win); + interactionRef.current = { + type: "rotate", + wordEl, + segmentId: box.segmentId, + startMX: e.clientX, + origTX: state.x, + origTY: state.y, + origRotation: state.rotation, + origScale: state.scale, + }; + }, + [iframeRef], + ); /** Get iframe contentWindow, needed for gsap calls */ const getIframeWin = useCallback((): Window | null => { - try { return iframeRef.current?.contentWindow ?? null; } catch { return null; } + try { + return iframeRef.current?.contentWindow ?? null; + } catch { + return null; + } }, [iframeRef]); // --- Unified pointer move --- - const handlePointerMove = useCallback((e: React.PointerEvent) => { - const i = interactionRef.current; - if (!i) return; - const win = getIframeWin(); - if (!win) return; + const handlePointerMove = useCallback( + (e: React.PointerEvent) => { + const i = interactionRef.current; + if (!i) return; + const win = getIframeWin(); + if (!win) return; - if (i.type === "move") { - const cssScale = getCssScale(); - const dx = (e.clientX - i.startMX) / cssScale; - const dy = (e.clientY - i.startMY) / cssScale; - writeTransform(i.wordEl, win, i.origTX + dx, i.origTY + dy, i.origScale, i.origRotation); - } else if (i.type === "scale") { - const dx = e.clientX - i.startMX; - const factor = 1 + dx / Math.max(i.startWidth, 50); - const newScale = Math.max(0.1, i.origScale * factor); - writeTransform(i.wordEl, win, i.origTX, i.origTY, newScale, i.origRotation); - } else if (i.type === "rotate") { - const angle = Math.atan2(e.clientY - i.centerY, e.clientX - i.centerX) * (180 / Math.PI); - const delta = angle - i.startAngle; - writeTransform(i.wordEl, win, i.origTX, i.origTY, i.origScale, i.origRotation + delta); - } - }, [getCssScale, getIframeWin]); + if (i.type === "move") { + const cssScale = getCssScale(); + const dx = (e.clientX - i.startMX) / cssScale; + const dy = (e.clientY - i.startMY) / cssScale; + writeTransform(i.wordEl, win, i.origTX + dx, i.origTY + dy, i.origScale, i.origRotation); + } else if (i.type === "scale") { + // Use distance from box center so dragging outward from ANY corner + // increases scale (not just right-side handles). + const cx = i.startMX - i.startDxFromCenter; + const startDist = Math.abs(i.startDxFromCenter); + const currentDist = Math.abs(e.clientX - cx); + const factor = startDist > 5 ? currentDist / startDist : 1; + const newScale = Math.max(0.1, i.origScale * factor); + writeTransform(i.wordEl, win, i.origTX, i.origTY, newScale, i.origRotation); + } else if (i.type === "rotate") { + // Horizontal drag maps to rotation: right = clockwise, left = counter-clockwise. + // 200px of horizontal movement = 90 degrees. + const dx = e.clientX - i.startMX; + const delta = (dx / 200) * 90; + writeTransform(i.wordEl, win, i.origTX, i.origTY, i.origScale, i.origRotation + delta); + } + }, + [getCssScale, getIframeWin], + ); // --- Unified pointer up — sync back to store --- const handlePointerUp = useCallback(() => { @@ -371,9 +510,12 @@ export const CaptionOverlay = memo(function CaptionOverlay({ } }, [getIframeWin]); - const handleBackgroundClick = useCallback((e: React.MouseEvent) => { - if (e.target === e.currentTarget) clearSelection(); - }, [clearSelection]); + const handleBackgroundClick = useCallback( + (e: React.MouseEvent) => { + if (e.target === e.currentTarget) clearSelection(); + }, + [clearSelection], + ); if (!isEditMode) return null; @@ -397,11 +539,18 @@ export const CaptionOverlay = memo(function CaptionOverlay({ isSelected ? "ring-2 ring-studio-accent" : "hover:ring-1 hover:ring-white/30", ].join(" ")} style={{ - left: box.x, top: box.y, width: box.width, height: box.height, + left: box.x, + top: box.y, + width: box.width, + height: box.height, cursor: isSelected ? "move" : "pointer", - touchAction: "none", borderRadius: 2, + touchAction: "none", + borderRadius: 2, + }} + onClick={(e) => { + e.stopPropagation(); + selectSegment(box.segmentId, e.shiftKey); }} - onClick={(e) => { e.stopPropagation(); selectSegment(box.segmentId, e.shiftKey); }} onPointerDown={(e) => { if (isSelected) startMove(box.groupIndex, box.wordIndex, box.segmentId, e); }} @@ -412,13 +561,16 @@ export const CaptionOverlay = memo(function CaptionOverlay({
startRotate(box, e)} /> @@ -426,11 +578,14 @@ export const CaptionOverlay = memo(function CaptionOverlay({
{/* Scale handles — four corners */} @@ -443,13 +598,18 @@ export const CaptionOverlay = memo(function CaptionOverlay({
startScale(box.groupIndex, box.wordIndex, box.segmentId, e)} + onPointerDown={(e) => + startScale(box.groupIndex, box.wordIndex, box.segmentId, e) + } /> ))} diff --git a/packages/studio/src/captions/components/CaptionPropertyPanel.tsx b/packages/studio/src/captions/components/CaptionPropertyPanel.tsx index ba75a8222..e345c3365 100644 --- a/packages/studio/src/captions/components/CaptionPropertyPanel.tsx +++ b/packages/studio/src/captions/components/CaptionPropertyPanel.tsx @@ -1,36 +1,7 @@ -import { memo, useCallback, useState } from "react"; +import { memo, useCallback } from "react"; import { useCaptionStore } from "../store"; import type { CaptionStyle } from "../types"; -import { CaptionAnimationPanel } from "./CaptionAnimationPanel"; - -// --------------------------------------------------------------------------- -// Helper Components -// --------------------------------------------------------------------------- - -function Section({ label, children }: { label: string; children: React.ReactNode }) { - return ( -
-
- - {label} - -
-
{children}
-
- ); -} - -function Row({ label, children }: { label: string; children: React.ReactNode }) { - return ( -
- {label} -
{children}
-
- ); -} - -const inputCls = - "w-full bg-neutral-900 border border-neutral-800 rounded px-1.5 py-0.5 text-2xs text-neutral-200 font-mono outline-none focus:border-neutral-600"; +import { Section, Row, inputCls } from "./shared"; // --------------------------------------------------------------------------- // Main component @@ -49,8 +20,6 @@ export const CaptionPropertyPanel = memo(function CaptionPropertyPanel({ const updateSelectedStyle = useCaptionStore((s) => s.updateSelectedStyle); const updateGroupStyle = useCaptionStore((s) => s.updateGroupStyle); - const [activeTab, setActiveTab] = useState<"style" | "animation">("style"); - // Resolve effective style for the first selected segment const firstSegmentId = selectedSegmentIds.size > 0 ? [...selectedSegmentIds][0] : undefined; const firstSegment = model?.segments.get(firstSegmentId ?? ""); @@ -76,7 +45,6 @@ export const CaptionPropertyPanel = memo(function CaptionPropertyPanel({ ...segmentOverrides, }; - /** * Apply a CSS style change to selected word elements in the iframe DOM in real time. * Maps CaptionStyle property names to CSS properties. @@ -112,9 +80,15 @@ export const CaptionPropertyPanel = memo(function CaptionPropertyPanel({ const c = child as HTMLElement; if (c.dataset.captionWrapper === "true") { const inner = c.querySelector(":scope > span"); - if (inner && idx === wi) { targetEls.push(inner); break; } + if (inner && idx === wi) { + targetEls.push(inner); + break; + } } else if (c.tagName === "SPAN") { - if (idx === wi) { targetEls.push(c); break; } + if (idx === wi) { + targetEls.push(c); + break; + } } idx++; } @@ -123,15 +97,23 @@ export const CaptionPropertyPanel = memo(function CaptionPropertyPanel({ } // Apply transform updates via gsap.set on the WRAPPER (not the word span) - const hasTransform = updates.x !== undefined || updates.y !== undefined || - updates.scaleX !== undefined || updates.scaleY !== undefined || updates.rotation !== undefined; + const hasTransform = + updates.x !== undefined || + updates.y !== undefined || + updates.scaleX !== undefined || + updates.scaleY !== undefined || + updates.rotation !== undefined; if (hasTransform) { try { - const iframeGsap = (iframeRef.current?.contentWindow as unknown as { - gsap?: { set: (el: HTMLElement, props: Record) => void; - getProperty: (el: HTMLElement, prop: string) => number }; - })?.gsap; + const iframeGsap = ( + iframeRef.current?.contentWindow as unknown as { + gsap?: { + set: (el: HTMLElement, props: Record) => void; + getProperty: (el: HTMLElement, prop: string) => number; + }; + } + )?.gsap; if (iframeGsap) { for (const el of targetEls) { // Get or create wrapper @@ -156,7 +138,9 @@ export const CaptionPropertyPanel = memo(function CaptionPropertyPanel({ }); } } - } catch { /* cross-origin */ } + } catch { + /* cross-origin */ + } } }, [iframeRef, model, selectedSegmentIds], @@ -194,101 +178,60 @@ export const CaptionPropertyPanel = memo(function CaptionPropertyPanel({ const scaleX = effectiveStyle.scaleX ?? 1; // Count label - const countLabel = selectedSegmentIds.size === 1 - ? "1 word" - : `${selectedSegmentIds.size} words`; + const countLabel = selectedSegmentIds.size === 1 ? "1 word" : `${selectedSegmentIds.size} words`; return (
{/* Header */}
-
- - {countLabel} - -
- {/* Tab switcher */} -
- - -
+ {countLabel}
- {/* Animation tab */} - {activeTab === "animation" && } +
+
+ + handleStyleChange({ x: Number(e.target.value) })} + className={inputCls} + /> + + + handleStyleChange({ y: Number(e.target.value) })} + className={inputCls} + /> + +
- {/* Style tab — Transform only */} - {activeTab === "style" && ( -
-
- - handleStyleChange({ x: Number(e.target.value) })} - className={inputCls} - /> - - - - handleStyleChange({ y: Number(e.target.value) })} - className={inputCls} - /> - -
- -
- - - handleStyleChange({ - scaleX: Number(e.target.value), - scaleY: Number(e.target.value), - }) - } - className={inputCls} - /> - - - - handleStyleChange({ rotation: Number(e.target.value) })} - className={inputCls} - /> - -
-
- )} +
+ + + handleStyleChange({ + scaleX: Number(e.target.value), + scaleY: Number(e.target.value), + }) + } + className={inputCls} + /> + + + handleStyleChange({ rotation: Number(e.target.value) })} + className={inputCls} + /> + +
+
); }); diff --git a/packages/studio/src/captions/components/shared.tsx b/packages/studio/src/captions/components/shared.tsx new file mode 100644 index 000000000..cb4bfe28b --- /dev/null +++ b/packages/studio/src/captions/components/shared.tsx @@ -0,0 +1,26 @@ +import type React from "react"; + +export const inputCls = + "w-full bg-neutral-900 border border-neutral-800 rounded px-1.5 py-0.5 text-2xs text-neutral-200 font-mono outline-none focus:border-neutral-600"; + +export function Section({ label, children }: { label: string; children: React.ReactNode }) { + return ( +
+
+ + {label} + +
+
{children}
+
+ ); +} + +export function Row({ label, children }: { label: string; children: React.ReactNode }) { + return ( +
+ {label} +
{children}
+
+ ); +} diff --git a/packages/studio/src/captions/generator.ts b/packages/studio/src/captions/generator.ts index 3228e0924..06fff5742 100644 --- a/packages/studio/src/captions/generator.ts +++ b/packages/studio/src/captions/generator.ts @@ -233,6 +233,10 @@ function hexToRgba(color: string, opacity: number): string { if (color.startsWith("rgb")) { return color; } + // Named colors and other non-hex values — return as-is + if (!color.startsWith("#")) { + return color; + } // Try to parse hex const hex = color.replace("#", ""); if (hex.length === 3 || hex.length === 6) { diff --git a/packages/studio/src/captions/hooks/useCaptionSync.ts b/packages/studio/src/captions/hooks/useCaptionSync.ts index 33bfa6839..08ee4c9e3 100644 --- a/packages/studio/src/captions/hooks/useCaptionSync.ts +++ b/packages/studio/src/captions/hooks/useCaptionSync.ts @@ -74,10 +74,11 @@ export function useCaptionSync(projectId: string | null) { const overrides = buildOverrides(state.model); - fetch( - `/api/projects/${pid}/files/${encodeURIComponent("caption-overrides.json")}`, - { method: "PUT", headers: { "Content-Type": "text/plain" }, body: JSON.stringify(overrides, null, 2) }, - ).catch(() => {}); + fetch(`/api/projects/${pid}/files/${encodeURIComponent("caption-overrides.json")}`, { + method: "PUT", + headers: { "Content-Type": "text/plain" }, + body: JSON.stringify(overrides, null, 2), + }).catch((err) => console.warn("[captions] auto-save failed:", err)); }, []); // Auto-save on model changes with 800ms debounce @@ -140,7 +141,10 @@ export function useCaptionSync(projectId: string | null) { const style: Partial = { ...seg.style }; if (override.x !== undefined) style.x = override.x; if (override.y !== undefined) style.y = override.y; - if (override.scale !== undefined) { style.scaleX = override.scale; style.scaleY = override.scale; } + if (override.scale !== undefined) { + style.scaleX = override.scale; + style.scaleY = override.scale; + } if (override.rotation !== undefined) style.rotation = override.rotation; if (override.activeColor !== undefined) style.activeColor = override.activeColor; if (override.dimColor !== undefined) style.dimColor = override.dimColor; diff --git a/packages/studio/src/captions/parser.ts b/packages/studio/src/captions/parser.ts index e8e17fd59..22acddea2 100644 --- a/packages/studio/src/captions/parser.ts +++ b/packages/studio/src/captions/parser.ts @@ -107,6 +107,8 @@ export function buildCaptionModel( export function extractTranscript(source: string): TranscriptWord[] { // Match: (const|let|var) (TRANSCRIPT|script) = [...] // The array may span multiple lines and contain trailing commas. + // The lazy [\s\S]*? anchors on the first `];` — assumes transcript word + // text never contains a literal `];` string (safe for speech transcripts). const varPattern = /(?:const|let|var)\s+(?:TRANSCRIPT|script)\s*=\s*(\[[\s\S]*?\]);/; const match = source.match(varPattern); diff --git a/packages/studio/src/captions/store.ts b/packages/studio/src/captions/store.ts index 48f9b1ca3..4c62c3b4e 100644 --- a/packages/studio/src/captions/store.ts +++ b/packages/studio/src/captions/store.ts @@ -7,6 +7,8 @@ import { CaptionStyle, } from "./types"; +let nextSplitId = 0; + interface CaptionState { isEditMode: boolean; model: CaptionModel | null; @@ -57,7 +59,7 @@ const initialState = { sourceFilePath: null, }; -export const useCaptionStore = create((set, get) => ({ +export const useCaptionStore = create((set) => ({ ...initialState, // Basic @@ -186,7 +188,7 @@ export const useCaptionStore = create((set, get) => ({ const firstIds = group.segmentIds.slice(0, splitIndex); const secondIds = group.segmentIds.slice(splitIndex); - const newGroupId = `group-${Date.now()}-${Math.random().toString(36).slice(2)}`; + const newGroupId = `group-split-${nextSplitId++}`; const groups = new Map(state.model.groups); groups.set(groupId, { ...group, segmentIds: firstIds }); groups.set(newGroupId, { ...group, id: newGroupId, segmentIds: secondIds }); @@ -232,7 +234,7 @@ export const useCaptionStore = create((set, get) => ({ }); // Clear selection if it referenced group2 - const selectedGroupId = get().selectedGroupId === groupId2 ? null : get().selectedGroupId; + const selectedGroupId = state.selectedGroupId === groupId2 ? null : state.selectedGroupId; return { model: { ...state.model, groups, segments, groupOrder }, selectedGroupId }; }), diff --git a/packages/studio/src/components/nle/NLELayout.tsx b/packages/studio/src/components/nle/NLELayout.tsx index 9e47a9e77..9b46ea1a1 100644 --- a/packages/studio/src/components/nle/NLELayout.tsx +++ b/packages/studio/src/components/nle/NLELayout.tsx @@ -376,8 +376,8 @@ export const NLELayout = memo(function NLELayout({ onDrillDown={handleDrillDown} renderClipContent={renderClipContent} /> - {timelineFooter}
+ {timelineFooter &&
{timelineFooter}
}
)} diff --git a/skills/hyperframes-captions/SKILL.md b/skills/hyperframes-captions/SKILL.md index dc8292feb..6072bbd3a 100644 --- a/skills/hyperframes-captions/SKILL.md +++ b/skills/hyperframes-captions/SKILL.md @@ -20,7 +20,6 @@ When transcribing: --- - Analyze the spoken content to determine caption style. If the user specifies a style, use that. Otherwise, detect tone from the transcript. ## Transcript Source diff --git a/skills/hyperframes-captions/transcript-guide.md b/skills/hyperframes-captions/transcript-guide.md index 34813834e..5bbf764a2 100644 --- a/skills/hyperframes-captions/transcript-guide.md +++ b/skills/hyperframes-captions/transcript-guide.md @@ -38,13 +38,13 @@ The CLI auto-detects and normalizes these formats: The default model (`small.en`) balances accuracy and speed. For better results, use a larger model: -| Model | Size | Speed | Accuracy | When to use | -| ----------- | ------ | -------- | --------- | ------------------------------------- | -| `tiny` | 75 MB | Fastest | Low | Quick previews, testing pipeline | -| `base` | 142 MB | Fast | Fair | Short clips, clear audio | -| `small` | 466 MB | Moderate | Good | **Default** — good for most content | -| `medium` | 1.5 GB | Slow | Very good | Important content, noisy audio, music | -| `large-v3` | 3.1 GB | Slowest | Best | Production quality | +| Model | Size | Speed | Accuracy | When to use | +| ---------- | ------ | -------- | --------- | ------------------------------------- | +| `tiny` | 75 MB | Fastest | Low | Quick previews, testing pipeline | +| `base` | 142 MB | Fast | Fair | Short clips, clear audio | +| `small` | 466 MB | Moderate | Good | **Default** — good for most content | +| `medium` | 1.5 GB | Slow | Very good | Important content, noisy audio, music | +| `large-v3` | 3.1 GB | Slowest | Best | Production quality | **Only add `.en` suffix when the user explicitly says the audio is English.** `.en` models are slightly more accurate for English but will TRANSLATE non-English audio instead of transcribing it.