mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 23:03:09 +00:00
feat(studio): drag assets from the sidebar onto the timeline (#464)
## Problem Studio still broke down in three concrete authoring flows around timeline assets: - you could import media into Assets, but not drag an already-imported asset from the Assets tab onto the timeline and persist it into source - dragging a file from outside the app onto the timeline only uploaded it into Assets instead of placing it at the dropped time/track - once a clip was on the timeline, there was no reliable keyboard delete flow for removing it safely from source While implementing direct external drops, another real bug showed up: - valid binary uploads like `raycast.mp4` from `Downloads` were being rejected as unsupported media in Studio dev because the Vite API bridge was corrupting multipart request bodies before they reached the upload route ## What this fixes ### Timeline asset placement from inside Studio - asset cards in the Assets tab are draggable - the timeline accepts asset drops even when it already has clips - dropping an asset onto the timeline inserts a new clip into the active composition source at the dropped time / track - asset paths are rewritten relative to the target composition file so drops into sub-compositions resolve correctly - the new clip is persisted immediately and the preview refreshes ### Direct external file drops onto the timeline - dropping a file from outside the app onto the timeline now uploads it and places it onto the dropped track/time in one shot - it no longer stops halfway by only adding the file into Assets - multiple dropped files are placed using the same drop start and successive tracks ### Delete key support - selected timeline clips can now be deleted with `Delete` / `Backspace` - deletion is persisted back to source, not just removed from local state - the delete path now uses a server-side DOM mutation helper with LinkeDOM for structural safety instead of client-side string surgery ### Binary upload fix for media files - the Studio Vite API bridge now forwards non-GET request bodies as raw bytes instead of decoding them as UTF-8 text - that preserves multipart uploads for binary media like MP4s - valid local videos from `Downloads` no longer get rejected as `Unsupported media skipped` just because the dev bridge corrupted the request body - upload validation now probes buffered media through a temp file path that preserves the file extension before saving into the project ## Root cause There were really two separate gaps: ### 1. Asset placement / deletion workflow gaps The timeline and asset systems already existed, but they were disconnected: - `AssetsTab` only supported copy/import flows - `Timeline` only handled raw file import, not positioned placement for existing assets - there was no utility layer for converting a dropped asset into persisted timeline HTML - there was no structurally safe deletion path for arbitrary selected timeline clips ### 2. Binary upload corruption in Studio dev The Studio Vite API bridge rebuilt non-GET request bodies like this: - read each request chunk - call `chunk.toString()` - concatenate into a string - construct the Fetch `Request` from that string body That works for text, but it corrupts multipart binary uploads. By the time the upload route wrote the received file and ran `ffprobe`, otherwise valid MP4s had already been mangled in-flight. ## Behavior - dropping on `index.html` inserts the asset into the root composition - dropping while drilled into a composition inserts into that composition file instead - drop X position maps to `data-start` - drop Y position maps to the current visible track row, with a new bottom track created if the drop lands below existing rows - images default to a short finite duration - audio/video default to their metadata duration when available, with a fallback duration if metadata cannot be read quickly - pressing `Delete` on a selected clip removes that clip from the underlying HTML source and clears selection in Studio - valid uploaded MP4s now survive the Studio dev API bridge intact instead of being rejected during upload validation ## Verification ### Local checks - `bunx oxlint packages/core/src/studio-api/helpers/sourceMutation.ts packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/core/src/studio-api/routes/files.ts packages/studio/src/App.tsx packages/studio/src/components/nle/NLELayout.tsx packages/studio/src/components/sidebar/AssetsTab.tsx packages/studio/src/player/components/Timeline.tsx packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.config.ts packages/studio/vite.request-body.ts packages/studio/vite.request-body.test.ts` - `bunx oxfmt --check` on the touched files - `bun run --filter @hyperframes/core typecheck` - `bun run --filter @hyperframes/studio typecheck` - `bun test packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.request-body.test.ts` ### Browser / live verification Verified against a live local Studio fixture: - dragging an existing asset from the Assets tab onto the timeline creates a persisted clip at the dropped position - dropping a file from outside the app directly onto the timeline uploads it and creates a persisted clip at the dropped position - selecting a dropped clip and pressing `Delete` removes it from both the live timeline and the saved source HTML - valid MP4 uploads like `raycast.mp4` now succeed through the live Studio upload route instead of being rejected as unsupported media ## Notes - the local `timeline-trio-verify` and `timeline-overlap-debug` projects used for verification are local-only and are not part of this PR - this PR is about asset placement, upload correctness, and deletion safety; it does not broaden into richer editing workflows beyond placing/removing clips from the timeline
This commit is contained in:
Miguel Ángel
@@ -1,9 +1,12 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import {
|
||||
generateTicks,
|
||||
getDefaultDroppedTrack,
|
||||
getTimelineCanvasHeight,
|
||||
resolveTimelineAssetDrop,
|
||||
getTimelinePlayheadLeft,
|
||||
getTimelineScrollLeftForZoomTransition,
|
||||
shouldHandleTimelineDeleteKey,
|
||||
shouldAutoScrollTimeline,
|
||||
} from "./Timeline";
|
||||
import { formatTime } from "../lib/time";
|
||||
@@ -162,3 +165,73 @@ describe("getTimelineCanvasHeight", () => {
|
||||
expect(getTimelineCanvasHeight(0)).toBeGreaterThan(24);
|
||||
});
|
||||
});
|
||||
|
||||
describe("shouldHandleTimelineDeleteKey", () => {
|
||||
it("handles Delete and Backspace when focus is not in an editor", () => {
|
||||
expect(shouldHandleTimelineDeleteKey({ key: "Delete" })).toBe(true);
|
||||
expect(shouldHandleTimelineDeleteKey({ key: "Backspace" })).toBe(true);
|
||||
});
|
||||
|
||||
it("ignores modifier shortcuts", () => {
|
||||
expect(shouldHandleTimelineDeleteKey({ key: "Delete", metaKey: true })).toBe(false);
|
||||
expect(shouldHandleTimelineDeleteKey({ key: "Backspace", ctrlKey: true })).toBe(false);
|
||||
});
|
||||
|
||||
it("ignores input and editable targets", () => {
|
||||
const input = { tagName: "INPUT", isContentEditable: false };
|
||||
const editable = { tagName: "DIV", isContentEditable: true };
|
||||
|
||||
expect(shouldHandleTimelineDeleteKey({ key: "Delete", target: input })).toBe(false);
|
||||
expect(shouldHandleTimelineDeleteKey({ key: "Delete", target: editable })).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("getDefaultDroppedTrack", () => {
|
||||
it("defaults to track 0 when there are no rows yet", () => {
|
||||
expect(getDefaultDroppedTrack([])).toBe(0);
|
||||
});
|
||||
|
||||
it("creates a new bottom track when dropped below existing rows", () => {
|
||||
expect(getDefaultDroppedTrack([0, 1, 5], 10)).toBe(6);
|
||||
});
|
||||
});
|
||||
|
||||
describe("resolveTimelineAssetDrop", () => {
|
||||
it("maps drop coordinates to a start time and visible track", () => {
|
||||
expect(
|
||||
resolveTimelineAssetDrop(
|
||||
{
|
||||
rectLeft: 100,
|
||||
rectTop: 200,
|
||||
scrollLeft: 0,
|
||||
scrollTop: 0,
|
||||
pixelsPerSecond: 100,
|
||||
duration: 10,
|
||||
trackHeight: 72,
|
||||
trackOrder: [0, 3, 7],
|
||||
},
|
||||
432,
|
||||
310,
|
||||
),
|
||||
).toEqual({ start: 3, track: 3 });
|
||||
});
|
||||
|
||||
it("can create a new bottom track when dropped below the last visible row", () => {
|
||||
expect(
|
||||
resolveTimelineAssetDrop(
|
||||
{
|
||||
rectLeft: 100,
|
||||
rectTop: 200,
|
||||
scrollLeft: 0,
|
||||
scrollTop: 0,
|
||||
pixelsPerSecond: 100,
|
||||
duration: 10,
|
||||
trackHeight: 72,
|
||||
trackOrder: [0, 3, 7],
|
||||
},
|
||||
250,
|
||||
600,
|
||||
),
|
||||
).toEqual({ start: 1.18, track: 8 });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -27,6 +27,7 @@ import {
|
||||
type TimelineTheme,
|
||||
} from "./timelineTheme";
|
||||
import { getTimelinePixelsPerSecond } from "./timelineZoom";
|
||||
import { TIMELINE_ASSET_MIME } from "../../utils/timelineAssetDrop";
|
||||
|
||||
/* ── Layout ─────────────────────────────────────────────────────── */
|
||||
const GUTTER = 32;
|
||||
@@ -140,6 +141,70 @@ export function getTimelineCanvasHeight(trackCount: number): number {
|
||||
return RULER_H + Math.max(0, trackCount) * TRACK_H + TIMELINE_SCROLL_BUFFER;
|
||||
}
|
||||
|
||||
export function shouldHandleTimelineDeleteKey(input: {
|
||||
key: string;
|
||||
metaKey?: boolean;
|
||||
ctrlKey?: boolean;
|
||||
altKey?: boolean;
|
||||
target?: EventTarget | null;
|
||||
}): boolean {
|
||||
if (input.key !== "Delete" && input.key !== "Backspace") return false;
|
||||
if (input.metaKey || input.ctrlKey || input.altKey) return false;
|
||||
const target =
|
||||
input.target && typeof input.target === "object"
|
||||
? (input.target as {
|
||||
tagName?: string;
|
||||
isContentEditable?: boolean;
|
||||
closest?: (selector: string) => Element | null;
|
||||
})
|
||||
: null;
|
||||
if (target) {
|
||||
const tag = target.tagName?.toLowerCase() ?? "";
|
||||
if (target.isContentEditable) return false;
|
||||
if (["input", "textarea", "select"].includes(tag)) return false;
|
||||
if (typeof target.closest === "function" && target.closest("[contenteditable='true']")) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
export function getDefaultDroppedTrack(trackOrder: number[], rowIndex?: number): number {
|
||||
if (trackOrder.length === 0) return 0;
|
||||
if (rowIndex == null || rowIndex < 0) return trackOrder[0];
|
||||
if (rowIndex >= trackOrder.length) {
|
||||
return Math.max(...trackOrder) + 1;
|
||||
}
|
||||
return trackOrder[rowIndex] ?? trackOrder[trackOrder.length - 1] ?? 0;
|
||||
}
|
||||
|
||||
export function resolveTimelineAssetDrop(
|
||||
input: {
|
||||
rectLeft: number;
|
||||
rectTop: number;
|
||||
scrollLeft: number;
|
||||
scrollTop: number;
|
||||
pixelsPerSecond: number;
|
||||
duration: number;
|
||||
trackHeight: number;
|
||||
trackOrder: number[];
|
||||
},
|
||||
clientX: number,
|
||||
clientY: number,
|
||||
): { start: number; track: number } {
|
||||
const x = clientX - input.rectLeft + input.scrollLeft - GUTTER;
|
||||
const y = clientY - input.rectTop + input.scrollTop - RULER_H;
|
||||
const start = Math.max(
|
||||
0,
|
||||
Math.min(input.duration, Math.round((x / Math.max(input.pixelsPerSecond, 1)) * 100) / 100),
|
||||
);
|
||||
const rowIndex = Math.floor(y / Math.max(input.trackHeight, 1));
|
||||
return {
|
||||
start,
|
||||
track: getDefaultDroppedTrack(input.trackOrder, rowIndex),
|
||||
};
|
||||
}
|
||||
|
||||
/* ── Component ──────────────────────────────────────────────────── */
|
||||
interface TimelineProps {
|
||||
/** Called when user seeks via ruler/track click or playhead drag */
|
||||
@@ -154,8 +219,19 @@ interface TimelineProps {
|
||||
/** Optional overlay renderer for clips (e.g. badges, cursors) */
|
||||
renderClipOverlay?: (element: import("../store/playerStore").TimelineElement) => ReactNode;
|
||||
/** Called when files are dropped onto the empty timeline */
|
||||
onFileDrop?: (files: File[]) => void;
|
||||
onFileDrop?: (
|
||||
files: File[],
|
||||
placement?: { start: number; track: number },
|
||||
) => Promise<void> | void;
|
||||
/** Called when an existing asset is dropped from the Assets tab */
|
||||
onAssetDrop?: (
|
||||
assetPath: string,
|
||||
placement: { start: number; track: number },
|
||||
) => Promise<void> | void;
|
||||
/** Persist a clip move back into source HTML */
|
||||
onDeleteElement?: (
|
||||
element: import("../store/playerStore").TimelineElement,
|
||||
) => Promise<void> | void;
|
||||
onMoveElement?: (
|
||||
element: import("../store/playerStore").TimelineElement,
|
||||
updates: Pick<import("../store/playerStore").TimelineElement, "start" | "track">,
|
||||
@@ -213,6 +289,8 @@ export const Timeline = memo(function Timeline({
|
||||
renderClipContent,
|
||||
renderClipOverlay,
|
||||
onFileDrop,
|
||||
onAssetDrop,
|
||||
onDeleteElement,
|
||||
onMoveElement,
|
||||
onResizeElement,
|
||||
onBlockedEditAttempt,
|
||||
@@ -263,10 +341,13 @@ export const Timeline = memo(function Timeline({
|
||||
const resizingClipRef = useRef<ResizingClipState | null>(null);
|
||||
resizingClipRef.current = resizingClip;
|
||||
const blockedClipRef = useRef<BlockedClipState | null>(null);
|
||||
const deleteInFlightRef = useRef(false);
|
||||
const onMoveElementRef = useRef(onMoveElement);
|
||||
onMoveElementRef.current = onMoveElement;
|
||||
const onResizeElementRef = useRef(onResizeElement);
|
||||
onResizeElementRef.current = onResizeElement;
|
||||
const onDeleteElementRef = useRef(onDeleteElement);
|
||||
onDeleteElementRef.current = onDeleteElement;
|
||||
const suppressClickRef = useRef(false);
|
||||
const [showPopover, setShowPopover] = useState(false);
|
||||
const [viewportWidth, setViewportWidth] = useState(0);
|
||||
@@ -337,6 +418,12 @@ export const Timeline = memo(function Timeline({
|
||||
}
|
||||
return [...trackOrder, draggedClip.previewTrack].sort((a, b) => a - b);
|
||||
}, [draggedClip, trackOrder]);
|
||||
const selectedElement = useMemo(
|
||||
() => elements.find((element) => (element.key ?? element.id) === selectedElementId) ?? null,
|
||||
[elements, selectedElementId],
|
||||
);
|
||||
const selectedElementRef = useRef<TimelineElement | null>(selectedElement);
|
||||
selectedElementRef.current = selectedElement;
|
||||
|
||||
// Calculate effective pixels per second
|
||||
// In fit mode, use clientWidth (excludes scrollbar) with a small padding
|
||||
@@ -743,6 +830,28 @@ export const Timeline = memo(function Timeline({
|
||||
};
|
||||
});
|
||||
|
||||
useMountEffect(() => {
|
||||
const handleKeyDown = (event: KeyboardEvent) => {
|
||||
if (!shouldHandleTimelineDeleteKey(event)) return;
|
||||
const selected = selectedElementRef.current;
|
||||
const onDelete = onDeleteElementRef.current;
|
||||
if (!selected || !onDelete || deleteInFlightRef.current) return;
|
||||
event.preventDefault();
|
||||
deleteInFlightRef.current = true;
|
||||
suppressClickRef.current = true;
|
||||
setShowPopover(false);
|
||||
setRangeSelection(null);
|
||||
Promise.resolve(onDelete(selected)).finally(() => {
|
||||
deleteInFlightRef.current = false;
|
||||
requestAnimationFrame(() => {
|
||||
suppressClickRef.current = false;
|
||||
});
|
||||
});
|
||||
};
|
||||
window.addEventListener("keydown", handleKeyDown);
|
||||
return () => window.removeEventListener("keydown", handleKeyDown);
|
||||
});
|
||||
|
||||
const handlePointerDown = useCallback(
|
||||
(e: React.PointerEvent) => {
|
||||
if (e.button !== 0) return;
|
||||
@@ -833,6 +942,74 @@ export const Timeline = memo(function Timeline({
|
||||
);
|
||||
|
||||
const [isDragOver, setIsDragOver] = useState(false);
|
||||
const handleAssetDragOver = useCallback((e: React.DragEvent) => {
|
||||
const hasFiles = e.dataTransfer.files.length > 0;
|
||||
const hasAsset = Array.from(e.dataTransfer.types).includes(TIMELINE_ASSET_MIME);
|
||||
if (!hasFiles && !hasAsset) return;
|
||||
e.preventDefault();
|
||||
if (hasAsset) {
|
||||
e.dataTransfer.dropEffect = "copy";
|
||||
}
|
||||
setIsDragOver(true);
|
||||
}, []);
|
||||
|
||||
const handleAssetDrop = useCallback(
|
||||
(e: React.DragEvent) => {
|
||||
e.preventDefault();
|
||||
setIsDragOver(false);
|
||||
if (onFileDrop && e.dataTransfer.files.length > 0) {
|
||||
const scroll = scrollRef.current;
|
||||
const rect = scroll?.getBoundingClientRect();
|
||||
const placement =
|
||||
scroll && rect
|
||||
? resolveTimelineAssetDrop(
|
||||
{
|
||||
rectLeft: rect.left,
|
||||
rectTop: rect.top,
|
||||
scrollLeft: scroll.scrollLeft,
|
||||
scrollTop: scroll.scrollTop,
|
||||
pixelsPerSecond: ppsRef.current,
|
||||
duration: durationRef.current,
|
||||
trackHeight: TRACK_H,
|
||||
trackOrder: trackOrderRef.current,
|
||||
},
|
||||
e.clientX,
|
||||
e.clientY,
|
||||
)
|
||||
: undefined;
|
||||
void onFileDrop(Array.from(e.dataTransfer.files), placement);
|
||||
return;
|
||||
}
|
||||
|
||||
const assetPayload = e.dataTransfer.getData(TIMELINE_ASSET_MIME);
|
||||
if (!assetPayload || !onAssetDrop) return;
|
||||
try {
|
||||
const parsed = JSON.parse(assetPayload) as { path?: string };
|
||||
if (!parsed.path) return;
|
||||
const scroll = scrollRef.current;
|
||||
const rect = scroll?.getBoundingClientRect();
|
||||
if (!scroll || !rect) return;
|
||||
const placement = resolveTimelineAssetDrop(
|
||||
{
|
||||
rectLeft: rect.left,
|
||||
rectTop: rect.top,
|
||||
scrollLeft: scroll.scrollLeft,
|
||||
scrollTop: scroll.scrollTop,
|
||||
pixelsPerSecond: ppsRef.current,
|
||||
duration: durationRef.current,
|
||||
trackHeight: TRACK_H,
|
||||
trackOrder: trackOrderRef.current,
|
||||
},
|
||||
e.clientX,
|
||||
e.clientY,
|
||||
);
|
||||
void onAssetDrop(parsed.path, placement);
|
||||
} catch {
|
||||
// ignore malformed drag payloads
|
||||
}
|
||||
},
|
||||
[onAssetDrop, onFileDrop],
|
||||
);
|
||||
|
||||
if (!timelineReady || elements.length === 0) {
|
||||
return (
|
||||
@@ -840,18 +1017,9 @@ export const Timeline = memo(function Timeline({
|
||||
className={`h-full border-t bg-[#0a0a0b] flex flex-col select-none transition-colors duration-150 ${
|
||||
isDragOver ? "border-studio-accent/50 bg-studio-accent/[0.03]" : "border-neutral-800/50"
|
||||
}`}
|
||||
onDragOver={(e) => {
|
||||
e.preventDefault();
|
||||
setIsDragOver(true);
|
||||
}}
|
||||
onDragOver={handleAssetDragOver}
|
||||
onDragLeave={() => setIsDragOver(false)}
|
||||
onDrop={(e) => {
|
||||
e.preventDefault();
|
||||
setIsDragOver(false);
|
||||
if (onFileDrop && e.dataTransfer.files.length > 0) {
|
||||
onFileDrop(Array.from(e.dataTransfer.files));
|
||||
}
|
||||
}}
|
||||
onDrop={handleAssetDrop}
|
||||
>
|
||||
{/* Ruler */}
|
||||
<div
|
||||
@@ -1015,6 +1183,9 @@ export const Timeline = memo(function Timeline({
|
||||
<div
|
||||
ref={scrollRef}
|
||||
className={`${zoomMode === "fit" ? "overflow-x-hidden" : "overflow-x-auto"} overflow-y-auto h-full`}
|
||||
onDragOver={handleAssetDragOver}
|
||||
onDragLeave={() => setIsDragOver(false)}
|
||||
onDrop={handleAssetDrop}
|
||||
onPointerDown={handlePointerDown}
|
||||
onPointerMove={handlePointerMove}
|
||||
onPointerUp={handlePointerUp}
|
||||
|
||||
Reference in New Issue
Block a user