mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-11 23:00:03 +00:00
feat(studio): drag assets from the sidebar onto the timeline (#464)
## Problem Studio still broke down in three concrete authoring flows around timeline assets: - you could import media into Assets, but not drag an already-imported asset from the Assets tab onto the timeline and persist it into source - dragging a file from outside the app onto the timeline only uploaded it into Assets instead of placing it at the dropped time/track - once a clip was on the timeline, there was no reliable keyboard delete flow for removing it safely from source While implementing direct external drops, another real bug showed up: - valid binary uploads like `raycast.mp4` from `Downloads` were being rejected as unsupported media in Studio dev because the Vite API bridge was corrupting multipart request bodies before they reached the upload route ## What this fixes ### Timeline asset placement from inside Studio - asset cards in the Assets tab are draggable - the timeline accepts asset drops even when it already has clips - dropping an asset onto the timeline inserts a new clip into the active composition source at the dropped time / track - asset paths are rewritten relative to the target composition file so drops into sub-compositions resolve correctly - the new clip is persisted immediately and the preview refreshes ### Direct external file drops onto the timeline - dropping a file from outside the app onto the timeline now uploads it and places it onto the dropped track/time in one shot - it no longer stops halfway by only adding the file into Assets - multiple dropped files are placed using the same drop start and successive tracks ### Delete key support - selected timeline clips can now be deleted with `Delete` / `Backspace` - deletion is persisted back to source, not just removed from local state - the delete path now uses a server-side DOM mutation helper with LinkeDOM for structural safety instead of client-side string surgery ### Binary upload fix for media files - the Studio Vite API bridge now forwards non-GET request bodies as raw bytes instead of decoding them as UTF-8 text - that preserves multipart uploads for binary media like MP4s - valid local videos from `Downloads` no longer get rejected as `Unsupported media skipped` just because the dev bridge corrupted the request body - upload validation now probes buffered media through a temp file path that preserves the file extension before saving into the project ## Root cause There were really two separate gaps: ### 1. Asset placement / deletion workflow gaps The timeline and asset systems already existed, but they were disconnected: - `AssetsTab` only supported copy/import flows - `Timeline` only handled raw file import, not positioned placement for existing assets - there was no utility layer for converting a dropped asset into persisted timeline HTML - there was no structurally safe deletion path for arbitrary selected timeline clips ### 2. Binary upload corruption in Studio dev The Studio Vite API bridge rebuilt non-GET request bodies like this: - read each request chunk - call `chunk.toString()` - concatenate into a string - construct the Fetch `Request` from that string body That works for text, but it corrupts multipart binary uploads. By the time the upload route wrote the received file and ran `ffprobe`, otherwise valid MP4s had already been mangled in-flight. ## Behavior - dropping on `index.html` inserts the asset into the root composition - dropping while drilled into a composition inserts into that composition file instead - drop X position maps to `data-start` - drop Y position maps to the current visible track row, with a new bottom track created if the drop lands below existing rows - images default to a short finite duration - audio/video default to their metadata duration when available, with a fallback duration if metadata cannot be read quickly - pressing `Delete` on a selected clip removes that clip from the underlying HTML source and clears selection in Studio - valid uploaded MP4s now survive the Studio dev API bridge intact instead of being rejected during upload validation ## Verification ### Local checks - `bunx oxlint packages/core/src/studio-api/helpers/sourceMutation.ts packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/core/src/studio-api/routes/files.ts packages/studio/src/App.tsx packages/studio/src/components/nle/NLELayout.tsx packages/studio/src/components/sidebar/AssetsTab.tsx packages/studio/src/player/components/Timeline.tsx packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.config.ts packages/studio/vite.request-body.ts packages/studio/vite.request-body.test.ts` - `bunx oxfmt --check` on the touched files - `bun run --filter @hyperframes/core typecheck` - `bun run --filter @hyperframes/studio typecheck` - `bun test packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.request-body.test.ts` ### Browser / live verification Verified against a live local Studio fixture: - dragging an existing asset from the Assets tab onto the timeline creates a persisted clip at the dropped position - dropping a file from outside the app directly onto the timeline uploads it and creates a persisted clip at the dropped position - selecting a dropped clip and pressing `Delete` removes it from both the live timeline and the saved source HTML - valid MP4 uploads like `raycast.mp4` now succeed through the live Studio upload route instead of being rejected as unsupported media ## Notes - the local `timeline-trio-verify` and `timeline-overlap-debug` projects used for verification are local-only and are not part of this PR - this PR is about asset placement, upload correctness, and deletion safety; it does not broaden into richer editing workflows beyond placing/removing clips from the timeline
This commit is contained in:
@@ -0,0 +1,87 @@
|
||||
import { AUDIO_EXT, IMAGE_EXT, VIDEO_EXT } from "./mediaTypes";
|
||||
|
||||
export const TIMELINE_ASSET_MIME = "application/x-hyperframes-asset";
|
||||
|
||||
export type TimelineAssetKind = "image" | "video" | "audio";
|
||||
|
||||
export function getTimelineAssetKind(assetPath: string): TimelineAssetKind | null {
|
||||
if (IMAGE_EXT.test(assetPath)) return "image";
|
||||
if (VIDEO_EXT.test(assetPath)) return "video";
|
||||
if (AUDIO_EXT.test(assetPath)) return "audio";
|
||||
return null;
|
||||
}
|
||||
|
||||
export function buildTimelineAssetId(assetPath: string, existingIds: Iterable<string>): string {
|
||||
const baseName = assetPath.split("/").pop() ?? "asset";
|
||||
const normalized = baseName
|
||||
.replace(/\.[^.]+$/, "")
|
||||
.replace(/[^a-zA-Z0-9_-]+/g, "_")
|
||||
.replace(/^_+|_+$/g, "")
|
||||
.toLowerCase();
|
||||
const baseId = normalized || "asset";
|
||||
const ids = new Set(existingIds);
|
||||
if (!ids.has(baseId)) return baseId;
|
||||
let suffix = 2;
|
||||
while (ids.has(`${baseId}_${suffix}`)) suffix += 1;
|
||||
return `${baseId}_${suffix}`;
|
||||
}
|
||||
|
||||
export function resolveTimelineAssetSrc(targetPath: string, assetPath: string): string {
|
||||
const targetDir = targetPath.includes("/")
|
||||
? targetPath.slice(0, targetPath.lastIndexOf("/"))
|
||||
: "";
|
||||
if (!targetDir) return assetPath;
|
||||
|
||||
const fromParts = targetDir.split("/").filter(Boolean);
|
||||
const toParts = assetPath.split("/").filter(Boolean);
|
||||
while (fromParts.length > 0 && toParts.length > 0 && fromParts[0] === toParts[0]) {
|
||||
fromParts.shift();
|
||||
toParts.shift();
|
||||
}
|
||||
|
||||
const up = fromParts.map(() => "..");
|
||||
const relative = [...up, ...toParts].join("/");
|
||||
return relative || assetPath.split("/").pop() || assetPath;
|
||||
}
|
||||
|
||||
export function buildTimelineFileDropPlacements(
|
||||
placement: { start: number; track: number },
|
||||
count: number,
|
||||
): Array<{ start: number; track: number }> {
|
||||
return Array.from({ length: Math.max(0, count) }, (_, index) => ({
|
||||
start: placement.start,
|
||||
track: placement.track + index,
|
||||
}));
|
||||
}
|
||||
|
||||
export function buildTimelineAssetInsertHtml(input: {
|
||||
id: string;
|
||||
assetPath: string;
|
||||
kind: TimelineAssetKind;
|
||||
start: number;
|
||||
duration: number;
|
||||
track: number;
|
||||
zIndex: number;
|
||||
}): string {
|
||||
const sharedAttrs = `id="${input.id}" class="clip" src="${input.assetPath}" data-start="${input.start}" data-duration="${input.duration}" data-track-index="${input.track}"`;
|
||||
|
||||
if (input.kind === "image") {
|
||||
return `<img ${sharedAttrs} style="position: absolute; inset: 0; width: 100%; height: 100%; object-fit: contain; z-index: ${input.zIndex}" />`;
|
||||
}
|
||||
|
||||
if (input.kind === "video") {
|
||||
return `<video ${sharedAttrs} muted playsinline style="position: absolute; inset: 0; width: 100%; height: 100%; object-fit: contain; z-index: ${input.zIndex}"></video>`;
|
||||
}
|
||||
|
||||
return `<audio ${sharedAttrs} style="z-index: ${input.zIndex}"></audio>`;
|
||||
}
|
||||
|
||||
export function insertTimelineAssetIntoSource(source: string, assetHtml: string): string {
|
||||
const rootOpenTag = /<[^>]*data-composition-id="[^"]+"[^>]*>/i;
|
||||
const match = rootOpenTag.exec(source);
|
||||
if (!match || match.index == null) {
|
||||
throw new Error("No composition root found in target source");
|
||||
}
|
||||
const insertAt = match.index + match[0].length;
|
||||
return `${source.slice(0, insertAt)}${assetHtml}${source.slice(insertAt)}`;
|
||||
}
|
||||
Reference in New Issue
Block a user