mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-04 07:19:52 +00:00
feat(studio): drag assets from the sidebar onto the timeline (#464)
## Problem Studio still broke down in three concrete authoring flows around timeline assets: - you could import media into Assets, but not drag an already-imported asset from the Assets tab onto the timeline and persist it into source - dragging a file from outside the app onto the timeline only uploaded it into Assets instead of placing it at the dropped time/track - once a clip was on the timeline, there was no reliable keyboard delete flow for removing it safely from source While implementing direct external drops, another real bug showed up: - valid binary uploads like `raycast.mp4` from `Downloads` were being rejected as unsupported media in Studio dev because the Vite API bridge was corrupting multipart request bodies before they reached the upload route ## What this fixes ### Timeline asset placement from inside Studio - asset cards in the Assets tab are draggable - the timeline accepts asset drops even when it already has clips - dropping an asset onto the timeline inserts a new clip into the active composition source at the dropped time / track - asset paths are rewritten relative to the target composition file so drops into sub-compositions resolve correctly - the new clip is persisted immediately and the preview refreshes ### Direct external file drops onto the timeline - dropping a file from outside the app onto the timeline now uploads it and places it onto the dropped track/time in one shot - it no longer stops halfway by only adding the file into Assets - multiple dropped files are placed using the same drop start and successive tracks ### Delete key support - selected timeline clips can now be deleted with `Delete` / `Backspace` - deletion is persisted back to source, not just removed from local state - the delete path now uses a server-side DOM mutation helper with LinkeDOM for structural safety instead of client-side string surgery ### Binary upload fix for media files - the Studio Vite API bridge now forwards non-GET request bodies as raw bytes instead of decoding them as UTF-8 text - that preserves multipart uploads for binary media like MP4s - valid local videos from `Downloads` no longer get rejected as `Unsupported media skipped` just because the dev bridge corrupted the request body - upload validation now probes buffered media through a temp file path that preserves the file extension before saving into the project ## Root cause There were really two separate gaps: ### 1. Asset placement / deletion workflow gaps The timeline and asset systems already existed, but they were disconnected: - `AssetsTab` only supported copy/import flows - `Timeline` only handled raw file import, not positioned placement for existing assets - there was no utility layer for converting a dropped asset into persisted timeline HTML - there was no structurally safe deletion path for arbitrary selected timeline clips ### 2. Binary upload corruption in Studio dev The Studio Vite API bridge rebuilt non-GET request bodies like this: - read each request chunk - call `chunk.toString()` - concatenate into a string - construct the Fetch `Request` from that string body That works for text, but it corrupts multipart binary uploads. By the time the upload route wrote the received file and ran `ffprobe`, otherwise valid MP4s had already been mangled in-flight. ## Behavior - dropping on `index.html` inserts the asset into the root composition - dropping while drilled into a composition inserts into that composition file instead - drop X position maps to `data-start` - drop Y position maps to the current visible track row, with a new bottom track created if the drop lands below existing rows - images default to a short finite duration - audio/video default to their metadata duration when available, with a fallback duration if metadata cannot be read quickly - pressing `Delete` on a selected clip removes that clip from the underlying HTML source and clears selection in Studio - valid uploaded MP4s now survive the Studio dev API bridge intact instead of being rejected during upload validation ## Verification ### Local checks - `bunx oxlint packages/core/src/studio-api/helpers/sourceMutation.ts packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/core/src/studio-api/routes/files.ts packages/studio/src/App.tsx packages/studio/src/components/nle/NLELayout.tsx packages/studio/src/components/sidebar/AssetsTab.tsx packages/studio/src/player/components/Timeline.tsx packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.config.ts packages/studio/vite.request-body.ts packages/studio/vite.request-body.test.ts` - `bunx oxfmt --check` on the touched files - `bun run --filter @hyperframes/core typecheck` - `bun run --filter @hyperframes/studio typecheck` - `bun test packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.request-body.test.ts` ### Browser / live verification Verified against a live local Studio fixture: - dragging an existing asset from the Assets tab onto the timeline creates a persisted clip at the dropped position - dropping a file from outside the app directly onto the timeline uploads it and creates a persisted clip at the dropped position - selecting a dropped clip and pressing `Delete` removes it from both the live timeline and the saved source HTML - valid MP4 uploads like `raycast.mp4` now succeed through the live Studio upload route instead of being rejected as unsupported media ## Notes - the local `timeline-trio-verify` and `timeline-overlap-debug` projects used for verification are local-only and are not part of this PR - this PR is about asset placement, upload correctness, and deletion safety; it does not broaden into richer editing workflows beyond placing/removing clips from the timeline
This commit is contained in:
Miguel Ángel
@@ -0,0 +1,126 @@
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import { initSandboxRuntimeModular } from "./init";
|
||||
import type { RuntimeTimelineLike } from "./types";
|
||||
|
||||
function createMockTimeline(duration: number): RuntimeTimelineLike {
|
||||
const state = { time: 0, paused: true };
|
||||
return {
|
||||
play: () => {
|
||||
state.paused = false;
|
||||
},
|
||||
pause: () => {
|
||||
state.paused = true;
|
||||
},
|
||||
seek: (time: number) => {
|
||||
state.time = time;
|
||||
},
|
||||
totalTime: (time: number) => {
|
||||
state.time = time;
|
||||
},
|
||||
time: () => state.time,
|
||||
duration: () => duration,
|
||||
add: () => {},
|
||||
paused: (value?: boolean) => {
|
||||
if (typeof value === "boolean") {
|
||||
state.paused = value;
|
||||
}
|
||||
return state.paused;
|
||||
},
|
||||
timeScale: () => {},
|
||||
set: () => {},
|
||||
getChildren: () => [],
|
||||
};
|
||||
}
|
||||
|
||||
describe("initSandboxRuntimeModular", () => {
|
||||
const originalRequestAnimationFrame = window.requestAnimationFrame;
|
||||
const originalCancelAnimationFrame = window.cancelAnimationFrame;
|
||||
|
||||
beforeEach(() => {
|
||||
document.body.innerHTML = "";
|
||||
(globalThis as typeof globalThis & { CSS?: { escape?: (value: string) => string } }).CSS ??= {};
|
||||
globalThis.CSS.escape ??= (value: string) => value;
|
||||
window.requestAnimationFrame = ((callback: FrameRequestCallback) => {
|
||||
callback(0);
|
||||
return 1;
|
||||
}) as typeof window.requestAnimationFrame;
|
||||
window.cancelAnimationFrame = (() => {}) as typeof window.cancelAnimationFrame;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
(window as Window & { __hfRuntimeTeardown?: (() => void) | null }).__hfRuntimeTeardown?.();
|
||||
document.body.innerHTML = "";
|
||||
delete (window as Window & { __timelines?: Record<string, RuntimeTimelineLike> }).__timelines;
|
||||
delete (window as Window & { __player?: unknown }).__player;
|
||||
delete (window as Window & { __playerReady?: boolean }).__playerReady;
|
||||
delete (window as Window & { __renderReady?: boolean }).__renderReady;
|
||||
window.requestAnimationFrame = originalRequestAnimationFrame;
|
||||
window.cancelAnimationFrame = originalCancelAnimationFrame;
|
||||
});
|
||||
|
||||
it("uses the shorter live child timeline when the authored window is longer", () => {
|
||||
const root = document.createElement("div");
|
||||
root.setAttribute("data-composition-id", "main");
|
||||
root.setAttribute("data-root", "true");
|
||||
root.setAttribute("data-width", "1920");
|
||||
root.setAttribute("data-height", "1080");
|
||||
document.body.appendChild(root);
|
||||
|
||||
const child = document.createElement("div");
|
||||
child.setAttribute("data-composition-id", "slide-1");
|
||||
child.setAttribute("data-start", "0");
|
||||
child.setAttribute("data-hf-authored-duration", "14");
|
||||
root.appendChild(child);
|
||||
|
||||
(window as Window & { __timelines?: Record<string, RuntimeTimelineLike> }).__timelines = {
|
||||
main: createMockTimeline(20),
|
||||
"slide-1": createMockTimeline(8),
|
||||
};
|
||||
|
||||
initSandboxRuntimeModular();
|
||||
|
||||
const player = (
|
||||
window as Window & {
|
||||
__player?: { renderSeek: (timeSeconds: number) => void };
|
||||
}
|
||||
).__player;
|
||||
expect(player).toBeDefined();
|
||||
|
||||
player?.renderSeek(9);
|
||||
|
||||
expect(child.style.visibility).toBe("hidden");
|
||||
});
|
||||
|
||||
it("uses the shorter authored host window when the child timeline is longer", () => {
|
||||
const root = document.createElement("div");
|
||||
root.setAttribute("data-composition-id", "main");
|
||||
root.setAttribute("data-root", "true");
|
||||
root.setAttribute("data-width", "1920");
|
||||
root.setAttribute("data-height", "1080");
|
||||
document.body.appendChild(root);
|
||||
|
||||
const child = document.createElement("div");
|
||||
child.setAttribute("data-composition-id", "slide-1");
|
||||
child.setAttribute("data-start", "0");
|
||||
child.setAttribute("data-hf-authored-duration", "2");
|
||||
root.appendChild(child);
|
||||
|
||||
(window as Window & { __timelines?: Record<string, RuntimeTimelineLike> }).__timelines = {
|
||||
main: createMockTimeline(20),
|
||||
"slide-1": createMockTimeline(8),
|
||||
};
|
||||
|
||||
initSandboxRuntimeModular();
|
||||
|
||||
const player = (
|
||||
window as Window & {
|
||||
__player?: { renderSeek: (timeSeconds: number) => void };
|
||||
}
|
||||
).__player;
|
||||
expect(player).toBeDefined();
|
||||
|
||||
player?.renderSeek(3);
|
||||
|
||||
expect(child.style.visibility).toBe("hidden");
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user