feat(studio): drag assets from the sidebar onto the timeline (#464)

## Problem

Studio still broke down in three concrete authoring flows around timeline assets:

- you could import media into Assets, but not drag an already-imported asset from the Assets tab onto the timeline and persist it into source
- dragging a file from outside the app onto the timeline only uploaded it into Assets instead of placing it at the dropped time/track
- once a clip was on the timeline, there was no reliable keyboard delete flow for removing it safely from source

While implementing direct external drops, another real bug showed up:

- valid binary uploads like `raycast.mp4` from `Downloads` were being rejected as unsupported media in Studio dev because the Vite API bridge was corrupting multipart request bodies before they reached the upload route

## What this fixes

### Timeline asset placement from inside Studio

- asset cards in the Assets tab are draggable
- the timeline accepts asset drops even when it already has clips
- dropping an asset onto the timeline inserts a new clip into the active composition source at the dropped time / track
- asset paths are rewritten relative to the target composition file so drops into sub-compositions resolve correctly
- the new clip is persisted immediately and the preview refreshes

### Direct external file drops onto the timeline

- dropping a file from outside the app onto the timeline now uploads it and places it onto the dropped track/time in one shot
- it no longer stops halfway by only adding the file into Assets
- multiple dropped files are placed using the same drop start and successive tracks

### Delete key support

- selected timeline clips can now be deleted with `Delete` / `Backspace`
- deletion is persisted back to source, not just removed from local state
- the delete path now uses a server-side DOM mutation helper with LinkeDOM for structural safety instead of client-side string surgery

### Binary upload fix for media files

- the Studio Vite API bridge now forwards non-GET request bodies as raw bytes instead of decoding them as UTF-8 text
- that preserves multipart uploads for binary media like MP4s
- valid local videos from `Downloads` no longer get rejected as `Unsupported media skipped` just because the dev bridge corrupted the request body
- upload validation now probes buffered media through a temp file path that preserves the file extension before saving into the project

## Root cause

There were really two separate gaps:

### 1. Asset placement / deletion workflow gaps

The timeline and asset systems already existed, but they were disconnected:

- `AssetsTab` only supported copy/import flows
- `Timeline` only handled raw file import, not positioned placement for existing assets
- there was no utility layer for converting a dropped asset into persisted timeline HTML
- there was no structurally safe deletion path for arbitrary selected timeline clips

### 2. Binary upload corruption in Studio dev

The Studio Vite API bridge rebuilt non-GET request bodies like this:

- read each request chunk
- call `chunk.toString()`
- concatenate into a string
- construct the Fetch `Request` from that string body

That works for text, but it corrupts multipart binary uploads. By the time the upload route wrote the received file and ran `ffprobe`, otherwise valid MP4s had already been mangled in-flight.

## Behavior

- dropping on `index.html` inserts the asset into the root composition
- dropping while drilled into a composition inserts into that composition file instead
- drop X position maps to `data-start`
- drop Y position maps to the current visible track row, with a new bottom track created if the drop lands below existing rows
- images default to a short finite duration
- audio/video default to their metadata duration when available, with a fallback duration if metadata cannot be read quickly
- pressing `Delete` on a selected clip removes that clip from the underlying HTML source and clears selection in Studio
- valid uploaded MP4s now survive the Studio dev API bridge intact instead of being rejected during upload validation

## Verification

### Local checks

- `bunx oxlint packages/core/src/studio-api/helpers/sourceMutation.ts packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/core/src/studio-api/routes/files.ts packages/studio/src/App.tsx packages/studio/src/components/nle/NLELayout.tsx packages/studio/src/components/sidebar/AssetsTab.tsx packages/studio/src/player/components/Timeline.tsx packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.config.ts packages/studio/vite.request-body.ts packages/studio/vite.request-body.test.ts`
- `bunx oxfmt --check` on the touched files
- `bun run --filter @hyperframes/core typecheck`
- `bun run --filter @hyperframes/studio typecheck`
- `bun test packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.request-body.test.ts`

### Browser / live verification

Verified against a live local Studio fixture:

- dragging an existing asset from the Assets tab onto the timeline creates a persisted clip at the dropped position
- dropping a file from outside the app directly onto the timeline uploads it and creates a persisted clip at the dropped position
- selecting a dropped clip and pressing `Delete` removes it from both the live timeline and the saved source HTML
- valid MP4 uploads like `raycast.mp4` now succeed through the live Studio upload route instead of being rejected as unsupported media

## Notes

- the local `timeline-trio-verify` and `timeline-overlap-debug` projects used for verification are local-only and are not part of this PR
- this PR is about asset placement, upload correctness, and deletion safety; it does not broaden into richer editing workflows beyond placing/removing clips from the timeline
This commit is contained in:
Miguel Ángel
2026-04-24 02:50:36 +02:00
committed by GitHub
parent 078a8b349b
commit 970b446c49
17 changed files with 1257 additions and 76 deletions
+126
View File
@@ -0,0 +1,126 @@
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { initSandboxRuntimeModular } from "./init";
import type { RuntimeTimelineLike } from "./types";
function createMockTimeline(duration: number): RuntimeTimelineLike {
const state = { time: 0, paused: true };
return {
play: () => {
state.paused = false;
},
pause: () => {
state.paused = true;
},
seek: (time: number) => {
state.time = time;
},
totalTime: (time: number) => {
state.time = time;
},
time: () => state.time,
duration: () => duration,
add: () => {},
paused: (value?: boolean) => {
if (typeof value === "boolean") {
state.paused = value;
}
return state.paused;
},
timeScale: () => {},
set: () => {},
getChildren: () => [],
};
}
describe("initSandboxRuntimeModular", () => {
const originalRequestAnimationFrame = window.requestAnimationFrame;
const originalCancelAnimationFrame = window.cancelAnimationFrame;
beforeEach(() => {
document.body.innerHTML = "";
(globalThis as typeof globalThis & { CSS?: { escape?: (value: string) => string } }).CSS ??= {};
globalThis.CSS.escape ??= (value: string) => value;
window.requestAnimationFrame = ((callback: FrameRequestCallback) => {
callback(0);
return 1;
}) as typeof window.requestAnimationFrame;
window.cancelAnimationFrame = (() => {}) as typeof window.cancelAnimationFrame;
});
afterEach(() => {
(window as Window & { __hfRuntimeTeardown?: (() => void) | null }).__hfRuntimeTeardown?.();
document.body.innerHTML = "";
delete (window as Window & { __timelines?: Record<string, RuntimeTimelineLike> }).__timelines;
delete (window as Window & { __player?: unknown }).__player;
delete (window as Window & { __playerReady?: boolean }).__playerReady;
delete (window as Window & { __renderReady?: boolean }).__renderReady;
window.requestAnimationFrame = originalRequestAnimationFrame;
window.cancelAnimationFrame = originalCancelAnimationFrame;
});
it("uses the shorter live child timeline when the authored window is longer", () => {
const root = document.createElement("div");
root.setAttribute("data-composition-id", "main");
root.setAttribute("data-root", "true");
root.setAttribute("data-width", "1920");
root.setAttribute("data-height", "1080");
document.body.appendChild(root);
const child = document.createElement("div");
child.setAttribute("data-composition-id", "slide-1");
child.setAttribute("data-start", "0");
child.setAttribute("data-hf-authored-duration", "14");
root.appendChild(child);
(window as Window & { __timelines?: Record<string, RuntimeTimelineLike> }).__timelines = {
main: createMockTimeline(20),
"slide-1": createMockTimeline(8),
};
initSandboxRuntimeModular();
const player = (
window as Window & {
__player?: { renderSeek: (timeSeconds: number) => void };
}
).__player;
expect(player).toBeDefined();
player?.renderSeek(9);
expect(child.style.visibility).toBe("hidden");
});
it("uses the shorter authored host window when the child timeline is longer", () => {
const root = document.createElement("div");
root.setAttribute("data-composition-id", "main");
root.setAttribute("data-root", "true");
root.setAttribute("data-width", "1920");
root.setAttribute("data-height", "1080");
document.body.appendChild(root);
const child = document.createElement("div");
child.setAttribute("data-composition-id", "slide-1");
child.setAttribute("data-start", "0");
child.setAttribute("data-hf-authored-duration", "2");
root.appendChild(child);
(window as Window & { __timelines?: Record<string, RuntimeTimelineLike> }).__timelines = {
main: createMockTimeline(20),
"slide-1": createMockTimeline(8),
};
initSandboxRuntimeModular();
const player = (
window as Window & {
__player?: { renderSeek: (timeSeconds: number) => void };
}
).__player;
expect(player).toBeDefined();
player?.renderSeek(3);
expect(child.style.visibility).toBe("hidden");
});
});
+22 -9
View File
@@ -368,13 +368,16 @@ export function initSandboxRuntimeModular(): void {
return resolver.resolveStartForElement(element, fallback);
};
const resolveDurationForElement = (element: Element): number | null => {
const resolveDurationForElement = (
element: Element,
opts?: { includeAuthoredTimingAttrs?: boolean },
): number | null => {
const resolver = createRuntimeStartTimeResolver({
timelineRegistry: (window.__timelines ?? {}) as Record<
string,
RuntimeTimelineLike | undefined
>,
includeAuthoredTimingAttrs: true,
includeAuthoredTimingAttrs: opts?.includeAuthoredTimingAttrs ?? true,
});
return resolver.resolveDurationForElement(element);
};
@@ -1234,18 +1237,28 @@ export function initSandboxRuntimeModular(): void {
}
const start = resolveStartForElement(rawNode, 0);
const duration = resolveDurationForElement(rawNode);
const end = duration != null && duration > 0 ? start + duration : Number.POSITIVE_INFINITY;
// For composition hosts, use the composition timeline's duration to compute end
let computedEnd = end;
let duration = resolveDurationForElement(rawNode);
const compId = rawNode.getAttribute("data-composition-id");
if (compId && !Number.isFinite(end)) {
if (compId) {
const compTimeline = (window.__timelines ?? {})[compId];
let liveDuration: number | null = null;
if (compTimeline && typeof compTimeline.duration === "function") {
const compDur = compTimeline.duration();
if (compDur > 0) computedEnd = start + compDur;
const compDur = Number(compTimeline.duration());
if (Number.isFinite(compDur) && compDur > 0) {
liveDuration = compDur;
}
}
// Composition hosts must respect both the authored clip window in the parent
// composition and the child composition's own live timeline duration.
if (duration != null && duration > 0 && liveDuration != null) {
duration = Math.min(duration, liveDuration);
} else if ((duration == null || duration <= 0) && liveDuration != null) {
duration = liveDuration;
}
}
const computedEnd =
duration != null && duration > 0 ? start + duration : Number.POSITIVE_INFINITY;
const isVisibleNow =
state.currentTime >= start &&
(Number.isFinite(computedEnd) ? state.currentTime < computedEnd : true);
@@ -0,0 +1,71 @@
import { describe, expect, it } from "vitest";
import { validateUploadedMedia, validateUploadedMediaBuffer } from "./mediaValidation.js";
describe("validateUploadedMedia", () => {
it("passes through non-media files", () => {
expect(
validateUploadedMedia("/tmp/test.svg", () => ({ status: 0, stdout: "", stderr: "" })),
).toEqual({
ok: true,
});
});
it("accepts video files with a video stream", () => {
expect(
validateUploadedMedia("/tmp/test.mp4", () => ({
status: 0,
stdout: JSON.stringify({ streams: [{ codec_type: "video" }] }),
stderr: "",
})),
).toEqual({ ok: true });
});
it("rejects video files with no supported video stream", () => {
expect(
validateUploadedMedia("/tmp/test.mp4", () => ({
status: 0,
stdout: JSON.stringify({ streams: [] }),
stderr: "",
})),
).toEqual({ ok: false, reason: "no supported video stream found" });
});
it("accepts audio files with an audio stream", () => {
expect(
validateUploadedMedia("/tmp/test.wav", () => ({
status: 0,
stdout: JSON.stringify({ streams: [{ codec_type: "audio" }] }),
stderr: "",
})),
).toEqual({ ok: true });
});
it("does not block upload when ffprobe is unavailable", () => {
expect(
validateUploadedMedia("/tmp/test.mp4", () => ({
status: null,
stdout: "",
stderr: "",
error: { code: "ENOENT" } as NodeJS.ErrnoException,
})),
).toEqual({ ok: true });
});
});
describe("validateUploadedMediaBuffer", () => {
it("validates media from a temp file that preserves the extension", () => {
let inspectedPath = "";
expect(
validateUploadedMediaBuffer("raycast.mp4", new Uint8Array([0, 1, 2]), (_command, args) => {
inspectedPath = args.at(-1) ?? "";
return {
status: 0,
stdout: JSON.stringify({ streams: [{ codec_type: "video" }] }),
stderr: "",
};
}),
).toEqual({ ok: true });
expect(inspectedPath).toMatch(/raycast\.mp4$/);
});
});
@@ -0,0 +1,80 @@
import { spawnSync } from "node:child_process";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { basename, join } from "node:path";
const VIDEO_EXT = /\.(mp4|webm|mov)$/i;
const AUDIO_EXT = /\.(mp3|wav|ogg|m4a|aac)$/i;
type FfprobeRunner = (
command: string,
args: string[],
) => {
status: number | null;
stdout: string | Buffer;
stderr: string | Buffer;
error?: NodeJS.ErrnoException;
};
export function validateUploadedMedia(
filePath: string,
runner: FfprobeRunner = spawnSync as unknown as FfprobeRunner,
): { ok: true } | { ok: false; reason: string } {
const isVideo = VIDEO_EXT.test(filePath);
const isAudio = AUDIO_EXT.test(filePath);
if (!isVideo && !isAudio) {
return { ok: true };
}
const result = runner("ffprobe", [
"-v",
"error",
"-show_entries",
"stream=codec_type",
"-of",
"json",
filePath,
]);
if (result.error?.code === "ENOENT") {
return { ok: true };
}
if (result.status !== 0) {
return { ok: false, reason: "ffprobe failed to read the media file" };
}
try {
const parsed = JSON.parse(String(result.stdout || "{}")) as {
streams?: Array<{ codec_type?: string }>;
};
const streams = parsed.streams ?? [];
const hasVideo = streams.some((stream) => stream.codec_type === "video");
const hasAudio = streams.some((stream) => stream.codec_type === "audio");
if (isVideo && !hasVideo) {
return { ok: false, reason: "no supported video stream found" };
}
if (isAudio && !hasAudio) {
return { ok: false, reason: "no supported audio stream found" };
}
return { ok: true };
} catch {
return { ok: false, reason: "ffprobe returned unreadable media metadata" };
}
}
export function validateUploadedMediaBuffer(
fileName: string,
buffer: Uint8Array,
runner: FfprobeRunner = spawnSync as unknown as FfprobeRunner,
): { ok: true } | { ok: false; reason: string } {
const tempDir = mkdtempSync(join(tmpdir(), "hyperframes-upload-"));
const tempPath = join(tempDir, basename(fileName));
try {
writeFileSync(tempPath, buffer);
return validateUploadedMedia(tempPath, runner);
} finally {
rmSync(tempDir, { recursive: true, force: true });
}
}
@@ -0,0 +1,30 @@
import { describe, expect, it } from "vitest";
import { removeElementFromHtml } from "./sourceMutation.js";
describe("removeElementFromHtml", () => {
it("removes a self-closing element by id", () => {
const html = `<!doctype html><html><body><div data-composition-id="main"><img id="photo" src="asset.png" /><div id="rest"></div></div></body></html>`;
const updated = removeElementFromHtml(html, { id: "photo" });
expect(updated).not.toContain(`id="photo"`);
expect(updated).toContain(`id="rest"`);
});
it("removes a matched composition host by selector", () => {
const html = `<!doctype html><html><body><div data-composition-id="main"><div data-composition-id="scene-a"><span>Scene A</span></div><div data-composition-id="scene-b"></div></div></body></html>`;
const updated = removeElementFromHtml(html, {
selector: '[data-composition-id="scene-a"]',
});
expect(updated).not.toContain(`data-composition-id="scene-a"`);
expect(updated).toContain(`data-composition-id="scene-b"`);
});
it("supports fragment html by returning updated body markup", () => {
const html = `<div id="photo"></div><div id="rest"></div>`;
expect(removeElementFromHtml(html, { id: "photo" })).toBe(`<div id="rest"></div>`);
});
});
@@ -0,0 +1,42 @@
import { parseHTML } from "linkedom";
export interface SourceMutationTarget {
id?: string | null;
selector?: string;
selectorIndex?: number;
}
function parseSourceDocument(source: string): { document: Document; wrappedFragment: boolean } {
const hasDocumentShell = /<!doctype|<html[\s>]/i.test(source);
if (hasDocumentShell) {
return { document: parseHTML(source).document, wrappedFragment: false };
}
return {
document: parseHTML(`<!DOCTYPE html><html><head></head><body>${source}</body></html>`).document,
wrappedFragment: true,
};
}
function findTargetElement(document: Document, target: SourceMutationTarget): Element | null {
if (target.id) {
const byId = document.getElementById(target.id);
if (byId) return byId;
}
if (!target.selector) return null;
try {
const matches = Array.from(document.querySelectorAll(target.selector));
return matches[target.selectorIndex ?? 0] ?? null;
} catch {
return null;
}
}
export function removeElementFromHtml(source: string, target: SourceMutationTarget): string {
const { document, wrappedFragment } = parseSourceDocument(source);
const element = findTargetElement(document, target);
if (!element) return source;
element.remove();
return wrappedFragment ? document.body.innerHTML || "" : document.toString();
}
+46 -1
View File
@@ -13,7 +13,9 @@ import {
} from "node:fs";
import { resolve, dirname, join } from "node:path";
import type { StudioApiAdapter } from "../types.js";
import { validateUploadedMediaBuffer } from "../helpers/mediaValidation.js";
import { isSafePath } from "../helpers/safePath.js";
import { removeElementFromHtml } from "../helpers/sourceMutation.js";
// ── Shared helpers ──────────────────────────────────────────────────────────
@@ -184,6 +186,43 @@ export function registerFileRoutes(api: Hono, adapter: StudioApiAdapter): void {
return c.json({ ok: true });
});
api.post("/projects/:id/file-mutations/remove-element/*", async (c) => {
const id = c.req.param("id");
const project = await adapter.resolveProject(id);
if (!project) return c.json({ error: "not found" }, 404);
const filePath = decodeURIComponent(
c.req.path.replace(`/projects/${project.id}/file-mutations/remove-element/`, ""),
);
if (filePath.includes("\0")) {
return c.json({ error: "forbidden" }, 403);
}
const absPath = resolve(project.dir, filePath);
if (!isSafePath(project.dir, absPath)) {
return c.json({ error: "forbidden" }, 403);
}
if (!existsSync(absPath)) {
return c.json({ error: "not found" }, 404);
}
const body = (await c.req.json().catch(() => null)) as {
target?: { id?: string | null; selector?: string; selectorIndex?: number };
} | null;
if (!body?.target) {
return c.json({ error: "target required" }, 400);
}
const originalContent = readFileSync(absPath, "utf-8");
const patchedContent = removeElementFromHtml(originalContent, body.target);
if (patchedContent === originalContent) {
return c.json({ ok: true, changed: false, content: originalContent });
}
writeFileSync(absPath, patchedContent, "utf-8");
return c.json({ ok: true, changed: true, content: patchedContent });
});
// ── Rename / Move ──
api.patch("/projects/:id/files/*", async (c) => {
@@ -263,6 +302,7 @@ export function registerFileRoutes(api: Hono, adapter: StudioApiAdapter): void {
const formData = await c.req.formData();
const uploaded: string[] = [];
const skipped: string[] = [];
const invalid: Array<{ name: string; reason: string }> = [];
for (const [, value] of formData.entries()) {
if (!(value instanceof File)) continue;
@@ -299,11 +339,16 @@ export function registerFileRoutes(api: Hono, adapter: StudioApiAdapter): void {
}
const buffer = Buffer.from(await value.arrayBuffer());
const validation = validateUploadedMediaBuffer(finalName, buffer);
if (!validation.ok) {
invalid.push({ name: finalName, reason: validation.reason });
continue;
}
writeFileSync(finalPath, buffer);
uploaded.push(subDir ? join(subDir, finalName) : finalName);
}
return c.json({ ok: true, files: uploaded, skipped }, 201);
return c.json({ ok: true, files: uploaded, skipped, invalid }, 201);
},
);
}