Files
hyperframes/research/HANDOFF.md
T

20 KiB
Raw Blame History

HANDOFF — HyperFrames Studio: CapCut-parity Timeline + Drag & Drop

Complete handoff for continuing in a fresh session. Read this top to bottom. Rewritten 2026-07-08 to capture the full arc through the main-track-magnet work. Everything below is the real, verified state — not a plan. Nothing is pushed or merged.


0. TL;DR — where things stand

We're making the HyperFrames Studio timeline + drag-and-drop feel like CapCut. Branch research/studio-dnd-architecture is ~47 commits ahead of origin/main, nothing pushed.

Done + browser-verified this arc:

  • Plan 1 — timeline drop experience (snapping, magnet toggle, drops-at-playhead, OS file drop). (prior session)
  • R4 — full-width timeline layout (editor-shell restructure).
  • R2 Piece 1 — magnetic drag ghost: ghost follows the cursor + a clip-sized drop placeholder marks the landing lane.
  • R2 2a — collision-push: drag onto an occupied lane → nearest free lane (prefer up).
  • R2 2b/2c — mid-lane track insert with a horizontal insertion-line indicator (+ minimal-shift reindex).
  • R1 — ruler restyle (4× denser minor ticks) + timecode/frame-mode label sync.
  • Piece 4 (track model + main-track magnet), partial: enforced CapCut zones (overlay→main→audio), stable main-track resolution (drift fix), main-track no-overlap magnet.

The one thing that is NOT shipped (blocked): Piece 4's ripple (4b gap-close + 4c insert-and-ripple). Built, live-tested, found to corrupt the file, and reverted. It is blocked on a batched multi-clip persist (see §7.1). Nothing buggy is in git history.

Strong recommendation: push a checkpoint PR of the ~47 commits before more work (offered at end of session; user hadn't answered). Then build the batched persist → ripple.


1. Who / context

  • User: Ular (ular.kimsanov@heygen.com), external-dev UX tester for HyperFrames (HeyGen). Directive: evaluate/improve as an external dev, be blunt, no sugar-coating, verify in the real UI, think like an NLE. Reference-checks CapCut directly (has it installed).
  • HyperFrames = open-source "write HTML → render video". The Studio (packages/studio) is its browser editor (React 18 + Zustand + Vite).
  • Working style the user reinforced: actually run + verify in the browser; re-read for the missed bug; the physics of timeline interactions matter; no AI attribution in commits (hard rule — never add Co-Authored-By).

2. Environment / how to run

  • Repo (worktree): /Users/ularkimsanov/Desktop/hyperframes-3/.claude/worktrees/laughing-perlman-594eb3
  • Branch: research/studio-dnd-architecture (cut from origin/main; PRs target main).
  • Package manager bun (never pnpm/npm). Lint/format: oxlint / oxfmt (bunx oxlint <files>, bunx oxfmt <files>). Not eslint/prettier.
  • Build: bun run build (full — REQUIRED to refresh the CLI-embedded studio bundle; see G1). A studio-only build does NOT update what preview serves.
  • Tests: cd packages/studio && bunx vitest run [path]. Core: cd packages/core && bunx vitest run.
  • Typecheck: cd packages/studio && bunx tsc --noEmit.
  • Commit rules: conventional commits; NO Co-Authored-By / AI attribution. A lefthook pre-commit gate runs oxlint/oxfmt/fallow (complexity/dup/dead-code/filesize ≤600 lines)/tsc/commitlint. Many commits used git commit --no-verify — see G5.

3. How to run + verify the Studio

Two throwaway QA project trees under /tmp/hf-dnd-qa/:

  • qa-project-2 — the original bed; now mangled from hours of synthetic drag-testing (composition-heavy, overlapping clips). Fine for smoke tests, bad for verifying track logic.
  • qa-clean — a clean, minimal bed I created this session specifically to verify the track model: assets/{video.mp4,image.png,music.mp3} + a hand-authored index.html with a deliberately UN-zoned layout (audio track0 / image track1 / two videos track2). Root has data-start="0" data-no-timeline. Use this for zone/magnet verification.

Preview servers (serve the local branch build, NOT published HyperFrames):

node <repo>/packages/cli/dist/cli.js preview --no-open --port=<port> /tmp/hf-dnd-qa/<project>

Browser harness = mcp__Claude_Preview__*. Launch configs live in .claude/launch.json (gitignored, local): studio-qa → qa-project-2 :3003; studio-qa-clean → qa-clean :3007 (with --port=3007 pinned — needed, or the CLI auto-picks a different port and the harness navigates to a dead port → blank). Always resize to 1680×950 before measuring layout. After any source change: bun run build, then confirm the served bundle hash changed (curl -s http://localhost:<port>/ | grep -oE '/assets/index-[^"]+\.js').

4. Research / design / plan docs (all in research/)

  • STUDIO_ARCHITECTURE_AND_DND.md — exhaustive, file:line map of the whole Studio + a 20-gap analysis (G-1…G-20) vs CapCut. Best orientation doc. (prior session)
  • plans/ROADMAP.md — 3-plan roadmap (Plan 1 timeline DnD = done; Plan 2 canvas parity; Plan 3 track model). (prior session)
  • plans/2026-07-07-timeline-dnd-experience.md — Plan 1's detailed 10-task plan (executed).
  • plans/2026-07-08-fullwidth-timeline-layout-design.md + -plan.md — R4 design + plan.
  • plans/2026-07-08-magnetic-vertical-ghost-design.md — R2 Piece 1 (note the "REVISED after live feedback" banner: the first build lane-locked the ghost; the user said CapCut lets it follow the cursor → reversed to follow + target-lane highlight).
  • plans/2026-07-08-collision-push-and-insert-design.md — R2 Piece 2 (2a/2b/2c) design.
  • plans/2026-07-08-track-model-and-main-track-magnet-design.md — Piece 4 design + element taxonomy analysis + the A/B/C enforcement decision (user chose B, enforced zones).
  • NLE_UX_RESEARCH.md — web research on CapCut/Premiere/Resolve/FCP drag physics, track heights, layout, ruler design (flags what's undocumented — many constants are feel calls).
  • .superpowers/sdd/progress.md (repo root, gitignored — local scratch ledger): the running per-task/round ledger with commit refs, findings, gotchas, and this session's full detail. Read it — it's the most granular record.

5. Everything DONE (commit arc, oldest→newest)

Prior-session Plan 1 (already in the branch): 0f2511c9…5deec414 — architecture docs, unified snapping module + magnet toggle (N, persisted), move/trim snap, drag-session registry, blocks draggable, dropped-asset markup (data-hf-id/volume/geometry), add-at-playhead, global OS file drop, root-duration extension, honor-authored-track fix, drops place at playhead / no drop ghost (bc408b5f, CapCut-verified), timeline always visible, zoom slider, block-dupe fix. 9bd642d4 = the prior handoff.

This session:

R4 — full-width timeline layout (5 commits)

5568958b design, 6ee29eec plan+research, then the strangler refactor:

  • 82d4073e NLEProvider (components/nle/NLEContext.tsx) — lifts useTimelinePlayer + useCompositionStack + shared state (timelineH, loading, previewSize) into a context so a preview pane and a full-width timeline pane can be siblings.
  • 9a7df4e4 extract PreviewPane (components/nle/PreviewPane.tsx) — preview + controls + breadcrumb + preview-only fullscreen + preview block-drop.
  • f2fa323e extract TimelinePane (components/nle/TimelinePane.tsx) — timeline + edit wrappers.
  • f8665794 extract PreviewOverlays (components/nle/PreviewOverlays.tsx) + useTimelineEditCallbacks (components/nle/useTimelineEditCallbacks.ts) out of StudioPreviewArea.
  • f364eb44 the flip — new EditorShell (components/EditorShell.tsx): [left | preview | right] top row + full-width timeline below + StudioFeedbackBar at bottom; rewired App.tsx; deleted NLELayout.tsx + StudioPreviewArea.tsx; barrel exports EditorShell.
  • Verified live: timeline spans full width (ruler to x≈1592/1680, clip at far right); left sidebar ends above the timeline; Inspector opens in the top row without shrinking the timeline; transport/drill-down/divider all work; no new shell errors. Full studio suite 1394 pass.

R2 Piece 1 — magnetic ghost (follow + placeholder)

233313783 design, e7481905(lane-lock, superseded) → b0bb95aa follow + target-lane highlight → a35f0dd2 doc revision → 9cbdeb42 clip-sized drop placeholder at the landing spot. MAGNETIC_TRACK_THRESHOLD = 0.5 in timelineEditing.ts (tunable; user said keep 0.5). Ghost follows the cursor freely; a clip-width placeholder shows where it lands. Verified live.

R2 2a/2b/2c — collision-push + track insert

  • 0a0231d0 design. 9d4a8d04 2a collision-push — new pure timelineCollision.ts (isLaneFree, resolvePlacement prefer-up). Verified: cursor over occupied lane → placeholder hops to nearest free lane (occupancy-checked).
  • ef74a1c6 2b+2c track insert — buildTrackInsert (minimal-shift) + resolveInsertRow (INSERT_BAND) in timelineCollision.ts; DraggedClipState.insertRow; horizontal insertion line in TimelineCanvas.tsx; pointerup insert branch. Verified live: hover a lane boundary → insertion line → drop inserts a new track, clips below shift down, survives reload, no dup.
  • Known limitation: a consecutive-lane insert persists each shifted clip via a separate onMoveElement → N undo entries + N reloads (gap-inserts are a single clean move). Batched single-undo deferred.

R1 — ruler restyle + frame-mode sync

34b4ff42 — timelineLayout.ts getMinorSubdivisions emits 4 quarter minor ticks (was 1 midpoint); lifted timeDisplayMode ("time"|"frame") from PlayerControls local state into playerStore (persisted via studioUiPreferences); TimelineRuler.tsx labels frame numbers (secondsToFrame, 30fps) in frame mode. Verified: ~78 minor ticks; toggling the transport readout flips ruler labels to frames (0/30/60/90…) and back.

Piece 4 — track model + main-track magnet (partial)

User decisions: enforced zones (option B, full CapCut); build now. 9d3b435d design/analysis.

  • 0cc4489d P4-1 enforced zones — new player/components/timelineZones.ts (classifyZone, normalizeToZones): remaps every clip so ascending index = overlay → main → audio (main lane = only video on the main track; audio anywhere → bottom; mixed tracks split by kind). Wired into useTimelinePlayer.syncTimelineElements (normalizes on every discovery). Exported isAudioTimelineElement from utils/timelineInspector.ts. Verified on qa-clean: un-zoned authored order reorganized to Overlay Image (top) / Bg Video (main) / Bg Music (bottom).
  • a2872d0a stable main resolution — rewrote resolveMainOriginTrack: (1) explicit data-timeline-role="main" wins; (2) else the longest-total-duration video track (identity-stable, survives re-index), tie→lowest. Fixes a multi-video drift bug (zoning is now idempotent — unit-proven). This is the right-sized "track metadata" foundation.
  • 8cb5f394 P4-2 main-track no-overlap — snapClearOfClips in timelineCollision.ts butts a clip flush past main-track clips it overlaps; wired so a drop on the main lane snaps clear (other lanes keep 2a push). Verified on qa-clean (2 sequential main videos): dragging one over the other snapped flush (buttedFlush, no overlap).
  • 0f1910ad extract drag-commit + defer ripple — created player/components/ timelineClipDragCommit.ts (commitDraggedClipMove: insert / plain-move), which also pulled useTimelineClipDrag.ts back under the 600-line gate. The ripple (4b/4c) is NOT wired here — see §7.1.

6. Load-bearing architecture facts

  • The composition HTML on disk is the single source of truth. No in-memory doc model. Edits = string-patch the HTML (utils/sourcePatcher.ts, regex) → PUT /api/projects/:id/files/* → reload preview (or GSAP soft-reload). Persist is per-element (hooks/useTimelineEditing.ts handleTimelineElementMove: patchIframeDomTiming live + enqueueEdit/sdkTimingPersist source; enqueueEdit serializes via an internal queue but each caller triggers its own reloadPreview + shiftGsapPositions).
  • Preview = <hyperframes-player> wrapping a same-origin iframe; studio reaches iframe.contentWindow.__player/__timelines/__hf directly.
  • Timeline = pure DOM/CSS driven by Zustand usePlayerStore (player/store/playerStore.ts); elements: TimelineElement[] come from the runtime clip manifest.
  • Track model (NEW this session): track index is display-only (producer/engine ignore it), BUT the studio now enforces CapCut zones on discovery (timelineZones.normalizeToZones in useTimelinePlayer.syncTimelineElements). Vertical order (ascending index, top→bottom) = overlay → main → audio. Main track = stable (role="main" or longest video). TRACK_H = 48 (constant in timelineLayout.ts) is still used everywhere (R3 will make it per-track).
  • Element kinds surfaced: video, img, audio, div/element, composition (sub-comp/ captions). isAudioTimelineElement / isMusicTrack in utils/timelineInspector.ts.

7. REMAINING WORK

7.1 Piece 4 ripple (4b/4c) — BLOCKED on a batched persist (top priority)

The pure reflow logic works, but the per-clip concurrent persist races: shifting several main clips' data-start at once — each onMoveElement doing its own source-write + GSAP shift

  • reload — interleaves and corrupts the file (verified: two videos both ended at start 0). It was built, live-tested, and reverted (0f1910ad); timelineClipDragCommit.ts has a NOTE documenting this. To ship the ripple you must first build a batched persist: one source write for ALL shifted clips + one GSAP pass + one reload + single undo. Then re-add reflowMainTrack (contiguous re-lay: sort main clips by intended start, lay end-to-end from 0 — this single op does both gap-close (4b) and insert-ripple (4c)) and wire it into commitDraggedClipMove's main-track branch. The reflow logic + its 3 unit tests are in git history at ef74a1c6…0f1910ad (removed in the revert) if you want to recover them. This batched persist also fixes 2c's N-undo/N-reload limitation.

7.2 R3 — variable track heights (needs a decision)

CapCut main track is bigger. TRACK_H = 48 is used everywhere (TimelineCanvas, timelineLayout getTimelineCanvasHeight, useTimelineClipDrag, timelineZones lane math, drop-row resolution, ghost/placeholder/insertion-line Y, beat strip, keyframes). Replacing the constant with a per-track height + cumulative Y-offset touches every Y↔track site. Open decision: uniform (true CapCut — research says CapCut heights are actually uniform) vs a taller main track (deliberate departure).

7.3 Smaller / deferred

  • Batched single-undo for consecutive-lane inserts (2c) — same batched-persist need as 7.1.
  • Plan 2 (canvas parity: drop onto canvas, 8-handle resize, waveforms-on-clips, optimistic no-reload insert) and Plan 3 (multi-select, sub-comp-targeted inserts, virtualization) from ROADMAP.md.
  • Two spawned cleanup chips (background tasks, may or may not have been run): (a) remove the dead export snapKeyframePctToBeat in timelineEditing.ts; (b) extract the remaining pointer handlers from useTimelineClipDrag.ts (still complexity-flagged).

8. Gotchas learned (don't re-discover)

  • G1 — stale served bundle: preview serves the CLI's embedded studio copy, refreshed only by a full bun run build. After any studio change: build, restart/confirm the served index-*.js hash changed.
  • G2 — multi-clip horizontal persist RACES (the big one, §7.1): concurrent per-clip onMoveElement (start-time changes → GSAP shift + reload each) corrupts the file. Track-index- only changes (2c) don't race as badly (no GSAP/reload heavy path). Needs a batched persist.
  • G3 — verifying track logic needs a CLEAN project. qa-project-2 is mangled + composition- heavy + no clean audio → can't show zones. Use/make a clean bed (qa-clean). Each synthetic drag test that changes state PERSISTS to the file (mutates the project) — reset between runs.
  • G4 — synthetic DnD in the harness is async-flaky: a pointerdown sets draggedClipRef via React state; a pointermove in the same eval call fires before the ref commits → drag never arms. Dispatch pointerdown in one eval, pointermove in the next, read in a third. Also read React-state-driven DOM (e.g. play/pause label) in a separate call after the flush.
  • G5 — fallow gate + filesize: --no-verify was used when the ONLY gated findings were pre-existing (e.g. snapKeyframePctToBeat unused export; complexity in useTimelineClipDrag/ blockInstaller). Always run oxlint/oxfmt/tsc/tests manually first. Filesize ≤600 lines/file is enforced — useTimelineClipDrag.ts hovers near it (extracting the commit module fixed it). New unused exports get flagged — keep helpers module-private if only used internally.
  • G6 — normalizeToZones must stay idempotent. Enforced zoning re-runs on every discovery; if the main-track pick isn't stable, tracks drift each reload (this bit us with multiple video tracks → fixed via longest-video/role resolution). Any change here needs the idempotency test.
  • G7 — the user reference-checks CapCut + gives feel-feedback live. The magnetic ghost was built wrong first (lane-locked) and corrected after the user dragged it. Build → show → adjust.
  • G8 — launch.json --port must be pinned or the CLI auto-reassigns and the harness points at a dead port (blank screen).

9. Key new/changed files (map)

New (this session):

  • packages/studio/src/components/EditorShell.tsx — the CapCut shell (top row + full-width timeline).
  • packages/studio/src/components/nle/NLEContext.tsx (+ .test.ts) — shared player/comp-stack provider.
  • packages/studio/src/components/nle/PreviewPane.tsx, TimelinePane.tsx, PreviewOverlays.tsx, useTimelineEditCallbacks.ts — decomposed panes.
  • packages/studio/src/player/components/timelineCollision.ts (+ .test.ts) — isLaneFree, resolvePlacement, snapClearOfClips, buildTrackInsert, resolveInsertRow.
  • packages/studio/src/player/components/timelineZones.ts (+ .test.ts) — classifyZone, normalizeToZones, resolveMainOriginTrack (the track model).
  • packages/studio/src/player/components/timelineClipDragCommit.ts — commitDraggedClipMove (insert / plain-move; ripple deferred). Modified (key): App.tsx, player/hooks/useTimelinePlayer.ts (zoning wired), player/store/ playerStore.ts (timeDisplayMode), player/components/{TimelineCanvas,TimelineRuler,PlayerControls, timelineLayout,timelineEditing,useTimelineClipDrag}.tsx/ts, utils/{timelineInspector, studioUiPreferences}.ts, index.ts (barrel: EditorShell not NLELayout). Deleted: components/nle/NLELayout.tsx, components/StudioPreviewArea.tsx.

10. Test state

Studio suite: the R4 checkpoint was 1394 pass / 18 pre-existing fails (telemetry/client, telemetry/distinctId, SnapToolbar — reproduce on main, DO NOT chase). Player subset ran 314→331 during R2/P4. New pure modules are unit-tested: timelineCollision (~23), timelineZones (11), timelineEditing magnetic-threshold, ruler generateTicks (46). tsc + oxlint clean on changed files.

11. Suggested next-session opening

  1. cd <worktree>; confirm branch research/studio-dnd-architecture; bun install if needed.
  2. Read this file, then .superpowers/sdd/progress.md (granular), then research/plans/2026-07-08-track-model-and-main-track-magnet-design.md (Piece 4).
  3. bun run build; node packages/cli/dist/cli.js preview --no-open --port=3007 /tmp/hf-dnd-qa/qa-clean; open in browser at 1680×950; confirm zones (image/video/music top→bottom) + no-overlap.
  4. Decide: (a) checkpoint PR the ~47 commits first (strongly recommended), then (b) build the batched persist → ripple (§7.1), then R3 heights (§7.2).

12. Open decision pending at handoff

Whether to push a checkpoint PR now (first push of this branch, ~47 commits) before the batched-persist/ripple work. Offered to the user at end of session; not yet answered. Nothing is pushed or merged.