Commit Graph
7 Commits
Author SHA1 Message Date
Vance IngallsandClaude Opus 5.5 e38585ef5f perf(studio): timeline thumbnails come back from disk in the next session (#6676)
* perf(studio): timeline thumbnails come back from disk in the next session

Reopening a project re-read every video's index and re-decoded every strip:
the frame cache and the source facts lived in memory, keyed by the session.

Now each source's facts (aspect, duration) and every decoded frame are kept
in Cache Storage under the file's ETag, the version the server says it is. A
new session checks each source once with a conditional one-byte read: a 304
means the stored frames are still that file's, and a strip they cover paints
without opening a reader. A new ETag drops what was stored for the old one.
Sources are dropped whole, least recently used first, past 256 MiB. A store
that is missing or fails (private window, quota) warns once and the strip
decodes as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(studio): a reopened project checks its videos again, and the disk index counts a frame once

The stored version a session validated was kept per video for the whole page,
so reopening a project after one of its videos was replaced at the same path
painted the old file's frames until a full reload. It is now kept per video
and project open (its contentVersion): each open re-validates once.

The index now counts a frame stored again under its name once, and merges
with what other tabs saved before writing, keeping caches this tab dropped
out, so the 256 MiB budget holds across tabs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 16:39:02 +00:00
Vance IngallsandClaude Opus 5.5 fa137cdfef fix(studio): a slow thumbnail job keeps a reader that other clips still decode from (#6643)
* fix(studio): a slow thumbnail job keeps a reader that other clips still decode from

A job that timed out judged the reader by its own progress only. A poster job that got
an already-open reader while a sibling was decoding on it, and took more than its
budget to reach its one frame (a slow link, Chrome's six connections held), logged a
timeout and retired a healthy reader, so the next clip re-read the index.

The pooled reader now counts the frames every job decodes from it, and a timed-out job
retires it only when no frame came from it while the job held it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(studio): the reader's frame count says whose frames it counts in its name, not a comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 09:18:55 +00:00
Vance IngallsandClaude Opus 5.5 a54921e0f6 perf(studio): share video sources and keep frames for timeline thumbnails (#6629)
* perf(studio): keep released timeline video frames for scroll-back

A video thumbnail frame was revoked the moment no mounted strip held it, so
scrolling back past the scheduler's cache decoded every clip again: 80 decode
passes for 40 clips on a 44-minute source, the whole first pass repeated.

Released frames now stay shared in a least-recently-released list capped by
the new sharedFrameCacheBytes budget (32 MiB of JPEG, about 11 frames for
each of a 473-clip episode's clips). Object URLs are revoked only on
eviction, and a frame any strip still shows is never evicted. Which frame a
strip shows is unchanged: the acceptance window is the same predicate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(studio): open each video source once for all its timeline clips

Every clip's thumbnail job opened its own mediabunny Input on the source URL,
so 40 clips cut from one 44-minute source opened it 80 times, re-reading and
re-parsing its 1.44 MB index each time (behind a 304, at 13-30 ms an open).

Jobs now share one pooled reader per source: the Input, its primary video
track and the track's size and duration, reference-counted and disposed
thumbnailSourceIdleMs (10 s) after the last job ends, or at once when the
source's contentVersion changes. Each job keeps its own CanvasSink: the sink
reuses one pooled canvas and a job blanks the canvases it saw on release, so
sharing one across jobs would corrupt a neighbour's frames.

A failed open leaves the pool at once, and a job that fails for any reason but
cancellation retires its reader, so the next clip opens afresh. Retries are
still counted per reader, as mediaUrlSourceOptions has done since #6477: two
jobs reading through one reader cannot tell whose read failed, and retiring a
reader whose retries are spent keeps that budget per reader rather than letting
a spent count fail later clips at once. The pool also owns the one
import("mediabunny"), which the decoder now takes from the opened source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(studio): start no timeline video decodes while the preview plays

A playing preview shares the machine's video decoder with timeline
thumbnails, and a long many-clip timeline queues hundreds of thumbnail decodes.
While the player store reports isPlaying, the scheduler now starts no new
visible or overscan video decode; a scrub's interaction request still starts,
decodes already running finish, and pausing drains the queue. Image,
composition and waveform work is unaffected.

The new thumbnailPlaybackHold module subscribes to the store's state rather
than wrapping setIsPlaying, so a reset or a project switch, which write
isPlaying: false directly, also release the held work. VideoThumbnail binds it
to the app's scheduler where video thumbnails load, which is also the path
HyperFrames Desktop runs (it embeds StudioApp, not Studio's main.tsx).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(studio): bound thumbnail range reads and their read cache

mediabunny reads a URL with open-ended `Range: bytes=N-` requests and cancels
each once it has the bytes it wanted, by when a local server has streamed
megabytes past them: about 4 MB sent for each thumbnail frame decoded. The
thumbnail reader's fetch now asks for at most 512 KiB from N; a shorter answer
than mediabunny wanted makes it ask again from where it stopped (its worker
resumes a capped response when the file size is known, which the first
206's Content-Range supplies). Closed ranges and the probe reader are untouched,
and so is Studio's playback, which does not read through mediaUrlSourceOptions.

Measured on two 44-minute 1080p sources, 40 clips each (a poster and a
4-frame strip per clip), against the same branch unbounded: bytes sent fall
13-22% one decode at a time and 33% two at a time; first-pass wall time falls
4-20%. The frames decoded are byte-identical. Most of what remains is real:
a strip slot whose keyframe is outside its window decodes from that keyframe
forward through its GOP.

A pooled reader also now caches at most 16 MiB of what it read, not
mediabunny's default 64 MiB, since it outlives its clip by up to
thumbnailSourceIdleMs. 8 MiB cost requests when a clip's poster and strip
decode together; 16 MiB measured the same as 64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(studio): bound thumbnail ranges only once Content-Range is readable

A thumbnail reader bounded even its first range request to 512 KiB. When the
answer's Content-Range cannot be read, mediabunny sizes the file as the request
start plus the answer's Content-Length, assuming the answer runs to the end of
the file, so a bounded first answer made a 10 MB file end at 512 KiB. Browsers
hide Content-Range cross-origin unless the server lists it in
Access-Control-Expose-Headers, and Studio reads remote https sources such as
HeyGen's gen-os-static bucket, which does not. With the header hidden, every
format failed: faststart and fragmented MP4 decoded 1 of 12 thumbnails, and
moov-at-end MP4, MOV and WebM none ("no decodable video track").

A reader now bounds its ranges only after an answer of its own was a 206 whose
Content-Range it could read and that starts where the request asked; its
first request stays open-ended, as before the bound. Any answer without such
a Content-Range, a whole-file 200 included, keeps that reader unbounded for
good. Same-origin readers, where Content-Range is readable, still bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(studio): count a thumbnail reader's read failures consecutively

A reader's retry budget (two retries, from #6477) counted every failure over
the reader's life. Pooled, a reader serves every clip of its source, so the
third transient failure anywhere on the source failed a clip with no retry.
The count now resets whenever a read delivers bytes, so a long-lived reader
keeps the per-clip tolerance a reader had when each clip opened its own, and
three failures in a row still give up. Bytes, not an ok status: a body that
breaks on its first read must still use up the budget. A reader that gives up
is retired, as before, so the next clip opens a fresh one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(studio): a read stalled mid-response no longer hangs its video source

The thumbnail job's signal reached only the source's open: keyframe lookups and
the decode loop ignore it, so a job whose response stalled mid-body never ended
and never released its pooled reader. The reader was never idle, so never
closed, and every later clip that needed the same bytes waited on the same
stalled read for the rest of the session. Before pooling, each clip had its own
reader, so only the stalled clip hung.

The scheduler's timeout now aborts with a TimeoutError reason. A job that times
out drops its reader at once; a job cancelled any other way that has still not
ended a load timeout later drops it then; a cancel the job answers, a scroll
away, leaves the reader pooled. Dropping disposes the reader whoever still uses
it, which fails their pending reads too.

Disposing a mediabunny Input does not end a read already waiting on a stalled
body (its orchestrator only flags the worker, which never wakes), so
mediaUrlSourceOptions gains cancelReads(), which aborts every fetch the reader
has in flight; the worker's read then fails and mediabunny rejects the reads
waiting on it. Measured in Chromium with a server that stalls one response:
the stalled clip settles at its timeout, and the same clip asked again, now
and after 11 s idle, decodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(studio): a clip's own failed result keeps its source's reader

A thumbnail job that failed for any reason but a cancel retired its pooled
reader, so one clip whose frames all failed to decode, or whose canvas would
not encode, made the next clip re-read and re-parse the source's index. Those
two failures are now a ClipResultError, which leaves the reader pooled; fetch,
demux and decoder failures still retire it.

Also pins four pool behaviours mutation testing found unguarded: a reader
closed after sitting idle has left the pool, so the next clip opens a live
one; a version change closes the old reader only once the job still decoding
on it ends; a job cancelled while the track is read stops the open there; and
a scroll-away cancel keeps the reader (the last landed with the stall fix).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(studio): keep a thumbnail reader's retry budget per start byte

ff16aed98 reset a pooled reader's retry count whenever any read delivered
bytes. That broke the #6477 give-up in two ways, both measured with real
mediabunny:

- A server that ignores Range answers 200 from byte 0, so mediabunny resumes
  at the bytes it got and is sent byte 0 again. With a body that breaks after
  64 KiB, those bytes reset the count every time: 61 fetches in 30 s and the
  read still pending.
- mediabunny reads with two workers. A healthy stream in one region kept
  forgiving a region that always answered 503: 41 fetches in 20 s, pending.

Failures are now counted per start byte, with no reset. A resume after
progress asks for a new start and gets its own two retries; asking for the
same start again uses them up whatever else the reader reads meanwhile. The
start is the byte the answer began at: the asked start for a 206 or a failed
request, 0 for a 200. Traced for the 200 case: requests at 0, 65536 and 65536
are all keyed 0, so the third failure gives up. Both repros now give up after
3 fetches ("network TypeError after 2 retries", "HTTP 503 after 2 retries"),
and failures at different starts, one per clip, never add up.

The fetch and the answer's typing move into typedFetch and typedResponse, so
fetchFn stays under the complexity gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(studio): fail a stalled thumbnail read on silence, not on a timeout

dbd820706 dropped a reader when the job using it timed out, but only once the
reader had opened: a timeout during the shared open released the job and left
the stalled open pooled, so every clip of that source joined it and timed out
in turn while clips kept arriving. Dropping on a timeout was also wrong when
bytes were still coming (a slow link's 25 s open threw its parsed index away),
and dropping a reader other clips were using failed them with "Media reads
cancelled", leaving them on "no preview" until they remounted.

A thumbnail read now fails only on silence. A request waiting 8 s for its
answer, or for its next bytes, is aborted and fails as a MediaFetchError
"stalled", which mediabunny retries like any read failure: a resume after
progress asks for a new start and continues, and three silent waits at one
start (8 + 0.5 + 8 + 1 + 8 = 25.5 s, inside the 30 s load timeout) give up,
which retires the reader. This covers the open and the decode alike, keeps a
reader whose bytes are still arriving, and never disposes a reader another
clip is using, so the job-level abandon and the scheduler's TimeoutError
reason are removed. Measured in Chromium against a stalling server:

- one-shot stall in the open: the clip decodes at 8.7 s (was: it and the next
  clip both timed out at 30 s);
- one-shot stall mid-decode: decodes at 8.7 s (main: hangs);
- a source that never sends a byte: each clip fails typed at 25.5 s and the
  next opens a fresh reader.

Each request gets its own AbortController, which follows mediabunny's signal
and the reader's cancel without AbortSignal.any; every way a request ends
unlinks it from the reader's long-lived cancel signal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(studio): pin the thumbnail behaviours mutation testing left unguarded

- A bounded answer's Content-Range keeps the reader bounded: a closed-range
  pattern that matched only open ranges flipped it unbounded after the first
  bounded request, undoing cf8f4554e.
- A frame that will not encode, through toBlob or OffscreenCanvas
  convertToBlob, is the clip's failure and keeps its reader. The
  convertToBlob rejection is now a ClipResultError too; it escaped as the
  browser's own error and retired the reader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(studio): pin the stall watchdog's guarantees about its own requests

Four guarantees of the thumbnail reader's per-request watchdog had no test;
each mutant that broke one passed the suite:

- a silent body fails as "stalled" and aborts the fetch under it, so the
  stalled connection is not left open;
- every way a request ends unlinks it from mediabunny's signal: an answered
  body, a rejected fetch, a 503, no body, a broken body, a cancel;
- a probe reader's body is never timed, however slow;
- a request made after the reader's reads were cancelled starts aborted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(studio): time a thumbnail read's silence only once its answer has started

The 8 s stall rule also timed the wait for a response's headers, and that
wait includes Chrome's queue for one of six HTTP/1.1 connections per host.
Desktop serves all media from one loopback host, where Studio's event stream
and streaming previews hold connections. Measured in Chromium against a
healthy server: with five of six connections held and 20 cancel/re-ask bursts
over the 40-clip pass, no false stall; with all six held for 40 s, two clips
failed as "stalled". The watchdog now times only a started body.

A server that never answers is now caught at the job's own timeout. The
scheduler's timeout again aborts with a TimeoutError reason. A job that timed
out without progress in its whole run (its source never finished opening, or
it decoded no frame) failed because of its source: a source that never
answers, or one that trickles bytes and never finishes. It logs one typed
MediaFetchError line ("network timeout") and retires its reader. A job that
waited for an open that finished, or decoded a frame, timed out on a slow link
and keeps the reader and its parsed index. Bytes alone are not progress, or a
trickling source would keep its reader for good.

An aborted job also ends its claim on the reader at the abort, rather than
when it settles: a job stuck in a read that ignores its signal no longer keeps
the reader open. A retired reader still waits for the clips decoding on it.

Measured in Chromium: a source that never answers fails each clip with one
typed line at 30 s and the next opens a fresh reader; with all six
connections held for 40 s, the two starved clips fail typed, none as a stall.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(studio): move the thumbnail reader's explanations into names

The comment ratchet (scripts/comment-ratchet.mjs) fails a source file whose
share of comment lines rises over its merge-base, and a new file above its
package's share. Five files of this branch did. The reasoning they carried is
in the commit messages and the PR description; what stays in code is named:
SILENT_BODY_STALL_MS, POOLED_READER_CACHE_BYTES, failuresByStart,
inFlightReads, decodedByJob, timeoutWithoutProgress and madeProgress. No
behaviour changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 06:37:40 +00:00
Miguel Ángel dfd2028f16 fix(studio): a clip that cannot be read gives up after two retries with one typed line (#6477)
* fix(studio): a clip that cannot be read gives up after two retries with one typed line

* fix(studio): a clip whose stream breaks mid-read also fails with the typed line
2026-10-26 21:08:46 +00:00
Miguel Ángel 9f09861d2e perf(studio): zooming a long timeline re-decodes and redraws less (#5142)
* perf(studio): reuse filmstrip frames when a zoom changes a strip's width

A zoom that changed a strip's frame count decoded the whole strip again
on the main thread, and showed the clip's single poster frame until it
landed. Frames are now sampled at each slice's left edge, so a strip
twice as long holds every frame of the shorter one; decoded frames are
shared per source and time, so zooming out decodes nothing and zooming in
decodes only the new half. The strip on screen stays until the new width
is ready.

* perf(studio): redraw only the waveforms on or near the screen after a zoom

A zoom resizes every mounted clip, and every mounted waveform redrew in
that same frame, on screen or not. A waveform now draws only while it is
on or near the screen (one shared observer, a screen of margin each side)
and draws when it comes near if a zoom changed it meanwhile.

* perf(studio): never start a beat-analysis stage while the timeline zooms

Analyzing a long music track runs for over a second on the main thread,
and when it started late it landed inside a zoom. The analysis now awaits
an optional pause before each long stage; Studio's waits for the timeline
to rest and the browser to go idle. Results are unchanged. The tempo
detector's stage is one call, so a zoom that starts inside it still waits
for it.

* fix(studio): reuse a filmstrip frame only where its own strip would show it

A shared frame was keyed by the time a strip asked for, not the time it
was decoded at, so a coarse strip's keyframe-early frame could land in a
finer strip and run its frames backwards. A frame now carries its decoded
time and is reused only when the strip would have accepted that keyframe
itself. The frame-count ladder now tops out at a power of two (32), so
its top step shares frames with the one below instead of none.

* feat(studio): show the clip's last frame at the end of every filmstrip

Left-edge sampling never showed a clip's final slice, so an end card or a
fade-out was missing from its strip. Each strip now also decodes the
clip's last frame (at its own time, never an earlier keyframe), shared
across zoom steps like the others, and the last tile shows it.

The strip on screen is now held per clip and range, not per props: a
clip crossing between on and off screen keeps it, and a clip narrowed to
one frame lets it go, so widening again never re-decodes a dropped width.

* docs(studio): keep the filmstrip files at or under main's comment share

The comment ratchet allows no file a higher share of comment lines than on
main; the added explanations move to the PR description.

* fix(studio): keep a held filmstrip leased so a full cache cannot evict it mid-zoom

The hold leased the strip on screen only once a new width was requested,
so for one commit it had no lease; with the cache over budget it was
evicted, the poster showed, and the old width decoded again. The strip on
screen is now leased for as long as it is the clip's, and shown only while
held. A frame that fails to decode takes the frame before it, so every
slot keeps its time and the last tile still shows the end.

* refactor(studio): one owner for which clip a held filmstrip belongs to

The hold checked the clip twice, in its lease and again before showing
it; the lease alone decides now, and the cross-clip test pins it. A test
now also pins that a filled frame is not charged to the cache.

* fix(core): hold the tempo detector until idle after a slow import

* fix(studio): scope shared video frames to timeline sessions

* chore(core): keep beat analysis comments within the ratchet
2026-10-07 07:38:13 +00:00
Miguel Ángel db42e60223 perf(studio): a clip's filmstrip decodes 9 frames instead of 272 and reaches the clip's end (#4526)
* perf(studio): build a video clip's filmstrip from its keyframes

* fix(studio): a filmstrip slot keeps its own time when its keyframe is far away

* perf(studio): a WebM filmstrip downloads each cluster once

Each keyframe lookup now runs just before its frame is decoded, from a
generator the canvas sink pulls, instead of all lookups before the first
decode. A WebM lookup loads its whole cluster, so resolving every lookup
first pushed the early clusters out of the source cache and the decode
pass fetched them again. A cancel during a lookup ends the generator
quietly so the decoder still closes; the cancel is raised after the loop.

* fix(studio): a trimmed video with no reported length fills its filmstrip

When the file reports no duration (a fragmented MP4), the source was
assumed to end at the trim length, so a trim that starts later collapsed
to zero length and every tile showed the trim start. It now ends at the
trim start plus the trim length.

* fix(studio): a video filmstrip reaches the end of the clip

The strip requests a power of two of frames spread over the whole clip,
but tile i showed frame i, so fewer tiles than frames showed only the
start of the clip and more tiles repeated the strip. Each tile now shows
the frame nearest its own place along the clip.
2026-09-26 08:52:21 +00:00
Miguel ÁngelandCodex 96861cbafc perf(studio-server): coordinate cancelable thumbnail generation (#2720)
* perf(studio): schedule adaptive timeline thumbnails

* perf(studio): bound thumbnail decoding resources

* perf(studio): virtualize timeline thumbnail media

* perf(studio): prioritize timeline thumbnail work

* perf(studio-server): coordinate cancelable thumbnail generation

---------

Co-authored-by: Codex <codex@local>
2026-08-05 20:41:35 -07:00