feat(studio): expose Studio's live state to an agentic browser (WebMCP) (#3511)

* feat(studio): expose Studio's live state to an agentic browser

Registers a `studio_look` tool on `document.modelContext`, so an agent in a
browser that supports it can read what Studio knows: the open project and
composition, the playhead, the human's current selection with its
capabilities, and the timeline's elements with a handle for each.

The API is `document.modelContext`, not `navigator.modelContext`. The latter
is a polyfill compatibility shim rather than a spec member, so feature
detecting it is wrong even where a published sample appears to work.

Three decisions worth knowing:

Registration happens ONCE per mount, with the dependencies held in a ref that
every render refreshes. Depending on the handlers instead re-runs on nearly
every interaction, because the DomEdit actions object changes identity with
the selection and the element list. Each re-run aborts the registration signal
and unregisters everything, and the spec warns that a quick unregister-then-
reregister can apply an old call's arguments against the new schema. The test
for this is the important one in the unit; breaking the empty dependency array
fails it and nothing else.

Tools resolve with a tagged result, they never reject. That is forced by the
spec: a rejected `execute` has its reason discarded and the caller sees a bare
UnknownError, so rejecting would guarantee the agent cannot learn why an edit
failed.

Elements are addressed by a minted handle, not by `TimelineElement.id`. That
id is a synthesised identity, so `getElementById` misses most elements; the
handle carries `data-hf-id`, else the DOM id, else a selector plus occurrence.

Mounted from `EditorShell` rather than `App`, because the DomEdit contexts are
only readable below `DomEditProvider` and `App.tsx` is three lines under the
600-line cap.

The undo signal is reported as the shell actually exposes it, `canUndo` and a
label, rather than as a revision counter. The depth lives in component-local
state and is not reachable without plumbing it through the shell context, so
the field says what it is instead of implying precision it does not have.

Writes are not in this change. `canWrite` is optimistic and the comment says
so; the write tools need a real guard against the paused-save and external-
conflict states, which are not on any context this component can reach yet.

* fix(studio): bound WebMCP look filters

* fix(studio): remove premature WebMCP write state

* docs(studio): name WebMCP singleton assumption

* fix(studio): surface WebMCP registration failures
This commit is contained in:
Miguel Ángel
2026-08-26 23:59:49 -04:00
committed by GitHub
parent 21bcd5745c
commit 94da403d6d
13 changed files with 1482 additions and 0 deletions
@@ -34,6 +34,14 @@ export interface StudioUiPreferences {
timelineZoomMode?: "fit" | "manual";
/** Manual timeline zoom percent, paired with `timelineZoomMode: "manual"`. */
timelineManualZoomPercent?: number;
/**
* Expose Studio's editing capabilities to an agentic browser as WebMCP tools.
* Absent means on: the browser still gates every actual call behind its own
* permission prompt, so "registered" is not "reachable without consent".
* Changes take effect on the next Studio reload because registration is
* intentionally scoped to one mount.
*/
agentToolsEnabled?: boolean;
}
const STUDIO_UI_PREFERENCES_KEY = "hf-studio-ui-preferences";
@@ -140,6 +148,9 @@ function readStorage(storage: Storage | null): StudioUiPreferences {
) {
preferences.timelineManualZoomPercent = parsed.timelineManualZoomPercent;
}
if (typeof parsed.agentToolsEnabled === "boolean") {
preferences.agentToolsEnabled = parsed.agentToolsEnabled;
}
return preferences;
} catch {
return {};