feat(studio): show the agent's storyboard in a Storyboard tab beside the preview (#7284)

* feat(studio): show the agent's storyboard in a Storyboard tab beside the preview

The board an agent writes per the storyboard recipe (boards/storyboard.html,
or a root storyboard.html) now shows in Studio the way HyperFrames Desktop
shows it: a Preview | Storyboard strip over the preview panel, only while the
project has a board. Studio reads the page's class="frame" boxes into a board
of stills, with acts, titles, times and words, and one frame at a time with
arrow keys. A page with no frames shows as written in a script-less iframe.

A board written while Studio is open brings its tab to the front, as Desktop
does at the agent's stop. A rewrite of the page, a still beside it or
STORYBOARD.md reloads it in place, and a dot marks a rewrite seen from the
preview. "Show in preview" goes back to the film at the frame's start.

Notes, drawings and the go belong to Desktop's agent, so the board's bar
carries the desktop-app button. Its funnel events now say which button was
pressed (placement: header or storyboard).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HebQXPiLcYKQfvPWB3Fgxy

* refactor(studio): split the storyboard view and its frame reader into smaller parts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HebQXPiLcYKQfvPWB3Fgxy

* docs(skills): the agent's storyboard stop points at the Storyboard tab, not the file

With the board shown beside the preview in Studio as in Desktop, the recipe's
stop and style-sampling's stop 1 tell the person to look in the Storyboard
tab, and to start a preview and give its link when none is running, instead
of handing over boards/storyboard.html to open from disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HebQXPiLcYKQfvPWB3Fgxy

* fix(studio): the storyboard listens on Studio's own file-change channel, and its notes are trimmed

The board opened its own /api/events stream, which the Vite dev server
behind the runtime smoke does not serve. It now subscribes through
subscribeFileChanges, the one channel picker the external-change
coordinator already uses (test adapter, Vite HMR, then SSE), so dev and
built Studio deliver the same changes to both.

The new files' comments are cut to the package's share, and the recipe's
Storyboard-tab line moves to the paragraph that names the board, leaving
the stop step as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HebQXPiLcYKQfvPWB3Fgxy

* fix(studio): the board hears file changes through the coordinator, not a second subscription

The external-change coordinator stays the one subscriber to the file-change
transport (HMR in dev, SSE when built), as externalFileChangeOwnership
asserts. It announces each decoded change through fileChangeFanout, and the
storyboard listens there. Mounting EditorShell alone, as its tests do, no
longer opens an EventSource, and the coordinator file stays under the line cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HebQXPiLcYKQfvPWB3Fgxy

* feat(studio): the storyboard's Open in desktop app asks the app to open on the board, and its card says why

- The storyboard bar's button posts `doc: "storyboard"` with its page id. The preview server passes only that value
  on, and `openInDesktop` writes it into `.hyperframes/desktop-cta.json` beside the page id, so the app can open the
  project on its Storyboard tab with Build in sight (hyperframes-internal reads it).
- The guide card opened from the storyboard says "Approve this storyboard in the desktop app" and "Comment on a frame,
  draw on it, then press Build", and its last step says the app opens on this storyboard. The header's card is
  unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HebQXPiLcYKQfvPWB3Fgxy

* feat(studio): approve the storyboard in Studio; comments on frames stay in the desktop app

A Studio user without the desktop app had no way to approve the board from the tab, and the bar said approval was
in the desktop app, so the tab read as a dead end. Studio talks to the person's own agent through their chat, so the
bar's Approve copies "The storyboard is approved. Build the video from it." for them to paste to it, and says so in
a toast (or spells the words out when the clipboard refuses). Commenting and drawing on frames stay in the desktop
app. The storyboard recipe tells the agent those pasted words are the person's go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HebQXPiLcYKQfvPWB3Fgxy

* fix(studio): the storyboard's desktop card offers commenting, since approving now happens in Studio

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HebQXPiLcYKQfvPWB3Fgxy

* fix(studio): hand-off intent before dispatch, steady film under the board, stills outside the board folder

- openInDesktop writes desktop-cta.json before sending the folder, so a running
  app reads the storyboard ask on arrival; a plain or header open drops an
  earlier storyboard ask; a failed open puts the earlier record back.
- PreviewStage keeps the film at one place in the tree with or without a board,
  so a board appearing never remounts it; the automatic switch goes through the
  same pause as a click, and a play while the board is up is paused.
- The board reloads when a still it draws changes, including a root board's
  stills/ and a boards/ page's ../assets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HebQXPiLcYKQfvPWB3Fgxy

* chore(cli): fit the Storyboard tab in the CLI's size budget; reword a record helper's doc

The CLI tarball carries Studio's build, which now holds the Storyboard tab; merged
with main it measures 7,982,002 bytes packed, 7 bytes past the 1% tolerance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HebQXPiLcYKQfvPWB3Fgxy

* fix(studio): hold the player while the storyboard covers the film

A pause after the fact let J's reverse loop schedule its next frame, so the
hidden film kept seeking. The stage now takes the player's own hold while the
board is up (and only when no one else holds it), so J, Space and playback
requests are refused before they start; Preview releases it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HebQXPiLcYKQfvPWB3Fgxy

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WaterrrForever
2026-11-06 16:16:21 +00:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 3e3ba8d0d0
commit 318724f5f4
25 changed files with 1453 additions and 66 deletions
+17 -5
View File
@@ -19,7 +19,9 @@ function server(
place: { platform?: NodeJS.Platform; installed?: () => boolean } = {},
) {
const app = new Hono();
const open = vi.fn((_dir: string, _cta?: string): DesktopOpenResult => OPENED);
const open = vi.fn(
(_dir: string, _cta?: string, _doc?: "storyboard"): DesktopOpenResult => OPENED,
);
mountDesktopRoutes(app, "/films/a", { ready, open, ...place });
const post = (headers: Record<string, string>, body?: unknown) =>
app.request("/api/open-in-desktop", {
@@ -118,7 +120,7 @@ describe("open-in-desktop routes", () => {
it("once live, opens this server's project for Studio's own POST", async () => {
const { open, post } = server(true);
expect(await (await post(STUDIO)).json()).toEqual(OPENED);
expect(open).toHaveBeenCalledWith("/films/a", undefined);
expect(open).toHaveBeenCalledWith("/films/a", undefined, undefined);
});
it("passes the press's page id on to the hand-off, and drops anything that is not one", async () => {
@@ -128,9 +130,19 @@ describe("open-in-desktop routes", () => {
await post(STUDIO, { cta: "../../etc/passwd" });
await post(STUDIO, { cta: 42 });
expect(open.mock.calls).toEqual([
["/films/a", id],
["/films/a", undefined],
["/films/a", undefined],
["/films/a", id, undefined],
["/films/a", undefined, undefined],
["/films/a", undefined, undefined],
]);
});
it("passes on only the storyboard as the page to open the project on", async () => {
const { open, post } = server(true);
await post(STUDIO, { doc: "storyboard" });
await post(STUDIO, { doc: "../index.html" });
expect(open.mock.calls).toEqual([
["/films/a", undefined, "storyboard"],
["/films/a", undefined, undefined],
]);
});
});
+10 -3
View File
@@ -27,7 +27,8 @@ export function mountDesktopRoutes(
projectDir: string,
{
ready = HANDOFF_READY,
open = (dir: string, cta?: string) => openInDesktop(dir, { cta, src: "browser-editor" }),
open = (dir: string, cta?: string, doc?: "storyboard") =>
openInDesktop(dir, { cta, doc, src: "browser-editor" }),
platform = process.platform,
installed = () => desktopInstalled({ platform }),
} = {},
@@ -57,7 +58,13 @@ export function mountDesktopRoutes(
downloadUrl,
});
const body: unknown = await c.req.json().catch(() => null);
const cta = (body as { cta?: unknown } | null)?.cta;
return c.json(open(projectDir, typeof cta === "string" && CTA_ID.test(cta) ? cta : undefined));
const { cta, doc } = (body ?? {}) as { cta?: unknown; doc?: unknown };
return c.json(
open(
projectDir,
typeof cta === "string" && CTA_ID.test(cta) ? cta : undefined,
doc === "storyboard" ? doc : undefined,
),
);
});
}
+78
View File
@@ -103,6 +103,84 @@ describe("openInDesktop", () => {
expect(existsSync(record)).toBe(false);
openInDesktop(dir, { ...LIVE, cta: "page-1", open: () => true });
expect(JSON.parse(readFileSync(record, "utf8"))).toMatchObject({ cta: "page-1" });
expect(JSON.parse(readFileSync(record, "utf8"))).not.toHaveProperty("doc");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("tells the app to open on the storyboard when the press came from the Storyboard tab", () => {
const dir = mkdtempSync(join(tmpdir(), "hf-cta-doc-"));
const record = join(dir, ".hyperframes", "desktop-cta.json");
try {
writeFileSync(join(dir, "index.html"), "<html></html>");
openInDesktop(dir, { ...LIVE, cta: "page-1", doc: "storyboard", open: () => true });
expect(JSON.parse(readFileSync(record, "utf8"))).toMatchObject({
cta: "page-1",
doc: "storyboard",
});
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("has the storyboard ask in place when the app is handed the folder, not after", () => {
const dir = mkdtempSync(join(tmpdir(), "hf-cta-order-"));
const record = join(dir, ".hyperframes", "desktop-cta.json");
try {
writeFileSync(join(dir, "index.html"), "<html></html>");
let seen: unknown = null;
openInDesktop(dir, {
...LIVE,
cta: "page-1",
doc: "storyboard",
open: () => {
seen = JSON.parse(readFileSync(record, "utf8"));
return true;
},
});
expect(seen).toMatchObject({ cta: "page-1", doc: "storyboard" });
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("doesn't reuse an earlier storyboard ask on a later header or plain open", () => {
const dir = mkdtempSync(join(tmpdir(), "hf-cta-later-"));
const record = join(dir, ".hyperframes", "desktop-cta.json");
const read = () => JSON.parse(readFileSync(record, "utf8"));
try {
writeFileSync(join(dir, "index.html"), "<html></html>");
openInDesktop(dir, { ...LIVE, cta: "page-1", doc: "storyboard", open: () => true });
let seen: unknown = null;
openInDesktop(dir, { ...LIVE, cta: "page-2", open: () => ((seen = read()), true) });
expect(seen).toMatchObject({ cta: "page-2" });
expect(seen).not.toHaveProperty("doc");
openInDesktop(dir, { ...LIVE, cta: "page-1", doc: "storyboard", open: () => true });
openInDesktop(dir, { ...LIVE, open: () => ((seen = read()), true) });
expect(seen).toMatchObject({ cta: "page-1" });
expect(seen).not.toHaveProperty("doc");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("puts the earlier record back when the app didn't take the folder", () => {
const dir = mkdtempSync(join(tmpdir(), "hf-cta-undo-"));
const record = join(dir, ".hyperframes", "desktop-cta.json");
try {
writeFileSync(join(dir, "index.html"), "<html></html>");
openInDesktop(dir, { ...LIVE, cta: "page-1", open: () => true });
const earlier = readFileSync(record, "utf8");
openInDesktop(dir, {
...LIVE,
cta: "page-2",
doc: "storyboard",
open: () => false,
installed: () => true,
});
expect(readFileSync(record, "utf8")).toBe(earlier);
} finally {
rmSync(dir, { recursive: true, force: true });
}
+44 -18
View File
@@ -7,7 +7,7 @@ import { addCatchUpNote, markSeen, readSeen, seenPath } from "./appHistory.js";
import { createProjectConfig } from "./projectConfig.js";
import { isCI } from "./attendedTerminal.js";
import { isAgentDriven } from "../telemetry/agent_runtime.js";
import { writeRecord } from "./projectRecords.js";
import { readRecord, removeRecord, writeRecord } from "./projectRecords.js";
// The HyperFrames desktop app, then builds named in HYPERFRAMES_DESKTOP_BUNDLE_IDS / _APP_NAMES (comma-separated).
const list = (name: string): string[] =>
@@ -180,10 +180,12 @@ export function openInDesktop(
installed = () => desktopInstalled({ platform }),
env = process.env,
cta,
doc,
src = "cli",
...where
}: AppLookup & {
cta?: string;
doc?: "storyboard";
src?: DownloadSource;
ready?: boolean;
open?: (bundleId: string, dir: string) => boolean;
@@ -198,26 +200,50 @@ export function openInDesktop(
reason,
downloadUrl: desktopDownloadUrl(platform, src) ?? STUDIO_PAGE,
});
const opened = (app: string): DesktopOpenResult => {
if (cta) writeRecord(dir, DESKTOP_CTA_RECORD, JSON.stringify({ cta, at: Date.now() }));
return {
opened: true,
app,
handedOver: leaveHandoff(dir, agentSession(env)),
};
const opened = (app: string): DesktopOpenResult => ({
opened: true,
app,
handedOver: leaveHandoff(dir, agentSession(env)),
});
const dispatch = (): DesktopOpenResult => {
if (platform === "darwin") {
const bundleId = bundleIds().find((id) => open(id, dir));
if (!bundleId) return notOpened(installed() ? "open-failed" : "not-installed");
return opened(bundleId === bundleIds()[0] ? DEFAULT_APP : bundleId);
}
const apps = desktopApps({ platform, env, ...where });
if (!apps) return notOpened("unsupported-platform");
if (!apps.length) return notOpened("not-installed");
const app = apps.find(({ executable }) => launch(executable, dir));
return app ? opened(app.name) : notOpened("open-failed");
};
if (!ready) return notOpened("handoff-unavailable");
markBareIndexAsProject(dir);
if (platform === "darwin") {
const bundleId = bundleIds().find((id) => open(id, dir));
if (!bundleId) return notOpened(installed() ? "open-failed" : "not-installed");
return opened(bundleId === bundleIds()[0] ? DEFAULT_APP : bundleId);
}
const apps = desktopApps({ platform, env, ...where });
if (!apps) return notOpened("unsupported-platform");
if (!apps.length) return notOpened("not-installed");
const app = apps.find(({ executable }) => launch(executable, dir));
return app ? opened(app.name) : notOpened("open-failed");
const undoIntent = leaveIntent(dir, cta, doc);
const result = dispatch();
if (!result.opened) undoIntent();
return result;
}
/** Written before the folder is sent, since a running app reads it on arrival; the undo puts back what was there. */
function leaveIntent(dir: string, cta?: string, doc?: "storyboard"): () => void {
const before = readRecord(dir, DESKTOP_CTA_RECORD, 4096);
const earlier = ((): { doc?: unknown } | null => {
try {
return JSON.parse(before);
} catch {
return null;
}
})();
if (cta || doc)
writeRecord(dir, DESKTOP_CTA_RECORD, JSON.stringify({ cta, at: Date.now(), doc }));
else if (earlier?.doc !== undefined)
writeRecord(dir, DESKTOP_CTA_RECORD, JSON.stringify({ ...earlier, doc: undefined }));
else return () => {};
return () => {
if (!before) removeRecord(dir, DESKTOP_CTA_RECORD);
else writeRecord(dir, DESKTOP_CTA_RECORD, before);
};
}
/** The `hyperframes open` line for a project, relative to where the person is. */
+22 -1
View File
@@ -1,4 +1,13 @@
import { closeSync, constants, fstatSync, lstatSync, mkdirSync, openSync, readSync } from "node:fs";
import {
closeSync,
constants,
fstatSync,
lstatSync,
mkdirSync,
openSync,
readSync,
unlinkSync,
} from "node:fs";
import { join } from "node:path";
import { replaceFileAtomically } from "@hyperframes/core/atomic-file";
@@ -81,3 +90,15 @@ export function writeRecord(projectDir: string, name: string, data: string): boo
const dir = recordsDir(projectDir, true);
return dir !== null && replaceFile(join(dir, name), data, 0o600);
}
/** Deletes a record that is a plain file; false when there is none to delete. */
export function removeRecord(projectDir: string, name: string): boolean {
const dir = recordsDir(projectDir);
try {
if (!dir || !lstatSync(join(dir, name)).isFile()) return false;
unlinkSync(join(dir, name));
return true;
} catch {
return false;
}
}
@@ -7,10 +7,17 @@ import { Button, cn } from "./ui";
// The desktop-app guide card under OpenInDesktopButton: Framey and three steps (download, install, open), all drawn
// from one `Progress`.
export type DesktopCtaPlacement = "header" | "storyboard";
export type DesktopRoute = {
handoff: boolean;
downloadUrl: string | null;
context: { os: string; app_installed: boolean; download_available: boolean };
context: {
os: string;
app_installed: boolean;
download_available: boolean;
placement: DesktopCtaPlacement;
};
};
/** The OS for the download, and what "this …" calls the machine. */
@@ -145,10 +152,14 @@ export function labelOf(p: Progress): string {
return isWaiting(p) ? "Finish installing" : "Install the desktop app";
}
const boardFirst = (p: Progress) => p.route.context.placement === "storyboard";
function subtitleOf(p: Progress): string {
if (p.failed) return "Almost there: the last step needs another try.";
if (!isInstalled(p))
return "Watch it, click a frame, tell Framey what to change. Three steps, about two minutes.";
return boardFirst(p)
? "Comment on a frame, draw on it, then press Build. Three steps, about two minutes."
: "Watch it, click a frame, tell Framey what to change. Three steps, about two minutes.";
return p.route.handoff
? "Everything is ready. One press and you're in."
: "Almost there: one last step.";
@@ -252,7 +263,11 @@ function OpenStep({ progress, projectDir, onOpen }: CardProps) {
);
return (
<Step n={3} state={installed ? NOW : TODO} title={title}>
<p className={NOTE}>Same project, same files, in the desktop app.</p>
<p className={NOTE}>
{boardFirst(progress)
? "Same project, same files. It opens on this storyboard."
: "Same project, same files, in the desktop app."}
</p>
<Button
variant="primary"
className="mt-2"
@@ -275,7 +290,9 @@ export function GuideCard(props: CardProps) {
<FrameyGlyph size={46} mood={moodOf(progress)} />
<div className="min-w-0">
<p className="text-step-14 font-semibold text-balance text-text-0">
Edit this video in the desktop app
{boardFirst(progress)
? "Comment on this storyboard in the desktop app"
: "Edit this video in the desktop app"}
</p>
<p className={NOTE}>
<Swap value={subtitle}>{subtitle}</Swap>
+10 -7
View File
@@ -1,6 +1,7 @@
import { memo, useCallback, type ReactNode } from "react";
import { matchesKey } from "../player/components/studioShortcuts";
import { PreviewPane } from "./nle/PreviewPane";
import { PreviewStage } from "./storyboard/PreviewStage";
import { TimelinePane } from "./nle/TimelinePane";
import { PreviewOverlays } from "./nle/PreviewOverlays";
import { PreviewReadOnlyProvider } from "./editor/previewReadOnlyContext";
@@ -313,13 +314,15 @@ function EditorShellBody({
<StudioAgentTools />
<Dock.Root projectId={projectId}>
<Dock.Panel id="preview">
<div className="relative flex h-full w-full min-w-0 flex-col">
<PreviewPane
previewOverlay={previewOverlay}
onSelectTimelineElement={onSelectTimelineElement}
onPreviewBlockDrop={onPreviewBlockDrop}
/>
</div>
<PreviewStage projectId={projectId}>
<div className="relative flex h-full w-full min-w-0 flex-col">
<PreviewPane
previewOverlay={previewOverlay}
onSelectTimelineElement={onSelectTimelineElement}
onPreviewBlockDrop={onPreviewBlockDrop}
/>
</div>
</PreviewStage>
</Dock.Panel>
<Dock.Panel id="timeline">
<TimelinePane
@@ -52,12 +52,12 @@ function serve(get: Response | null, post?: unknown) {
);
}
async function mount(): Promise<HTMLElement> {
async function mount(placement?: "storyboard"): Promise<HTMLElement> {
const host = document.createElement("div");
document.body.append(host);
const root = createRoot(host);
mounted = { root, host };
await act(async () => root.render(<OpenInDesktopButton />));
await act(async () => root.render(<OpenInDesktopButton placement={placement} />));
return host;
}
@@ -86,7 +86,7 @@ beforeEach(() => {
const events = (name: string) =>
track.mock.calls.filter(([event]) => event === name).map(([, props]) => props);
const MAC = { os: "darwin", app_installed: true, download_available: true };
const MAC = { os: "darwin", app_installed: true, download_available: true, placement: "header" };
afterEach(() => {
vi.unstubAllGlobals();
@@ -140,7 +140,7 @@ const stepState = (n: number) =>
guide()?.querySelector<HTMLElement>(`[data-step="${n}"]`)?.dataset.state ?? null;
const cardButton = (text: string) =>
[...(guide()?.querySelectorAll("button") ?? [])].find((b) => b.textContent === text) ?? null;
const noApp = { os: "darwin", app_installed: false, download_available: true };
const noApp = { os: "darwin", app_installed: false, download_available: true, placement: "header" };
const routeBody = (over: Record<string, unknown> = {}) => ({
...noApp,
available: true,
@@ -244,7 +244,13 @@ it("finds the install only from the preview server, asking only while the card i
(e) => (e as { step: string }).step === "install_detected",
);
expect(detected).toEqual([
{ step: "install_detected", os: "darwin", app_installed: true, download_available: true },
{
step: "install_detected",
os: "darwin",
app_installed: true,
download_available: true,
placement: "header",
},
]);
const settled = gets;
await act(async () => vi.advanceTimersByTime(10_000));
@@ -360,7 +366,7 @@ it("a hand-off that finds the app gone goes back to the download step", async ()
});
it("where the app is installed but cannot take a project (Linux), step 3 shows the folder to drop in", async () => {
const linux = { os: "linux", app_installed: true, download_available: true };
const linux = { os: "linux", app_installed: true, download_available: true, placement: "header" };
serve(
Response.json({
...linux,
@@ -399,7 +405,13 @@ it("on Windows, step 1 names the Windows download and opens it tagged", async ()
await act(async () => cardButton("Download for Windows")!.click());
expect(opened).toEqual([tagged(`${DOWNLOAD}?os=windows`)]);
expect(events("desktop_cta_eligibility")).toEqual([
{ eligibility: "download", os: "win32", app_installed: false, download_available: true },
{
eligibility: "download",
os: "win32",
app_installed: false,
download_available: true,
placement: "header",
},
]);
});
@@ -436,11 +448,50 @@ it("counts where the button can show, the app here, and platforms with no app",
os: "freebsd",
app_installed: false,
download_available: false,
placement: "header",
},
]);
expect(events("desktop_cta_impression")).toEqual([]);
});
it("tells the storyboard's button from the header's in every funnel event", async () => {
serve(
Response.json({
...MAC,
available: true,
handoff: true,
downloadUrl: DOWNLOAD,
installed: true,
}),
{ opened: true },
);
await mount("storyboard");
const storyboardButton = document.querySelector<HTMLButtonElement>(
'[data-testid="storyboard-open-in-desktop"]',
);
expect(storyboardButton).not.toBeNull();
await act(async () => storyboardButton!.click());
const placements = track.mock.calls
.filter(([event]) => String(event).startsWith("desktop_cta"))
.map(([, props]) => (props as { placement?: string }).placement);
expect(placements.length).toBeGreaterThan(0);
expect(new Set(placements)).toEqual(new Set(["storyboard"]));
expect(postBodies).toEqual([{ cta: resolveTabId(), doc: "storyboard" }]);
});
it("asks the storyboard's visitor to comment on the board in the app, not to edit the video", async () => {
serve(Response.json({ ...MAC, available: true, handoff: false, downloadUrl: DOWNLOAD }));
await mount("storyboard");
await act(async () =>
document
.querySelector<HTMLButtonElement>('[data-testid="storyboard-open-in-desktop"]')!
.click(),
);
expect(guide()!.textContent).toContain("Comment on this storyboard in the desktop app");
expect(guide()!.textContent).toContain("then press Build");
expect(guide()!.textContent).not.toContain("Edit this video");
});
it("records one impression, only once the button is on screen", async () => {
serve(
Response.json({
@@ -512,7 +563,12 @@ it("records a press the preview server never answered as failed", async () => {
});
it("records the download press as sent to the download, never as an install", async () => {
const linux = { os: "linux", app_installed: false, download_available: true };
const linux = {
os: "linux",
app_installed: false,
download_available: true,
placement: "header",
};
vi.stubGlobal("open", () => ({ opener: null }));
serve(
Response.json({
@@ -10,6 +10,7 @@ import {
isWaiting,
labelOf,
moodOf,
type DesktopCtaPlacement,
type DesktopRoute,
type Progress,
} from "./DesktopGuideCard";
@@ -47,7 +48,7 @@ const eligibilityOf = (route: DesktopRoute): Eligibility =>
route.handoff ? "open_in_app" : route.downloadUrl ? "download" : "no_app_for_platform";
/** The preview server's answer, or null when there is none (the desktop's own Studio, or no server). */
async function readRoute(): Promise<DesktopRoute | null> {
async function readRoute(placement: DesktopCtaPlacement): Promise<DesktopRoute | null> {
const res = await studioApiFetch(ROUTE);
if (!res.ok) return null;
const body = (await res.json()) as {
@@ -66,16 +67,17 @@ async function readRoute(): Promise<DesktopRoute | null> {
os: typeof body.os === "string" ? body.os : "unknown",
app_installed: body.installed === true,
download_available: downloadUrl !== null,
placement,
},
};
}
/** Null hides the button: no route, or nothing here to open or download. `setRoute` takes a fresher answer. */
function useDesktopRoute() {
function useDesktopRoute(placement: DesktopCtaPlacement) {
const [route, setRoute] = useState<DesktopRoute | null>(null);
useEffect(() => {
let live = true;
readRoute()
readRoute(placement)
.then((answered) => {
if (!live || !answered) return;
trackStudioEvent("desktop_cta_eligibility", {
@@ -88,7 +90,7 @@ function useDesktopRoute() {
return () => {
live = false;
};
}, []);
}, [placement]);
return [route, setRoute] as const;
}
@@ -145,12 +147,16 @@ function useGaze(
}
/** While the card is open and the app is not here, ask the preview server again; stops on close or once found. */
function useInstallWatch(watching: boolean, onAnswer: (route: DesktopRoute) => void): void {
function useInstallWatch(
watching: boolean,
placement: DesktopCtaPlacement,
onAnswer: (route: DesktopRoute) => void,
): void {
useEffect(() => {
if (!watching) return;
let live = true;
const ask = () =>
readRoute()
readRoute(placement)
.then((answered) => live && answered && onAnswer(answered))
.catch(() => {});
void ask();
@@ -159,7 +165,7 @@ function useInstallWatch(watching: boolean, onAnswer: (route: DesktopRoute) => v
live = false;
clearInterval(timer);
};
}, [watching, onAnswer]);
}, [watching, placement, onAnswer]);
}
const LOOK =
@@ -167,7 +173,11 @@ const LOOK =
const LOOK_FAILED = "border-danger/45 bg-danger/8 text-danger-ink enabled:hover:bg-danger/14";
/** The guide's state and the two actions that move it: the download and the hand-off. */
function useGuide(route: DesktopRoute | null, setRoute: (route: DesktopRoute) => void) {
function useGuide(
route: DesktopRoute | null,
setRoute: (route: DesktopRoute) => void,
placement: DesktopCtaPlacement,
) {
const { showToast } = useStudioShellContext();
const [cardOpen, setCardOpen] = useState(false);
const [opening, setOpening] = useState(false);
@@ -190,7 +200,7 @@ function useGuide(route: DesktopRoute | null, setRoute: (route: DesktopRoute) =>
},
[setRoute],
);
useInstallWatch(cardOpen && route !== null && !installed, onAnswer);
useInstallWatch(cardOpen && route !== null && !installed, placement, onAnswer);
useEffect(() => {
if (!downloaded || installed) return;
@@ -256,7 +266,10 @@ function useGuide(route: DesktopRoute | null, setRoute: (route: DesktopRoute) =>
const res = await studioApiFetch(ROUTE, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ cta: resolveTabId() }),
body: JSON.stringify({
cta: resolveTabId(),
...(placement === "storyboard" ? { doc: "storyboard" } : {}),
}),
});
await landed((await res.json()) as OpenResult);
} catch {
@@ -271,14 +284,18 @@ function useGuide(route: DesktopRoute | null, setRoute: (route: DesktopRoute) =>
return { progress, cardOpen, showCard, download, open };
}
export function OpenInDesktopButton() {
export function OpenInDesktopButton({
placement = "header",
}: {
placement?: DesktopCtaPlacement;
} = {}) {
const projectDir = useFileManagerContextOptional()?.projectDir ?? null;
const [route, setRoute] = useDesktopRoute();
const [route, setRoute] = useDesktopRoute(placement);
const trigger = useRef<HTMLButtonElement>(null);
const eye = useRef<SVGGElement>(null);
useGaze(trigger, eye);
useImpression(trigger, route);
const guide = useGuide(route, setRoute);
const guide = useGuide(route, setRoute, placement);
if (!guide) return null;
const { progress, cardOpen, showCard, download, open } = guide;
const label = labelOf(progress);
@@ -300,7 +317,7 @@ export function OpenInDesktopButton() {
<Button
ref={trigger}
variant="ghost"
data-testid="header-open-in-desktop"
data-testid={`${placement}-open-in-desktop`}
data-opening={progress.opening || undefined}
aria-busy={progress.opening || waiting || undefined}
title={
@@ -0,0 +1,48 @@
// @vitest-environment happy-dom
import React, { act } from "react";
import { createRoot } from "react-dom/client";
import { afterEach, expect, it, vi } from "vitest";
(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
const showToast = vi.fn();
vi.mock("../../contexts/StudioContext", () => ({ useStudioShellContext: () => ({ showToast }) }));
const track = vi.fn();
vi.mock("../../utils/studioTelemetry", () => ({ trackStudioEvent: track }));
const copy = vi.fn(async (_text: string) => true);
vi.mock("../../utils/clipboard", () => ({ copyTextToClipboard: copy }));
const { APPROVE_WORDS, ApproveBoardButton } = await import("./ApproveBoardButton");
afterEach(() => {
document.body.innerHTML = "";
vi.clearAllMocks();
});
async function press() {
const host = document.createElement("div");
document.body.append(host);
const root = createRoot(host);
await act(async () => root.render(<ApproveBoardButton />));
await act(async () =>
document.querySelector<HTMLButtonElement>('[data-testid="storyboard-approve"]')!.click(),
);
await act(async () => root.unmount());
}
it("approves without the desktop app: copies the go for the person's own agent", async () => {
await press();
expect(copy).toHaveBeenCalledWith(APPROVE_WORDS);
expect(showToast).toHaveBeenCalledWith(
"Copied. Paste it to your agent to start the build.",
"info",
);
expect(track).toHaveBeenCalledWith("storyboard_action", { action: "approve_copied" });
});
it("spells the go out when the clipboard refuses, so the person can still type it", async () => {
copy.mockResolvedValueOnce(false);
await press();
expect(showToast).toHaveBeenCalledWith(expect.stringContaining(APPROVE_WORDS), "info");
expect(track).toHaveBeenCalledWith("storyboard_action", { action: "approve_copy_failed" });
});
@@ -0,0 +1,33 @@
import { useStudioShellContext } from "../../contexts/StudioContext";
import { copyTextToClipboard } from "../../utils/clipboard";
import { trackStudioEvent } from "../../utils/studioTelemetry";
import { Button } from "../ui";
export const APPROVE_WORDS = "The storyboard is approved. Build the video from it.";
/** Studio talks to the person's own agent through their chat: Approve copies the go for them to paste there. */
export function ApproveBoardButton() {
const { showToast } = useStudioShellContext();
const approve = async () => {
const copied = await copyTextToClipboard(APPROVE_WORDS);
trackStudioEvent("storyboard_action", {
action: copied ? "approve_copied" : "approve_copy_failed",
});
showToast(
copied
? "Copied. Paste it to your agent to start the build."
: `Couldn't copy. Tell your agent: "${APPROVE_WORDS}"`,
"info",
);
};
return (
<Button
variant="primary"
size="sm"
onClick={() => void approve()}
data-testid="storyboard-approve"
>
Approve
</Button>
);
}
@@ -0,0 +1,114 @@
// @vitest-environment happy-dom
import React, { act, useEffect } from "react";
import { createRoot, type Root } from "react-dom/client";
import { create } from "zustand";
import { afterEach, beforeEach, expect, it, vi } from "vitest";
import type { StoryboardBoard } from "./useStoryboardBoard";
(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
const pauses = vi.fn();
const player = create<{
isPlaying: boolean;
playLocked: boolean;
setPlayLocked: (locked: boolean) => void;
setIsPlaying: (playing: boolean) => void;
requestPlayback: (playing: boolean) => void;
requestSeek: (seconds: number) => void;
}>((set) => ({
isPlaying: false,
playLocked: false,
setPlayLocked: (playLocked) => set({ playLocked }),
setIsPlaying: (isPlaying) => set({ isPlaying }),
requestPlayback: (playing) => {
if (!playing) pauses();
},
requestSeek: () => {},
}));
vi.mock("../../player", () => ({ usePlayerStore: player }));
vi.mock("../../utils/studioTelemetry", () => ({ trackStudioEvent: vi.fn() }));
vi.mock("./StoryboardBoardView", () => ({
StoryboardBoardView: () => <div data-testid="board-view" />,
}));
let board: StoryboardBoard;
vi.mock("./useStoryboardBoard", () => ({ useStoryboardBoard: () => board }));
const { PreviewStage } = await import("./PreviewStage");
const noBoard: StoryboardBoard = {
path: null,
revision: 0,
read: { state: "loading" },
appeared: false,
};
const written: StoryboardBoard = { ...noBoard, path: "boards/storyboard.html", appeared: true };
let mounts = 0;
function Film() {
useEffect(() => {
mounts += 1;
}, []);
return <div data-testid="film" />;
}
let root: Root;
const render = () =>
act(async () =>
root.render(
<PreviewStage projectId="p1">
<Film />
</PreviewStage>,
),
);
const boardShown = () => document.querySelector('[data-testid="board-view"]') !== null;
beforeEach(() => {
mounts = 0;
board = noBoard;
player.setState({ isPlaying: false, playLocked: false });
const host = document.createElement("div");
document.body.append(host);
root = createRoot(host);
});
afterEach(async () => {
await act(async () => root.unmount());
document.body.innerHTML = "";
vi.clearAllMocks();
});
it("keeps the film mounted and pauses it when the agent's board first appears", async () => {
player.setState({ isPlaying: true });
await render();
board = written;
await render();
expect(boardShown()).toBe(true);
expect(mounts).toBe(1);
expect(pauses).toHaveBeenCalled();
});
it("keeps the film mounted when the board goes and comes back", async () => {
board = written;
await render();
board = noBoard;
await render();
expect(boardShown()).toBe(false);
board = written;
await render();
expect(mounts).toBe(1);
});
it("holds the player while the board covers the film, and leaves another owner's hold alone", async () => {
board = written;
await render();
expect(player.getState().playLocked).toBe(true);
board = noBoard;
await render();
expect(player.getState().playLocked).toBe(false);
player.setState({ playLocked: true });
board = written;
await render();
board = noBoard;
await render();
expect(player.getState().playLocked).toBe(true);
});
@@ -0,0 +1,72 @@
// @vitest-environment happy-dom
import React, { act } from "react";
import { createRoot, type Root } from "react-dom/client";
import { afterEach, expect, it, vi } from "vitest";
import {
makeAdapterWindow,
makeFakeIframe,
renderTimelinePlayerHarness,
resetPlayerStore,
} from "../../player/hooks/timelinePlayerTestHarness";
import { usePlayerStore } from "../../player/store/playerStore";
import type { StoryboardBoard } from "./useStoryboardBoard";
(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
vi.mock("../../utils/studioTelemetry", () => ({ trackStudioEvent: vi.fn() }));
vi.mock("./StoryboardBoardView", () => ({
StoryboardBoardView: ({ onShowInPreview }: { onShowInPreview: (s: number | null) => void }) => (
<button data-testid="back-to-film" onClick={() => onShowInPreview(null)} />
),
}));
const board: StoryboardBoard = {
path: "boards/storyboard.html",
revision: 0,
read: { state: "loading" },
appeared: true,
};
vi.mock("./useStoryboardBoard", () => ({ useStoryboardBoard: () => board }));
const { PreviewStage } = await import("./PreviewStage");
let stage: Root | null = null;
afterEach(() => {
act(() => stage?.unmount());
stage = null;
document.body.innerHTML = "";
resetPlayerStore();
usePlayerStore.setState({ playLocked: false });
vi.restoreAllMocks();
});
const pressJ = () =>
act(() => {
window.dispatchEvent(new KeyboardEvent("keydown", { key: "j", code: "KeyJ" }));
});
it("J under the board leaves the hidden film where it was, and works again back on Preview", () => {
const frames: FrameRequestCallback[] = [];
vi.spyOn(window, "requestAnimationFrame").mockImplementation((tick) => frames.push(tick));
const { api, root } = renderTimelinePlayerHarness();
const { adapter, win } = makeAdapterWindow();
act(() => {
api.iframeRef.current = makeFakeIframe(win);
api.onIframeLoad();
usePlayerStore.setState({ timelineReady: true });
});
adapter.seek(10);
const host = document.createElement("div");
document.body.append(host);
stage = createRoot(host);
act(() => stage!.render(<PreviewStage projectId="p1">{null}</PreviewStage>));
pressJ();
for (const tick of frames.splice(0)) act(() => tick(1000));
expect(adapter.getTime()).toBe(10);
expect(frames).toHaveLength(0);
act(() => document.querySelector<HTMLButtonElement>('[data-testid="back-to-film"]')!.click());
pressJ();
expect(frames.length).toBeGreaterThan(0);
act(() => root.unmount());
});
@@ -0,0 +1,93 @@
import { useCallback, useEffect, useRef, useState, type ReactNode } from "react";
import { usePlayerStore } from "../../player";
import { trackStudioEvent } from "../../utils/studioTelemetry";
import { Tab, Tabs, TabsList, cn } from "../ui";
import { StoryboardBoardView } from "./StoryboardBoardView";
import { useStoryboardBoard } from "./useStoryboardBoard";
type StageTab = "preview" | "storyboard";
/**
* The preview panel's stage: the film, and beside it the storyboard once the project has one. With no board there
* is no strip at all. The film keeps one place in the tree whether or not a board exists, so a board appearing or
* going never remounts it, and it stays paused under the board so going back keeps its place.
*/
export function PreviewStage({ projectId, children }: { projectId: string; children: ReactNode }) {
const board = useStoryboardBoard(projectId);
const [tab, setTab] = useState<StageTab>("preview");
const [dotted, setDotted] = useState(false);
const shownRevision = useRef(board.revision);
const tabRef = useRef(tab);
tabRef.current = tab;
const show = useCallback((next: StageTab, action?: string) => {
if (next === tabRef.current) return;
if (next === "storyboard") usePlayerStore.getState().requestPlayback(false);
trackStudioEvent("storyboard_tab", {
action: action ?? (next === "storyboard" ? "open" : "close"),
});
setTab(next);
}, []);
// A board that appears while Studio is open is the agent's stop: show it, as Desktop does.
useEffect(() => {
if (board.appeared) show("storyboard", "shown_on_write");
}, [board.appeared, show]);
// The film is hidden under the board: the player's own hold refuses a play from anywhere (J, Space, a request)
// before it starts. Another owner's hold is left as it is.
useEffect(() => {
if (tab !== "storyboard" || usePlayerStore.getState().playLocked) return;
usePlayerStore.getState().setPlayLocked(true);
return () => usePlayerStore.getState().setPlayLocked(false);
}, [tab]);
useEffect(() => {
if (tab === "storyboard") {
shownRevision.current = board.revision;
setDotted(false);
} else if (board.revision !== shownRevision.current) setDotted(true);
}, [tab, board.revision]);
useEffect(() => {
if (!board.path) setTab("preview");
}, [board.path]);
const path = board.path;
const onBoard = path !== null && tab === "storyboard";
return (
<div className="flex h-full w-full min-w-0 flex-col">
{path !== null && (
<Tabs value={tab} onValueChange={(value) => show(value as StageTab)}>
<div className="flex h-9 shrink-0 items-center border-b border-border-strong bg-surface px-2">
<TabsList aria-label="Preview pages" className="bg-transparent p-0">
<Tab value="preview">Preview</Tab>
<Tab value="storyboard" data-testid="stage-tab-storyboard">
Storyboard
{dotted && (
<span
className="ml-1.5 inline-block h-1.5 w-1.5 rounded-full bg-accent"
aria-label=", updated"
/>
)}
</Tab>
</TabsList>
</div>
</Tabs>
)}
<div className={cn("relative min-h-0 flex-1", onBoard && "hidden")}>{children}</div>
{onBoard && (
<div className="min-h-0 flex-1">
<StoryboardBoardView
projectId={projectId}
board={{ ...board, path }}
onShowInPreview={(seconds) => {
setTab("preview");
if (seconds !== null) usePlayerStore.getState().requestSeek(seconds);
}}
/>
</div>
)}
</div>
);
}
@@ -0,0 +1,322 @@
import { useCallback, useEffect, useState } from "react";
import { trackStudioEvent } from "../../utils/studioTelemetry";
import { OpenInDesktopButton } from "../OpenInDesktopButton";
import { ApproveBoardButton } from "./ApproveBoardButton";
import { Button, cn } from "../ui";
import { groupByAct, stillPath, type BoardFrame } from "./storyboardBoard";
import { projectFileUrl, type StoryboardBoard } from "./useStoryboardBoard";
interface Props {
projectId: string;
board: StoryboardBoard & { path: string };
/** Back to the film, at a moment of it. */
onShowInPreview: (seconds: number | null) => void;
}
/** A frame's start in seconds, from its "a to b" range; null when it has none. */
export const frameStart = (time: string): number | null => {
const at = /(\d+(?:\.\d+)?)/.exec(time)?.[1];
return at === undefined ? null : Number(at);
};
/** What the board area shows: the page as written, a wait, or the frames; and why, when it is not the frames. */
export function boardMode(
read: StoryboardBoard["read"],
asWritten: boolean,
): { mode: "page" | "loading" | "frames"; frames: BoardFrame[]; reason: string | null } {
if (read.state === "failed")
return { mode: "page", frames: [], reason: "Couldn't read the page, shown as written" };
if (read.state === "loading") return { mode: "loading", frames: [], reason: null };
if (!read.frames.length)
return { mode: "page", frames: [], reason: "No frames found, shown as written" };
return { mode: asWritten ? "page" : "frames", frames: read.frames, reason: null };
}
/**
* The storyboard, drawn by Studio from the frames the agent's page holds: a board of stills, and one frame at a time.
* Notes, drawings and the go are Desktop's (they go to its agent), so the bar hands the project there.
*/
export function StoryboardBoardView({ projectId, board, onShowInPreview }: Props) {
const [asWritten, setAsWritten] = useState(false);
const { mode, frames, reason } = boardMode(board.read, asWritten);
const still = (frame: BoardFrame): string | null => {
const path = stillPath(board.path, frame.still);
return path ? projectFileUrl(projectId, path, board.revision) : null;
};
return (
<div
className="flex h-full w-full min-w-0 flex-col bg-neutral-950"
data-testid="storyboard-board"
>
<BoardBar
path={board.path}
count={frames.length}
asWritten={mode === "page"}
canToggle={reason === null && mode !== "loading"}
onToggle={() => {
trackStudioEvent("storyboard_action", { action: asWritten ? "board" : "as_written" });
setAsWritten((v) => !v);
}}
reason={reason}
/>
<div className="min-h-0 flex-1 overflow-auto">
{mode === "page" && (
<PageAsWritten url={projectFileUrl(projectId, board.path, board.revision)} />
)}
{mode === "loading" && (
<p className="p-6 text-step-11 text-text-4">Reading the storyboard…</p>
)}
{mode === "frames" && (
<FramesBody frames={frames} still={still} onShowInPreview={onShowInPreview} />
)}
</div>
</div>
);
}
/** The board of stills, or one frame of it. */
function FramesBody({
frames,
still,
onShowInPreview,
}: {
frames: BoardFrame[];
still: (frame: BoardFrame) => string | null;
onShowInPreview: (seconds: number | null) => void;
}) {
const [open, setOpen] = useState<number | null>(null);
useEffect(() => {
if (open !== null && open >= frames.length) setOpen(frames.length ? frames.length - 1 : null);
}, [frames.length, open]);
const frame = open === null ? undefined : frames[open];
if (!frame || open === null)
return (
<BoardGrid
frames={frames}
still={still}
onOpen={(i) => {
trackStudioEvent("storyboard_action", { action: "open_frame" });
setOpen(i);
}}
/>
);
return (
<OpenFrame
frame={frame}
index={open}
count={frames.length}
src={still(frame)}
onStep={(to) => setOpen(Math.max(0, Math.min(frames.length - 1, to)))}
onClose={() => setOpen(null)}
onShowInPreview={() => {
trackStudioEvent("storyboard_action", { action: "show_in_preview" });
onShowInPreview(frameStart(frame.time));
}}
/>
);
}
function BoardBar({
path,
count,
asWritten,
canToggle,
onToggle,
reason,
}: {
path: string;
count: number;
asWritten: boolean;
canToggle: boolean;
onToggle: () => void;
reason: string | null;
}) {
return (
<div className="flex min-h-10 shrink-0 flex-wrap items-center justify-between gap-x-4 gap-y-1 border-b border-border-strong bg-surface px-3 py-1">
<div className="flex min-w-0 items-center gap-2 text-step-11">
<span className="font-semibold text-text-0">Storyboard</span>
{count > 0 && !asWritten && <span className="text-text-3">{count} frames</span>}
<span className="truncate text-text-4" title={path}>
{reason ?? path}
</span>
{canToggle && (
<Button variant="ghost" size="sm" onClick={onToggle}>
{asWritten ? "Show as board" : "Show page as written"}
</Button>
)}
</div>
<div className="flex items-center gap-2">
<span className="text-step-10 text-text-3 max-[1100px]:hidden">
Approve to have your agent build it. Comments on frames are in the desktop app
</span>
<ApproveBoardButton />
<OpenInDesktopButton placement="storyboard" />
</div>
</div>
);
}
function BoardGrid({
frames,
still,
onOpen,
}: {
frames: BoardFrame[];
still: (frame: BoardFrame) => string | null;
onOpen: (index: number) => void;
}) {
let index = 0;
return (
<div className="flex flex-col gap-6 p-4">
{groupByAct(frames).map((group, g) => (
<section key={`${g}:${group.act}`} className="flex flex-col gap-3">
{group.act && (
<h3 className="text-step-10 font-semibold tracking-wide text-accent uppercase">
{group.act}
</h3>
)}
<div className="grid gap-3 [grid-template-columns:repeat(auto-fill,minmax(220px,1fr))]">
{group.frames.map((frame) => {
const at = index++;
const src = still(frame);
return (
<button
key={`${at}:${frame.number}`}
type="button"
onClick={() => onOpen(at)}
className="group flex flex-col overflow-hidden rounded-lg border border-border-strong bg-neutral-900 text-left outline-hidden hover:border-accent focus-visible:border-accent"
aria-label={`Frame ${frame.number}: ${frame.title}`}
>
<div className="flex aspect-video w-full items-center justify-center bg-black">
{src ? (
<img
src={src}
alt=""
className="h-full w-full object-contain"
loading="lazy"
/>
) : (
<span className="text-step-11 text-text-4">Not sketched yet</span>
)}
</div>
<div className="flex flex-col gap-0.5 p-2.5">
<span className="truncate text-step-11 font-semibold text-text-0">
{frame.number}: {frame.title || "Untitled"}
</span>
{frame.time && (
<span className="text-step-10 tabular-nums text-text-3">{frame.time}</span>
)}
{frame.words && (
<span className="line-clamp-2 text-step-10 text-text-2">{frame.words}</span>
)}
</div>
</button>
);
})}
</div>
</section>
))}
</div>
);
}
function OpenFrame({
frame,
index,
count,
src,
onStep,
onClose,
onShowInPreview,
}: {
frame: BoardFrame;
index: number;
count: number;
src: string | null;
onStep: (to: number) => void;
onClose: () => void;
onShowInPreview: () => void;
}) {
const onKey = useCallback(
(e: KeyboardEvent) => {
const target = e.target as HTMLElement | null;
if (target?.closest("input, textarea, [contenteditable='true']")) return;
if (e.key === "ArrowLeft") onStep(index - 1);
else if (e.key === "ArrowRight") onStep(index + 1);
else if (e.key === "Escape") onClose();
else return;
e.preventDefault();
e.stopPropagation();
},
[index, onStep, onClose],
);
useEffect(() => {
window.addEventListener("keydown", onKey, true);
return () => window.removeEventListener("keydown", onKey, true);
}, [onKey]);
return (
<div className="flex h-full min-h-0 flex-col gap-3 p-4">
<div className="flex items-center justify-between gap-2">
<Button variant="ghost" size="sm" onClick={onClose}>
← Whole board
</Button>
<div className="flex items-center gap-1">
<Button
variant="ghost"
size="sm"
disabled={index === 0}
onClick={() => onStep(index - 1)}
>
Back
</Button>
<span className="px-1 text-step-11 tabular-nums text-text-2">
{index + 1} / {count}
</span>
<Button
variant="ghost"
size="sm"
disabled={index >= count - 1}
onClick={() => onStep(index + 1)}
>
Next
</Button>
</div>
<Button variant="ghost" size="sm" onClick={onShowInPreview}>
Show in preview
</Button>
</div>
<div className="flex min-h-0 flex-1 items-center justify-center rounded-lg bg-black">
{src ? (
<img src={src} alt="" className="max-h-full max-w-full object-contain" />
) : (
<span className="text-step-12 text-text-4">Not sketched yet</span>
)}
</div>
<div className="flex flex-col gap-1">
<span className="text-step-12 font-semibold text-text-0">
{frame.number}: {frame.title || "Untitled"}
{frame.time && (
<span className="ml-2 font-normal tabular-nums text-text-3">{frame.time}</span>
)}
</span>
{frame.words && <p className="text-step-11 text-text-2">{frame.words}</p>}
</div>
</div>
);
}
/** The agent's page itself, its scripts off: what a board Studio cannot read into frames still looks like. */
function PageAsWritten({ url }: { url: string }) {
return (
<iframe
key={url}
title="Storyboard page"
src={url}
sandbox=""
referrerPolicy="no-referrer"
className={cn("h-full w-full border-0 bg-white")}
/>
);
}
@@ -0,0 +1,183 @@
// @vitest-environment happy-dom
import { describe, expect, it } from "vitest";
import {
findBoardFile,
groupByAct,
readBoardFrames,
stillPath,
touchesBoard,
} from "./storyboardBoard";
import { boardMode, frameStart } from "./StoryboardBoardView";
const page = (body: string) => new DOMParser().parseFromString(`<body>${body}</body>`, "text/html");
describe("findBoardFile", () => {
it("finds the board in boards/, whatever its case", () => {
expect(findBoardFile(["index.html", "boards/StoryBoard.html"])).toBe("boards/StoryBoard.html");
});
it("prefers a root copy over the one in boards/, as Desktop does", () => {
expect(findBoardFile(["boards/storyboard.html", "storyboard.html"])).toBe("storyboard.html");
});
it("does not take a board from any other folder", () => {
expect(findBoardFile(["old/boards/storyboard.html", "compositions/storyboard.html"])).toBe(
null,
);
});
});
describe("touchesBoard", () => {
it("counts the page, a still beside it and the plan, and nothing else", () => {
const board = "boards/storyboard.html";
expect(touchesBoard("boards/storyboard.html", board)).toBe(true);
expect(touchesBoard("boards/stills/frame-03-at-9.4s.png", board)).toBe(true);
expect(touchesBoard("STORYBOARD.md", board)).toBe(true);
expect(touchesBoard("index.html", board)).toBe(false);
expect(touchesBoard("compositions/s1.html", board)).toBe(false);
});
it("does not take every project write as the board's when the board is at the root", () => {
expect(touchesBoard("index.html", "storyboard.html")).toBe(false);
});
it("counts a still the board draws from outside its own folder: a root board, or ../assets", () => {
const rootStills = new Set([stillPath("storyboard.html", "stills/frame-01.png")!]);
expect(touchesBoard("stills/frame-01.png", "storyboard.html", rootStills)).toBe(true);
expect(touchesBoard("stills/frame-02.png", "storyboard.html", rootStills)).toBe(false);
expect(touchesBoard("index.html", "storyboard.html", rootStills)).toBe(false);
const board = "boards/storyboard.html";
const upStills = new Set([stillPath(board, "../assets/a.png")!]);
expect(touchesBoard("assets/a.png", board, upStills)).toBe(true);
expect(touchesBoard("assets/b.png", board, upStills)).toBe(false);
});
});
describe("stillPath", () => {
it("resolves a still against the page's own folder", () => {
expect(stillPath("boards/storyboard.html", "stills/frame-00-at-1.5s.png")).toBe(
"boards/stills/frame-00-at-1.5s.png",
);
expect(stillPath("boards/storyboard.html", "../assets/a%20b.png?v=2")).toBe("assets/a b.png");
});
it("refuses a still outside the project or on another host", () => {
expect(stillPath("boards/storyboard.html", "../../etc/passwd")).toBe(null);
expect(stillPath("boards/storyboard.html", "https://example.com/a.png")).toBe(null);
expect(stillPath("boards/storyboard.html", "data:image/png;base64,AAAA")).toBe(null);
expect(stillPath("boards/storyboard.html", "")).toBe(null);
});
});
describe("readBoardFrames", () => {
it("reads the recipe's page: act, number, title, time, still and words", () => {
const frames = readBoardFrames(
page(`
<section class="act">Act 1: Cold open</section>
<div class="frame">
<img src="stills/frame-00-at-1.5s.png" alt="" />
<div class="title">01: Three exits</div>
<div class="time">0.0 to 5.6s</div>
<p class="words">Badges 1, 2, 3 land on the words.</p>
</div>
<section class="act">Act 2: The filing</section>
<div class="frame">
<img src="stills/frame-01-at-6s.png" alt="" />
<div class="title">02: One goes to OpenAI</div>
<div class="time">5.6–9.4s</div>
</div>`),
);
expect(frames).toEqual([
{
number: "01",
title: "Three exits",
act: "Act 1: Cold open",
time: "0.0 to 5.6s",
still: "stills/frame-00-at-1.5s.png",
words: "Badges 1, 2, 3 land on the words.",
},
{
number: "02",
title: "One goes to OpenAI",
act: "Act 2: The filing",
time: "5.6 to 9.4s",
still: "stills/frame-01-at-6s.png",
words: "",
},
]);
});
it("numbers frames by place when the page gives none, and keeps a nested frame inside its parent", () => {
const frames = readBoardFrames(
page(`
<div class="frame"><div class="title">The hook</div><div class="frame"><img src="a.png"></div></div>
<div class="frame"><div class="title">The turn</div></div>`),
);
expect(frames.map((f) => [f.number, f.title])).toEqual([
["01", "The hook"],
["02", "The turn"],
]);
});
it("finds no frames on a page that marks them another way", () => {
expect(
readBoardFrames(page(`<div class="cell" id="frame-01"><img src="a.png"></div>`)),
).toEqual([]);
});
});
describe("groupByAct", () => {
it("keeps film order and starts a group at each act change", () => {
const frame = (act: string, number: string) => ({
act,
number,
title: "",
time: "",
still: "",
words: "",
});
const groups = groupByAct([
frame("", "01"),
frame("A", "02"),
frame("A", "03"),
frame("B", "04"),
]);
expect(groups.map((g) => [g.act, g.frames.map((f) => f.number)])).toEqual([
["", ["01"]],
["A", ["02", "03"]],
["B", ["04"]],
]);
});
});
describe("frameStart", () => {
it("reads a frame's start from its range", () => {
expect(frameStart("9.4 to 15.7s")).toBe(9.4);
expect(frameStart("")).toBe(null);
});
});
describe("boardMode", () => {
const frame = { number: "01", title: "A", act: "", time: "", still: "", words: "" };
it("shows the page as written, with the reason, when it holds no frames or cannot be read", () => {
expect(boardMode({ state: "read", frames: [] }, false)).toMatchObject({
mode: "page",
reason: "No frames found, shown as written",
});
expect(boardMode({ state: "failed" }, false)).toMatchObject({
mode: "page",
reason: "Couldn't read the page, shown as written",
});
});
it("shows the frames, or the page when the person asks for it", () => {
expect(boardMode({ state: "read", frames: [frame] }, false)).toEqual({
mode: "frames",
frames: [frame],
reason: null,
});
expect(boardMode({ state: "read", frames: [frame] }, true).mode).toBe("page");
});
});
@@ -0,0 +1,148 @@
// Read the way HyperFrames Desktop's Storyboard tab reads the page, so a board one shows is a board the other shows.
/** Where the page may sit, in the order a copy wins: the project root, then boards/. */
const BOARD_DIRS = ["", "boards/"];
const BOARD_FILE = "storyboard.html";
const PLAN_FILE = "storyboard.md";
export function findBoardFile(files: readonly string[]): string | null {
for (const dir of BOARD_DIRS) {
const found = files.find((path) => path.toLowerCase() === dir + BOARD_FILE);
if (found) return found;
}
return null;
}
const folderOf = (path: string): string => path.slice(0, path.lastIndexOf("/") + 1);
/** Whether a write to `changed` changes what the board at `board` shows: the page, anything beside it, a still
* it draws (`stills`, as project paths), or the plan. */
export function touchesBoard(
changed: string,
board: string,
stills: ReadonlySet<string> = new Set(),
): boolean {
const path = changed.replace(/\\/g, "/").replace(/^\.\//, "");
if (path === board || path.toLowerCase() === PLAN_FILE || stills.has(path)) return true;
const folder = folderOf(board);
return folder !== "" && path.startsWith(folder);
}
/** A still's project path, from its `src` as the page wrote it; null for one that is not a project file. */
export function stillPath(board: string, src: string): string | null {
if (!src || /^[a-z][a-z0-9+.-]*:/i.test(src) || src.startsWith("//")) return null;
const clean = src.split(/[?#]/)[0] ?? "";
const parts = (clean.startsWith("/") ? clean.slice(1) : folderOf(board) + clean).split("/");
const out: string[] = [];
for (const part of parts) {
if (part === "" || part === ".") continue;
if (part === "..") {
if (!out.length) return null;
out.pop();
} else out.push(part);
}
return out.length ? out.map(safeDecode).join("/") : null;
}
const safeDecode = (part: string): string => {
try {
return decodeURIComponent(part);
} catch {
return part;
}
};
export interface BoardFrame {
number: string;
title: string;
act: string;
/** Its range as "a to b" ("0.0 to 3.0s"), or "". */
time: string;
still: string;
words: string;
}
const TIME = /\d+(?:[.:]\d+)?\s*s?\s*[–-]\s*\d+(?:[.:]\d+)?\s*s?/;
// "01 — The dial", "00a · Cold open — the line": a frame's number, then its name.
const NUMBERED = /^\s*(\d{1,3}[a-z]?)\s*[—–·.:-]\s*(.+)$/;
const NOTE_PLACES = [".desc", ".description", ".caption", ".note", ".words", "p"];
const text = (el: Element | null | undefined): string =>
(el?.textContent ?? "").replace(/\s+/g, " ").trim();
const tidy = (words: string): string =>
words
.replace(TIME, "")
.replace(/[\s·—–:-]+$/, "")
.trim();
const pad = (n: number): string => String(n).padStart(2, "0");
/** The `class="frame"` boxes the page draws, in order; a frame nested in another is part of it. */
export function readBoardFrames(doc: Document): BoardFrame[] {
const boxes = [...doc.querySelectorAll<HTMLElement>(".frame")].filter(
(el) => !el.parentElement?.closest(".frame"),
);
return boxes.map((el, i) => frameInfo(doc, el, i));
}
/** The frame's own box: its highest ancestor holding no other frame (a scene card, a contact-sheet cell). */
function ownBox(el: HTMLElement): HTMLElement {
let box = el;
while (box.parentElement && box.parentElement !== el.ownerDocument.body) {
if (box.parentElement.querySelectorAll(".frame").length > 1) break;
box = box.parentElement;
}
return box;
}
function frameInfo(doc: Document, el: HTMLElement, i: number): BoardFrame {
const box = ownBox(el);
const heading = box.querySelector(".scene-title, .label, [class*='title']");
const first = heading?.firstElementChild ? text(heading.firstElementChild) : text(heading);
const named = NUMBERED.exec(tidy(first));
return {
number: named?.[1] ?? badgeNumber(box) ?? pad(i + 1),
title: named?.[2]?.trim() ?? tidy(first),
act: actBefore(doc, el),
time: timeOf(box, heading),
still: el.querySelector("img")?.getAttribute("src") ?? "",
// First by this order, not the page's: a .words box can hold the title and time before its .note.
words: text(NOTE_PLACES.map((place) => box.querySelector(place)).find(Boolean)),
};
}
function timeOf(box: HTMLElement, heading: Element | null): string {
const raw = text(box.querySelector(".time")) || (text(heading).match(TIME)?.[0] ?? "");
return raw.replace(/\s*[\u2013-]\s*/, " to ");
}
function badgeNumber(box: HTMLElement): string | undefined {
const badge = /\d+/.exec(text(box.querySelector(".scene-badge")))?.[0];
return badge === undefined ? undefined : pad(Number(badge));
}
/** The act heading last before the frame, by its own words (not the times beside them). */
function actBefore(doc: Document, el: HTMLElement): string {
const acts = [...doc.querySelectorAll(".actbar, .act, [class*='act-bar']")].filter(
(act) => act.compareDocumentPosition(el) & Node.DOCUMENT_POSITION_FOLLOWING,
);
const act = acts.at(-1);
if (!act) return "";
const own = [...act.childNodes]
.filter((node) => node.nodeType === Node.TEXT_NODE)
.map((node) => node.textContent)
.join(" ")
.replace(/\s+/g, " ")
.trim();
return tidy(own || text(act));
}
/** The frames in their acts, in film order; frames before any act heading share one untitled group. */
export function groupByAct(frames: readonly BoardFrame[]): { act: string; frames: BoardFrame[] }[] {
const groups: { act: string; frames: BoardFrame[] }[] = [];
for (const frame of frames) {
const last = groups.at(-1);
if (last && last.act === frame.act) last.frames.push(frame);
else groups.push({ act: frame.act, frames: [frame] });
}
return groups;
}
@@ -0,0 +1,99 @@
import { useEffect, useRef, useState } from "react";
import { encodeUrlPath } from "@hyperframes/parsers";
import { useFileManagerContextOptional } from "../../contexts/FileManagerContext";
import { onExternalFileChange } from "../../hooks/fileChangeFanout";
import { buildProjectApiPath } from "../../utils/projectRouting";
import { studioApiFetch } from "../../utils/studioApiFetch";
import {
findBoardFile,
readBoardFrames,
stillPath,
touchesBoard,
type BoardFrame,
} from "./storyboardBoard";
export type BoardRead =
| { state: "loading" }
| { state: "read"; frames: BoardFrame[] }
| { state: "failed" };
export interface StoryboardBoard {
/** The board's project path, or null while the project has none. */
path: string | null;
/** Bumps on every write that changes what the board shows; key a still or an iframe on it to reload it. */
revision: number;
read: BoardRead;
/** True once a board appears after the first file list: the agent just wrote it and is waiting on a go. */
appeared: boolean;
}
/** A project file's URL on the preview server. */
export const projectFileUrl = (projectId: string, path: string, revision = 0): string =>
`${buildProjectApiPath(projectId, `/preview/${encodeUrlPath(path)}`)}${revision ? `?v=${revision}` : ""}`;
function changeOf(payload: unknown): { path: string; projectId?: string } | null {
const { path, projectId } = (payload ?? {}) as { path?: unknown; projectId?: unknown };
if (typeof path !== "string") return null;
return { path, projectId: typeof projectId === "string" ? projectId : undefined };
}
const isBoardName = (path: string): boolean => /(^|[\\/])storyboard\.html$/i.test(path);
/** The project's storyboard page, read into frames, and reread on every write that changes it. */
export function useStoryboardBoard(projectId: string): StoryboardBoard {
const files = useFileManagerContextOptional();
const fileTree = files?.fileTree;
const loaded = files?.fileTreeLoaded === true;
const path = fileTree ? findBoardFile(fileTree) : null;
const [revision, setRevision] = useState(0);
const [read, setRead] = useState<BoardRead>({ state: "loading" });
const [appeared, setAppeared] = useState(false);
const firstListing = useRef<boolean | null>(null);
if (loaded && firstListing.current === null) firstListing.current = path !== null;
useEffect(() => {
if (path && firstListing.current === false) setAppeared(true);
if (!path) setAppeared(false);
}, [path]);
const pathRef = useRef(path);
pathRef.current = path;
const stillsRef = useRef<ReadonlySet<string>>(new Set());
const refreshRef = useRef(files?.refreshFileTree);
refreshRef.current = files?.refreshFileTree;
useEffect(() => {
return onExternalFileChange((payload) => {
const change = changeOf(payload);
if (!change || (change.projectId && change.projectId !== projectId)) return;
const board = pathRef.current;
if (board && touchesBoard(change.path, board, stillsRef.current)) setRevision((n) => n + 1);
// A board written or removed changes the file list the tab is found in.
if (isBoardName(change.path)) void refreshRef.current?.();
});
}, [projectId]);
useEffect(() => {
if (!path) return;
let live = true;
setRead((prev) => (prev.state === "read" ? prev : { state: "loading" }));
studioApiFetch(projectFileUrl(projectId, path, revision), { cache: "no-store" })
.then((res) => (res.ok ? res.text() : Promise.reject(new Error(String(res.status)))))
.then((html) => {
if (!live) return;
// DOMParser builds an inert document: the page's scripts never run here.
const doc = new DOMParser().parseFromString(html, "text/html");
const frames = readBoardFrames(doc);
stillsRef.current = new Set(
frames.map((frame) => stillPath(path, frame.still)).filter((still) => still !== null),
);
setRead({ state: "read", frames });
})
.catch(() => live && setRead({ state: "failed" }));
return () => {
live = false;
};
}, [projectId, path, revision]);
return { path, revision, read, appeared };
}
@@ -0,0 +1,20 @@
import { describe, expect, it, vi } from "vitest";
import { announceExternalFileChange, onExternalFileChange } from "./fileChangeFanout";
describe("fileChangeFanout", () => {
it("hands every announced change to each listener until it unsubscribes", () => {
const first = vi.fn();
const second = vi.fn();
const stopFirst = onExternalFileChange(first);
const stopSecond = onExternalFileChange(second);
announceExternalFileChange({ path: "boards/storyboard.html" });
stopFirst();
announceExternalFileChange({ path: "index.html" });
stopSecond();
expect(first.mock.calls).toEqual([[{ path: "boards/storyboard.html" }]]);
expect(second.mock.calls).toEqual([
[{ path: "boards/storyboard.html" }],
[{ path: "index.html" }],
]);
});
});
@@ -0,0 +1,13 @@
type FileChangeListener = (payload: unknown) => void;
const listeners = new Set<FileChangeListener>();
/** Hears every decoded external file change the coordinator receives, without a transport of its own. */
export function onExternalFileChange(listener: FileChangeListener): () => void {
listeners.add(listener);
return () => listeners.delete(listener);
}
export function announceExternalFileChange(payload: unknown): void {
for (const listener of listeners) listener(payload);
}
@@ -1,3 +1,4 @@
import { announceExternalFileChange } from "./fileChangeFanout";
import { useCallback, useEffect, useRef, useState, type MutableRefObject } from "react";
import {
mergeFileChangePayloads,
@@ -477,7 +478,11 @@ export function useExternalFileChangeCoordinator({
processChangeRef.current = processChange;
useEffect(() => {
// One decoder for all three transports; the rungs only choose the channel.
const handler = (delivery?: unknown) => processChangeRef.current(decodeFileChange(delivery));
const handler = (delivery?: unknown) => {
const payload = decodeFileChange(delivery);
announceExternalFileChange(payload);
processChangeRef.current(payload);
};
const adapter = testHotAdapter();
if (adapter) {
adapter.on("hf:file-change", handler);
+1 -1
View File
@@ -4,7 +4,7 @@
"unpacked": 290508
},
"@hyperframes/cli": {
"packed": 7902966,
"packed": 7982002,
"unpacked": 25159107
},
"@hyperframes/core": {
+2 -2
View File
@@ -6,7 +6,7 @@
"files": 2
},
"hyperframes": {
"hash": "fcbd52e834332599",
"hash": "17e33f68596da873",
"files": 78
},
"hyperframes-cli": {
@@ -34,7 +34,7 @@
"files": 2
},
"style-sampling": {
"hash": "5d26c6211fdfc062",
"hash": "dcd3765bed6b5c77",
"files": 2
}
}
+1 -1
View File
@@ -4,7 +4,7 @@ Use it whenever a run makes or changes a storyboard: when the person asks for on
A film's shots are planned in the SHOTS line of `SPEC.md` (`spec.md`); a storyboard turns those shots into stills when the person wants to approve them first, one frame per shot, carrying its `reads` and `sound` (`storyboard-format.md`).
The person sees the plan as stills on the board, comments or draws on a frame, and says go. The board is `boards/storyboard.html`, and tools that show it find its frames by the classes in § 2; boxes marked any other way (`cell` boxes, `id="frame-NN"` markers) are not found.
The person sees the plan as stills on the board, comments or draws on a frame, and says go. The board is `boards/storyboard.html`, and tools that show it find its frames by the classes in § 2; boxes marked any other way (`cell` boxes, `id="frame-NN"` markers) are not found. HyperFrames Desktop and the Studio of `npx hyperframes preview` show the board in a Storyboard tab beside the preview and bring it to the front when it is written, so hand it over as that tab: with no preview running, start one and give its link, never the file's path. In that Studio, Approve copies "The storyboard is approved. Build the video from it." for the person to paste to you: that is their go.
## 1. The run
+1 -1
View File
@@ -33,7 +33,7 @@ The person's message carries what it carries. Ask once, together, only what is s
- Facts first, per REMIX.md: every number, date and quote with its source. A research sub-agent is fine.
- Write the script to voice.md and the series' script anatomy; run the script checks REMIX.md names.
- **Stop 1:** pose the script and the shot plan as stills on the board, written as the hyperframes skill's `references/storyboard.md` says: one still per beat, its line under it, what each shot shows and which component makes it. Never pay for a voice to get a board. A series kit builds its film from the voice's timing, so first give it estimated timing with the free route REMIX.md names for stop 1 (most kits: `python3 tools/voice.py --silent`, silent stand-ins at the series' pace), assemble, and pose in the `index.html` that builds, as `storyboard.md` says. A drawn series poses in the test shot its REMIX.md names instead. Every time on the board is an estimate until the voice exists; say so. Point the person to `boards/storyboard.html` and wait for their go.
- **Stop 1:** pose the script and the shot plan as stills on the board, written as the hyperframes skill's `references/storyboard.md` says: one still per beat, its line under it, what each shot shows and which component makes it. Never pay for a voice to get a board. A series kit builds its film from the voice's timing, so first give it estimated timing with the free route REMIX.md names for stop 1 (most kits: `python3 tools/voice.py --silent`, silent stand-ins at the series' pace), assemble, and pose in the `index.html` that builds, as `storyboard.md` says. A drawn series poses in the test shot its REMIX.md names instead. Every time on the board is an estimate until the voice exists; say so. Tell the person the board is in the Storyboard tab beside the preview, as `storyboard.md` says, and wait for their go.
## 5. Voice and face