From f5bb3e5519bf9a76ac1d770234fb65fe7f152832 Mon Sep 17 00:00:00 2001 From: Michael Wang Date: Tue, 17 Nov 2026 05:12:31 +0000 Subject: [PATCH] fix: a received-files record follows Studio renames, moves and deletes, and a nested record can't be moved out of reach (#7989) * fix(cli): a received folder's record is read wherever Studio moves it The search for received-files records now walks every folder a render, preview or publish can load files from, including renders/, dist/, .next/, coverage/ and Studio's own folders in the project's .hyperframes/. A received folder moved there by Studio keeps its files received. Studio's folder delete and move also refuse a folder holding a received-files record that lists a file outside its own folder, since that file would stay behind with no record. A received folder whose record lists only its own files can still be deleted or moved. * fix(cli): a renamed or moved received file stays listed under its new name The CLI-hosted Studio now follows every rename, move or delete it completes in the received-files records, whichever client sent it. A renamed file, or every file in a moved folder, stays listed under its new path; a file moved out of a received folder is listed in the project's own record. Changes are made under the record's lock, and a record that cannot be changed is left as it was. The per-file readers used by publish, cloud render, feedback and add now count every file as received when a record lists a file that is no longer on disk, since that file may be there under another name. This matches the local render check. Projects with no record are unaffected. * fix(cli): received content that leaves the record still counts as received When a Studio rename, move or delete takes a received file off a listed path, the sha256 of its content is kept in .hyperframes/received-content.json, the file and format the desktop app uses; other keys in it are kept. A file holding the same bytes counts as received for publish, cloud render, feedback, add and the local render check, so content brought back by undo or renamed outside Studio stays received. Studio's own backups do not count. A listed file that is gone from disk now makes only pages, stylesheets, scripts, SVG and links out of the project count as received, and the upload warning names the gone file. The record follows a request spelled in another letter case or Unicode form by matching the name the disk holds, and waits for the record's lock without blocking the server. Studio's folder delete and move no longer refuse a folder whose record lists a file outside it; that file's content is kept as received instead. * fix(cli): a folder delete or move keeps what a record inside it lists elsewhere Before Studio deletes or moves a folder, every file that a received-files record inside the folder lists outside it is listed in the project's own record by its path from the project folder. Such a file stays received after it is edited, and a listed link stays received too. When the project's record cannot be written, the delete or move is refused. Received content that cannot be read is never written over, so the desktop app's keys in it are kept. The warning about a listed file that is gone now says how to clear it. * fix(cli): received-content checks skip large non-code files and name a skipped search A file larger than 16 MB that is not a page, stylesheet, script or SVG is no longer read to compare its content with received content, so a large archive is not read on every check. When the project has too many folders to search for received-files records, the upload warning now says so. The case-matching publish test now expects the gone-file rule on a disk that tells letter case apart, and a comment no longer names a file path. * fix(cli): name a gone received file from the project folder on every system The gone-file note now names the file from the same project path the record search starts at. On Windows, a short (8.3) folder name in the temp path made the two spellings differ, so the note showed a long relative path instead of the file's name. Content hashing opens a file once, checks it through that handle and reads from the same handle, and the received-content file is read before it is classified, so nothing is checked by path and then used by path. * fix(cli): received files stay received through undo, saves, and partial folder deletes An undo, redo or restore lists again every file it writes that holds received content, and a Studio save lists such a file before it changes it. So a Studio delete followed by an undo and an edit leaves the file received. A Studio rename, move or delete now records the received side first, under the record's lock, and is refused when that fails: the content leaving listed paths is kept, a renamed file is listed under its new path, and every line of a record inside a moved or deleted folder is copied into the project's record. Once the files have moved, only lines whose file was there and whose content was kept are taken off. Lines at a rename's destination and lines whose file was already gone stay. The search for records inside the folder has no size limit, and nothing is locked when nothing is listed. * fix(cli): unreadable received content counts every file as received The readers treat a received-content file they cannot read the way they treat an unreadable received-files record: every file counts as received, and the local render check names the file it could not read. A record line that points outside the project is ignored, so a record in a received folder cannot make the readers look at or report files outside the project. * fix(cli): the CLI's own history undo and restore keep received files listed `hyperframes history undo` and `restore` with no Studio running now list again every file they write that holds received content, the same way Studio's history routes do. A delete in Studio, then an undo or restore from the CLI, then an edit, leaves the file received. A Studio edit that names its file in the request body, such as a freeze-frame edit, now lists that file first when it holds received content. --- .../cli/src/server/followReceivedRecords.ts | 155 ++++++ .../studioServer.receivedRecord.test.ts | 484 ++++++++++++++++++ packages/cli/src/server/studioServer.ts | 2 + packages/cli/src/utils/historyOwner.ts | 20 +- .../cli/src/utils/privateProjectFile.test.ts | 93 +++- packages/cli/src/utils/privateProjectFile.ts | 264 +++++++--- .../utils/publishProject.absolutePath.test.ts | 36 +- packages/cli/src/utils/publishProject.ts | 25 +- .../cli/src/utils/receivedContent.test.ts | 25 + packages/cli/src/utils/receivedContent.ts | 98 ++++ .../cli/src/utils/receivedRecordFollow.ts | 212 ++++++++ packages/cli/src/utils/receivedRender.test.ts | 53 ++ packages/cli/src/utils/receivedRender.ts | 51 +- .../src/routes/files.pathSafety.test.ts | 47 ++ 14 files changed, 1493 insertions(+), 72 deletions(-) create mode 100644 packages/cli/src/server/followReceivedRecords.ts create mode 100644 packages/cli/src/server/studioServer.receivedRecord.test.ts create mode 100644 packages/cli/src/utils/receivedContent.test.ts create mode 100644 packages/cli/src/utils/receivedContent.ts create mode 100644 packages/cli/src/utils/receivedRecordFollow.ts diff --git a/packages/cli/src/server/followReceivedRecords.ts b/packages/cli/src/server/followReceivedRecords.ts new file mode 100644 index 0000000000..e3c853f592 --- /dev/null +++ b/packages/cli/src/server/followReceivedRecords.ts @@ -0,0 +1,155 @@ +import { existsSync, statSync } from "node:fs"; +import type { Context, Hono } from "hono"; +import { requestSubPath } from "@hyperframes/studio-server"; +import { resolveWritableWithinProject } from "@hyperframes/studio-server/safe-path"; +import { + dropReceivedArrivals, + dropReceivedLeavers, + keepListedInFolder, + onDiskPath, + plannedReceivedChange, + type ReceivedChange, + recordReceivedArrivals, + relistHistoryWrites, + relistReceivedContent, +} from "../utils/receivedRecordFollow.js"; + +const isObject = (value: unknown): value is Record => + value !== null && typeof value === "object" && !Array.isArray(value); + +const textAt = (value: unknown, key: string): string[] => + isObject(value) && typeof value[key] === "string" ? [value[key]] : []; + +const listAt = (value: unknown, key: string): unknown[] => + isObject(value) && Array.isArray(value[key]) ? value[key] : []; + +const warn = (error: unknown) => + console.warn(`[studio] Could not update the received-files record: ${String(error)}`); + +const busy = (c: Context) => + c.json( + { error: "Could not update the received-files record; try again.", why: "record_busy" }, + 409, + ); + +const jsonBody = (c: Context): Promise => + c.req.raw + .clone() + .json() + .catch(() => null); + +/** A project file the request may change, spelled as it is on disk; null when the route will refuse the path. */ +function writablePath(projectDir: string, path: string): string | null { + const found = path ? resolveWritableWithinProject(projectDir, path) : null; + return found && existsSync(found) ? onDiskPath(found) : null; +} + +const isFile = (path: string) => statSync(path, { throwIfNoEntry: false })?.isFile() === true; + +async function requestedChange( + c: Context, + projectDir: string, +): Promise<{ from: string; to?: string } | null> { + const from = writablePath(projectDir, requestSubPath(c.req.url, "projects/:id/files")); + if (from === null) return null; + if (c.req.method !== "PATCH") return { from }; + const newPath = textAt(await jsonBody(c), "newPath")[0]; + const to = newPath ? resolveWritableWithinProject(projectDir, newPath) : null; + return to === null ? null : { from, to: onDiskPath(to) }; +} + +async function startedChange( + projectDir: string, + { from, to }: { from: string; to?: string }, +): Promise { + if (!(await keepListedInFolder(projectDir, from))) return null; + return plannedReceivedChange(projectDir, from, to) + .then(recordReceivedArrivals) + .catch(() => null); +} + +/** + * Follows a Studio rename, move or delete in the received-files records. Before the files move, every file listed in + * a record inside a moved or deleted folder is listed in the project's own record, the received content leaving a + * listed path is kept, and a renamed file is listed under its new path; when that fails, the request is refused. + * Once the files have moved, their old lines go. + */ +function followMovesAndDeletes(app: Hono, projectDir: string): void { + app.on(["PATCH", "DELETE"], "/api/projects/:id/files/*", async (c, next) => { + const requested = await requestedChange(c, projectDir); + if (requested === null) return next(); + const change = await startedChange(projectDir, requested); + if (change === null) return busy(c); + await next(); + await (c.res.ok ? dropReceivedLeavers(change) : dropReceivedArrivals(change)).catch(warn); + }); +} + +/** Undo, redo and restore: every file they write that holds received content is listed again. */ +function followHistory(app: Hono, projectDir: string): void { + const routes = ["step", "undo", "restore"].map((name) => `/api/projects/:id/history/${name}`); + app.on("POST", routes, async (c, next) => { + await next(); + if (!c.res.ok) return; + const sent: unknown = await c.res + .clone() + .json() + .catch(() => null); + const entry = isObject(sent) && isObject(sent.entry) ? sent.entry : sent; + const files = listAt(entry, "files").flatMap((file) => + textAt(file, "path").map((path) => ({ path })), + ); + await relistHistoryWrites(projectDir, { files }).catch(warn); + }); +} + +const SAVES_NAMING_THE_FILE = [ + "files", + "file-mutations/:operation", + "gsap-mutations", + "gsap-mutations-batch", + "gsap-mutation-rollback", +]; + +async function savedPaths(c: Context): Promise { + const route = SAVES_NAMING_THE_FILE.find((prefix) => + c.req.routePath.startsWith(`/api/projects/:id/${prefix}/`), + ); + const named = route === undefined ? "" : requestSubPath(c.req.url, `projects/:id/${route}`); + if (named) return [named]; + const body = await jsonBody(c); + return [ + ...textAt(body, "path"), + ...listAt(body, "files").flatMap((file) => textAt(file, "path")), + ...listAt(body, "batches").flatMap((batch) => textAt(batch, "sourceFile")), + ]; +} + +/** + * A Studio save to a file that holds received content no record lists lists that file first, so the edit leaves it + * received; when the record cannot take it, the save is refused. + */ +function followSaves(app: Hono, projectDir: string): void { + const relistBeforeSave = async (c: Context, next: () => Promise) => { + const paths = (await savedPaths(c)).flatMap((path) => writablePath(projectDir, path) ?? []); + try { + await relistReceivedContent(projectDir, paths.filter(isFile)); + } catch { + return busy(c); + } + return next(); + }; + app.put("/api/projects/:id/files/*", relistBeforeSave); + app.on( + "POST", + SAVES_NAMING_THE_FILE.slice(1).map((prefix) => `/api/projects/:id/${prefix}/*`), + relistBeforeSave, + ); +} + +/** Keeps the project's received-files records true to Studio's file requests, whichever client sent them. */ +export function followReceivedRecords(app: Hono, projectDir: string): void { + followMovesAndDeletes(app, projectDir); + followHistory(app, projectDir); + followSaves(app, projectDir); +} diff --git a/packages/cli/src/server/studioServer.receivedRecord.test.ts b/packages/cli/src/server/studioServer.receivedRecord.test.ts new file mode 100644 index 0000000000..7956aa7774 --- /dev/null +++ b/packages/cli/src/server/studioServer.receivedRecord.test.ts @@ -0,0 +1,484 @@ +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + readdirSync, + readFileSync, + rmSync, + symlinkSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { fileContentVersion } from "@hyperframes/studio-server"; +import { historyDeps, withOwner } from "../utils/historyOwner.js"; +import { receivedFiles } from "../utils/privateProjectFile.js"; +import { contentHash } from "../utils/receivedContent.js"; +import { outsideAssetsAllowed } from "../utils/receivedRender.js"; +import { createStudioServer, type StudioServer } from "./studioServer.js"; +import { cleanupStudioServerRoot, makeStudioServerRoot } from "./studioServerTestFixture.js"; + +const walk = vi.hoisted(() => ({ exhausted: false })); + +vi.mock("../utils/walkBudget.js", async (importOriginal) => { + const real = await importOriginal(); + return { walkBudget: () => (walk.exhausted ? () => false : real.walkBudget()) }; +}); + +let root: string; +let server: StudioServer | undefined; + +afterEach(() => { + walk.exhausted = false; + if (server) { + chmodAll(root); + cleanupStudioServerRoot(server, root); + } + server = undefined; +}); + +function chmodAll(dir: string): void { + for (const entry of readdirSync(dir, { withFileTypes: true })) + if (entry.isDirectory() && !entry.isSymbolicLink()) { + chmodSync(join(dir, entry.name), 0o755); + chmodAll(join(dir, entry.name)); + } +} + +function caseInsensitiveDisk(): boolean { + const probe = mkdtempSync(join(tmpdir(), "hf-case-probe-")); + try { + return existsSync(probe.toUpperCase()) && existsSync(probe.toLowerCase()); + } finally { + rmSync(probe, { recursive: true, force: true }); + } +} + +function write(path: string, text: string | Buffer): void { + mkdirSync(join(path, ".."), { recursive: true }); + writeFileSync(path, text); +} + +function studio(): { projectDir: string; app: StudioServer["app"] } { + const fixture = makeStudioServerRoot("hf-received-record-"); + root = fixture.root; + write(join(root, "outside", "pic.png"), "x"); + server = createStudioServer({ + projectDir: fixture.projectDir, + projectName: "film", + historyRoot: join(root, "history"), + }); + return { projectDir: fixture.projectDir, app: server.app }; +} + +const fileUrl = (path: string) => `/api/projects/film/files/${encodeURIComponent(path)}`; +const move = (app: StudioServer["app"], from: string, newPath: string) => + app.request(fileUrl(from), { + method: "PATCH", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ newPath }), + }); +const allowed = (projectDir: string) => outsideAssetsAllowed(projectDir, "index.html", () => {}); + +describe("a received folder with its own record, in a project with none", () => { + function withReceivedFolder(projectDir: string): void { + write(join(projectDir, "index.html"), `
`); + write(join(projectDir, "sub", "scene.html"), ``); + write( + join(projectDir, "sub", ".hyperframes", "received-files.json"), + JSON.stringify({ files: ["scene.html"] }), + ); + } + + it.each([["dist"], ["renders/sub"], [".hyperframes/requests/sub"]])( + "keeps its files received when Studio moves it to %s", + async (to) => { + const { projectDir, app } = studio(); + withReceivedFolder(projectDir); + expect(allowed(projectDir)).toBe(false); + expect((await move(app, "sub", to)).status).toBe(200); + expect(receivedFiles(projectDir)(`${to}/scene.html`)).toBe(true); + expect(allowed(projectDir)).toBe(false); + }, + ); + + it("takes its received files along when Studio deletes it, keeping their content", async () => { + const { projectDir, app } = studio(); + withReceivedFolder(projectDir); + const scene = contentHash(join(projectDir, "sub", "scene.html")); + expect((await app.request(fileUrl("sub"), { method: "DELETE" })).status).toBe(200); + expect(existsSync(join(projectDir, "sub"))).toBe(false); + const own = join(projectDir, ".hyperframes"); + expect(JSON.parse(readFileSync(join(own, "received-files.json"), "utf-8")).files).toEqual([]); + const content = JSON.parse(readFileSync(join(own, "received-content.json"), "utf-8")); + expect(content.hashes).toEqual([scene]); + const received = receivedFiles(projectDir); + expect(received.gone).toBeUndefined(); + expect(received("index.html")).toBe(false); + }); + + it.skipIf(process.platform === "win32" || process.getuid?.() === 0)( + "keeps its files listed when a folder delete stops partway", + async () => { + const { projectDir, app } = studio(); + withReceivedFolder(projectDir); + write(join(projectDir, "sub", "locked", "keep.txt"), "x"); + chmodSync(join(projectDir, "sub", "locked"), 0o555); + const deleted = await app.request(fileUrl("sub"), { method: "DELETE" }); + expect(deleted.ok).toBe(false); + const listed = JSON.parse( + readFileSync(join(projectDir, ".hyperframes", "received-files.json"), "utf-8"), + ).files; + expect(listed).toContain("sub/scene.html"); + expect(receivedFiles(projectDir)("sub/scene.html")).toBe(true); + }, + ); + + function withOutsideEntry(projectDir: string, entry: string): void { + withReceivedFolder(projectDir); + rmSync(join(projectDir, "sub", ".hyperframes"), { recursive: true }); + write( + join(projectDir, "sub", "notes", ".hyperframes", "received-files.json"), + JSON.stringify({ files: [entry] }), + ); + } + + it("keeps a file its record lists outside the folder received after the folder is deleted and the file edited", async () => { + const { projectDir, app } = studio(); + withOutsideEntry(projectDir, "../scene.html"); + expect((await app.request(fileUrl("sub/notes"), { method: "DELETE" })).status).toBe(200); + write(join(projectDir, "sub", "scene.html"), `

edited

`); + expect(receivedFiles(projectDir)("sub/scene.html")).toBe(true); + expect(allowed(projectDir)).toBe(false); + }); + + it.skipIf(process.platform === "win32")( + "keeps a listed link out of the project received after its record's folder is deleted", + async () => { + const { projectDir, app } = studio(); + withOutsideEntry(projectDir, "../link.png"); + symlinkSync(join(root, "outside", "pic.png"), join(projectDir, "sub", "link.png")); + expect((await app.request(fileUrl("sub/notes"), { method: "DELETE" })).status).toBe(200); + expect(receivedFiles(projectDir)("sub/link.png")).toBe(true); + }, + ); + + it("deletes a folder past the project search's size limit, keeping what its record lists outside it", async () => { + const { projectDir, app } = studio(); + withOutsideEntry(projectDir, "../scene.html"); + walk.exhausted = true; + expect((await app.request(fileUrl("sub/notes"), { method: "DELETE" })).status).toBe(200); + walk.exhausted = false; + write(join(projectDir, "sub", "scene.html"), "

edited

"); + expect(receivedFiles(projectDir)("sub/scene.html")).toBe(true); + }); + + it("refuses a folder delete when the project's record cannot take the files listed outside it", async () => { + const { projectDir, app } = studio(); + withOutsideEntry(projectDir, "../scene.html"); + write(join(projectDir, ".hyperframes", "received-files.json.lock"), ""); + expect((await app.request(fileUrl("sub/notes"), { method: "DELETE" })).status).toBe(409); + expect(existsSync(join(projectDir, "sub", "notes", ".hyperframes"))).toBe(true); + expect(receivedFiles(projectDir)("sub/scene.html")).toBe(true); + }); +}); + +describe("a Studio rename, move or delete in a project with a received-files record", () => { + const record = (folder: string) => join(folder, ".hyperframes", "received-files.json"); + const listed = (folder: string) => JSON.parse(readFileSync(record(folder), "utf-8")).files; + function withRecord(projectDir: string): void { + write(join(projectDir, "index.html"), `
`); + write(join(projectDir, "scenes", "intro.html"), ``); + write(join(projectDir, "scenes", "logo.png"), "x"); + write(record(projectDir), JSON.stringify({ files: ["scenes/intro.html", "scenes/logo.png"] })); + } + + it("keeps a renamed received file received under its new name", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + expect((await move(app, "scenes/intro.html", "scenes/opening.html")).status).toBe(200); + expect(listed(projectDir)).toEqual(["scenes/logo.png", "scenes/opening.html"]); + const received = receivedFiles(projectDir); + expect(received("scenes/opening.html")).toBe(true); + expect(received("index.html")).toBe(false); + }); + + it("keeps every received file in a moved folder received under the new folder", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + expect((await move(app, "scenes", "parts/scenes")).status).toBe(200); + expect(listed(projectDir)).toEqual(["parts/scenes/intro.html", "parts/scenes/logo.png"]); + expect(receivedFiles(projectDir)("parts/scenes/intro.html")).toBe(true); + expect(receivedFiles(projectDir)("index.html")).toBe(false); + }); + + it("takes a deleted received file off the record, so the person's own files stay theirs", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + const deleted = await app.request(fileUrl("scenes/intro.html"), { method: "DELETE" }); + expect(deleted.status).toBe(200); + expect(listed(projectDir)).toEqual(["scenes/logo.png"]); + expect(receivedFiles(projectDir)("index.html")).toBe(false); + }); + + it("counts a deleted received file as received again when its content comes back", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + const before = readFileSync(join(projectDir, "scenes", "intro.html")); + expect((await app.request(fileUrl("scenes/intro.html"), { method: "DELETE" })).status).toBe( + 200, + ); + const content = JSON.parse( + readFileSync(join(projectDir, ".hyperframes", "received-content.json"), "utf-8"), + ); + expect(content.hashes).toHaveLength(1); + writeFileSync(join(projectDir, "scenes", "intro.html"), before); + writeFileSync(join(projectDir, "restored.html"), before); + const received = receivedFiles(projectDir); + expect(received("scenes/intro.html")).toBe(true); + expect(received("restored.html")).toBe(true); + expect(received("index.html")).toBe(false); + expect(allowed(projectDir)).toBe(false); + }); + + it("never writes over received content it cannot read, and keeps the deleted file listed", async () => { + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + const { projectDir, app } = studio(); + withRecord(projectDir); + write(join(projectDir, ".hyperframes", "received-content.json"), "{not json"); + expect((await app.request(fileUrl("scenes/intro.html"), { method: "DELETE" })).status).toBe( + 200, + ); + expect(readFileSync(join(projectDir, ".hyperframes", "received-content.json"), "utf-8")).toBe( + "{not json", + ); + expect(listed(projectDir)).toContain("scenes/intro.html"); + warn.mockRestore(); + }); + + it("keeps the desktop app's own keys in the received content", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + const desktop = { hashes: [], unlisted: { "a.html": 1 }, copies: { "b.html": [] } }; + write(join(projectDir, ".hyperframes", "received-content.json"), JSON.stringify(desktop)); + expect((await app.request(fileUrl("scenes/intro.html"), { method: "DELETE" })).status).toBe( + 200, + ); + const content = JSON.parse( + readFileSync(join(projectDir, ".hyperframes", "received-content.json"), "utf-8"), + ); + expect(content).toMatchObject({ unlisted: desktop.unlisted, copies: desktop.copies }); + expect(content.hashes).toHaveLength(1); + }); + + it.runIf(caseInsensitiveDisk())( + "follows a rename the request spells in another letter case", + async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + expect((await move(app, "Scenes/INTRO.html", "scenes/opening.html")).status).toBe(200); + expect(listed(projectDir)).toEqual(["scenes/logo.png", "scenes/opening.html"]); + }, + ); + + it("answers other requests while it waits for the record's lock, then refuses the rename", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + writeFileSync(`${record(projectDir)}.lock`, ""); + const renaming = move(app, "scenes/intro.html", "scenes/opening.html"); + const started = Date.now(); + await new Promise((resolve) => setTimeout(resolve, 200)); + await app.request(fileUrl("index.html")); + expect(Date.now() - started).toBeLessThan(1_000); + expect((await renaming).status).toBe(409); + expect(existsSync(join(projectDir, "scenes", "intro.html"))).toBe(true); + expect(existsSync(join(projectDir, "scenes", "opening.html"))).toBe(false); + expect(listed(projectDir)).toEqual(["scenes/intro.html", "scenes/logo.png"]); + }); + + it("renames a file no record lists while the record is busy", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + writeFileSync(`${record(projectDir)}.lock`, ""); + expect((await move(app, "index.html", "main.html")).status).toBe(200); + }); + + it("keeps a file moved out of a received folder received in the project's own record", async () => { + const { projectDir, app } = studio(); + write(join(projectDir, "sub", "page.html"), "
"); + write(join(projectDir, "sub", "scene.html"), "
"); + write(record(join(projectDir, "sub")), JSON.stringify({ files: ["page.html", "scene.html"] })); + expect((await move(app, "sub/page.html", "page.html")).status).toBe(200); + expect((await move(app, "sub/scene.html", "sub/opening.html")).status).toBe(200); + expect(listed(projectDir)).toEqual(["page.html"]); + expect(listed(join(projectDir, "sub"))).toEqual(["opening.html"]); + const received = receivedFiles(projectDir); + expect(received("page.html")).toBe(true); + expect(received("sub/opening.html")).toBe(true); + }); + + it("keeps a received file received through an edit after a delete and an undo", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + expect((await app.request("/api/projects/film/history")).status).toBe(200); + expect((await app.request(fileUrl("scenes/intro.html"), { method: "DELETE" })).status).toBe( + 200, + ); + expect(listed(projectDir)).toEqual(["scenes/logo.png"]); + const restored = await app.request("/api/projects/film/history/restore", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ point: "start" }), + }); + expect(restored.status).toBe(200); + expect(existsSync(join(projectDir, "scenes", "intro.html"))).toBe(true); + write(join(projectDir, "scenes", "intro.html"), `

2

`); + expect(receivedFiles(projectDir)("scenes/intro.html")).toBe(true); + expect(allowed(projectDir)).toBe(false); + }); + + it("lists a file holding received content again before a Studio save changes it", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + const intro = join(projectDir, "scenes", "intro.html"); + const before = readFileSync(intro); + expect((await app.request(fileUrl("scenes/intro.html"), { method: "DELETE" })).status).toBe( + 200, + ); + writeFileSync(intro, before); + const saved = await app.request(fileUrl("scenes/intro.html"), { + method: "PUT", + headers: { "If-Match": fileContentVersion(before) }, + body: `

2

`, + }); + expect(saved.status).toBe(200); + expect(listed(projectDir)).toContain("scenes/intro.html"); + expect(receivedFiles(projectDir)("scenes/intro.html")).toBe(true); + const own = readFileSync(join(projectDir, "index.html")); + const ownSave = await app.request(fileUrl("index.html"), { + method: "PUT", + headers: { "If-Match": fileContentVersion(own) }, + body: "

mine

", + }); + expect(ownSave.status).toBe(200); + expect(listed(projectDir)).not.toContain("index.html"); + }); + + it.each([ + ["freeze-frame", { path: "scenes/intro.html" }], + ["split-batch", { files: [{ path: "scenes/intro.html" }] }], + ["patch-element-batches", { batches: [{ sourceFile: "scenes/intro.html" }] }], + ])( + "lists a file holding received content again before a %s edit names it", + async (operation, body) => { + const { projectDir, app } = studio(); + withRecord(projectDir); + const intro = join(projectDir, "scenes", "intro.html"); + const before = readFileSync(intro); + expect((await app.request(fileUrl("scenes/intro.html"), { method: "DELETE" })).status).toBe( + 200, + ); + writeFileSync(intro, before); + await app.request(`/api/projects/film/file-mutations/${operation}`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + }); + expect(listed(projectDir)).toContain("scenes/intro.html"); + }, + ); + + it("refuses a Studio save to a file holding received content when the record is busy", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + const intro = join(projectDir, "scenes", "intro.html"); + const before = readFileSync(intro); + expect((await app.request(fileUrl("scenes/intro.html"), { method: "DELETE" })).status).toBe( + 200, + ); + writeFileSync(intro, before); + writeFileSync(`${record(projectDir)}.lock`, ""); + const saved = await app.request(fileUrl("scenes/intro.html"), { + method: "PUT", + headers: { "If-Match": fileContentVersion(before) }, + body: "

2

", + }); + expect(saved.status).toBe(409); + expect(readFileSync(intro)).toEqual(before); + }); + + it("keeps a received file received through an edit after a delete and Cmd+Z", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + expect((await app.request("/api/projects/film/history")).status).toBe(200); + expect((await app.request(fileUrl("scenes/intro.html"), { method: "DELETE" })).status).toBe( + 200, + ); + const post = (path: string, body: object) => + app.request(`/api/projects/film/history/${path}`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + }); + expect((await post("checkpoint", {})).status).toBe(200); + const stepped = await post("step", { direction: "back" }); + expect(await stepped.json()).toMatchObject({ ok: true, entry: { files: expect.any(Array) } }); + expect(existsSync(join(projectDir, "scenes", "intro.html"))).toBe(true); + write(join(projectDir, "scenes", "intro.html"), "

edited

"); + expect(receivedFiles(projectDir)("scenes/intro.html")).toBe(true); + }); + + it("keeps a received file received through an edit after a delete and an undo from the CLI with Studio closed", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + expect((await app.request("/api/projects/film/history")).status).toBe(200); + expect((await app.request(fileUrl("scenes/intro.html"), { method: "DELETE" })).status).toBe( + 200, + ); + expect(listed(projectDir)).toEqual(["scenes/logo.png"]); + await server?.shutdown(); + const saved = { ...historyDeps }; + Object.assign(historyDeps, { + historyRoot: join(root, "history"), + findServer: async () => null, + }); + try { + const restored = await withOwner(projectDir, (owner) => + owner.restore("start", { kind: "agent", name: "Agent" }), + ); + expect(restored).not.toBeNull(); + } finally { + Object.assign(historyDeps, saved); + } + write(join(projectDir, "scenes", "intro.html"), `

2

`); + expect(receivedFiles(projectDir)("scenes/intro.html")).toBe(true); + expect(allowed(projectDir)).toBe(false); + }); + + it("keeps a line whose file is already gone when Studio deletes or renames onto its folder", async () => { + const { projectDir, app } = studio(); + withRecord(projectDir); + write( + record(projectDir), + JSON.stringify({ files: ["scenes/intro.html", "scenes/old.html", "parts/gone.html"] }), + ); + expect((await app.request(fileUrl("scenes"), { method: "DELETE" })).status).toBe(200); + expect(listed(projectDir)).toEqual(["parts/gone.html", "scenes/old.html"]); + write(join(projectDir, "draft.html"), "

"); + expect((await move(app, "draft.html", "parts/gone.html")).status).toBe(200); + expect(listed(projectDir)).toEqual(["parts/gone.html", "scenes/old.html"]); + expect(receivedFiles(projectDir).gone).toBe("scenes/old.html"); + }); + + it("moves a line its record spells in decomposed Unicode", async () => { + const { projectDir, app } = studio(); + const folder = "cafe\u0301"; + write(join(projectDir, folder, "intro.html"), "

"); + write(record(projectDir), JSON.stringify({ files: [`${folder}/intro.html`] })); + expect((await move(app, folder, `parts/${folder}`)).status).toBe(200); + expect(listed(projectDir)).toEqual(["parts/caf\u00e9/intro.html"]); + expect(receivedFiles(projectDir)("parts/caf\u00e9/intro.html")).toBe(true); + }); +}); diff --git a/packages/cli/src/server/studioServer.ts b/packages/cli/src/server/studioServer.ts index 3c26949487..34e2f86459 100644 --- a/packages/cli/src/server/studioServer.ts +++ b/packages/cli/src/server/studioServer.ts @@ -38,6 +38,7 @@ import { import { emitStudioRenderComplete, emitStudioRenderError } from "./studioRenderTelemetry.js"; import { startRenderCpuSampler } from "../telemetry/renderCpu.js"; import { mountDesktopRoutes } from "./desktopRoutes.js"; +import { followReceivedRecords } from "./followReceivedRecords.js"; import { hostHeaderAllowed, PREVIEW_ALLOWED_HOSTS_ENV, @@ -1157,6 +1158,7 @@ export function createStudioServer(options: StudioServerOptions): StudioServer { mountDesktopRoutes(app, projectDir); mountShareRoutes(app, projectDir); + followReceivedRecords(app, projectDir); // ── Pre-flight checks for render ──────────────────────────────────────── // Intercept render requests before they reach the shared API so we can diff --git a/packages/cli/src/utils/historyOwner.ts b/packages/cli/src/utils/historyOwner.ts index 3b560fec00..721835bad5 100644 --- a/packages/cli/src/utils/historyOwner.ts +++ b/packages/cli/src/utils/historyOwner.ts @@ -211,6 +211,14 @@ function previewOwner(route: (path: string) => string): Owner { }; } +async function relistWrites(projectDir: string, entry: HistoryEntry | null): Promise { + if (!entry?.files.length) return; + const { relistHistoryWrites } = await import("./receivedRecordFollow.js"); + await relistHistoryWrites(projectDir, entry).catch((error: unknown) => + console.warn(`Could not update the received-files record: ${String(error)}`), + ); +} + async function directOwner( projectDir: string, ownerWaitMs = historyDeps.ownerWaitMs, @@ -243,8 +251,16 @@ async function directOwner( list: async () => history.list(), peek: async (point) => history.peek(point), blob: (hash) => history.readBlob(hash), - undo: (id, who, mode) => history.undo(id, { who, ...(mode && { mode }) }), - restore: (point, who) => history.restore(point, who), + undo: async (id, who, mode) => { + const undone = await history.undo(id, { who, ...(mode && { mode }) }); + if (undone.ok) await relistWrites(projectDir, undone.entry); + return undone; + }, + restore: async (point, who) => { + const restored = await history.restore(point, who); + await relistWrites(projectDir, restored); + return restored; + }, pin: async (id, pinned) => history.pin(id, pinned), // Opening filed every earlier write; the turn's own writes are filed to it when the next open passes it. begin: async () => ({ id: randomUUID(), startedAt: historyDeps.now() }), diff --git a/packages/cli/src/utils/privateProjectFile.test.ts b/packages/cli/src/utils/privateProjectFile.test.ts index b8c968aa15..acf3ff00a9 100644 --- a/packages/cli/src/utils/privateProjectFile.test.ts +++ b/packages/cli/src/utils/privateProjectFile.test.ts @@ -10,6 +10,7 @@ import { writeFileSync, } from "node:fs"; import { spawn } from "node:child_process"; +import { createHash } from "node:crypto"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { isCredentialOrToolStatePath } from "@hyperframes/core/safe-path"; @@ -26,7 +27,7 @@ vi.mock("node:fs", async (importOriginal) => { const actual = await importOriginal(); const lstatSync = ((path: string, options?: { bigint?: boolean }) => { const stat = actual.lstatSync(path, options as never); - return zeroFileIds.on ? Object.assign(stat, { dev: 0n, ino: 0n }) : stat; + return zeroFileIds.on && stat ? Object.assign(stat, { dev: 0n, ino: 0n }) : stat; }) as typeof actual.lstatSync; return { ...actual, default: { ...actual, lstatSync }, lstatSync }; }); @@ -137,6 +138,77 @@ describe("receivedFiles", () => { }); }); +describe("receivedFiles when a listed file is gone", () => { + function project(listed: string[]): string { + dir = mkdtempSync(join(tmpdir(), "hf-received-gone-")); + mkdirSync(join(dir, ".hyperframes")); + writeFileSync(join(dir, "mine.html"), ""); + writeFileSync( + join(dir, ".hyperframes", "received-files.json"), + JSON.stringify({ files: listed }), + ); + return dir; + } + + it("counts every page, stylesheet, script and SVG as received, and names the gone file", () => { + const root = project(["theirs.html"]); + writeFileSync(join(root, "logo.png"), "png"); + const received = receivedFiles(root); + for (const code of ["mine.html", "renamed.html", "a.css", "b.js", "c.mjs", "d.svg"]) + expect(received(code)).toBe(true); + expect(received("logo.png")).toBe(false); + expect(received.gone).toBe("theirs.html"); + }); + + it("counts a file with the content of a received file that left the record as received", () => { + const root = project([]); + writeFileSync(join(root, "logo.png"), "theirs"); + writeFileSync( + join(root, ".hyperframes", "received-content.json"), + JSON.stringify({ hashes: [createHash("sha256").update("theirs").digest("hex")] }), + ); + const received = receivedFiles(root); + expect(received("logo.png")).toBe(true); + expect(received("mine.html")).toBe(false); + expect(received.gone).toBeUndefined(); + }); + + it("counts every file as received when the received content cannot be read", () => { + const root = project([]); + writeFileSync(join(root, ".hyperframes", "received-content.json"), "{not json"); + expect(receivedFiles(root)("mine.html")).toBe(true); + }); + + it("ignores a gone file a record lists outside the project", () => { + const root = project([]); + mkdirSync(join(root, "sub", ".hyperframes"), { recursive: true }); + writeFileSync( + join(root, "sub", ".hyperframes", "received-files.json"), + JSON.stringify({ files: ["../../not-in-project/gone.html"] }), + ); + const received = receivedFiles(root); + expect(received.gone).toBeUndefined(); + expect(received("mine.html")).toBe(false); + }); + + it("counts every page as received for a gone file listed by a received folder's own record", () => { + const root = project([]); + mkdirSync(join(root, "sub", ".hyperframes"), { recursive: true }); + writeFileSync( + join(root, "sub", ".hyperframes", "received-files.json"), + JSON.stringify({ files: ["gone.html"] }), + ); + expect(receivedFiles(root)("mine.html")).toBe(true); + }); + + it("still tells the person's own files apart when every listed file is there", () => { + const root = project(["theirs.html"]); + writeFileSync(join(root, "theirs.html"), ""); + expect(receivedFiles(root)("theirs.html")).toBe(true); + expect(receivedFiles(root)("mine.html")).toBe(false); + }); +}); + describe("receivedFiles for a received project copied into a folder", () => { function withNested(record?: string): string { dir = mkdtempSync(join(tmpdir(), "hf-received-nested-")); @@ -165,6 +237,25 @@ describe("receivedFiles for a received project copied into a folder", () => { expect(received("compositions/x/index.html")).toBe(false); expect(received("index.html")).toBe(false); }); + + it.each([ + "renders", + "dist", + ".next", + "coverage", + "renders/x", + ".hyperframes/requests/x", + ".hyperframes/prepared-assets/x", + ])("names the copy's files when it sits in %s", (folder) => { + dir = mkdtempSync(join(tmpdir(), "hf-received-nested-")); + mkdirSync(join(dir, folder, ".hyperframes"), { recursive: true }); + writeFileSync(join(dir, folder, "index.html"), ""); + writeFileSync( + join(dir, folder, ".hyperframes", "received-files.json"), + JSON.stringify({ files: ["index.html"] }), + ); + expect(receivedFiles(dir)(`${folder}/index.html`)).toBe(true); + }); }); describe("recordReceivedFiles", () => { diff --git a/packages/cli/src/utils/privateProjectFile.ts b/packages/cli/src/utils/privateProjectFile.ts index 36fbc0d204..bc0264312c 100644 --- a/packages/cli/src/utils/privateProjectFile.ts +++ b/packages/cli/src/utils/privateProjectFile.ts @@ -1,5 +1,6 @@ import { closeSync, + type Dirent, lstatSync, openSync, readdirSync, @@ -8,6 +9,7 @@ import { rmSync, statSync, } from "node:fs"; +import { readdir } from "node:fs/promises"; import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; import { isEnvOrGitPath, @@ -15,10 +17,22 @@ import { linksOutside, } from "@hyperframes/studio-server/outside-file"; import { isPrivateProjectFile, realFilePath } from "@hyperframes/studio-server/safe-path"; -import { isCredentialOrToolStatePath } from "@hyperframes/core/safe-path"; +import { + isCredentialOrToolStatePath, + isStudioPrivatePath, + projectPathSegments, +} from "@hyperframes/core/safe-path"; import { registryRoot } from "../registry/publication.js"; import { recordFolder, writeRecord } from "./projectRecords.js"; import { walkBudget } from "./walkBudget.js"; +import { + contentHash, + isCodeFile, + readReceivedContent, + type ReceivedContent, + receivedContentFile, + writeReceivedContent, +} from "./receivedContent.js"; export { isEnvOrGitPath, isPrivateOutsideFile, linksOutside }; @@ -71,43 +85,75 @@ function sleepSync(ms: number): void { Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); } -function takeRecordLock(lock: string): void { - const deadline = Date.now() + LOCK_WAIT_MS; - for (;;) { - try { - closeSync(openSync(lock, "wx")); - return; - } catch (error) { - if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; - } - const heldFor = Date.now() - (statSync(lock, { throwIfNoEntry: false })?.mtimeMs ?? Date.now()); - if (heldFor > LOCK_STALE_MS) rmSync(lock, { force: true }); - else if (Date.now() < deadline) sleepSync(25); - else throw new Error(`Another program is changing ${RECEIVED_RECORD}; try again.`); +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +function tryRecordLock(lock: string, deadline: number): "taken" | "wait" { + try { + closeSync(openSync(lock, "wx")); + return "taken"; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; } + const heldFor = Date.now() - (statSync(lock, { throwIfNoEntry: false })?.mtimeMs ?? Date.now()); + if (heldFor > LOCK_STALE_MS) rmSync(lock, { force: true }); + else if (Date.now() >= deadline) + throw new Error(`Another program is changing ${RECEIVED_RECORD}; try again.`); + return "wait"; } -function changeReceivedRecord(projectDir: string, change: (files: Set) => boolean): void { +function takeRecordLock(lock: string): void { + const deadline = Date.now() + LOCK_WAIT_MS; + while (tryRecordLock(lock, deadline) === "wait") sleepSync(25); +} + +async function takeRecordLockAsync(lock: string): Promise { + const deadline = Date.now() + LOCK_WAIT_MS; + while (tryRecordLock(lock, deadline) === "wait") await sleep(25); +} + +type RecordChange = (files: Set, content: ReceivedContent) => boolean; + +function recordLock(projectDir: string): { root: string; lock: string } { const root = registryRoot(projectDir); if (!recordFolder(root, "")) throw new Error( `${join(root, ".hyperframes")} is a link or not a folder; remove it and try again.`, ); - const lock = join(root, RECEIVED_RECORD_LOCK); + return { root, lock: join(root, RECEIVED_RECORD_LOCK) }; +} + +function applyRecordChange(root: string, change: RecordChange): void { + const record = readRecord(root, true); + if (record && !Array.isArray(record.listed)) return; + const files = new Set(((record?.listed ?? []) as unknown[]).map(String)); + const content = readReceivedContent(root); + const known = content.hashes.size; + const changed = change(files, content); + const json = `${JSON.stringify({ files: [...files].sort() })}\n`; + if (changed && !writeRecord(root, basename(RECEIVED_RECORD), json)) + throw new Error(`Could not write ${join(root, RECEIVED_RECORD)}.`); + if (content.hashes.size > known && !writeReceivedContent(root, content)) + throw new Error(`Could not write the received content beside ${join(root, RECEIVED_RECORD)}.`); +} + +function changeReceivedRecord(projectDir: string, change: RecordChange): void { + const { root, lock } = recordLock(projectDir); takeRecordLock(lock); try { - const record = readRecord(root, true); - if (record && !Array.isArray(record.listed)) return; - const files = new Set(((record?.listed ?? []) as unknown[]).map(String)); - if (!change(files)) return; - if ( - !writeRecord( - root, - basename(RECEIVED_RECORD), - `${JSON.stringify({ files: [...files].sort() })}\n`, - ) - ) - throw new Error(`Could not write ${join(root, RECEIVED_RECORD)}.`); + applyRecordChange(root, change); + } finally { + rmSync(lock, { force: true }); + } +} + +export async function changeReceivedRecordAsync( + projectDir: string, + change: RecordChange, +): Promise { + const { root, lock } = recordLock(projectDir); + await takeRecordLockAsync(lock); + try { + applyRecordChange(root, change); } finally { rmSync(lock, { force: true }); } @@ -131,6 +177,32 @@ export function forgetReceivedFiles(root: string, paths: string[]): void { changeReceivedRecord(root, (files) => paths.map((path) => files.delete(path)).some(Boolean)); } +export const receivedLinesAtOrAbove = (dir: string): { folder: string; lines: string[] }[] => + receivedRecordsAtOrAbove(dir).flatMap((record) => + Array.isArray(record.listed) ? [{ folder: record.root, lines: record.listed.map(String) }] : [], + ); + +const listedPaths = (record: ReceivedRecord): string[] => + Array.isArray(record.listed) + ? record.listed.map((entry) => join(record.root, String(entry))) + : []; + +const isAtOrUnder = (path: string, folder: string) => { + const inside = relative(folder, path); + return ( + inside === "" || (inside !== ".." && !inside.startsWith(`..${sep}`) && !isAbsolute(inside)) + ); +}; + +export async function filesListedInFolder(target: string): Promise { + if (!statSync(target, { throwIfNoEntry: false })?.isDirectory()) return []; + const own = readRecord(target, true); + const below = await receivedRecordsBelowAsync(target); + if (below === null) return null; + const records = own ? [nestedRecord(target, target, own), ...below] : below; + return records.flatMap(listedPaths); +} + function receivedRecordsAtOrAbove(start: string): ReceivedRecord[] { const records: ReceivedRecord[] = []; for (let dir = start; ; ) { @@ -151,20 +223,26 @@ function receivedRecordsAtOrAbove(start: string): ReceivedRecord[] { function nestedRecord(start: string, dir: string, record: { listed: unknown }): ReceivedRecord { const folder = relative(start, dir).split(sep).join("/"); const listed = Array.isArray(record.listed) - ? record.listed.map((entry) => `${folder}/${String(entry)}`) + ? record.listed.map((entry) => (folder ? `${folder}/${String(entry)}` : String(entry))) : undefined; return { file: join(dir, RECEIVED_RECORD), root: start, projectDirFromRoot: "", listed }; } -const NOT_WALKED = new Set([ - ".git", - ".hyperframes", - "node_modules", - "renders", - "dist", - ".next", - "coverage", -]); +const GIT_AND_PACKAGE_FOLDERS = new Set([".git", "node_modules"]); + +export function mayHoldLoadedFiles(start: string, folder: string): boolean { + const rel = relative(start, folder).split(sep).join("/"); + const segments = projectPathSegments(rel); + if (GIT_AND_PACKAGE_FOLDERS.has(segments.at(-1) ?? "")) return false; + if (!segments.includes(".hyperframes")) return true; + return segments.length === 1 || !isStudioPrivatePath(rel); +} + +const subfolders = (start: string, dir: string, entries: Dirent[]): string[] => + entries + .filter((entry) => entry.isDirectory()) + .map((entry) => join(dir, entry.name)) + .filter((folder) => mayHoldLoadedFiles(start, folder)); function receivedRecordsBelow(start: string): ReceivedRecord[] | null { const withinBudget = walkBudget(); @@ -173,9 +251,7 @@ function receivedRecordsBelow(start: string): ReceivedRecord[] | null { for (let dir = pending.pop(); dir !== undefined; dir = pending.pop()) { let folders: string[]; try { - folders = readdirSync(dir, { withFileTypes: true }) - .filter((entry) => entry.isDirectory() && !NOT_WALKED.has(entry.name.toLowerCase())) - .map((entry) => join(dir, entry.name)); + folders = subfolders(start, dir, readdirSync(dir, { withFileTypes: true })); } catch { return null; } @@ -187,13 +263,33 @@ function receivedRecordsBelow(start: string): ReceivedRecord[] | null { return records; } -function receivedRecords(projectDir: string): ReceivedRecord[] | null { - let start: string; - try { - start = realpathSync(projectDir); - } catch { - start = resolve(projectDir); +async function receivedRecordsBelowAsync(start: string): Promise { + const records: ReceivedRecord[] = []; + const pending = [start]; + for (let dir = pending.pop(); dir !== undefined; dir = pending.pop()) { + let folders: string[]; + try { + folders = subfolders(start, dir, await readdir(dir, { withFileTypes: true })); + } catch { + return null; + } + const record = dir === start ? null : readRecord(dir, true); + if (record) records.push(nestedRecord(start, dir, record)); + pending.push(...folders); } + return records; +} + +function walkStart(projectDir: string): string { + try { + return realpathSync(projectDir); + } catch { + return resolve(projectDir); + } +} + +function receivedRecords(projectDir: string): ReceivedRecord[] | null { + const start = walkStart(projectDir); const below = receivedRecordsBelow(start); return below && [...receivedRecordsAtOrAbove(start), ...below]; } @@ -207,7 +303,11 @@ function fileIdentity(path: string): string | null { } } -function listedIn(record: ReceivedRecord, projectDir: string): (path: string) => boolean { +function listedIn( + record: ReceivedRecord, + projectDir: string, + start: string, +): (path: string) => boolean { const listed = (record.listed as unknown[]).map((path) => String(path).normalize("NFC")); const files = new Set(listed); const keyOf = (path: string) => @@ -215,7 +315,10 @@ function listedIn(record: ReceivedRecord, projectDir: string): (path: string) => let listedOnDisk: Set | undefined; const sameFileAsListed = (path: string) => { listedOnDisk ??= new Set( - listed.flatMap((entry) => fileIdentity(join(record.root, entry)) ?? []), + listed + .map((entry) => join(record.root, entry)) + .filter((file) => isAtOrUnder(file, start)) + .flatMap((file) => fileIdentity(file) ?? []), ); const identity = fileIdentity(join(projectDir, path)); return identity !== null && listedOnDisk.has(identity); @@ -223,12 +326,57 @@ function listedIn(record: ReceivedRecord, projectDir: string): (path: string) => return (path) => files.has(keyOf(path)) || sameFileAsListed(path); } -export function receivedFiles(projectDir: string): (path: string) => boolean { +function onDisk(path: string): boolean { + try { + return lstatSync(path, { throwIfNoEntry: false }) !== undefined; + } catch { + return false; + } +} + +export type ReceivedFiles = ((path: string) => boolean) & { gone?: string; unchecked?: boolean }; + +const contentFolder = (record: ReceivedRecord) => dirname(dirname(record.file)); + +function receivedContentOf( + records: ReceivedRecord[], +): { hashes: string[] } | { unreadable: string } { + const hashes: string[] = []; + for (const folder of new Set(records.map(contentFolder))) { + const content = readReceivedContent(folder); + if (!content.readable) return { unreadable: receivedContentFile(folder) }; + hashes.push(...content.hashes); + } + return { hashes }; +} + +function holdsReceivedContent(projectDir: string, hashes: Set) { + const seen = new Map(); + return (path: string): boolean => { + if (hashes.size === 0) return false; + if (!seen.has(path)) seen.set(path, hashes.has(contentHash(join(projectDir, path)) ?? "")); + return seen.get(path) === true; + }; +} + +export function receivedFiles(projectDir: string): ReceivedFiles { const records = receivedRecords(projectDir); - if (records === null || records.some((record) => !Array.isArray(record.listed))) - return () => true; - const listedInAnyRecord = records.map((record) => listedIn(record, projectDir)); - return (path) => listedInAnyRecord.some((listed) => listed(path)); + if (records === null) return Object.assign(() => true, { unchecked: true }); + if (records.some((record) => !Array.isArray(record.listed))) return () => true; + const content = receivedContentOf(records); + if ("unreadable" in content) return () => true; + const start = walkStart(projectDir); + const listedInProject = records.flatMap(listedPaths).filter((path) => isAtOrUnder(path, start)); + const goneFile = listedInProject.find((path) => !onDisk(path)); + const mayBeGoneFile = (path: string) => + goneFile !== undefined && + (isCodeFile(path) || linksOutside(projectDir, join(projectDir, path))); + const sameContent = holdsReceivedContent(projectDir, new Set(content.hashes)); + const listedInAnyRecord = records.map((record) => listedIn(record, projectDir, start)); + const received = (path: string) => + listedInAnyRecord.some((listed) => listed(path)) || mayBeGoneFile(path) || sameContent(path); + const gone = goneFile && relative(start, goneFile).split(sep).join("/"); + return Object.assign(received, gone === undefined ? {} : { gone }); } function listedUnder(record: ReceivedRecord): string[] { @@ -253,14 +401,16 @@ function listedUnder(record: ReceivedRecord): string[] { export function receivedRecordStatus( projectDir: string, -): { path: string; listed: string[] | null } | "unchecked" | null { +): { path: string; listed: string[] | null; hashes: string[] } | "unchecked" | null { const records = receivedRecords(projectDir); if (records === null) return "unchecked"; const [nearest] = records; if (!nearest) return null; const unreadable = records.find((record) => !Array.isArray(record.listed)); - if (unreadable) return { path: unreadable.file, listed: null }; - return { path: nearest.file, listed: records.flatMap(listedUnder) }; + if (unreadable) return { path: unreadable.file, listed: null, hashes: [] }; + const content = receivedContentOf(records); + if ("unreadable" in content) return { path: content.unreadable, listed: null, hashes: [] }; + return { path: nearest.file, listed: records.flatMap(listedUnder), hashes: content.hashes }; } const leavesProject = (path: string): boolean => diff --git a/packages/cli/src/utils/publishProject.absolutePath.test.ts b/packages/cli/src/utils/publishProject.absolutePath.test.ts index 976760578f..0dd7740ff7 100644 --- a/packages/cli/src/utils/publishProject.absolutePath.test.ts +++ b/packages/cli/src/utils/publishProject.absolutePath.test.ts @@ -3,6 +3,7 @@ import { existsSync, mkdirSync, mkdtempSync, + renameSync, rmSync, symlinkSync, writeFileSync, @@ -387,6 +388,22 @@ describe("outside references in files that came in with an imported project", () ); }); + it("leaves out the outside reference of a listed file renamed outside Studio", () => { + const { root, ext, warnings } = stageReceived( + { "index.html": "", "scene.html": `` }, + { files: ["index.html", "scene.html"] }, + ); + expect(ext).toEqual([]); + expect(warnings).toHaveLength(1); + renameSync(join(root, "p", "scene.html"), join(root, "p", "opening.html")); + const after: string[] = []; + const staged = stagePublishFiles(join(root, "p"), Infinity, (w) => after.push(w)); + expect([...staged.keys()].filter((key) => key.startsWith("_ext/"))).toEqual([]); + expect(after).toEqual([ + "opening.html references ../shared/logo.png, which leads outside the project and came in with an imported project; it was left out of the upload. scene.html is listed as received but is gone, so every page, stylesheet, script and SVG counts as received. To clear this, put the file back, or take its line out of .hyperframes/received-files.json once you know where its content went.", + ]); + }); + it("still copies the same reference from a file the person wrote", () => { const { ext, warnings } = stageReceived( { @@ -513,10 +530,25 @@ describe("outside references in files that came in with an imported project", () { files: ["index.html", "Assets/a.html"] }, ); const caseInsensitive = existsSync(join(root, "p", "ASSETS", "a.html")); - expect(ext).toEqual(caseInsensitive ? [] : ["_ext/shared/logo.png"]); - expect(warnings).toHaveLength(caseInsensitive ? 1 : 0); + expect(ext).toEqual([]); + expect(warnings).toHaveLength(1); + const gone = "Assets/a.html is listed as received but is gone"; + if (caseInsensitive) expect(warnings[0]).not.toContain(gone); + else expect(warnings[0]).toContain(gone); }); + it("says so when the project has too many folders to search for records", () => { + const { root } = stageReceived({ "index.html": "" }, { files: ["index.html"] }); + writeFileSync(join(root, "p", "mine.html"), ``); + for (let i = 0; i <= 20_000; i++) + mkdirSync(join(root, "p", "many", `d${i}`), { recursive: true }); + const warnings: string[] = []; + stagePublishFiles(join(root, "p"), Infinity, (w) => warnings.push(w)); + expect(warnings).toEqual([ + "mine.html references ../shared/logo.png, which leads outside the project and came in with an imported project; it was left out of the upload. The project has too many folders to search them all for received-files records, so every file counts as received.", + ]); + }, 30_000); + it("treats every file as received when the record cannot be read", () => { const { ext, warnings } = stageReceived( { "index.html": `` }, diff --git a/packages/cli/src/utils/publishProject.ts b/packages/cli/src/utils/publishProject.ts index be8be29b13..d516c1d4ca 100644 --- a/packages/cli/src/utils/publishProject.ts +++ b/packages/cli/src/utils/publishProject.ts @@ -14,7 +14,7 @@ import { parseHTML } from "linkedom"; import AdmZip from "adm-zip"; import type { Ignore } from "ignore"; import { projectIgnore } from "@hyperframes/lint"; -import { isNonRelativeUrl, isPathInside, rewriteSrcsetUrls } from "@hyperframes/core"; +import { isNonRelativeUrl, isPathInside, rewriteSrcsetUrls, terminalText } from "@hyperframes/core"; import { cssUrlMatches, replaceMatches } from "@hyperframes/parsers/css-url-scans"; import { decodeUrlPathVariants, encodeUrlPath } from "@hyperframes/parsers"; import { buildAuthHeaders } from "../auth/client.js"; @@ -30,6 +30,7 @@ import { linksOutside, linksToPrivateOutsideFile, receivedFiles, + type ReceivedFiles, } from "./privateProjectFile.js"; import { INCLUDE_OUTSIDE_FILES_HINT, @@ -419,7 +420,7 @@ interface ExternalAssetContext { notApproved: Set; onLeftOut?: OnLeftOut; onOutsideFile?: OnOutsideFile; - received: (path: string) => boolean; + received: ReceivedFiles; walkedLeftOut: ReadonlySet; filmPages: Set | null; } @@ -477,7 +478,7 @@ function referencedFileLeftOut( ): LeftOutReason | null { if (linksToPrivateOutsideFile(ctx.absProjectDir, absPath)) return "blocklisted"; if (isPrivateCompositionFile(ctx.absProjectDir, absPath)) return "private"; - if (ctx.received(entryPath) && linksOutside(ctx.absProjectDir, absPath)) return "received"; + if (linksOutside(ctx.absProjectDir, absPath) && ctx.received(entryPath)) return "received"; if (!loadableReference(ctx.absProjectDir, absPath)) return "private"; if (!linkOutsideApproved(ctx.absProjectDir, rel, absPath, ctx.onOutsideFile)) return "not_approved"; @@ -529,6 +530,18 @@ const HARD_REFERENCES = new Set(["video[src]", "audio[src]", "img[src]", "video[ const relativeToProject = (ctx: ExternalAssetContext, absPath: string) => encodeUrlPath(relative(ctx.absProjectDir, absPath).replaceAll("\\", "/")); +const UNCHECKED_NOTE = + " The project has too many folders to search them all for received-files records, so every file counts as received."; + +function strictNote(received: ReceivedFiles, reason: LeftOutReason): string { + if (reason !== "received") return ""; + if (received.unchecked) return UNCHECKED_NOTE; + if (received.gone === undefined) return ""; + return terminalText( + ` ${received.gone} is listed as received but is gone, so every page, stylesheet, script and SVG counts as received. To clear this, put the file back, or take its line out of .hyperframes/received-files.json once you know where its content went.`, + ); +} + function leaveOut( ctx: ExternalAssetContext, entryPath: string, @@ -536,7 +549,7 @@ function leaveOut( absPath: string, reason: LeftOutReason = "blocklisted", ): void { - const warning = leftOutWarning(entryPath, rawPath, reason); + const warning = `${leftOutWarning(entryPath, rawPath, reason)}${strictNote(ctx.received, reason)}`; if (ctx.leftOut.has(warning)) return; ctx.leftOut.add(warning); ctx.onLeftOut?.( @@ -898,7 +911,7 @@ function walkedFileLeftOut( ): WalkedLeftOutReason | null { if (linksToPrivateOutsideFile(absProjectDir, absPath)) return "blocklisted"; if (isPrivateCompositionFile(absProjectDir, absPath)) return "private"; - if (received(filePath) && linksOutside(absProjectDir, absPath)) return "received"; + if (linksOutside(absProjectDir, absPath) && received(filePath)) return "received"; return null; } @@ -931,7 +944,7 @@ export function stagePublishFiles( const reason = walkedFileLeftOut(absProjectDir, filePath, absPath, received); if (reason) { onLeftOut?.( - `${filePath} ${WALKED_LEFT_OUT[reason]}; it was left out of the upload.`, + `${filePath} ${WALKED_LEFT_OUT[reason]}; it was left out of the upload.${strictNote(received, reason)}`, leftOutReference(absProjectDir, filePath, filePath, absPath, reason), ); walkedLeftOut.add(archiveKey(filePath)); diff --git a/packages/cli/src/utils/receivedContent.test.ts b/packages/cli/src/utils/receivedContent.test.ts new file mode 100644 index 0000000000..023fa55da4 --- /dev/null +++ b/packages/cli/src/utils/receivedContent.test.ts @@ -0,0 +1,25 @@ +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { contentHash } from "./receivedContent.js"; + +let dir: string | undefined; + +afterEach(() => { + if (dir) rmSync(dir, { recursive: true, force: true }); + dir = undefined; +}); + +describe("contentHash", () => { + it("skips a large file that names no others, and still hashes a large script", () => { + dir = mkdtempSync(join(tmpdir(), "hf-received-content-")); + const large = Buffer.alloc(16 * 1024 * 1024 + 1, "x"); + writeFileSync(join(dir, "archive.zip"), large); + writeFileSync(join(dir, "bundle.js"), large); + writeFileSync(join(dir, "small.zip"), "x"); + expect(contentHash(join(dir, "archive.zip"))).toBeNull(); + expect(contentHash(join(dir, "bundle.js"))).toMatch(/^[0-9a-f]{64}$/); + expect(contentHash(join(dir, "small.zip"))).toMatch(/^[0-9a-f]{64}$/); + }); +}); diff --git a/packages/cli/src/utils/receivedContent.ts b/packages/cli/src/utils/receivedContent.ts new file mode 100644 index 0000000000..f3f456fef0 --- /dev/null +++ b/packages/cli/src/utils/receivedContent.ts @@ -0,0 +1,98 @@ +import { createHash } from "node:crypto"; +import { closeSync, constants, fstatSync, openSync, readFileSync, type Stats } from "node:fs"; +import { type FileHandle, open } from "node:fs/promises"; +import { extname, join } from "node:path"; +import { readRecordTail, recordKind, writeRecord } from "./projectRecords.js"; + +/** + * The sha256s of received content that left a listed path, kept beside the project's received-files record in the + * file and format the desktop app keeps. A file holding the same bytes counts as received. Keys this module does + * not use are written back as they were read. + */ +const RECEIVED_CONTENT = "received-content.json"; +const MAX_CONTENT_BYTES = 16 * 1024 * 1024; +const SHA256 = /^[0-9a-f]{64}$/; + +const isObject = (value: unknown): value is Record => + value !== null && typeof value === "object" && !Array.isArray(value); + +export const receivedContentFile = (dir: string): string => + join(dir, ".hyperframes", RECEIVED_CONTENT); + +export interface ReceivedContent { + hashes: Set; + saved: Record; + readable: boolean; +} + +/** The received content beside the record in `dir`; empty when it is missing. One that cannot be read is never written over. */ +export function readReceivedContent(dir: string): ReceivedContent { + const empty = (readable: boolean) => ({ hashes: new Set(), saved: {}, readable }); + const read = readRecordTail(dir, RECEIVED_CONTENT, MAX_CONTENT_BYTES); + if (read === null) return empty(recordKind(dir, RECEIVED_CONTENT) === "absent"); + let saved: unknown; + try { + saved = read?.whole ? JSON.parse(read.text) : null; + } catch { + saved = null; + } + if (!isObject(saved)) return empty(false); + const listed = Array.isArray(saved.hashes) ? saved.hashes : []; + const hashes = listed.filter( + (hash): hash is string => typeof hash === "string" && SHA256.test(hash), + ); + return { hashes: new Set(hashes), saved, readable: true }; +} + +export function writeReceivedContent(dir: string, content: ReceivedContent): boolean { + if (!content.readable) return false; + const json = { ...content.saved, hashes: [...content.hashes].sort() }; + return writeRecord(dir, RECEIVED_CONTENT, `${JSON.stringify(json)}\n`); +} + +const sha256 = (bytes: Buffer) => createHash("sha256").update(bytes).digest("hex"); + +const MAX_HASHED_BYTES = 16 * 1024 * 1024; +const READ_PLAIN = constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0) | (constants.O_NONBLOCK ?? 0); + +/** Pages, stylesheets, scripts and SVG of any size, and other plain files up to 16 MB, as `stat` reads them. */ +export const hashable = (path: string, stat: Stats) => + stat.isFile() && (stat.size <= MAX_HASHED_BYTES || isCodeFile(path)); + +/** The sha256 of a plain file's bytes; null for a link, a folder, a large file that names no others, or one that cannot be read. */ +export function contentHash(path: string): string | null { + let fd: number; + try { + fd = openSync(path, READ_PLAIN); + } catch { + return null; + } + try { + return hashable(path, fstatSync(fd)) ? sha256(readFileSync(fd)) : null; + } catch { + return null; + } finally { + closeSync(fd); + } +} + +export async function contentHashAsync(path: string): Promise { + let handle: FileHandle; + try { + handle = await open(path, READ_PLAIN); + } catch { + return null; + } + try { + return hashable(path, await handle.stat()) ? sha256(await handle.readFile()) : null; + } catch { + return null; + } finally { + await handle.close(); + } +} + +const CODE_FILE = new Set([".html", ".htm", ".xhtml", ".css", ".js", ".mjs", ".cjs", ".svg"]); + +/** Pages, stylesheets, scripts and SVG: the files that can name other files for a render or upload to load. */ +export const isCodeFile = (path: string): boolean => CODE_FILE.has(extname(path).toLowerCase()); diff --git a/packages/cli/src/utils/receivedRecordFollow.ts b/packages/cli/src/utils/receivedRecordFollow.ts new file mode 100644 index 0000000000..defda9e28d --- /dev/null +++ b/packages/cli/src/utils/receivedRecordFollow.ts @@ -0,0 +1,212 @@ +import { existsSync, readdirSync, statSync } from "node:fs"; +import { lstat } from "node:fs/promises"; +import { basename, dirname, isAbsolute, join, relative, sep } from "node:path"; +import { realFilePath, resolveWithinProject } from "@hyperframes/studio-server/safe-path"; +import { registryRoot } from "../registry/publication.js"; +import { + changeReceivedRecordAsync, + filesListedInFolder, + receivedLinesAtOrAbove, +} from "./privateProjectFile.js"; +import { contentHashAsync, hashable, readReceivedContent } from "./receivedContent.js"; +import { isMediaFontOrImage } from "./receivedRender.js"; + +type Lines = Map; + +/** A listed file a Studio rename, move or delete takes off its path, and its sha256 when its content is checked. */ +interface Leaving { + line: string; + hash?: string; +} + +/** Worked out while the files are on disk. A line whose file is already gone stays, as the sign it is gone. */ +export interface ReceivedChange { + root: string; + leaving: Map; + arriving: Lines; +} + +const swapCase = (path: string) => + [...path].map((c) => (c === c.toLowerCase() ? c.toUpperCase() : c.toLowerCase())).join(""); + +/** Whether `folder` sits on a disk that takes a name in any letter case; false when that cannot be told. */ +function caseInsensitive(folder: string): boolean { + const swapped = swapCase(folder); + return swapped !== folder && existsSync(swapped); +} + +/** `path` spelled as its folder lists it, so a request in another letter case or Unicode form names the listed file. */ +export function onDiskPath(path: string): string { + const folder = realFilePath(dirname(path)); + const name = basename(path); + const folded = name.normalize("NFC").toLowerCase(); + try { + const names = readdirSync(folder); + const found = + names.find((entry) => entry === name) ?? + names.find((entry) => entry.normalize("NFC").toLowerCase() === folded); + return join(folder, found ?? name); + } catch { + return join(folder, name); + } +} + +const keyIn = (folder: string, path: string) => + relative(folder, path).split(sep).join("/").normalize("NFC"); + +const leavesFolder = (key: string) => key === ".." || key.startsWith("../") || isAbsolute(key); + +function entriesAt(folder: string, key: string): (entry: string) => boolean { + const fold = caseInsensitive(folder) + ? (text: string) => text.normalize("NFC").toLowerCase() + : (text: string) => text.normalize("NFC"); + const at = fold(key); + return (entry) => fold(entry) === at || fold(entry).startsWith(`${at}/`); +} + +const push = (map: Map, folder: string, item: T) => + map.set(folder, [...(map.get(folder) ?? []), item]); + +async function addLines(lines: Lines): Promise { + const added: Lines = new Map(); + for (const [folder, wanted] of lines) + await changeReceivedRecordAsync(folder, (files) => { + const fresh = wanted.filter((line) => !files.has(line)); + fresh.forEach((line) => files.add(line)); + if (fresh.length > 0) added.set(folder, fresh); + return fresh.length > 0; + }); + return added; +} + +async function dropLines(lines: Lines): Promise { + for (const [folder, stale] of lines) + await changeReceivedRecordAsync(folder, (files) => + stale.map((line) => files.delete(line)).some(Boolean), + ); +} + +/** + * Lists in the project's own record, by their paths from the project folder, every file that the records in the + * folder `target` list, so a folder delete that stops partway, or a move, leaves each of them listed. False when that + * cannot be done, and the folder must then stay as it is. + */ +export async function keepListedInFolder(projectDir: string, target: string): Promise { + const root = registryRoot(projectDir); + const listed = await filesListedInFolder(target); + if (listed === null) return false; + const lines = listed + .flatMap((path) => [keyIn(root, path), keyIn(root, realFilePath(path))]) + .filter((key) => !leavesFolder(key)); + if (lines.length === 0) return true; + try { + await addLines(new Map([[root, lines]])); + return true; + } catch { + return false; + } +} + +/** Null for a gone file or one whose content cannot be read: its line stays. */ +async function leaving(line: string, path: string): Promise { + const stat = await lstat(path).catch(() => null); + if (stat === null) return null; + if (isMediaFontOrImage(path) || (stat.isFile() && !hashable(path, stat))) return { line }; + const hash = await contentHashAsync(path); + return hash === null ? null : { line, hash }; +} + +export async function plannedReceivedChange( + projectDir: string, + from: string, + to?: string, +): Promise { + const root = registryRoot(projectDir); + const change: ReceivedChange = { root, leaving: new Map(), arriving: new Map() }; + for (const { folder, lines } of receivedLinesAtOrAbove(dirname(from))) { + const fromKey = keyIn(folder, from); + const at = entriesAt(folder, fromKey); + for (const line of lines.filter(at)) { + const left = await leaving(line, join(folder, line)); + if (left === null) continue; + push(change.leaving, folder, left); + if (to === undefined) continue; + const rest = line.normalize("NFC").slice(fromKey.length); + const toKey = keyIn(folder, to); + if (leavesFolder(toKey)) push(change.arriving, root, keyIn(root, to) + rest); + else push(change.arriving, folder, toKey + rest); + } + } + return change; +} + +/** False when the received content cannot be read; it is then left as it is. */ +async function keepLeftContent(change: ReceivedChange): Promise { + const hashes = [...change.leaving.values()].flat().flatMap((left) => left.hash ?? []); + if (hashes.length === 0) return true; + if (!readReceivedContent(change.root).readable) return false; + await changeReceivedRecordAsync(change.root, (files, content) => { + hashes.forEach((hash) => content.hashes.add(hash)); + return files.size === 0; + }); + return true; +} + +/** + * Made before the files move; throws when a record cannot be changed. The result holds only the lines it newly listed, + * so a refused rename can take them back, and keeps listed a file whose content could not be kept. + */ +export async function recordReceivedArrivals(change: ReceivedChange): Promise { + const contentKept = await keepLeftContent(change); + const arriving = await addLines(change.arriving); + const leaving = new Map( + [...change.leaving].map(([folder, lines]) => [ + folder, + lines.filter((left) => contentKept || left.hash === undefined), + ]), + ); + return { ...change, leaving, arriving }; +} + +export const dropReceivedLeavers = (change: ReceivedChange): Promise => + dropLines( + new Map([...change.leaving].map(([folder, lines]) => [folder, lines.map((left) => left.line)])), + ); + +export const dropReceivedArrivals = (change: ReceivedChange): Promise => + dropLines(change.arriving); + +/** + * Lists in the project's own record each of `paths` that no record lists and that holds content kept as received, so + * an undo, a redo, a restore or a save that brings such content back keeps it received through later edits. + */ +export async function relistReceivedContent(projectDir: string, paths: string[]): Promise { + const root = registryRoot(projectDir); + const lines: string[] = []; + for (const path of paths) { + const records = receivedLinesAtOrAbove(dirname(path)); + const listed = records.some(({ folder, lines: listedLines }) => + listedLines.some((line) => line.normalize("NFC") === keyIn(folder, path)), + ); + if (listed) continue; + const hashes = new Set( + records.flatMap(({ folder }) => [...readReceivedContent(folder).hashes]), + ); + if (hashes.size === 0) continue; + const hash = await contentHashAsync(path); + const line = keyIn(root, realFilePath(path)); + if (hash !== null && hashes.has(hash) && !leavesFolder(line)) lines.push(line); + } + if (lines.length > 0) await addLines(new Map([[root, lines]])); +} + +/** Lists again each file a history undo, redo or restore wrote that holds received content. */ +export function relistHistoryWrites( + projectDir: string, + entry: { files: readonly { path: string }[] } | null, +): Promise { + const written = (entry?.files ?? []) + .flatMap((file) => resolveWithinProject(projectDir, file.path) ?? []) + .filter((path) => statSync(path, { throwIfNoEntry: false })?.isFile() === true); + return relistReceivedContent(projectDir, written); +} diff --git a/packages/cli/src/utils/receivedRender.test.ts b/packages/cli/src/utils/receivedRender.test.ts index 708dd2dc2b..608e6ef335 100644 --- a/packages/cli/src/utils/receivedRender.test.ts +++ b/packages/cli/src/utils/receivedRender.test.ts @@ -10,6 +10,7 @@ import { symlinkSync, writeFileSync, } from "node:fs"; +import { createHash } from "node:crypto"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { compileForRender } from "../../../producer/src/services/htmlCompiler.js"; @@ -294,6 +295,21 @@ describe("outsideAssetsAllowed", () => { expect(line).toContain(`Could not read ${path}`); }); + it.each(["renders/x", "dist", ".next/x", "coverage/x", ".hyperframes/requests/x"])( + "refuses outside files when the copy sits in %s", + (folder) => { + const dir = project(); + write(join(dir, "index.html"), `
`); + const up = "../".repeat(folder.split("/").length + 1); + write(join(dir, folder, "index.html"), ``); + write(join(dir, folder, ".hyperframes", "received-files.json"), received("index.html")); + const { allowed, line } = decide(dir); + expect(allowed).toBe(false); + expect(line).toContain(`${join(folder, "index.html")} ${SOMEONE_ELSE}`); + expect(line).toContain(`${up}outside/pic.png`); + }, + ); + it("keeps outside files for a folder with no record of its own", () => { const { allowed, warn } = decide(withNested()); expect(allowed).toBe(true); @@ -322,6 +338,43 @@ describe("outsideAssetsAllowed", () => { }); }); + describe("content of a received file that left the record", () => { + const hashOf = (text: string) => createHash("sha256").update(text).digest("hex"); + function withLeftContent(text: string): string { + const dir = project(received()); + write( + join(dir, ".hyperframes", "received-content.json"), + JSON.stringify({ hashes: [hashOf(text)] }), + ); + return dir; + } + + it("refuses outside files when a project file holds it, and names that file", () => { + const dir = withLeftContent(``); + const { allowed, line } = decide(dir); + expect(allowed).toBe(false); + expect(line).toContain( + `${join("compositions", "scene.html")} holds the content of a file that came from someone else`, + ); + }); + + it("refuses outside files and names the received content when it cannot be read", () => { + const dir = project(received()); + write(join(dir, ".hyperframes", "received-content.json"), "{not json"); + const { allowed, line } = decide(dir); + expect(allowed).toBe(false); + expect(line).toContain( + `Could not read ${join(realpathSync(dir), ".hyperframes", "received-content.json")}`, + ); + }); + + it("keeps outside files when only Studio's backups hold it", () => { + const dir = withLeftContent("

theirs

"); + write(join(dir, ".hyperframes", "backup", "scene.html"), "

theirs

"); + expect(decide(dir).allowed).toBe(true); + }); + }); + it("keeps outside files for a project with no received-files record", () => { const { allowed, warn } = decide(project()); expect(allowed).toBe(true); diff --git a/packages/cli/src/utils/receivedRender.ts b/packages/cli/src/utils/receivedRender.ts index bfaa75e097..871454c32c 100644 --- a/packages/cli/src/utils/receivedRender.ts +++ b/packages/cli/src/utils/receivedRender.ts @@ -2,7 +2,8 @@ import { closeSync, openSync, readdirSync, readSync, statSync, type Dirent } fro import { extname, join, relative, resolve } from "node:path"; import { isPathInside, terminalText } from "@hyperframes/core"; import { namesAnotherNetworkRoot } from "@hyperframes/parsers/asset-resolution"; -import { linksOutside, receivedRecordStatus } from "./privateProjectFile.js"; +import { linksOutside, mayHoldLoadedFiles, receivedRecordStatus } from "./privateProjectFile.js"; +import { contentHash } from "./receivedContent.js"; import { walkBudget } from "./walkBudget.js"; import { htmlReferences, @@ -157,7 +158,7 @@ function firstBytes(path: string): Buffer | null { } } -function isMediaFontOrImage(path: string): boolean { +export function isMediaFontOrImage(path: string): boolean { const matches = MEDIA_SIGNATURES[extname(path).toLowerCase()]; const bytes = matches && isFile(path) ? firstBytes(path) : null; return bytes !== null && matches !== undefined && matches(bytes); @@ -264,9 +265,50 @@ interface Refusal { const READS_NOTHING_OUTSIDE = "so the render reads nothing outside the project."; +function holderIn( + dir: string, + listing: Dirent[], + walk: { projectDir: string; known: Set; pending: string[] }, +) { + for (const entry of listing) { + const path = join(dir, entry.name); + if (entry.isDirectory()) { + if (mayHoldLoadedFiles(walk.projectDir, path)) walk.pending.push(path); + } else if (entry.isFile() && !isMediaName(path) && walk.known.has(contentHash(path) ?? "")) { + return path; + } + } + return undefined; +} + +function receivedContentHolder(projectDir: string, hashes: string[]): string | null | undefined { + if (hashes.length === 0) return undefined; + const withinBudget = walkBudget(); + const walk = { projectDir, known: new Set(hashes), pending: [projectDir] }; + for (let dir = walk.pending.pop(); dir !== undefined; dir = walk.pending.pop()) { + const listing = listDir(dir); + if (listing === null || !withinBudget(listing.length)) return null; + const holder = holderIn(dir, listing, walk); + if (holder !== undefined) return holder; + } + return undefined; +} + +function receivedContentRefusal(projectDir: string, hashes: string[]): Refusal | null { + const holder = receivedContentHolder(projectDir, hashes); + if (holder === undefined) return null; + if (holder === null) { + const reason = `Could not check every project file for content that came from someone else, ${READS_NOTHING_OUTSIDE}`; + return { reason, alwaysWarn: true }; + } + const shown = relative(realOrNull(projectDir) ?? projectDir, holder); + const reason = `${shown} holds the content of a file that came from someone else, ${READS_NOTHING_OUTSIDE}`; + return { reason, alwaysWarn: false }; +} + function receivedRefusal( projectDir: string, - record: { path: string; listed: string[] | null } | "unchecked", + record: { path: string; listed: string[] | null; hashes: string[] } | "unchecked", ): Refusal | null { const shown = (path: string) => relative(realOrNull(projectDir) ?? projectDir, path); if (record === "unchecked") { @@ -276,12 +318,13 @@ function receivedRefusal( if (record.listed === null) { return { reason: `Could not read ${record.path}, ${READS_NOTHING_OUTSIDE}`, alwaysWarn: true }; } - if (record.listed.length === 0) return null; const nonMedia = record.listed.find((path) => !isMediaFontOrImage(path)); if (nonMedia !== undefined) { const reason = `${shown(nonMedia)} came from someone else and is not a media, font or image file, ${READS_NOTHING_OUTSIDE}`; return { reason, alwaysWarn: false }; } + const contentRefusal = receivedContentRefusal(projectDir, record.hashes); + if (contentRefusal !== null || record.listed.length === 0) return contentRefusal; const received = receivedIdentities(record.listed); const aliases = received === null ? null : nonMediaAliases(projectDir, received); if (aliases === null) { diff --git a/packages/studio-server/src/routes/files.pathSafety.test.ts b/packages/studio-server/src/routes/files.pathSafety.test.ts index f910fcc9cf..9290d5e79f 100644 --- a/packages/studio-server/src/routes/files.pathSafety.test.ts +++ b/packages/studio-server/src/routes/files.pathSafety.test.ts @@ -359,6 +359,53 @@ describe("files other programs run or trust", () => { expect(existsSync(join(project, "compositions"))).toBe(false); }); + describe("a folder holding a received-files record", () => { + const record = (project: string, folder: string, files: string[]) => { + mkdirSync(join(project, folder, ".hyperframes"), { recursive: true }); + writeFileSync( + join(project, folder, ".hyperframes", "received-files.json"), + JSON.stringify({ files }), + ); + }; + const move = (app: Hono, from: string, newPath: string) => + app.request(fileUrl(from), { + method: "PATCH", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ newPath }), + }); + + it("deletes or moves a folder whose record lists a file outside it", async () => { + const { app, project } = fixture(); + mkdirSync(join(project, "sub")); + writeFileSync(join(project, "sub", "page.html"), "
"); + record(project, "sub/notes", ["../page.html"]); + expect((await move(app, "sub/notes", "sub/kept")).status).toBe(200); + expect((await app.request(fileUrl("sub/kept"), { method: "DELETE" })).status).toBe(200); + expect(existsSync(join(project, "sub", "page.html"))).toBe(true); + }); + + it("deletes or moves a received folder whose record lists only its own files", async () => { + const { app, project } = fixture(); + mkdirSync(join(project, "sub")); + writeFileSync(join(project, "sub", "page.html"), "
"); + record(project, "sub", ["page.html", "gone.html"]); + expect((await move(app, "sub", "dist")).status).toBe(200); + expect(existsSync(join(project, "dist", ".hyperframes", "received-files.json"))).toBe(true); + expect((await app.request(fileUrl("dist"), { method: "DELETE" })).status).toBe(200); + expect(existsSync(join(project, "dist"))).toBe(false); + }); + + it("refuses a delete or move whose target is a records folder", async () => { + const { app, project } = fixture(); + writeFileSync(join(project, "page.html"), "
"); + record(project, "sub", ["page.html"]); + expect((await app.request(fileUrl("sub/.hyperframes"), { method: "DELETE" })).status).toBe( + 403, + ); + expect((await move(app, "sub/.hyperframes", "dist")).status).toBe(403); + }); + }); + it("still writes compositions, stylesheets, script assets and uploads", async () => { const { app, project } = fixture(); for (const path of [