mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-18 07:50:13 +00:00
* fix(cli): a received folder's record is read wherever Studio moves it The search for received-files records now walks every folder a render, preview or publish can load files from, including renders/, dist/, .next/, coverage/ and Studio's own folders in the project's .hyperframes/. A received folder moved there by Studio keeps its files received. Studio's folder delete and move also refuse a folder holding a received-files record that lists a file outside its own folder, since that file would stay behind with no record. A received folder whose record lists only its own files can still be deleted or moved. * fix(cli): a renamed or moved received file stays listed under its new name The CLI-hosted Studio now follows every rename, move or delete it completes in the received-files records, whichever client sent it. A renamed file, or every file in a moved folder, stays listed under its new path; a file moved out of a received folder is listed in the project's own record. Changes are made under the record's lock, and a record that cannot be changed is left as it was. The per-file readers used by publish, cloud render, feedback and add now count every file as received when a record lists a file that is no longer on disk, since that file may be there under another name. This matches the local render check. Projects with no record are unaffected. * fix(cli): received content that leaves the record still counts as received When a Studio rename, move or delete takes a received file off a listed path, the sha256 of its content is kept in .hyperframes/received-content.json, the file and format the desktop app uses; other keys in it are kept. A file holding the same bytes counts as received for publish, cloud render, feedback, add and the local render check, so content brought back by undo or renamed outside Studio stays received. Studio's own backups do not count. A listed file that is gone from disk now makes only pages, stylesheets, scripts, SVG and links out of the project count as received, and the upload warning names the gone file. The record follows a request spelled in another letter case or Unicode form by matching the name the disk holds, and waits for the record's lock without blocking the server. Studio's folder delete and move no longer refuse a folder whose record lists a file outside it; that file's content is kept as received instead. * fix(cli): a folder delete or move keeps what a record inside it lists elsewhere Before Studio deletes or moves a folder, every file that a received-files record inside the folder lists outside it is listed in the project's own record by its path from the project folder. Such a file stays received after it is edited, and a listed link stays received too. When the project's record cannot be written, the delete or move is refused. Received content that cannot be read is never written over, so the desktop app's keys in it are kept. The warning about a listed file that is gone now says how to clear it. * fix(cli): received-content checks skip large non-code files and name a skipped search A file larger than 16 MB that is not a page, stylesheet, script or SVG is no longer read to compare its content with received content, so a large archive is not read on every check. When the project has too many folders to search for received-files records, the upload warning now says so. The case-matching publish test now expects the gone-file rule on a disk that tells letter case apart, and a comment no longer names a file path. * fix(cli): name a gone received file from the project folder on every system The gone-file note now names the file from the same project path the record search starts at. On Windows, a short (8.3) folder name in the temp path made the two spellings differ, so the note showed a long relative path instead of the file's name. Content hashing opens a file once, checks it through that handle and reads from the same handle, and the received-content file is read before it is classified, so nothing is checked by path and then used by path. * fix(cli): received files stay received through undo, saves, and partial folder deletes An undo, redo or restore lists again every file it writes that holds received content, and a Studio save lists such a file before it changes it. So a Studio delete followed by an undo and an edit leaves the file received. A Studio rename, move or delete now records the received side first, under the record's lock, and is refused when that fails: the content leaving listed paths is kept, a renamed file is listed under its new path, and every line of a record inside a moved or deleted folder is copied into the project's record. Once the files have moved, only lines whose file was there and whose content was kept are taken off. Lines at a rename's destination and lines whose file was already gone stay. The search for records inside the folder has no size limit, and nothing is locked when nothing is listed. * fix(cli): unreadable received content counts every file as received The readers treat a received-content file they cannot read the way they treat an unreadable received-files record: every file counts as received, and the local render check names the file it could not read. A record line that points outside the project is ignored, so a record in a received folder cannot make the readers look at or report files outside the project. * fix(cli): the CLI's own history undo and restore keep received files listed `hyperframes history undo` and `restore` with no Studio running now list again every file they write that holds received content, the same way Studio's history routes do. A delete in Studio, then an undo or restore from the CLI, then an edit, leaves the file received. A Studio edit that names its file in the request body, such as a freeze-frame edit, now lists that file first when it holds received content.
439 lines
15 KiB
TypeScript
439 lines
15 KiB
TypeScript
import {
|
|
closeSync,
|
|
type Dirent,
|
|
lstatSync,
|
|
openSync,
|
|
readdirSync,
|
|
readFileSync,
|
|
realpathSync,
|
|
rmSync,
|
|
statSync,
|
|
} from "node:fs";
|
|
import { readdir } from "node:fs/promises";
|
|
import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
|
|
import {
|
|
isEnvOrGitPath,
|
|
isPrivateOutsideFile,
|
|
linksOutside,
|
|
} from "@hyperframes/studio-server/outside-file";
|
|
import { isPrivateProjectFile, realFilePath } from "@hyperframes/studio-server/safe-path";
|
|
import {
|
|
isCredentialOrToolStatePath,
|
|
isStudioPrivatePath,
|
|
projectPathSegments,
|
|
} from "@hyperframes/core/safe-path";
|
|
import { registryRoot } from "../registry/publication.js";
|
|
import { recordFolder, writeRecord } from "./projectRecords.js";
|
|
import { walkBudget } from "./walkBudget.js";
|
|
import {
|
|
contentHash,
|
|
isCodeFile,
|
|
readReceivedContent,
|
|
type ReceivedContent,
|
|
receivedContentFile,
|
|
writeReceivedContent,
|
|
} from "./receivedContent.js";
|
|
|
|
export { isEnvOrGitPath, isPrivateOutsideFile, linksOutside };
|
|
|
|
export function linksToPrivateOutsideFile(projectDir: string, filePath: string): boolean {
|
|
return linksOutside(projectDir, filePath) && isPrivateOutsideFile(projectDir, filePath);
|
|
}
|
|
|
|
const RECEIVED_RECORD = join(".hyperframes", "received-files.json");
|
|
const RECEIVED_RECORD_LOCK = `${RECEIVED_RECORD}.lock`;
|
|
const LOCK_WAIT_MS = 2_000;
|
|
const LOCK_STALE_MS = 10_000;
|
|
|
|
interface ReceivedRecord {
|
|
file: string;
|
|
root: string;
|
|
projectDirFromRoot: string;
|
|
listed: unknown;
|
|
}
|
|
|
|
type RecordKind = "absent" | "file" | "other" | "unknown";
|
|
|
|
function recordKind(dir: string): RecordKind {
|
|
const path = join(dir, RECEIVED_RECORD);
|
|
let found;
|
|
try {
|
|
found = lstatSync(path, { throwIfNoEntry: false });
|
|
} catch (error) {
|
|
return (error as NodeJS.ErrnoException).code === "ENOTDIR" ? "absent" : "unknown";
|
|
}
|
|
if (!found) return "absent";
|
|
try {
|
|
return statSync(path).isFile() ? "file" : "other";
|
|
} catch {
|
|
return "other";
|
|
}
|
|
}
|
|
|
|
function readRecord(dir: string, ownFolder: boolean): { listed: unknown } | null {
|
|
const kind = recordKind(dir);
|
|
if (kind === "absent" || (kind === "unknown" && !ownFolder)) return null;
|
|
if (kind !== "file") return { listed: undefined };
|
|
try {
|
|
return { listed: JSON.parse(readFileSync(join(dir, RECEIVED_RECORD), "utf-8")).files };
|
|
} catch {
|
|
return { listed: undefined };
|
|
}
|
|
}
|
|
|
|
function sleepSync(ms: number): void {
|
|
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
|
|
}
|
|
|
|
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
|
|
|
|
function tryRecordLock(lock: string, deadline: number): "taken" | "wait" {
|
|
try {
|
|
closeSync(openSync(lock, "wx"));
|
|
return "taken";
|
|
} catch (error) {
|
|
if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error;
|
|
}
|
|
const heldFor = Date.now() - (statSync(lock, { throwIfNoEntry: false })?.mtimeMs ?? Date.now());
|
|
if (heldFor > LOCK_STALE_MS) rmSync(lock, { force: true });
|
|
else if (Date.now() >= deadline)
|
|
throw new Error(`Another program is changing ${RECEIVED_RECORD}; try again.`);
|
|
return "wait";
|
|
}
|
|
|
|
function takeRecordLock(lock: string): void {
|
|
const deadline = Date.now() + LOCK_WAIT_MS;
|
|
while (tryRecordLock(lock, deadline) === "wait") sleepSync(25);
|
|
}
|
|
|
|
async function takeRecordLockAsync(lock: string): Promise<void> {
|
|
const deadline = Date.now() + LOCK_WAIT_MS;
|
|
while (tryRecordLock(lock, deadline) === "wait") await sleep(25);
|
|
}
|
|
|
|
type RecordChange = (files: Set<string>, content: ReceivedContent) => boolean;
|
|
|
|
function recordLock(projectDir: string): { root: string; lock: string } {
|
|
const root = registryRoot(projectDir);
|
|
if (!recordFolder(root, ""))
|
|
throw new Error(
|
|
`${join(root, ".hyperframes")} is a link or not a folder; remove it and try again.`,
|
|
);
|
|
return { root, lock: join(root, RECEIVED_RECORD_LOCK) };
|
|
}
|
|
|
|
function applyRecordChange(root: string, change: RecordChange): void {
|
|
const record = readRecord(root, true);
|
|
if (record && !Array.isArray(record.listed)) return;
|
|
const files = new Set(((record?.listed ?? []) as unknown[]).map(String));
|
|
const content = readReceivedContent(root);
|
|
const known = content.hashes.size;
|
|
const changed = change(files, content);
|
|
const json = `${JSON.stringify({ files: [...files].sort() })}\n`;
|
|
if (changed && !writeRecord(root, basename(RECEIVED_RECORD), json))
|
|
throw new Error(`Could not write ${join(root, RECEIVED_RECORD)}.`);
|
|
if (content.hashes.size > known && !writeReceivedContent(root, content))
|
|
throw new Error(`Could not write the received content beside ${join(root, RECEIVED_RECORD)}.`);
|
|
}
|
|
|
|
function changeReceivedRecord(projectDir: string, change: RecordChange): void {
|
|
const { root, lock } = recordLock(projectDir);
|
|
takeRecordLock(lock);
|
|
try {
|
|
applyRecordChange(root, change);
|
|
} finally {
|
|
rmSync(lock, { force: true });
|
|
}
|
|
}
|
|
|
|
export async function changeReceivedRecordAsync(
|
|
projectDir: string,
|
|
change: RecordChange,
|
|
): Promise<void> {
|
|
const { root, lock } = recordLock(projectDir);
|
|
await takeRecordLockAsync(lock);
|
|
try {
|
|
applyRecordChange(root, change);
|
|
} finally {
|
|
rmSync(lock, { force: true });
|
|
}
|
|
}
|
|
|
|
export function recordReceivedFiles(root: string, paths: string[]): string[] {
|
|
const added: string[] = [];
|
|
if (paths.length === 0) return added;
|
|
changeReceivedRecord(root, (files) => {
|
|
for (const path of paths.map((one) => one.normalize("NFC"))) {
|
|
if (!files.has(path)) added.push(path);
|
|
files.add(path);
|
|
}
|
|
return added.length > 0;
|
|
});
|
|
return added;
|
|
}
|
|
|
|
export function forgetReceivedFiles(root: string, paths: string[]): void {
|
|
if (paths.length === 0) return;
|
|
changeReceivedRecord(root, (files) => paths.map((path) => files.delete(path)).some(Boolean));
|
|
}
|
|
|
|
export const receivedLinesAtOrAbove = (dir: string): { folder: string; lines: string[] }[] =>
|
|
receivedRecordsAtOrAbove(dir).flatMap((record) =>
|
|
Array.isArray(record.listed) ? [{ folder: record.root, lines: record.listed.map(String) }] : [],
|
|
);
|
|
|
|
const listedPaths = (record: ReceivedRecord): string[] =>
|
|
Array.isArray(record.listed)
|
|
? record.listed.map((entry) => join(record.root, String(entry)))
|
|
: [];
|
|
|
|
const isAtOrUnder = (path: string, folder: string) => {
|
|
const inside = relative(folder, path);
|
|
return (
|
|
inside === "" || (inside !== ".." && !inside.startsWith(`..${sep}`) && !isAbsolute(inside))
|
|
);
|
|
};
|
|
|
|
export async function filesListedInFolder(target: string): Promise<string[] | null> {
|
|
if (!statSync(target, { throwIfNoEntry: false })?.isDirectory()) return [];
|
|
const own = readRecord(target, true);
|
|
const below = await receivedRecordsBelowAsync(target);
|
|
if (below === null) return null;
|
|
const records = own ? [nestedRecord(target, target, own), ...below] : below;
|
|
return records.flatMap(listedPaths);
|
|
}
|
|
|
|
function receivedRecordsAtOrAbove(start: string): ReceivedRecord[] {
|
|
const records: ReceivedRecord[] = [];
|
|
for (let dir = start; ; ) {
|
|
const record = readRecord(dir, dir === start);
|
|
if (record)
|
|
records.push({
|
|
file: join(dir, RECEIVED_RECORD),
|
|
root: dir,
|
|
projectDirFromRoot: relative(dir, start).split(sep).join("/"),
|
|
...record,
|
|
});
|
|
const parent = dirname(dir);
|
|
if (parent === dir) return records;
|
|
dir = parent;
|
|
}
|
|
}
|
|
|
|
function nestedRecord(start: string, dir: string, record: { listed: unknown }): ReceivedRecord {
|
|
const folder = relative(start, dir).split(sep).join("/");
|
|
const listed = Array.isArray(record.listed)
|
|
? record.listed.map((entry) => (folder ? `${folder}/${String(entry)}` : String(entry)))
|
|
: undefined;
|
|
return { file: join(dir, RECEIVED_RECORD), root: start, projectDirFromRoot: "", listed };
|
|
}
|
|
|
|
const GIT_AND_PACKAGE_FOLDERS = new Set([".git", "node_modules"]);
|
|
|
|
export function mayHoldLoadedFiles(start: string, folder: string): boolean {
|
|
const rel = relative(start, folder).split(sep).join("/");
|
|
const segments = projectPathSegments(rel);
|
|
if (GIT_AND_PACKAGE_FOLDERS.has(segments.at(-1) ?? "")) return false;
|
|
if (!segments.includes(".hyperframes")) return true;
|
|
return segments.length === 1 || !isStudioPrivatePath(rel);
|
|
}
|
|
|
|
const subfolders = (start: string, dir: string, entries: Dirent[]): string[] =>
|
|
entries
|
|
.filter((entry) => entry.isDirectory())
|
|
.map((entry) => join(dir, entry.name))
|
|
.filter((folder) => mayHoldLoadedFiles(start, folder));
|
|
|
|
function receivedRecordsBelow(start: string): ReceivedRecord[] | null {
|
|
const withinBudget = walkBudget();
|
|
const records: ReceivedRecord[] = [];
|
|
const pending = [start];
|
|
for (let dir = pending.pop(); dir !== undefined; dir = pending.pop()) {
|
|
let folders: string[];
|
|
try {
|
|
folders = subfolders(start, dir, readdirSync(dir, { withFileTypes: true }));
|
|
} catch {
|
|
return null;
|
|
}
|
|
if (!withinBudget(folders.length)) return null;
|
|
const record = dir === start ? null : readRecord(dir, true);
|
|
if (record) records.push(nestedRecord(start, dir, record));
|
|
pending.push(...folders);
|
|
}
|
|
return records;
|
|
}
|
|
|
|
async function receivedRecordsBelowAsync(start: string): Promise<ReceivedRecord[] | null> {
|
|
const records: ReceivedRecord[] = [];
|
|
const pending = [start];
|
|
for (let dir = pending.pop(); dir !== undefined; dir = pending.pop()) {
|
|
let folders: string[];
|
|
try {
|
|
folders = subfolders(start, dir, await readdir(dir, { withFileTypes: true }));
|
|
} catch {
|
|
return null;
|
|
}
|
|
const record = dir === start ? null : readRecord(dir, true);
|
|
if (record) records.push(nestedRecord(start, dir, record));
|
|
pending.push(...folders);
|
|
}
|
|
return records;
|
|
}
|
|
|
|
function walkStart(projectDir: string): string {
|
|
try {
|
|
return realpathSync(projectDir);
|
|
} catch {
|
|
return resolve(projectDir);
|
|
}
|
|
}
|
|
|
|
function receivedRecords(projectDir: string): ReceivedRecord[] | null {
|
|
const start = walkStart(projectDir);
|
|
const below = receivedRecordsBelow(start);
|
|
return below && [...receivedRecordsAtOrAbove(start), ...below];
|
|
}
|
|
|
|
function fileIdentity(path: string): string | null {
|
|
try {
|
|
const stat = lstatSync(path, { bigint: true });
|
|
return stat.ino === 0n ? null : `${stat.dev}:${stat.ino}`;
|
|
} catch {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
function listedIn(
|
|
record: ReceivedRecord,
|
|
projectDir: string,
|
|
start: string,
|
|
): (path: string) => boolean {
|
|
const listed = (record.listed as unknown[]).map((path) => String(path).normalize("NFC"));
|
|
const files = new Set(listed);
|
|
const keyOf = (path: string) =>
|
|
(record.projectDirFromRoot ? `${record.projectDirFromRoot}/${path}` : path).normalize("NFC");
|
|
let listedOnDisk: Set<string> | undefined;
|
|
const sameFileAsListed = (path: string) => {
|
|
listedOnDisk ??= new Set(
|
|
listed
|
|
.map((entry) => join(record.root, entry))
|
|
.filter((file) => isAtOrUnder(file, start))
|
|
.flatMap((file) => fileIdentity(file) ?? []),
|
|
);
|
|
const identity = fileIdentity(join(projectDir, path));
|
|
return identity !== null && listedOnDisk.has(identity);
|
|
};
|
|
return (path) => files.has(keyOf(path)) || sameFileAsListed(path);
|
|
}
|
|
|
|
function onDisk(path: string): boolean {
|
|
try {
|
|
return lstatSync(path, { throwIfNoEntry: false }) !== undefined;
|
|
} catch {
|
|
return false;
|
|
}
|
|
}
|
|
|
|
export type ReceivedFiles = ((path: string) => boolean) & { gone?: string; unchecked?: boolean };
|
|
|
|
const contentFolder = (record: ReceivedRecord) => dirname(dirname(record.file));
|
|
|
|
function receivedContentOf(
|
|
records: ReceivedRecord[],
|
|
): { hashes: string[] } | { unreadable: string } {
|
|
const hashes: string[] = [];
|
|
for (const folder of new Set(records.map(contentFolder))) {
|
|
const content = readReceivedContent(folder);
|
|
if (!content.readable) return { unreadable: receivedContentFile(folder) };
|
|
hashes.push(...content.hashes);
|
|
}
|
|
return { hashes };
|
|
}
|
|
|
|
function holdsReceivedContent(projectDir: string, hashes: Set<string>) {
|
|
const seen = new Map<string, boolean>();
|
|
return (path: string): boolean => {
|
|
if (hashes.size === 0) return false;
|
|
if (!seen.has(path)) seen.set(path, hashes.has(contentHash(join(projectDir, path)) ?? ""));
|
|
return seen.get(path) === true;
|
|
};
|
|
}
|
|
|
|
export function receivedFiles(projectDir: string): ReceivedFiles {
|
|
const records = receivedRecords(projectDir);
|
|
if (records === null) return Object.assign(() => true, { unchecked: true });
|
|
if (records.some((record) => !Array.isArray(record.listed))) return () => true;
|
|
const content = receivedContentOf(records);
|
|
if ("unreadable" in content) return () => true;
|
|
const start = walkStart(projectDir);
|
|
const listedInProject = records.flatMap(listedPaths).filter((path) => isAtOrUnder(path, start));
|
|
const goneFile = listedInProject.find((path) => !onDisk(path));
|
|
const mayBeGoneFile = (path: string) =>
|
|
goneFile !== undefined &&
|
|
(isCodeFile(path) || linksOutside(projectDir, join(projectDir, path)));
|
|
const sameContent = holdsReceivedContent(projectDir, new Set(content.hashes));
|
|
const listedInAnyRecord = records.map((record) => listedIn(record, projectDir, start));
|
|
const received = (path: string) =>
|
|
listedInAnyRecord.some((listed) => listed(path)) || mayBeGoneFile(path) || sameContent(path);
|
|
const gone = goneFile && relative(start, goneFile).split(sep).join("/");
|
|
return Object.assign(received, gone === undefined ? {} : { gone });
|
|
}
|
|
|
|
function listedUnder(record: ReceivedRecord): string[] {
|
|
const entries = (record.listed as unknown[]).map(String);
|
|
const prefix = record.projectDirFromRoot ? `${record.projectDirFromRoot}/`.normalize("NFC") : "";
|
|
const folder = prefix ? fileIdentity(join(record.root, record.projectDirFromRoot)) : null;
|
|
const identities = new Map<string, string | null>();
|
|
const inFolderOnDisk = (path: string) => {
|
|
for (let dir = dirname(path); folder && dir.length > record.root.length; dir = dirname(dir)) {
|
|
if (!identities.has(dir)) identities.set(dir, fileIdentity(dir));
|
|
if (identities.get(dir) === folder) return true;
|
|
}
|
|
return false;
|
|
};
|
|
return entries
|
|
.filter(
|
|
(entry) =>
|
|
entry.normalize("NFC").startsWith(prefix) || inFolderOnDisk(join(record.root, entry)),
|
|
)
|
|
.map((entry) => join(record.root, entry));
|
|
}
|
|
|
|
export function receivedRecordStatus(
|
|
projectDir: string,
|
|
): { path: string; listed: string[] | null; hashes: string[] } | "unchecked" | null {
|
|
const records = receivedRecords(projectDir);
|
|
if (records === null) return "unchecked";
|
|
const [nearest] = records;
|
|
if (!nearest) return null;
|
|
const unreadable = records.find((record) => !Array.isArray(record.listed));
|
|
if (unreadable) return { path: unreadable.file, listed: null, hashes: [] };
|
|
const content = receivedContentOf(records);
|
|
if ("unreadable" in content) return { path: content.unreadable, listed: null, hashes: [] };
|
|
return { path: nearest.file, listed: records.flatMap(listedUnder), hashes: content.hashes };
|
|
}
|
|
|
|
const leavesProject = (path: string): boolean =>
|
|
isAbsolute(path) || path.split(/[\\/]/)[0] === "..";
|
|
|
|
const projectRelatives = (projectDir: string, filePath: string): string[] => [
|
|
relative(projectDir, filePath),
|
|
relative(realFilePath(projectDir), realFilePath(filePath)),
|
|
];
|
|
|
|
/** Project files no composition loads, so none is served or uploaded: env, Git, credentials, tool and agent state. */
|
|
export function isPrivateCompositionFile(projectDir: string, filePath: string): boolean {
|
|
const paths = projectRelatives(projectDir, filePath);
|
|
return (
|
|
paths.some(isEnvOrGitPath) ||
|
|
paths.some((path) => !leavesProject(path) && isCredentialOrToolStatePath(path)) ||
|
|
isPrivateProjectFile(projectDir, filePath)
|
|
);
|
|
}
|
|
|
|
export function isHiddenProjectFile(projectDir: string, filePath: string): boolean {
|
|
return projectRelatives(projectDir, filePath).some((path) =>
|
|
path.split(/[\\/]/).some((segment) => segment.startsWith(".") && segment !== ".."),
|
|
);
|
|
}
|