mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-19 19:36:04 +00:00
feat(share): attach editable project sources to reviews (#7137)
This commit is contained in:
@@ -0,0 +1,91 @@
|
||||
---
|
||||
title: "Editable source in a share"
|
||||
description: "Choose whether a CLI review includes the project files used by its rendered video."
|
||||
---
|
||||
|
||||
`hyperframes share` can attach editable project files to the same review page
|
||||
used by Desktop. Source is available when the workspace's sharing and
|
||||
Composition Sync permissions allow it. The recipient's access still depends on
|
||||
the share's settings and their plan.
|
||||
|
||||
```bash
|
||||
hyperframes share --public # Video and source when eligible
|
||||
hyperframes share --update # Update the existing review
|
||||
hyperframes share --update --video-only # Explicitly omit editable source
|
||||
```
|
||||
|
||||
The five-free-project sharing limit is unchanged. A workspace that cannot use
|
||||
Composition Sync can still share its rendered video. The command reports
|
||||
`source.at: "video_only"` and a reason in JSON; this is a successful video share.
|
||||
A source upload does not grant anyone editing access by itself. Manage access
|
||||
through `hyperframes share --manage`.
|
||||
|
||||
## What travels
|
||||
|
||||
Source uses the same portable project policy as Desktop: authored HTML, scripts,
|
||||
styles, assets, and supported Studio edit metadata. Known credential filenames
|
||||
(such as `.env`, `.npmrc`, and `*.pem`), dependency directories, chat
|
||||
history, local sharing state, and recorded render outputs stay out.
|
||||
|
||||
These are filename rules, not a secret scanner. `.gitignore` and
|
||||
`.hyperframesignore` do not control source capture. Other regular files can be
|
||||
included even if Git ignores them. Keep private material outside the project,
|
||||
or use `--video-only` when you do not want to upload project files. Unsupported symlinks and invalid file names stop source
|
||||
preparation rather than silently producing an incomplete editable project.
|
||||
|
||||
The upload uses the workspace's file-count, per-file, project-size and manifest
|
||||
limits. Large media uploads in parts. Only a verified published snapshot that
|
||||
still matches the rendered project can be attached to the video version. The CLI
|
||||
checks the contents of rendered assets, including changes that preserve file
|
||||
sizes and timestamps. Deployments must include the exact source storage origins
|
||||
in `HYPERFRAMES_REVIEW_UPLOAD_ORIGINS` alongside the video storage origins.
|
||||
|
||||
## Interrupted uploads and unavailable source
|
||||
|
||||
Retry the same command after a network interruption. Its local record preserves
|
||||
the composition identity, upload idempotency key and completed file parts.
|
||||
A storage upload that makes no progress for 60 seconds is stopped and retried
|
||||
within the retry limit. The source record lives at `.hyperframes/share-sync.<backend-hash>.json` and
|
||||
contains no sign-in token. Do not copy that record into another project or edit
|
||||
its account, workspace or path: mismatches stop sharing to avoid overwriting
|
||||
another project's source.
|
||||
|
||||
If source changed, conflicts with another snapshot, or remains under verification,
|
||||
the command explains why it stopped. Retry after resolving the problem, or pass
|
||||
`--video-only` to share just the rendered video. An unexpected source upload
|
||||
failure never silently turns an eligible source share into video-only. If the
|
||||
server confirms an upload expired or failed, a retry starts a new upload. If
|
||||
the files changed while an earlier upload was still pending, the CLI cancels
|
||||
that upload before starting a new one. It waits for uploads already under
|
||||
verification and preserves conflicts for explicit resolution.
|
||||
|
||||
A video-only retry does not resume an interrupted video upload that already
|
||||
includes source. Check its status before retrying. To publish the same render as
|
||||
a new video-only version after the earlier upload finishes, use
|
||||
`hyperframes share --update --video-only`. Previous versions retain their files.
|
||||
|
||||
`--status` reads the review without uploading anything. Its source state is
|
||||
`not_checked`; it does not promise that local files match the shared version.
|
||||
The upload result uses `ready` only after source verification and attachment
|
||||
preparation; video readiness remains the separate top-level `ready` field.
|
||||
|
||||
## Integrating another local host
|
||||
|
||||
`@hyperframes/share` exports `shareSources` and
|
||||
`createProjectShareSourceStore`. Call `prepare(projectPath, renderedFingerprint)`
|
||||
before `addVersion`, then pass `filesForExport` and `requireSource: true` for a
|
||||
ready source result. `requireSource` prevents a later mismatch from degrading to
|
||||
a video-only version. Pass `omitSource: true` for a video-only operation that
|
||||
must not reuse a source-bearing version or pending upload.
|
||||
|
||||
A host may inject its existing binding store, or supply a read-only
|
||||
`existingBinding` resolver for Desktop bindings. Exact snapshots can be reused
|
||||
only in the same account and owner workspace, without a pending sync operation.
|
||||
Unsynced changes in a Desktop-owned binding require a Desktop sync first. The
|
||||
standalone CLI keeps its own project-local sync record; it does not discover or
|
||||
modify Desktop's application data directory.
|
||||
|
||||
## Related topics
|
||||
|
||||
- [CLI reference](/packages/cli#share) — audiences, updates, and processing status.
|
||||
- [CLI guide](/developers/cli) — choose the command for your task.
|
||||
@@ -196,6 +196,7 @@
|
||||
"pages": [
|
||||
"developers/cli",
|
||||
"packages/cli",
|
||||
"developers/share-source",
|
||||
"developers/catalog-search",
|
||||
"developers/harness-tool-context",
|
||||
"packages/lint"
|
||||
|
||||
@@ -38,6 +38,10 @@ export default defineCommand({
|
||||
type: "boolean",
|
||||
description: "Render and add a new video version to the existing share",
|
||||
},
|
||||
"video-only": {
|
||||
type: "boolean",
|
||||
description: "Share only the rendered video, without editable source",
|
||||
},
|
||||
status: { type: "boolean", description: "Read status without rendering or uploading" },
|
||||
manage: { type: "boolean", description: "Open this share's web management page" },
|
||||
space: { type: "string", description: "Owning workspace for a new share" },
|
||||
@@ -87,6 +91,7 @@ interface ShareArgs {
|
||||
json?: boolean;
|
||||
"wait-seconds"?: string;
|
||||
space?: string;
|
||||
"video-only"?: boolean;
|
||||
}
|
||||
function parseInput(args: ShareArgs): ShareAction {
|
||||
const waitSeconds = Number(args["wait-seconds"]);
|
||||
@@ -97,7 +102,7 @@ function parseInput(args: ShareArgs): ShareAction {
|
||||
const audience = requestedAudience(args);
|
||||
const action = args.status ? "status" : args.manage ? "manage" : args.update ? "update" : "share";
|
||||
|
||||
return { action, audience, space: args.space, waitSeconds };
|
||||
return { action, audience, space: args.space, waitSeconds, videoOnly: args["video-only"] };
|
||||
}
|
||||
|
||||
function requestedAudience(args: ShareArgs) {
|
||||
@@ -137,6 +142,8 @@ async function chooseAudience(input: ShareAction, args: ShareArgs, service: Shar
|
||||
}
|
||||
|
||||
async function presentResult(result: ShareState, input: ShareAction, json: boolean) {
|
||||
if (!json && result.source.at === "video_only")
|
||||
console.error(`Sharing video only (${result.source.reason}).`);
|
||||
if (json) console.log(JSON.stringify({ ok: true, ...result }));
|
||||
else if (input.action === "manage") {
|
||||
console.log(result.manageUrl);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { ReviewFailed, reviewSentence } from "@hyperframes/share";
|
||||
import { ShareSourceError } from "./sources.js";
|
||||
import { isAuthError } from "../auth/errors.js";
|
||||
import { CliUsageError } from "../utils/commandResult.js";
|
||||
|
||||
@@ -20,6 +21,8 @@ const HINTS: Record<string, string> = {
|
||||
};
|
||||
export function shareError(error: unknown): { code: string; message: string } {
|
||||
if (error instanceof CliUsageError) return { code: error.code, message: error.message };
|
||||
if (error instanceof ShareSourceError)
|
||||
return { code: "source_unavailable", message: error.message };
|
||||
if (isAuthError(error))
|
||||
return {
|
||||
code: error.code.toLowerCase(),
|
||||
|
||||
@@ -5,7 +5,10 @@ import { join } from "node:path";
|
||||
import type { RenderExecutionDependencies } from "../commands/render/execute.js";
|
||||
import type { RenderOptions } from "../commands/render.js";
|
||||
const mocks = vi.hoisted(() => ({ render: vi.fn(), plan: vi.fn(), execute: vi.fn() }));
|
||||
vi.mock("@hyperframes/share", () => ({ projectFingerprint: async () => "unchanged" }));
|
||||
vi.mock("@hyperframes/share", () => ({
|
||||
projectFingerprint: async () => "unchanged",
|
||||
shareSourceFingerprint: async () => "unchanged-content",
|
||||
}));
|
||||
vi.mock("../commands/render/plan.js", () => ({ createRenderPlan: mocks.plan }));
|
||||
vi.mock("../commands/render/execute.js", () => ({ executeRenderPlan: mocks.execute }));
|
||||
vi.mock("../commands/render.js", () => ({
|
||||
|
||||
@@ -1,8 +1,16 @@
|
||||
import { mkdtempSync, mkdirSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { latestShareRender, prepareShareRender } from "./render.js";
|
||||
import { dirname, join } from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { projectFingerprint } from "@hyperframes/share";
|
||||
import { latestShareRender, prepareShareRender, shareContentFingerprint } from "./render.js";
|
||||
const rendering = vi.hoisted(() => ({ execute: vi.fn() }));
|
||||
vi.mock("../commands/render/plan.js", () => ({ createRenderPlan: (plan: unknown) => plan }));
|
||||
vi.mock("../commands/render/execute.js", () => ({ executeRenderPlan: rendering.execute }));
|
||||
vi.mock("../commands/render.js", () => ({
|
||||
renderLocal: vi.fn(),
|
||||
checkRenderResolutionPreflight: vi.fn(),
|
||||
}));
|
||||
const dirs: string[] = [];
|
||||
function folder() {
|
||||
const dir = realpathSync(mkdtempSync(join(tmpdir(), "hf-share-")));
|
||||
@@ -41,4 +49,26 @@ describe("local sharing render boundary", () => {
|
||||
);
|
||||
expect(latestShareRender(dir)).toEqual({ id, path: output, fingerprint: "f" });
|
||||
});
|
||||
it.each([".media/voice.mp3", ".hyperframes/studio-manual-edits.json"])(
|
||||
"invalidates a cached render when %s changes",
|
||||
async (path) => {
|
||||
const dir = folder();
|
||||
writeFileSync(join(dir, "index.html"), '<audio src=".media/voice.mp3"></audio>');
|
||||
mkdirSync(dirname(join(dir, path)), { recursive: true });
|
||||
writeFileSync(join(dir, path), "original");
|
||||
rendering.execute.mockImplementation(async ({ output }: { output: string }) => {
|
||||
writeFileSync(output, "rendered video");
|
||||
});
|
||||
await prepareShareRender(dir);
|
||||
const original = latestShareRender(dir);
|
||||
const watcher = await projectFingerprint(dir);
|
||||
await prepareShareRender(dir);
|
||||
expect(latestShareRender(dir)?.id).toBe(original?.id);
|
||||
writeFileSync(join(dir, path), "changed");
|
||||
expect(await projectFingerprint(dir)).toBe(watcher);
|
||||
expect(await shareContentFingerprint(dir)).not.toBe(original?.contentFingerprint);
|
||||
await prepareShareRender(dir);
|
||||
expect(latestShareRender(dir)?.id).not.toBe(original?.id);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
@@ -2,12 +2,15 @@ import { randomUUID } from "node:crypto";
|
||||
import { existsSync, readFileSync, realpathSync } from "node:fs";
|
||||
import { mkdir, writeFile, rename, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { projectFingerprint } from "@hyperframes/share";
|
||||
import { projectFingerprint, shareSourceFingerprint } from "@hyperframes/share";
|
||||
|
||||
export const shareContentFingerprint = shareSourceFingerprint;
|
||||
|
||||
export interface ShareRender {
|
||||
path: string;
|
||||
id: string;
|
||||
fingerprint: string;
|
||||
contentFingerprint?: string;
|
||||
}
|
||||
|
||||
export function latestShareRender(dir: string): ShareRender | null {
|
||||
@@ -33,16 +36,29 @@ export function latestShareRender(dir: string): ShareRender | null {
|
||||
const expected = join(realpathSync(dir), ".hyperframes", "share-renders", `${value.id}.mp4`);
|
||||
if (value.path !== expected || !existsSync(expected) || realpathSync(expected) !== expected)
|
||||
return null;
|
||||
return { path: expected, id: value.id, fingerprint: value.fingerprint };
|
||||
const contentFingerprint =
|
||||
"contentFingerprint" in value && typeof value.contentFingerprint === "string"
|
||||
? value.contentFingerprint
|
||||
: undefined;
|
||||
return {
|
||||
path: expected,
|
||||
id: value.id,
|
||||
fingerprint: value.fingerprint,
|
||||
...(contentFingerprint && { contentFingerprint }),
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function prepareShareRender(dir: string): Promise<void> {
|
||||
const before = await projectFingerprint(dir);
|
||||
const [before, content] = await Promise.all([
|
||||
projectFingerprint(dir),
|
||||
shareContentFingerprint(dir),
|
||||
]);
|
||||
const previous = latestShareRender(dir);
|
||||
if (previous?.fingerprint === before) return;
|
||||
if (content && previous?.fingerprint === before && previous.contentFingerprint === content)
|
||||
return;
|
||||
const stateFolder = join(dir, ".hyperframes");
|
||||
if (existsSync(stateFolder) && realpathSync(stateFolder) !== stateFolder)
|
||||
throw new Error("Sharing state must stay inside this project, without symlinks.");
|
||||
@@ -79,11 +95,18 @@ export async function prepareShareRender(dir: string): Promise<void> {
|
||||
});
|
||||
if (!existsSync(output))
|
||||
throw new Error("The render did not produce a video. Run hyperframes render to diagnose it.");
|
||||
if ((await projectFingerprint(dir)) !== before)
|
||||
const [after, afterContent] = await Promise.all([
|
||||
projectFingerprint(dir),
|
||||
shareContentFingerprint(dir),
|
||||
]);
|
||||
if (after !== before || afterContent !== content)
|
||||
throw new Error(
|
||||
"The project changed while rendering. Share again to capture the current project.",
|
||||
);
|
||||
await writeFile(temporary, JSON.stringify({ id, path: output, fingerprint: before }));
|
||||
await writeFile(
|
||||
temporary,
|
||||
JSON.stringify({ id, path: output, fingerprint: before, contentFingerprint: content }),
|
||||
);
|
||||
await rename(temporary, target);
|
||||
} catch (error) {
|
||||
await Promise.allSettled([rm(output, { force: true }), rm(temporary, { force: true })]);
|
||||
|
||||
@@ -214,6 +214,18 @@ describe("share operation consent and identity", () => {
|
||||
"Video rejected.",
|
||||
);
|
||||
});
|
||||
it("keeps the kernel's default source callback for a video-only update", async () => {
|
||||
mocks.status.mockResolvedValue(shared);
|
||||
mocks.flags.mockResolvedValue({ versionSharing: true, shareFiles: true });
|
||||
mocks.add.mockResolvedValue(null);
|
||||
|
||||
const result = await createShareService("/project").run({ action: "update", videoOnly: true });
|
||||
|
||||
expect(result.source).toEqual({ at: "video_only", reason: "requested" });
|
||||
expect(mocks.add).toHaveBeenCalledOnce();
|
||||
expect(mocks.add.mock.calls[0]?.[0]).not.toHaveProperty("filesForExport");
|
||||
expect(mocks.add).toHaveBeenCalledWith(expect.objectContaining({ omitSource: true }));
|
||||
});
|
||||
it("does not silently upload changed edits while resuming first share", async () => {
|
||||
mocks.status.mockResolvedValue(shared);
|
||||
mocks.project.mockResolvedValue({ latest_ready_version_id: null });
|
||||
|
||||
@@ -20,12 +20,14 @@ import {
|
||||
import { createShareAccess, shareSite } from "./auth.js";
|
||||
import { assertReviewUploadStorage, bindCliShareProject } from "./host.js";
|
||||
import { prepareShareRender } from "./render.js";
|
||||
import { prepareCliShareSource, uncheckedSource, type PreparedShareSource } from "./sources.js";
|
||||
|
||||
export interface ShareAction {
|
||||
action: "share" | "update" | "status" | "manage";
|
||||
audience?: AccessMode;
|
||||
space?: string;
|
||||
waitSeconds?: number;
|
||||
videoOnly?: boolean;
|
||||
}
|
||||
function linkState(share: ReviewShare | null, ready: boolean): string {
|
||||
if (!share) return "not_shared";
|
||||
@@ -49,6 +51,7 @@ async function readStatus(dir: string, client: ReviewClient, expectedProjectId?:
|
||||
pending,
|
||||
state: linkState(share, ready),
|
||||
ready,
|
||||
source: uncheckedSource(),
|
||||
...statusLinks(activeUrl, ready, processing, project),
|
||||
};
|
||||
}
|
||||
@@ -77,11 +80,13 @@ function selectSpace(spaces: Awaited<ReturnType<ReviewClient["spaces"]>>, id?: s
|
||||
if (id) return spaces.find((space) => space.space_id === id);
|
||||
return spaces.find((space) => space.bound) ?? spaces.find((space) => space.kind === "personal");
|
||||
}
|
||||
function assertUploadProject(share: ReviewShare | null, expectedId?: string): string {
|
||||
function assertUploadProject(
|
||||
share: ReviewShare | null,
|
||||
expectedId?: string,
|
||||
): asserts share is ReviewShare & { projectId: string } {
|
||||
const id = share?.projectId;
|
||||
if (!id || (expectedId && id !== expectedId))
|
||||
throw new ReviewFailed(409, "review_binding_changed");
|
||||
return id;
|
||||
}
|
||||
function assertExistingIntent(share: ReviewShare | null, input: ShareAction) {
|
||||
if (!share) {
|
||||
@@ -145,7 +150,7 @@ export function createShareService(
|
||||
await auth.access();
|
||||
return readStatus(dir, client, expectedProjectId);
|
||||
};
|
||||
const upload = async (input: ShareAction, projectId: string) => {
|
||||
const upload = async (input: ShareAction, projectId: string, sources: PreparedShareSource) => {
|
||||
const end = Date.now() + (input.waitSeconds ?? 300) * 1000;
|
||||
let added = false;
|
||||
const failure = await addVersion({
|
||||
@@ -155,13 +160,16 @@ export function createShareService(
|
||||
client,
|
||||
onProgress: progress,
|
||||
allowPending: true,
|
||||
...(sources.filesForExport && { filesForExport: sources.filesForExport }),
|
||||
requireSource: sources.requireSource,
|
||||
omitSource: sources.source.at === "video_only",
|
||||
onAdded: () => {
|
||||
added = true;
|
||||
},
|
||||
open: () => Date.now() < end,
|
||||
});
|
||||
if (failure) throw new Error(failure);
|
||||
const result = await status(projectId);
|
||||
const result = { ...(await status(projectId)), source: sources.source };
|
||||
if (result.version.failure) throw new Error(result.version.failure);
|
||||
if (!added && !result.version.last?.current)
|
||||
return { ...result, state: "processing", ready: false, url: null };
|
||||
@@ -177,29 +185,45 @@ export function createShareService(
|
||||
spaceId: space,
|
||||
audience: input.audience,
|
||||
});
|
||||
const projectId = assertUploadProject(share, current?.projectId);
|
||||
if (share?.takedown) throw new ReviewFailed(409, "moderation_hold");
|
||||
assertUploadProject(share, current?.projectId);
|
||||
if (share.takedown) throw new ReviewFailed(409, "moderation_hold");
|
||||
assertExistingIntent(share, {
|
||||
...input,
|
||||
space: input.space ?? space,
|
||||
audience: input.audience ?? current?.access,
|
||||
});
|
||||
return projectId;
|
||||
return { projectId: share.projectId, spaceId: share.space?.id, audience: share.access };
|
||||
};
|
||||
const checkPreparedTarget = async (target: Awaited<ReturnType<typeof uploadTarget>>) => {
|
||||
const share = await reviewStatus({ projectPath: dir, client });
|
||||
assertUploadProject(share, target.projectId);
|
||||
if (share.takedown) throw new ReviewFailed(409, "moderation_hold");
|
||||
assertExistingIntent(share, {
|
||||
action: "update",
|
||||
space: target.spaceId,
|
||||
audience: target.audience,
|
||||
});
|
||||
};
|
||||
const run = async (input: ShareAction) => {
|
||||
const current = await status();
|
||||
if (input.action === "status" || input.action === "manage") return current;
|
||||
assertExistingIntent(current.share, input);
|
||||
if (retrievesExisting(current, input)) return current;
|
||||
if (!(await client.reviewFlags()).versionSharing)
|
||||
throw new ReviewFailed(403, "feature_unavailable");
|
||||
const flags = await client.reviewFlags();
|
||||
if (!flags.versionSharing) throw new ReviewFailed(403, "feature_unavailable");
|
||||
const space = current.share
|
||||
? current.share.space?.id
|
||||
: await preflightFirst(dir, client, input);
|
||||
assertResumeSource(current, input);
|
||||
assertReviewUploadStorage();
|
||||
await prepareShareRender(dir);
|
||||
return upload(input, await uploadTarget(current.share, input, space));
|
||||
const target = await uploadTarget(current.share, input, space);
|
||||
const sources = await prepareCliShareSource(dir, auth.access, target.spaceId, {
|
||||
videoOnly: input.videoOnly,
|
||||
shareFiles: flags.shareFiles,
|
||||
});
|
||||
if (sources.requireSource) await checkPreparedTarget(target);
|
||||
return upload(input, target.projectId, sources);
|
||||
};
|
||||
return {
|
||||
status,
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
import { beforeEach, expect, it, vi } from "vitest";
|
||||
const mocks = vi.hoisted(() => ({
|
||||
prepare: vi.fn(),
|
||||
filesForExport: vi.fn(),
|
||||
factory: vi.fn(),
|
||||
content: vi.fn(),
|
||||
}));
|
||||
vi.mock("@hyperframes/share", () => ({
|
||||
shareSources: mocks.factory,
|
||||
projectFingerprint: async () => "rendered",
|
||||
}));
|
||||
vi.mock("./render.js", () => ({
|
||||
latestShareRender: () => ({ fingerprint: "rendered", contentFingerprint: "original-content" }),
|
||||
shareContentFingerprint: mocks.content,
|
||||
}));
|
||||
import { prepareCliShareSource, ShareSourceError } from "./sources.js";
|
||||
import { shareError } from "./errors.js";
|
||||
const access = async () => ({
|
||||
token: "test",
|
||||
heygen: "https://app.test",
|
||||
heygenApi: "https://api.test",
|
||||
owner: "owner",
|
||||
});
|
||||
beforeEach(() => {
|
||||
vi.resetAllMocks();
|
||||
mocks.content.mockResolvedValue("original-content");
|
||||
mocks.factory.mockReturnValue({ prepare: mocks.prepare, filesForExport: mocks.filesForExport });
|
||||
});
|
||||
it("eligible source requires attachment on the version", async () => {
|
||||
const snapshot = { space_id: "space", composition_id: "composition", snapshot_id: "snapshot" };
|
||||
mocks.prepare.mockResolvedValue({ at: "ready", snapshot });
|
||||
expect(await prepareCliShareSource("/project", access, "space", { shareFiles: true })).toEqual({
|
||||
source: { at: "ready", snapshot },
|
||||
requireSource: true,
|
||||
filesForExport: mocks.filesForExport,
|
||||
});
|
||||
expect(mocks.prepare).toHaveBeenCalledWith("/project", "rendered");
|
||||
});
|
||||
it("refuses source bytes that differ from the rendered assets even when metadata matches", async () => {
|
||||
mocks.prepare.mockImplementation(async (dir: string) => {
|
||||
const fingerprint = mocks.factory.mock.calls[0]?.[0].fingerprint;
|
||||
expect(await fingerprint(dir)).toBe("rendered");
|
||||
mocks.content.mockResolvedValue("changed-content");
|
||||
expect(await fingerprint(dir)).toBeNull();
|
||||
return { at: "unavailable", reason: "source_render_changed" };
|
||||
});
|
||||
await expect(
|
||||
prepareCliShareSource("/project", access, "space", { shareFiles: true }),
|
||||
).rejects.toThrow("source_render_changed");
|
||||
});
|
||||
it("free plan ineligibility preserves video sharing and reports source unavailable", async () => {
|
||||
mocks.prepare.mockResolvedValue({ at: "unavailable", reason: "source_plan_required" });
|
||||
expect(await prepareCliShareSource("/project", access, "space", { shareFiles: true })).toEqual({
|
||||
source: { at: "video_only", reason: "source_plan_required" },
|
||||
requireSource: false,
|
||||
});
|
||||
});
|
||||
it("explicit video-only skips source network requests", async () => {
|
||||
await prepareCliShareSource("/project", access, "space", { shareFiles: true, videoOnly: true });
|
||||
expect(mocks.factory).not.toHaveBeenCalled();
|
||||
});
|
||||
it("unexpected upload errors and stale source block with an explicit video-only retry", async () => {
|
||||
mocks.prepare.mockRejectedValueOnce(new Error("network failed"));
|
||||
await expect(
|
||||
prepareCliShareSource("/project", access, "space", { shareFiles: true }),
|
||||
).rejects.toThrow("--video-only");
|
||||
mocks.prepare.mockResolvedValue({ at: "unavailable", reason: "source_render_changed" });
|
||||
await expect(
|
||||
prepareCliShareSource("/project", access, "space", { shareFiles: true }),
|
||||
).rejects.toThrow("--video-only");
|
||||
});
|
||||
|
||||
it("source failure has a stable bridge and JSON error code", () => {
|
||||
expect(shareError(new ShareSourceError("Retry or choose video-only"))).toEqual({
|
||||
code: "source_unavailable",
|
||||
message: "Retry or choose video-only",
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,60 @@
|
||||
import { projectFingerprint, shareSources, type VersionRequest } from "@hyperframes/share";
|
||||
import type { createShareAccess } from "./auth.js";
|
||||
import { latestShareRender, shareContentFingerprint } from "./render.js";
|
||||
|
||||
export class ShareSourceError extends Error {}
|
||||
|
||||
export type ShareSourceState =
|
||||
| { at: "not_checked" }
|
||||
| { at: "video_only"; reason: string }
|
||||
| { at: "ready"; snapshot: { space_id: string; composition_id: string; snapshot_id: string } };
|
||||
export interface PreparedShareSource {
|
||||
source: ShareSourceState;
|
||||
filesForExport?: VersionRequest["filesForExport"];
|
||||
requireSource: boolean;
|
||||
}
|
||||
const unavailablePlans = new Set([
|
||||
"source_disabled",
|
||||
"source_plan_required",
|
||||
"source_composition_limit",
|
||||
]);
|
||||
export function uncheckedSource(): ShareSourceState {
|
||||
return { at: "not_checked" };
|
||||
}
|
||||
export async function prepareCliShareSource(
|
||||
dir: string,
|
||||
access: ReturnType<typeof createShareAccess>["access"],
|
||||
spaceId: string | undefined,
|
||||
options: { videoOnly?: boolean; shareFiles: boolean },
|
||||
): Promise<PreparedShareSource> {
|
||||
if (options.videoOnly) return videoOnly("requested");
|
||||
if (!options.shareFiles) return videoOnly("source_disabled");
|
||||
if (!spaceId) throw new Error("The owning workspace could not be verified. Retry sharing.");
|
||||
try {
|
||||
const render = latestShareRender(dir);
|
||||
const source = shareSources({
|
||||
access,
|
||||
spaceId,
|
||||
fingerprint: async (path) =>
|
||||
render?.contentFingerprint === (await shareContentFingerprint(path))
|
||||
? projectFingerprint(path)
|
||||
: null,
|
||||
});
|
||||
const prepared = await source.prepare(dir, render?.fingerprint ?? null);
|
||||
if (prepared.at === "ready")
|
||||
return { source: prepared, filesForExport: source.filesForExport, requireSource: true };
|
||||
if (unavailablePlans.has(prepared.reason)) return videoOnly(prepared.reason);
|
||||
throw new Error(`Source could not be attached (${prepared.reason}).`);
|
||||
} catch (error) {
|
||||
const reason = error instanceof Error ? error.message : "Source upload failed.";
|
||||
throw new ShareSourceError(
|
||||
`${reason} Retry sharing, or use --video-only to share just the video.`,
|
||||
{
|
||||
cause: error,
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
function videoOnly(reason: string): PreparedShareSource {
|
||||
return { source: { at: "video_only", reason }, requireSource: false };
|
||||
}
|
||||
@@ -35,7 +35,7 @@ const [current, version] = await Promise.all([
|
||||
|
||||
`access(refusedToken?)` returns `{ token, heygen, heygenApi, owner }` or null. It is read for every request. A refused bearer is passed back so the host can refresh it. `heygen` is the review API base. A host can inject its own `signedInRequest` to retain network tracing. `execFile` can likewise be injected to retain native process policy.
|
||||
|
||||
Video uploads require exact storage origins from trusted host configuration. Set `reviewUploadOrigins` on the host or supply a comma-separated `HYPERFRAMES_REVIEW_UPLOAD_ORIGINS` environment variable. Configure the bucket-specific HTTPS origins used by the signer. Use URL origins without paths, trailing slashes, or wildcards. Local tests can explicitly allow an HTTP origin on `127.0.0.1`. An empty list refuses every video upload.
|
||||
Video and source uploads require exact storage origins from trusted host configuration. Set `reviewUploadOrigins` on the host or supply a comma-separated `HYPERFRAMES_REVIEW_UPLOAD_ORIGINS` environment variable. Configure the bucket-specific HTTPS origins used by the signer. Use URL origins without paths, trailing slashes, or wildcards. Local tests can explicitly allow an HTTP origin on `127.0.0.1`. An empty list refuses every upload.
|
||||
|
||||
Before first registration the host must obtain explicit audience consent and prepare a current MP4. `shareForReview` registers/enables the link; it does not imply playable media. Await `addVersion`, then check the server project/latest ready version before reporting success. `addFirstVersion` starts background work and is for long-lived hosts only. `open: () => boolean` on `addVersion` bounds the polling lifetime without discarding the saved operation. `filesForExport(projectPath, renderedFingerprint)` optionally supplies an already-synced matching snapshot. Omit it for ordinary video sharing.
|
||||
|
||||
@@ -50,3 +50,60 @@ Metadata mutations and version uploads use separate cross-process locks under `.
|
||||
`snapshotProject` skips symlinks by default, retaining Desktop watcher, change-set and saved-render compatibility; `snapshotFiles` also skips symlink entries. Set `followSymlinks: true` for a link-aware full snapshot. The CLI-facing `projectFingerprint` opts into following linked media (with cycle detection), so edited linked files invalidate its cached render. Both comparisons ignore HTML ID stamps. Export/history metadata, caches, hidden folders and installed packages are excluded. Hosts must record the fingerprint of the actual rendered source and must not relabel an old render with a newer fingerprint.
|
||||
|
||||
Public type exports include `ShareHost`, `ReviewClient`, `ReviewShare`, `VersionRequest`, and `Wire`, plus `@hyperframes/share/types/reviewWire` and `types/share`. Per-module entry points preserve Desktop imports during adoption.
|
||||
|
||||
## Source snapshots
|
||||
|
||||
`shareSources({access, spaceId})` prepares editable source for a rendered version:
|
||||
|
||||
```js
|
||||
const sources = shareSources({ access, spaceId });
|
||||
const result = await sources.prepare(projectPath, renderedFingerprint);
|
||||
if (result.at === "ready") {
|
||||
await addVersion({
|
||||
projectPath,
|
||||
title,
|
||||
client,
|
||||
filesForExport: sources.filesForExport,
|
||||
requireSource: true,
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
Handle `unavailable.reason` before creating the video version. Plan ineligibility
|
||||
can use video-only; changed source, conflicts and unexpected upload failures need
|
||||
an explicit choice to omit source. The CLI exposes `--video-only` for that choice.
|
||||
`filesForExport` only validates already-prepared source; it does not upload.
|
||||
|
||||
The adapter uses the portable-source-v4 capture/manifest rules and
|
||||
multipart transport. The pinned policy, Unicode 15.1 registry (with its license),
|
||||
and manifest test vectors are checked into this repository. Source is captured through
|
||||
verified file handles; upload bytes are streamed and checked against their hash.
|
||||
Workspace limits, upload receipts and published snapshot identity are checked.
|
||||
Pending uploads retain their original idempotency key across retries. A conflict
|
||||
is never auto-resolved. When local source changes, the adapter cancels an older
|
||||
pending upload before starting another; uploads under verification are preserved.
|
||||
A stalled storage PUT stops after 60 seconds without progress and can retry with
|
||||
the same upload identity.
|
||||
|
||||
Default state uses `.hyperframes/share-sync.<backend-hash>.json`, with strict
|
||||
account/backend/realpath binding, private atomic writes, and a process lock.
|
||||
Hosts can inject a compatible `store` or read-only `existingBinding` resolver.
|
||||
An external Desktop binding is reused only when exact and settled; this adapter
|
||||
never writes Desktop's binding or starts a second upload on its behalf.
|
||||
|
||||
Source sharing requires an authorized account and a workspace that supports editable project files.
|
||||
Source-plan/feature eligibility remains server controlled. Automated fixtures do
|
||||
not establish live entitlement, moderation, source download or outside-editor
|
||||
acceptance. See [the source guide](https://hyperframes.heygen.com/developers/share-source).
|
||||
|
||||
Tests run locally without an account or service credentials:
|
||||
|
||||
```bash
|
||||
node --test packages/share/tests/*.test.mjs
|
||||
bun run --cwd packages/share build
|
||||
```
|
||||
|
||||
The bundled Unicode data records SHA-256 hashes for its inputs in `sources`.
|
||||
Those files are published in the [Unicode 15.1 database](https://www.unicode.org/Public/15.1.0/ucd/).
|
||||
Keep the included Unicode license with redistributed data. The pinned version
|
||||
keeps path comparison consistent across Node versions and operating systems.
|
||||
|
||||
@@ -1,3 +1,6 @@
|
||||
import { cp } from "node:fs/promises";
|
||||
await cp("src/types", "dist/types", { recursive: true });
|
||||
await cp("src/hosts.d.ts", "dist/hosts.d.ts");
|
||||
|
||||
await cp("src/sources/types.d.ts", "dist/sources/types.d.ts");
|
||||
await cp("src/sources/adapterTypes.d.ts", "dist/sources/adapterTypes.d.ts");
|
||||
|
||||
@@ -80,3 +80,9 @@ export { bindShareHost } from "./host.mjs";
|
||||
/** @typedef {import("./types/share.js").AccessMode} AccessMode */
|
||||
/** @typedef {import("./types/share.js").ReviewExportProgress} ReviewExportProgress */
|
||||
export { pendingReviewExport } from "./reviewExport.mjs";
|
||||
|
||||
export {
|
||||
shareSources,
|
||||
shareSourceFingerprint,
|
||||
createProjectShareSourceStore,
|
||||
} from "./shareSources.mjs";
|
||||
|
||||
@@ -80,7 +80,7 @@ async function claim(candidate, lock, owner) {
|
||||
}
|
||||
}
|
||||
}
|
||||
/** @template T @param {string} projectPath @param {"review" | "export"} lane @param {() => Promise<T>} run
|
||||
/** @template T @param {string} projectPath @param {"review" | "export" | "source"} lane @param {() => Promise<T>} run
|
||||
* @param {{ timeoutMs?: number, signal?: AbortSignal }} [options] @returns {Promise<T>} */
|
||||
export async function withProjectShareLock(
|
||||
projectPath,
|
||||
@@ -90,7 +90,14 @@ export async function withProjectShareLock(
|
||||
) {
|
||||
const root = await realpath(projectPath);
|
||||
const folder = await mkdirInside(root, ".hyperframes");
|
||||
const lock = join(folder, lane === "review" ? "review.lock" : "review-export.lock");
|
||||
const lock = join(
|
||||
folder,
|
||||
lane === "review"
|
||||
? "review.lock"
|
||||
: lane === "source"
|
||||
? "share-source.lock"
|
||||
: "review-export.lock",
|
||||
);
|
||||
const owner = { token: randomUUID(), pid: process.pid, hostname: hostname() };
|
||||
const candidate = `${lock}.${owner.token}.tmp`;
|
||||
await writeFile(candidate, JSON.stringify(owner), { flag: "wx", mode: 0o600 });
|
||||
|
||||
@@ -31,7 +31,7 @@ async function snapshotKey(full, byContent) {
|
||||
const stat = await handle.stat();
|
||||
if (!stat.isFile()) return undefined;
|
||||
if (full.endsWith(".html")) return contentKey(await handle.readFile("utf8"));
|
||||
if (byContent) return sha256(await handle.readFile());
|
||||
if (byContent) return await hashFile(handle);
|
||||
return `${stat.mtimeMs}:${stat.size}`;
|
||||
} catch {
|
||||
return undefined;
|
||||
@@ -40,6 +40,13 @@ async function snapshotKey(full, byContent) {
|
||||
}
|
||||
}
|
||||
|
||||
/** @param {import("node:fs/promises").FileHandle} handle */
|
||||
async function hashFile(handle) {
|
||||
const hash = createHash("sha256");
|
||||
for await (const chunk of handle.createReadStream({ autoClose: false })) hash.update(chunk);
|
||||
return hash.digest("hex");
|
||||
}
|
||||
|
||||
/** @param {import("node:fs").Dirent} entry @param {string} full @param {boolean} followSymlinks */
|
||||
async function entryKind(entry, full, followSymlinks) {
|
||||
if (!entry.isSymbolicLink()) return entry;
|
||||
@@ -115,6 +122,6 @@ export function diffChangedFiles(before, after) {
|
||||
return changed;
|
||||
}
|
||||
|
||||
/** @param {string} projectPath */
|
||||
export const projectFingerprint = async (projectPath) =>
|
||||
snapshotFingerprint(await snapshotProject(projectPath, { followSymlinks: true }));
|
||||
/** @param {string} projectPath @param {{ byContent?: boolean }} [options] */
|
||||
export const projectFingerprint = async (projectPath, options) =>
|
||||
snapshotFingerprint(await snapshotProject(projectPath, { ...options, followSymlinks: true }));
|
||||
|
||||
@@ -40,7 +40,8 @@ import { versionSentence } from "./reviewVersionSentences.mjs";
|
||||
* known; `last`: the newest version.
|
||||
* @typedef {{ pending: { body: ExportRequest, videoId: string | null, projectId?: string | null,
|
||||
* checkingSince?: number, snapshotRefused?: boolean, unanswered?: boolean, reviewSent?: ReviewSent } | null,
|
||||
* last: { fingerprint: string, versionId: string, exportedAt: string, projectId: string, files?: boolean, checksum?: string }
|
||||
* last: { fingerprint: string, versionId: string, exportedAt: string, projectId: string, files?: boolean,
|
||||
* snapshot?: ExportRequest["snapshot"], checksum?: string }
|
||||
* | null }} VersionRecord */
|
||||
|
||||
/** `onAdded`: a version went ready, with whether it carries files and whether it is the page's first from here.
|
||||
@@ -53,7 +54,7 @@ import { versionSentence } from "./reviewVersionSentences.mjs";
|
||||
* @typedef {VersionHooks & { projectPath: string, title: string, client: ReviewClient, put?: typeof putFile,
|
||||
* sleep?: (ms: number) => Promise<void>, probe?: (file: string) => Promise<VideoFacts | null>, now?: () => number,
|
||||
* projectId?: string | null, filesForExport?: FilesForRender, fromReview?: FromReview | null,
|
||||
* allowPending?: boolean }} VersionRequest */
|
||||
* allowPending?: boolean, requireSource?: boolean, omitSource?: boolean }} VersionRequest */
|
||||
|
||||
/** The snapshot of `rendered`, the project as the export was rendered; null when the folder is not that render.
|
||||
* @typedef {(projectPath: string, rendered: string | null) => Promise<import("./types/reviewWire.js").SnapshotRef | null>}
|
||||
@@ -178,7 +179,15 @@ function isLast(value) {
|
||||
isSavedId(value.versionId) &&
|
||||
isSavedId(value.exportedAt) &&
|
||||
isSavedId(value.projectId) &&
|
||||
isLastMetadata(value)
|
||||
);
|
||||
}
|
||||
|
||||
/** @param {Record<string, unknown>} value */
|
||||
function isLastMetadata(value) {
|
||||
return (
|
||||
isOptionalBoolean(value.files) &&
|
||||
isOptionalSnapshot(value.snapshot) &&
|
||||
(value.checksum === undefined || isSavedId(value.checksum))
|
||||
);
|
||||
}
|
||||
@@ -280,8 +289,8 @@ export async function reviewVersion(projectPath, server = {}) {
|
||||
|
||||
/** The newest MP4 and the body that adds it; a stale, non-MP4, too big or too long export never sends anything.
|
||||
* @param {string} dir @param {string} title @param {(file: string) => Promise<VideoFacts | null>} probe
|
||||
* @param {FilesForRender} filesForExport */
|
||||
async function exportSource(dir, title, probe, filesForExport) {
|
||||
* @param {FilesForRender} filesForExport @param {boolean} requireSource */
|
||||
async function exportSource(dir, title, probe, filesForExport, requireSource) {
|
||||
const latest = latestExportFile(dir);
|
||||
if (!latest || sourceOf(dir, await projectFingerprint(dir)) !== "ready")
|
||||
return { failure: local("Export an MP4 of the project as it is now first.") };
|
||||
@@ -308,18 +317,34 @@ async function exportSource(dir, title, probe, filesForExport) {
|
||||
content_type: "video/mp4",
|
||||
},
|
||||
};
|
||||
const snapshot = await filesForExport(dir, body.source_fingerprint).catch((error) => {
|
||||
// expected: the version still goes up, and Share's files line says it has no files and offers Sync now.
|
||||
console.error(
|
||||
"[review files] couldn't read the synced snapshot; the version goes up without files:",
|
||||
error,
|
||||
);
|
||||
return null;
|
||||
});
|
||||
const snapshot = await sourceSnapshot(
|
||||
dir,
|
||||
body.source_fingerprint,
|
||||
filesForExport,
|
||||
requireSource,
|
||||
);
|
||||
if (missingRequiredSource(requireSource, snapshot))
|
||||
return {
|
||||
failure: local(
|
||||
"Source changed after preparation. Retry sharing, or choose video-only explicitly.",
|
||||
),
|
||||
};
|
||||
if (snapshot) body.snapshot = snapshot;
|
||||
return { file: latest.path, body };
|
||||
}
|
||||
|
||||
/** @param {boolean} required @param {import("./types/reviewWire.js").SnapshotRef | null} snapshot */
|
||||
const missingRequiredSource = (required, snapshot) => required && !snapshot;
|
||||
|
||||
/** @param {string} dir @param {string | null} rendered @param {FilesForRender} filesForExport @param {boolean} requireSource */
|
||||
async function sourceSnapshot(dir, rendered, filesForExport, requireSource) {
|
||||
return filesForExport(dir, rendered).catch((error) => {
|
||||
if (requireSource) throw error;
|
||||
console.error("[review files] could not read source; adding video only:", error);
|
||||
return null;
|
||||
});
|
||||
}
|
||||
|
||||
/** @param {ExportRequest["snapshot"]} a @param {ExportRequest["snapshot"]} b */
|
||||
const sameSnapshot = (a, b) =>
|
||||
a?.space_id === b?.space_id &&
|
||||
@@ -369,7 +394,7 @@ async function createdExport(r, id, pending, kept) {
|
||||
return await sendCreate(r, id, pending, kept);
|
||||
} catch (error) {
|
||||
const refused = error instanceof ReviewFailed && error.reason === "snapshot_mismatch";
|
||||
if (!refused || !pending.body.snapshot || pending.unanswered) throw error;
|
||||
if (!refused || !pending.body.snapshot || pending.unanswered || r.requireSource) throw error;
|
||||
console.error("[review] snapshot refused; the version goes up without files:", error);
|
||||
const body = { ...pending.body, client_operation_id: `export-${randomUUID()}` };
|
||||
delete body.snapshot;
|
||||
@@ -571,6 +596,7 @@ async function finish(dir, state, pending, id, done, fromReview) {
|
||||
exportedAt: done.exported_at,
|
||||
projectId: id,
|
||||
files: Boolean(pending.body.snapshot),
|
||||
...(pending.body.snapshot && { snapshot: pending.body.snapshot }),
|
||||
checksum: pending.body.local_upload.checksum_sha256,
|
||||
};
|
||||
await writeState(dir, { pending: null, last: ready ? last : state.last });
|
||||
@@ -583,13 +609,32 @@ async function finish(dir, state, pending, id, done, fromReview) {
|
||||
async function run(r, report) {
|
||||
report({ step: "preparing", sent: 0, total: 0 });
|
||||
const state = await readState(r.projectPath);
|
||||
const source = await exportSource(r.projectPath, r.title, r.probe, r.filesForExport);
|
||||
const source = await exportSource(
|
||||
r.projectPath,
|
||||
r.title,
|
||||
r.probe,
|
||||
r.omitSource ? noSource : r.filesForExport,
|
||||
r.requireSource,
|
||||
);
|
||||
if ("failure" in source) return source.failure ?? null;
|
||||
const id = r.projectId ?? (await reviewProjectId(r.projectPath));
|
||||
if (!id) return local("Share the project first, then add a version.");
|
||||
if (await alreadyUploaded(r.client, id, state, source.body)) return null;
|
||||
if (
|
||||
sameSourceMode(r, state.last, source.body.snapshot) &&
|
||||
(await alreadyUploaded(r.client, id, state, source.body))
|
||||
)
|
||||
return null;
|
||||
return sendVersion(r, report, source, id, state);
|
||||
}
|
||||
const noSource = async () => null;
|
||||
|
||||
/** @param {Required<VersionRequest>} r @param {VersionRecord["last"]} last @param {ExportRequest["snapshot"]} snapshot */
|
||||
function sameSourceMode(r, last, snapshot) {
|
||||
if (r.requireSource) return sameSnapshot(last?.snapshot, snapshot);
|
||||
if (r.omitSource) return last?.files !== true;
|
||||
return true;
|
||||
}
|
||||
|
||||
/** @param {ReviewClient} client @param {string} id @param {VersionRecord} state @param {ExportRequest} body */
|
||||
async function alreadyUploaded(client, id, state, body) {
|
||||
const last = state.last;
|
||||
@@ -606,6 +651,8 @@ async function alreadyUploaded(client, id, state, body) {
|
||||
async function sendVersion(r, report, source, id, state) {
|
||||
const first = state.last?.projectId !== id;
|
||||
let pending = bodyToSend(state, source.body, id);
|
||||
const sourceConflict = pendingSourceConflict(r, pending, source.body);
|
||||
if (sourceConflict) return local(sourceConflict);
|
||||
await writeState(r.projectPath, (state = { ...state, pending }));
|
||||
try {
|
||||
const answer = await createdExport(r, id, pending, (next) =>
|
||||
@@ -638,6 +685,17 @@ async function sendVersion(r, report, source, id, state) {
|
||||
}
|
||||
}
|
||||
|
||||
/** @param {Required<VersionRequest>} r @param {NonNullable<VersionRecord["pending"]>} pending @param {ExportRequest} fresh */
|
||||
function pendingSourceConflict(r, pending, fresh) {
|
||||
if (r.requireSource && !pending.body.snapshot)
|
||||
return "A pending video-only upload must finish before attaching source. Retry with --video-only, then share source again.";
|
||||
if (r.omitSource && pending.body.snapshot)
|
||||
return "A pending upload includes source. Check its status or retry with source before creating a video-only version.";
|
||||
if (r.requireSource && !sameSnapshot(pending.body.snapshot, fresh.snapshot))
|
||||
return "A pending upload includes a different source snapshot. Check its status before sharing again.";
|
||||
return null;
|
||||
}
|
||||
|
||||
/** @param {Required<VersionRequest>} r @param {VersionRecord} state
|
||||
* @param {NonNullable<VersionRecord["pending"]>} pending @param {string} id @param {ExportDTO} done @param {boolean} first */
|
||||
async function finishRun(r, state, pending, id, done, first) {
|
||||
@@ -665,8 +723,10 @@ export async function addVersion(request) {
|
||||
onProgress: () => {},
|
||||
open: () => true,
|
||||
allowPending: false,
|
||||
filesForExport: async () => null,
|
||||
filesForExport: noSource,
|
||||
fromReview: null,
|
||||
requireSource: false,
|
||||
omitSource: false,
|
||||
onAdded: () => {},
|
||||
...request,
|
||||
projectId: request.projectId ?? null,
|
||||
|
||||
@@ -18,6 +18,22 @@ export function fileSha256Base64(file) {
|
||||
|
||||
/** @typedef {"sent" | "expired" | "refused" | "offline"} PutOutcome */
|
||||
|
||||
/** @param {string} url */
|
||||
export function trustedUploadUrl(url) {
|
||||
if (!URL.canParse(url)) return null;
|
||||
const target = new URL(url);
|
||||
const loopback = target.protocol === "http:" && target.hostname === "127.0.0.1";
|
||||
if (
|
||||
(target.protocol !== "https:" && !loopback) ||
|
||||
target.username ||
|
||||
target.password ||
|
||||
target.hash ||
|
||||
!reviewUploadOrigins().includes(target.origin)
|
||||
)
|
||||
return null;
|
||||
return target;
|
||||
}
|
||||
|
||||
/** Streams the file to the signed URL through main's fetch (the app's proxy), the server's headers as given plus
|
||||
* the length, and no bearer. Nothing sent or answered for `stallMs` ends it as offline, never "Uploading" forever.
|
||||
* @param {{ put_url: string, headers: Record<string, string> }} upload @param {string} file @param {number} size
|
||||
@@ -29,22 +45,8 @@ export async function putFile(
|
||||
onSent,
|
||||
{ stallMs = 60_000, fetchImpl = fetch } = {},
|
||||
) {
|
||||
let target;
|
||||
try {
|
||||
target = new URL(upload.put_url);
|
||||
} catch {
|
||||
return "refused";
|
||||
}
|
||||
const loopback = target.protocol === "http:" && target.hostname === "127.0.0.1";
|
||||
if (
|
||||
(target.protocol !== "https:" && !loopback) ||
|
||||
target.username ||
|
||||
target.password ||
|
||||
target.hash ||
|
||||
!reviewUploadOrigins().includes(target.origin)
|
||||
) {
|
||||
return "refused";
|
||||
}
|
||||
const target = trustedUploadUrl(upload.put_url);
|
||||
if (!target) return "refused";
|
||||
const stall = new AbortController();
|
||||
/** @type {ReturnType<typeof setTimeout> | undefined} */
|
||||
let timer;
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
// @ts-check
|
||||
import { realpath } from "node:fs/promises";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { basename } from "node:path";
|
||||
import { projectFingerprint } from "./projectSnapshot.mjs";
|
||||
import { withProjectShareLock } from "./projectLock.mjs";
|
||||
import { captureProject } from "./sources/capture.mjs";
|
||||
import { sourceApi } from "./sources/api.mjs";
|
||||
import { createProjectShareSourceStore } from "./sources/store.mjs";
|
||||
import { realWait } from "./sources/transfer.mjs";
|
||||
import { recoverSourceUpload, uploadSource } from "./sources/upload.mjs";
|
||||
|
||||
/** @typedef {import("./sources/adapterTypes.js").SourceResult} SourceResult
|
||||
* @typedef {import("./sources/types.js").Binding} Binding */
|
||||
/** @param {string} reason @returns {SourceResult} */
|
||||
const unavailable = (reason) => ({ at: "unavailable", reason });
|
||||
/** Content identity for sharing, including portable hidden render inputs. The Desktop watch inventory stays separate.
|
||||
* @param {string} projectPath @returns {Promise<string | null>} */
|
||||
export async function shareSourceFingerprint(projectPath) {
|
||||
const capture = await captureProject(await realpath(projectPath));
|
||||
return capture.at === "captured" && !captureRefusal(capture) ? capture.manifest.sha256 : null;
|
||||
}
|
||||
/** @param {Binding} binding @param {string} space */
|
||||
function bindingRefusal(binding, space) {
|
||||
if (binding.ownerSpaceId !== space) return "source_other_workspace";
|
||||
if (binding.sharedFrom) return "source_owned_by_someone_else";
|
||||
if (binding.gettingFromCloud || binding.duplicate) return "source_operation_in_progress";
|
||||
return null;
|
||||
}
|
||||
/** @param {import("./sources/types.js").Capture} capture */
|
||||
function captureRefusal(capture) {
|
||||
if (capture.at === "too-big") return `source_${capture.over.limit}_exceeded`;
|
||||
if (capture.at === "problems") return `source_${capture.problems[0]?.problem ?? "invalid"}`;
|
||||
if (capture.settled.agentTurnOpen || capture.settled.projectLocked || capture.waiting.length)
|
||||
return "source_edit_in_progress";
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Prepare source explicitly before adding the video version. No unexpected network or filesystem failure is
|
||||
* converted to video-only. `filesForExport` revalidates the prepared bytes and never starts another upload.
|
||||
* @param {import("./sources/adapterTypes.js").SourceOptions} options */
|
||||
export function shareSources(options) {
|
||||
const store = options.store ?? createProjectShareSourceStore();
|
||||
const fingerprint = options.fingerprint ?? projectFingerprint;
|
||||
const signal = options.signal ?? new AbortController().signal;
|
||||
/** @type {Map<string, { rendered: string, digest: string, snapshot: import("./types/reviewWire.js").SnapshotRef, account: string, backend: string }>} */
|
||||
const prepared = new Map();
|
||||
/** @param {string} projectPath @param {string | null} rendered @returns {Promise<SourceResult>} */
|
||||
const prepare = async (projectPath, rendered) => {
|
||||
const root = await realpath(projectPath);
|
||||
prepared.delete(root);
|
||||
if (!enabled(options)) return unavailable("source_disabled");
|
||||
if (!rendered || (await fingerprint(root)) !== rendered)
|
||||
return unavailable("source_render_changed");
|
||||
return withProjectShareLock(
|
||||
root,
|
||||
"source",
|
||||
async () => {
|
||||
const signed = await options.access();
|
||||
if (!signed?.owner) return unavailable("source_signed_out");
|
||||
const key = { backend: signed.heygenApi, account: signed.owner, projectPath: root };
|
||||
const guardedAccess = async (/** @type {string | undefined} */ refused) => {
|
||||
const next = await options.access(refused);
|
||||
if (!next || next.owner !== key.account || next.heygenApi !== key.backend)
|
||||
throw new Error("Source account or backend changed while sharing.");
|
||||
return next;
|
||||
};
|
||||
const api =
|
||||
options.api ?? sourceApi({ access: guardedAccess, fetchImpl: options.fetchImpl });
|
||||
const { own, existing, binding } = await selectBinding(key, options, store);
|
||||
const refused = bindingRefusal(binding, options.spaceId);
|
||||
if (refused) return unavailable(refused);
|
||||
const result = await prepareBinding({
|
||||
binding,
|
||||
own: Boolean(own),
|
||||
external: Boolean(existing),
|
||||
api,
|
||||
store,
|
||||
options,
|
||||
signal,
|
||||
fingerprint,
|
||||
rendered,
|
||||
});
|
||||
const manifest = result.manifest;
|
||||
await guardedAccess(undefined);
|
||||
if (result.at !== "ready" || !manifest) return result;
|
||||
if (!(await stillMatches(root, rendered, manifest.sha256, fingerprint)))
|
||||
return unavailable("source_render_changed");
|
||||
prepared.set(root, {
|
||||
rendered,
|
||||
digest: manifest.sha256,
|
||||
snapshot: result.snapshot,
|
||||
account: key.account,
|
||||
backend: key.backend,
|
||||
});
|
||||
return { at: "ready", snapshot: result.snapshot };
|
||||
},
|
||||
{ signal },
|
||||
);
|
||||
};
|
||||
/** @param {string} projectPath @param {string | null} rendered */
|
||||
const filesForExport = async (projectPath, rendered) => {
|
||||
const root = await realpath(projectPath);
|
||||
const held = prepared.get(root);
|
||||
const signed = await options.access();
|
||||
if (
|
||||
!held ||
|
||||
!rendered ||
|
||||
held.rendered !== rendered ||
|
||||
!sameOwner(signed, held) ||
|
||||
!enabled(options)
|
||||
)
|
||||
return null;
|
||||
return (await stillMatches(root, rendered, held.digest, fingerprint)) ? held.snapshot : null;
|
||||
};
|
||||
return { prepare, filesForExport };
|
||||
}
|
||||
|
||||
/** @param {import("./sources/adapterTypes.js").SourceOptions} options */
|
||||
const enabled = (options) => options.enabled?.() ?? true;
|
||||
/** @param {Awaited<ReturnType<import("./signedInRequest.mjs").SignedInAccess>>} signed
|
||||
* @param {{account: string, backend: string}} held */
|
||||
const sameOwner = (signed, held) =>
|
||||
signed?.owner === held.account && signed?.heygenApi === held.backend;
|
||||
/** @typedef {{ binding: Binding, own: boolean, external: boolean, api: import("./sources/api.mjs").SourceApi,
|
||||
* store: import("./sources/adapterTypes.js").SourceStore, options: import("./sources/adapterTypes.js").SourceOptions,
|
||||
* signal: AbortSignal, fingerprint: (path: string) => Promise<string | null>, rendered: string }} Preparation */
|
||||
/** @param {Preparation} job @returns {Promise<SourceResult & {manifest?: import("./sources/types.js").Manifest}>} */
|
||||
async function prepareBinding(job) {
|
||||
const { binding, api, fingerprint, rendered } = job;
|
||||
const limits = await api.limits(binding.ownerSpaceId);
|
||||
if (!limits.syncAllowed) return unavailable("source_plan_required");
|
||||
const capture = await captureProject(binding.projectPath, { limits: limits.push });
|
||||
const bad = captureRefusal(capture);
|
||||
if (bad) return unavailable(bad);
|
||||
if (capture.at !== "captured") return unavailable("source_invalid");
|
||||
if ((await fingerprint(binding.projectPath)) !== rendered)
|
||||
return unavailable("source_render_changed");
|
||||
const result = await snapshotFor(job, capture.manifest, limits);
|
||||
return { ...result, manifest: capture.manifest };
|
||||
}
|
||||
/** @param {Preparation} job @param {import("./sources/types.js").Manifest} manifest
|
||||
* @param {Extract<import("./sources/types.js").SyncLimitsRead, {at: "read"}>} limits @returns {Promise<SourceResult>} */
|
||||
async function snapshotFor(job, manifest, limits) {
|
||||
const { binding, api, store, options, signal } = job;
|
||||
const blocked = await recoverEarlierCapture(job, manifest.sha256);
|
||||
if (blocked) return blocked;
|
||||
if (binding.base?.manifestSha256 === manifest.sha256 && !binding.inflight)
|
||||
return reuseSnapshot(api, binding, manifest.sha256);
|
||||
if (job.external) return unavailable("source_sync_required");
|
||||
if (!job.own && limits.compositions >= limits.maxCompositions)
|
||||
return unavailable("source_composition_limit");
|
||||
await store.write(binding);
|
||||
binding.composition = await api.create(
|
||||
binding.ownerSpaceId,
|
||||
binding.compositionId,
|
||||
basename(binding.projectPath),
|
||||
);
|
||||
await store.write(binding);
|
||||
return uploadSource({ binding, manifest, api, store, signal, ...uploadOptions(options) });
|
||||
}
|
||||
/** @param {Preparation} job @param {string} digest */
|
||||
function recoverEarlierCapture({ external, binding, api, store }, digest) {
|
||||
const inflight = binding.inflight;
|
||||
if (external || !inflight || inflight.manifestSha256 === digest) return null;
|
||||
return recoverSourceUpload({ binding, api, store });
|
||||
}
|
||||
|
||||
/** @param {import("./sources/adapterTypes.js").SourceOptions} options */
|
||||
function uploadOptions(options) {
|
||||
return {
|
||||
fetchImpl: options.fetchImpl ?? fetch,
|
||||
wait: options.wait ?? realWait,
|
||||
timeoutMs: options.verificationTimeoutMs ?? 120_000,
|
||||
onProgress: options.onProgress ?? (() => {}),
|
||||
};
|
||||
}
|
||||
|
||||
/** @param {import("./sources/types.js").BindingKey} key @param {import("./sources/adapterTypes.js").SourceOptions} options
|
||||
* @param {import("./sources/adapterTypes.js").SourceStore} store */
|
||||
async function selectBinding(key, options, store) {
|
||||
const own = await store.read(key);
|
||||
const existing = own ? null : await options.existingBinding?.(key);
|
||||
return { own, existing, binding: own ?? existing ?? newBinding(key, options.spaceId) };
|
||||
}
|
||||
/** @param {import("./sources/types.js").BindingKey} key @param {string} space @returns {Binding} */
|
||||
function newBinding(key, space) {
|
||||
return {
|
||||
...key,
|
||||
version: 1,
|
||||
compositionId: randomUUID(),
|
||||
ownerSpaceId: space,
|
||||
boundAt: Date.now(),
|
||||
composition: null,
|
||||
etag: null,
|
||||
base: null,
|
||||
inflight: null,
|
||||
};
|
||||
}
|
||||
/** @param {import("./sources/api.mjs").SourceApi} api @param {Binding} binding @param {string} digest @returns {Promise<SourceResult>} */
|
||||
async function reuseSnapshot(api, binding, digest) {
|
||||
if (!binding.base) return unavailable("source_sync_required");
|
||||
const remote = await api.snapshot(
|
||||
binding.ownerSpaceId,
|
||||
binding.compositionId,
|
||||
binding.base.snapshotId,
|
||||
);
|
||||
if (
|
||||
remote.id !== binding.base.snapshotId ||
|
||||
remote.compositionId !== binding.compositionId ||
|
||||
remote.manifestSha256 !== digest ||
|
||||
remote.manifest?.sha256 !== digest ||
|
||||
remote.publicationOutcome !== "published"
|
||||
)
|
||||
return unavailable("source_snapshot_mismatch");
|
||||
return {
|
||||
at: "ready",
|
||||
snapshot: {
|
||||
space_id: binding.ownerSpaceId,
|
||||
composition_id: binding.compositionId,
|
||||
snapshot_id: remote.id,
|
||||
},
|
||||
};
|
||||
}
|
||||
/** @param {string} root @param {string} rendered @param {string} digest @param {(path: string) => Promise<string | null>} fingerprint */
|
||||
async function stillMatches(root, rendered, digest, fingerprint) {
|
||||
if ((await fingerprint(root)) !== rendered) return false;
|
||||
const capture = await captureProject(root);
|
||||
return (
|
||||
!captureRefusal(capture) &&
|
||||
capture.at === "captured" &&
|
||||
capture.manifest.sha256 === digest &&
|
||||
(await fingerprint(root)) === rendered
|
||||
);
|
||||
}
|
||||
export { createProjectShareSourceStore } from "./sources/store.mjs";
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
import type { Binding, BindingKey, Manifest } from "./types.js";
|
||||
import type { SnapshotRef } from "../types/reviewWire.js";
|
||||
import type { SignedInAccess } from "../signedInRequest.mjs";
|
||||
import type { SourceApi } from "./api.mjs";
|
||||
|
||||
export interface SourceStore {
|
||||
read(key: BindingKey): Promise<Binding | null>;
|
||||
write(binding: Binding): Promise<void>;
|
||||
}
|
||||
export type SourceResult =
|
||||
| { at: "ready"; snapshot: SnapshotRef }
|
||||
| { at: "unavailable"; reason: string };
|
||||
export interface SourceOptions {
|
||||
access: SignedInAccess;
|
||||
spaceId: string;
|
||||
store?: SourceStore;
|
||||
enabled?: () => boolean;
|
||||
fingerprint?: (path: string) => Promise<string | null>;
|
||||
/** Host-owned Desktop bindings may be read for exact reuse, never overwritten. */
|
||||
existingBinding?: (key: BindingKey) => Promise<Binding | null>;
|
||||
fetchImpl?: typeof fetch;
|
||||
signal?: AbortSignal;
|
||||
verificationTimeoutMs?: number;
|
||||
onProgress?: (bytes: number) => void;
|
||||
/** Test/host seams; the default is the authenticated v3 client. */
|
||||
api?: SourceApi;
|
||||
wait?: (ms: number, signal?: AbortSignal) => Promise<void>;
|
||||
}
|
||||
export interface SourceRun {
|
||||
binding: Binding;
|
||||
manifest: Manifest;
|
||||
api: SourceApi;
|
||||
store: SourceStore;
|
||||
fetchImpl: typeof fetch;
|
||||
signal: AbortSignal;
|
||||
wait: (ms: number, signal?: AbortSignal) => Promise<void>;
|
||||
timeoutMs: number;
|
||||
onProgress: (bytes: number) => void;
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
// @ts-check
|
||||
import { signedInRequest } from "../signedInRequest.mjs";
|
||||
import { SyncApiError, refusalOf } from "./errors.mjs";
|
||||
import {
|
||||
beginOf,
|
||||
grantsOf,
|
||||
objectOf,
|
||||
uploadOf,
|
||||
syncLimitsOf,
|
||||
compositionOf,
|
||||
snapshotOf,
|
||||
} from "./wire.mjs";
|
||||
|
||||
/** @param {{ access: import("../signedInRequest.mjs").SignedInAccess, fetchImpl?: typeof fetch }} deps */
|
||||
export function sourceApi({ access, fetchImpl = fetch }) {
|
||||
const request = signedInRequest({
|
||||
access,
|
||||
fetchImpl,
|
||||
timeoutMs: 120_000,
|
||||
tag: "share-source",
|
||||
host: "heygenApi",
|
||||
offline: () => new SyncApiError("offline"),
|
||||
signedOut: () => new SyncApiError("signed-out"),
|
||||
});
|
||||
/** @param {string} method @param {string} route @param {unknown} [body] @param {Record<string,string>} [headers] */
|
||||
const send = async (method, route, body, headers) => {
|
||||
const { reply } = await request(method, route, body, headers);
|
||||
if (!reply.ok) throw await refusalOf(reply);
|
||||
return objectOf(await reply.json(), "envelope").data;
|
||||
};
|
||||
/** @param {string} space @param {string} composition */
|
||||
const path = (space, composition) =>
|
||||
`/v3/hyperframes/workspaces/${encodeURIComponent(space)}/compositions/${encodeURIComponent(composition)}`;
|
||||
return {
|
||||
/** @param {string} space */
|
||||
limits: async (space) =>
|
||||
syncLimitsOf(
|
||||
await send(
|
||||
"GET",
|
||||
`/v3/hyperframes/workspaces/${encodeURIComponent(space)}/composition-sync/limits`,
|
||||
),
|
||||
),
|
||||
/** @param {string} space @param {string} composition @param {string} title */
|
||||
create: async (space, composition, title) =>
|
||||
compositionOf(await send("PUT", path(space, composition), { title })),
|
||||
/** @param {string} space @param {string} composition @param {string} snapshot */
|
||||
snapshot: async (space, composition, snapshot) =>
|
||||
snapshotOf(
|
||||
await send(
|
||||
"GET",
|
||||
`${path(space, composition)}/snapshots/${encodeURIComponent(snapshot)}?include=manifest`,
|
||||
),
|
||||
),
|
||||
/** @param {string} space @param {string} composition @param {import("./types.js").BeginRequest} body */
|
||||
begin: async (space, composition, body) =>
|
||||
beginOf(
|
||||
await send(
|
||||
"POST",
|
||||
`${path(space, composition)}/uploads`,
|
||||
{
|
||||
base_snapshot_id: body.baseSnapshotId,
|
||||
resolves_conflict_snapshot_ids: [],
|
||||
client_type: "cli",
|
||||
manifest_sha256: body.manifest.sha256,
|
||||
manifest: body.manifest.document,
|
||||
},
|
||||
{ "Idempotency-Key": body.idempotencyKey },
|
||||
),
|
||||
false,
|
||||
),
|
||||
/** @param {string} space @param {string} composition @param {string} upload @param {import("./types.js").GrantRequest[]} files */
|
||||
uploadUrls: async (space, composition, upload, files) =>
|
||||
grantsOf(
|
||||
await send(
|
||||
"POST",
|
||||
`${path(space, composition)}/uploads/${encodeURIComponent(upload)}/upload-urls`,
|
||||
{
|
||||
files: files.map((file) => ({
|
||||
sha256: file.sha256,
|
||||
...(file.multipartUploadId && { multipart_upload_id: file.multipartUploadId }),
|
||||
...(file.parts && {
|
||||
parts: file.parts.map((part) => ({
|
||||
part_number: part.partNumber,
|
||||
sha256: part.sha256,
|
||||
})),
|
||||
}),
|
||||
})),
|
||||
},
|
||||
),
|
||||
),
|
||||
/** @param {string} space @param {string} composition @param {string} upload */
|
||||
status: async (space, composition, upload) =>
|
||||
uploadOf(
|
||||
await send("GET", `${path(space, composition)}/uploads/${encodeURIComponent(upload)}`),
|
||||
),
|
||||
/** @param {string} space @param {string} composition @param {string} upload */
|
||||
cancel: async (space, composition, upload) =>
|
||||
uploadOf(
|
||||
await send(
|
||||
"POST",
|
||||
`${path(space, composition)}/uploads/${encodeURIComponent(upload)}/cancel`,
|
||||
),
|
||||
),
|
||||
/** @param {string} space @param {string} composition @param {string} upload */
|
||||
submit: async (space, composition, upload) =>
|
||||
uploadOf(
|
||||
await send(
|
||||
"POST",
|
||||
`${path(space, composition)}/uploads/${encodeURIComponent(upload)}/verification`,
|
||||
),
|
||||
),
|
||||
};
|
||||
}
|
||||
/** @typedef {ReturnType<typeof sourceApi>} SourceApi */
|
||||
@@ -0,0 +1,262 @@
|
||||
// @ts-check
|
||||
// Captures a project folder as the canonical manifest the server accepts: every portable regular file with its
|
||||
// SHA-256 and length. A file that cannot be read, a link, a special file or a name no client can store fails the
|
||||
// capture with a list of them; it never becomes a deletion. Limits are checked from sizes alone, before any hashing.
|
||||
import { createHash } from "node:crypto";
|
||||
import { constants } from "node:fs";
|
||||
import { lstat, open, readdir, realpath } from "node:fs/promises";
|
||||
import { dirname, join } from "node:path";
|
||||
|
||||
import { ENTRYPOINT, ManifestInvalid, canonicalBytes, parseManifest } from "./manifest.mjs";
|
||||
import { V1_LIMITS, overLimit } from "./limits.mjs";
|
||||
import {
|
||||
PathViolation,
|
||||
checkInventoryPaths,
|
||||
classifyPath,
|
||||
excludedFolder,
|
||||
prefiltered,
|
||||
skippedSymlink,
|
||||
validatePath,
|
||||
} from "./pathPolicy.mjs";
|
||||
import { readRenderOutputs, settledState } from "./projectState.mjs";
|
||||
|
||||
/** @typedef {import("./types.js").CaptureProblem} CaptureProblem @typedef {import("./types.js").Found} Found
|
||||
* @typedef {import("./types.js").Walk} Walk @typedef {import("./types.js").Capture} Capture */
|
||||
|
||||
/** Files up to this size use one bounded buffer; larger files stream from the verified handle. */
|
||||
const INLINE_HASH_BYTES = 4 * 1024 * 1024;
|
||||
const HASHING_AT_ONCE = 4;
|
||||
|
||||
/** @param {string} root @param {string} rel @param {import("node:fs").Dirent} entry @param {Walk} walk
|
||||
* @param {string[]} dirs */
|
||||
async function visit(root, rel, entry, walk, dirs) {
|
||||
if (entry.isDirectory()) {
|
||||
if (!excludedFolder(rel)) dirs.push(rel);
|
||||
return;
|
||||
}
|
||||
const action = classifyPath(rel).client_action;
|
||||
if (action === "exclude") return;
|
||||
if (action === "wait_then_exclude") return void walk.waiting.push(rel);
|
||||
if (entry.isSymbolicLink()) {
|
||||
if (skippedSymlink(rel)) walk.skippedLinks.push(rel);
|
||||
else walk.problems.push({ path: rel, problem: "symlink" });
|
||||
return;
|
||||
}
|
||||
if (!entry.isFile()) return void walk.problems.push({ path: rel, problem: "special-file" });
|
||||
await visitFile(root, rel, walk);
|
||||
}
|
||||
/** @param {string} root @param {string} rel @param {Walk} walk */
|
||||
async function visitFile(root, rel, walk) {
|
||||
try {
|
||||
validatePath(rel);
|
||||
} catch (error) {
|
||||
if (!(error instanceof PathViolation)) throw error;
|
||||
return void walk.problems.push({ path: rel, problem: "invalid-name", code: error.code });
|
||||
}
|
||||
const full = join(root, rel);
|
||||
try {
|
||||
const { size, mtimeMs, ino, dev } = await lstat(full);
|
||||
walk.files.push({ path: rel, full, size, mtimeMs, ino, dev });
|
||||
} catch {
|
||||
walk.problems.push({ path: rel, problem: "unreadable" });
|
||||
}
|
||||
}
|
||||
|
||||
/** A folder that was a link, or another folder, by the time it was read. */
|
||||
class FolderMoved extends Error {}
|
||||
|
||||
/** A folder's entries, read only while the same real folder stands at its path before and after: one swapped for a
|
||||
* link to somewhere else is never listed. @param {string} root @param {string} dir */
|
||||
async function listFolder(root, dir) {
|
||||
const full = join(root, dir);
|
||||
const before = await lstat(full);
|
||||
if (!before.isDirectory()) throw new FolderMoved();
|
||||
const entries = await readdir(full, { withFileTypes: true });
|
||||
const after = await lstat(full);
|
||||
if (!after.isDirectory() || after.ino !== before.ino || after.dev !== before.dev)
|
||||
throw new FolderMoved();
|
||||
return entries;
|
||||
}
|
||||
|
||||
/** Every file a capture would list, and what keeps one from being listed. @param {string} root
|
||||
* @param {(rel: string) => boolean} renderOutput @returns {Promise<Walk>} */
|
||||
async function walkProject(root, renderOutput) {
|
||||
/** @type {Walk} */
|
||||
const walk = { files: [], problems: [], waiting: [], skippedLinks: [] };
|
||||
const dirs = [""];
|
||||
for (let dir = dirs.pop(); dir !== undefined; dir = dirs.pop()) {
|
||||
let entries;
|
||||
try {
|
||||
entries = await listFolder(root, dir);
|
||||
} catch (error) {
|
||||
const changed = error instanceof FolderMoved;
|
||||
walk.problems.push({ path: dir, problem: changed ? "changed" : "unreadable" });
|
||||
continue;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
const rel = dir ? `${dir}/${entry.name}` : entry.name;
|
||||
if (prefiltered(entry.name) || renderOutput(rel)) continue;
|
||||
await visit(root, rel, entry, walk, dirs);
|
||||
}
|
||||
}
|
||||
return walk;
|
||||
}
|
||||
|
||||
/** The manifest's canonical length, from paths and sizes alone: every SHA-256 is 64 characters.
|
||||
* @param {Found[]} files */
|
||||
const manifestBytes = (files) =>
|
||||
canonicalBytes({
|
||||
format_version: 1,
|
||||
entrypoint: ENTRYPOINT,
|
||||
files: files.map(({ path, size }) => ({ path, sha256: "0".repeat(64), size })),
|
||||
}).length;
|
||||
|
||||
/** A file changed this recently can be written again within its time's granularity, keeping its size and time. */
|
||||
const SETTLED_MS = 2_000;
|
||||
|
||||
/** A file's SHA-256 by path, size, time and inode, kept for the launch so a check after a focus reads no bytes.
|
||||
* Only a settled file's hash is kept. */
|
||||
export function hashMemo(now = Date.now) {
|
||||
/** @type {Map<string, { size: number, mtimeMs: number, ino: number, sha256: string }>} */
|
||||
const known = new Map();
|
||||
return {
|
||||
/** @param {Found} file */
|
||||
get: (file) => {
|
||||
const kept = known.get(file.full);
|
||||
const same =
|
||||
kept?.size === file.size && kept.mtimeMs === file.mtimeMs && kept.ino === file.ino;
|
||||
return same ? kept.sha256 : null;
|
||||
},
|
||||
/** @param {Found} file @param {string} sha256 */
|
||||
set: (file, sha256) => {
|
||||
if (now() - file.mtimeMs > SETTLED_MS) known.set(file.full, { ...file, sha256 });
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** The file the walk listed, as an open handle: never through a link at its own name (O_NOFOLLOW), and only when the
|
||||
* opened file is the same regular file (inode, device, size) found inside the project, so a file or folder swapped
|
||||
* for a link after the walk is refused rather than followed. Null when it is not. @param {string} root
|
||||
* @param {Found} file @param {typeof realpath} [resolve] */
|
||||
async function openListed(root, file, resolve = realpath) {
|
||||
const handle = await open(file.full, constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0)).catch(
|
||||
(/** @type {NodeJS.ErrnoException} */ error) => {
|
||||
if (error.code === "ELOOP" || error.code === "EMLINK") return null;
|
||||
throw error;
|
||||
},
|
||||
);
|
||||
if (!handle) return null;
|
||||
try {
|
||||
const opened = await handle.stat();
|
||||
const same =
|
||||
opened.isFile() &&
|
||||
opened.ino === file.ino &&
|
||||
opened.dev === file.dev &&
|
||||
opened.size === file.size &&
|
||||
(await resolve(dirname(file.full))) === join(root, dirname(file.path));
|
||||
if (same) return handle;
|
||||
} catch (error) {
|
||||
await handle.close();
|
||||
throw error;
|
||||
}
|
||||
await handle.close();
|
||||
return null;
|
||||
}
|
||||
|
||||
/** The file's SHA-256, read through the handle `openListed` checked; null when the file is not the one listed, or its
|
||||
* bytes changed while being read. `root` is the project's real path; `resolve` finds a folder's real path.
|
||||
* @param {string} root @param {Found} file @param {typeof realpath} [resolve] */
|
||||
export async function hashListedFile(root, file, resolve = realpath) {
|
||||
const handle = await openListed(root, file, resolve);
|
||||
if (!handle) return null;
|
||||
try {
|
||||
const sha256 =
|
||||
file.size <= INLINE_HASH_BYTES
|
||||
? await handle
|
||||
.readFile()
|
||||
.then((bytes) =>
|
||||
bytes.length === file.size ? createHash("sha256").update(bytes).digest("hex") : null,
|
||||
)
|
||||
: await hashHandle(handle);
|
||||
const after = await handle.stat();
|
||||
return after.size === file.size && after.mtimeMs === file.mtimeMs ? sha256 : null;
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** @param {string} root @param {Found[]} files @param {ReturnType<typeof hashMemo>} memo
|
||||
* @returns {Promise<{ hashed: import("./types.js").ManifestFile[], problems: CaptureProblem[] }>} */
|
||||
async function hashAll(root, files, memo) {
|
||||
/** @type {import("./types.js").ManifestFile[]} */
|
||||
const hashed = [];
|
||||
/** @type {CaptureProblem[]} */
|
||||
const problems = [];
|
||||
const queue = [...files];
|
||||
const worker = async () => {
|
||||
for (let file = queue.shift(); file; file = queue.shift()) {
|
||||
const sha256 = memo.get(file) ?? (await hashListedFile(root, file).catch(() => undefined));
|
||||
if (sha256 === undefined) problems.push({ path: file.path, problem: "unreadable" });
|
||||
else if (sha256 === null) problems.push({ path: file.path, problem: "changed" });
|
||||
else {
|
||||
memo.set(file, sha256);
|
||||
hashed.push({ path: file.path, sha256, size: file.size });
|
||||
}
|
||||
}
|
||||
};
|
||||
await Promise.all(Array.from({ length: HASHING_AT_ONCE }, worker));
|
||||
return { hashed, problems };
|
||||
}
|
||||
|
||||
/** Problems that keep the files found from being one manifest: two names one disk stores as one, no index.html.
|
||||
* @param {Found[]} files @returns {CaptureProblem[]} */
|
||||
function inventoryProblems(files) {
|
||||
if (!files.some((file) => file.path === ENTRYPOINT))
|
||||
return [{ path: ENTRYPOINT, problem: "no-entrypoint" }];
|
||||
try {
|
||||
checkInventoryPaths(files.map((file) => file.path));
|
||||
return [];
|
||||
} catch (error) {
|
||||
if (!(error instanceof PathViolation)) throw error;
|
||||
return [{ path: error.path, problem: "invalid-name", code: error.code }];
|
||||
}
|
||||
}
|
||||
|
||||
/** Captures `root` (a project's real path). `memo` reuses hashes of unchanged files; a send should pass a fresh one.
|
||||
* @param {string} root @param {{ memo?: ReturnType<typeof hashMemo>, limits?: import("./types.js").Limits }} [options]
|
||||
* @returns {Promise<Capture>} */
|
||||
export async function captureProject(root, { memo = hashMemo(), limits = V1_LIMITS } = {}) {
|
||||
const renderOutput = await readRenderOutputs(root);
|
||||
const walk = await walkProject(root, renderOutput);
|
||||
const problems = walk.problems.length ? walk.problems : inventoryProblems(walk.files);
|
||||
if (problems.length) return { at: "problems", problems };
|
||||
const over = overLimit(walk.files, manifestBytes(walk.files), limits);
|
||||
if (over) return { at: "too-big", over };
|
||||
const { hashed, problems: unhashed } = await hashAll(await realpath(root), walk.files, memo);
|
||||
if (unhashed.length) return { at: "problems", problems: unhashed };
|
||||
try {
|
||||
const manifest = parseManifest({ format_version: 1, entrypoint: ENTRYPOINT, files: hashed });
|
||||
const settled = await settledState(root);
|
||||
return {
|
||||
at: "captured",
|
||||
manifest,
|
||||
settled,
|
||||
waiting: walk.waiting,
|
||||
skippedLinks: walk.skippedLinks,
|
||||
};
|
||||
} catch (error) {
|
||||
if (!(error instanceof ManifestInvalid)) throw error;
|
||||
const first = error.violations[0];
|
||||
return {
|
||||
at: "problems",
|
||||
problems: [{ path: first?.path ?? "", problem: "invalid-name", code: error.code }],
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/** @param {import("node:fs/promises").FileHandle} handle */
|
||||
async function hashHandle(handle) {
|
||||
const hash = createHash("sha256");
|
||||
for await (const chunk of handle.createReadStream({ autoClose: false })) hash.update(chunk);
|
||||
return hash.digest("hex");
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
// @ts-check
|
||||
// Load the bundled snapshot policy and Unicode registries from JSON or Brotli.
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { brotliDecompressSync } from "node:zlib";
|
||||
|
||||
/** A registry's parsed JSON. @param {string} name a file in contract/, such as "path-policy.json" @returns {any} */
|
||||
export function readContract(name) {
|
||||
const packed = new URL(`./contract/${name}.br`, import.meta.url);
|
||||
const text = existsSync(packed)
|
||||
? brotliDecompressSync(readFileSync(packed)).toString("utf8")
|
||||
: readFileSync(new URL(`./contract/${name}`, import.meta.url), "utf8");
|
||||
return JSON.parse(text);
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
UNICODE LICENSE V3
|
||||
|
||||
COPYRIGHT AND PERMISSION NOTICE
|
||||
|
||||
Copyright © 1991-2026 Unicode, Inc.
|
||||
|
||||
NOTICE TO USER: Carefully read the following legal agreement. BY
|
||||
DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR
|
||||
SOFTWARE, YOU UNEQUIVOCALLY ACCEPT, AND AGREE TO BE BOUND BY, ALL OF THE
|
||||
TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE, DO NOT
|
||||
DOWNLOAD, INSTALL, COPY, DISTRIBUTE OR USE THE DATA FILES OR SOFTWARE.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a
|
||||
copy of data files and any associated documentation (the "Data Files") or
|
||||
software and any associated documentation (the "Software") to deal in the
|
||||
Data Files or Software without restriction, including without limitation
|
||||
the rights to use, copy, modify, merge, publish, distribute, and/or sell
|
||||
copies of the Data Files or Software, and to permit persons to whom the
|
||||
Data Files or Software are furnished to do so, provided that either (a)
|
||||
this copyright and permission notice appear with all copies of the Data
|
||||
Files or Software, or (b) this copyright and permission notice appear in
|
||||
associated Documentation.
|
||||
|
||||
THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
|
||||
KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
|
||||
THIRD PARTY RIGHTS.
|
||||
|
||||
IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE
|
||||
BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES,
|
||||
OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
|
||||
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
|
||||
ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA
|
||||
FILES OR SOFTWARE.
|
||||
|
||||
Except as contained in this notice, the name of a copyright holder shall
|
||||
not be used in advertising or otherwise to promote the sale, use or other
|
||||
dealings in these Data Files or Software without prior written
|
||||
authorization of the copyright holder.
|
||||
@@ -0,0 +1,311 @@
|
||||
{
|
||||
"policy_version": "portable-source-v4",
|
||||
"manifest": {
|
||||
"format_version": 1,
|
||||
"entrypoint": "index.html",
|
||||
"fields": ["format_version", "entrypoint", "files"],
|
||||
"file_fields": ["path", "sha256", "size"],
|
||||
"sha256": "64 lowercase hex characters: the SHA-256 of the file's exact bytes",
|
||||
"size": "the file's byte length, an integer from 0 to 9007199254740991 (2^53 - 1, the largest integer every JSON client reads exactly)",
|
||||
"same_digest_same_size": "two entries with the same sha256 must have the same size",
|
||||
"not_in_manifest": "upload state, latest pointer, authors, comments, timestamps, URLs, S3 keys and sync binding metadata"
|
||||
},
|
||||
"canonical_encoding": {
|
||||
"files_order": "ascending by the UTF-8 bytes of path (not UTF-16 code units: JavaScript's default sort differs above U+FFFF)",
|
||||
"object_keys": "sorted; every key is ASCII, so byte, code-point and UTF-16 order agree",
|
||||
"whitespace": "none outside strings: separators are ',' and ':'",
|
||||
"strings": "UTF-8; only '\"' and '\\' are escaped, because policy-valid paths contain no control characters or lone surrogates",
|
||||
"numbers": "integers in decimal with no sign, fraction or exponent",
|
||||
"manifest_sha256": "lowercase hex SHA-256 of the canonical bytes"
|
||||
},
|
||||
"unicode": {
|
||||
"version": "15.1.0",
|
||||
"table": "unicode-15.1.0.json",
|
||||
"assigned_only": "every code point must be assigned in Unicode 15.1; unassigned and future code points are refused",
|
||||
"normalization": "every path must already be NFC",
|
||||
"comparison_key": "NFC(full_casefold(NFC(segment))), with default full case folding (CaseFolding.txt status C and F)"
|
||||
},
|
||||
"path": {
|
||||
"max_path_bytes": 1024,
|
||||
"max_segment_bytes": 240,
|
||||
"separator": "/",
|
||||
"forbidden_characters": "<>:\"|?*\\",
|
||||
"reserved_segment_regex": "(?:con|prn|aux|nul|com[1-9\u00b9\u00b2\u00b3]|lpt[1-9\u00b9\u00b2\u00b3])(?:\\..*)?",
|
||||
"reserved_segment_match": "full match against each segment's comparison key, so CON, Con.txt and nul.tar.gz are all reserved",
|
||||
"canonical_spelling_roots": [".media", ".hyperframes"],
|
||||
"checks": [
|
||||
{
|
||||
"code": "path_empty",
|
||||
"rule": "the path is not the empty string"
|
||||
},
|
||||
{
|
||||
"code": "path_invalid_character",
|
||||
"rule": "no C0 or C1 control character, DEL, lone surrogate or forbidden character"
|
||||
},
|
||||
{
|
||||
"code": "path_unassigned_code_point",
|
||||
"rule": "every code point is assigned in Unicode 15.1"
|
||||
},
|
||||
{
|
||||
"code": "path_not_nfc",
|
||||
"rule": "the path equals its NFC form"
|
||||
},
|
||||
{
|
||||
"code": "path_too_long",
|
||||
"rule": "at most max_path_bytes UTF-8 bytes"
|
||||
},
|
||||
{
|
||||
"code": "path_not_relative",
|
||||
"rule": "does not start with '/'"
|
||||
},
|
||||
{
|
||||
"code": "path_empty_segment",
|
||||
"rule": "no empty segment ('a//b', a trailing '/')"
|
||||
},
|
||||
{
|
||||
"code": "path_dot_segment",
|
||||
"rule": "no '.' or '..' segment"
|
||||
},
|
||||
{
|
||||
"code": "segment_too_long",
|
||||
"rule": "each segment is at most max_segment_bytes UTF-8 bytes"
|
||||
},
|
||||
{
|
||||
"code": "segment_trailing_dot_or_space",
|
||||
"rule": "no segment ends in '.' or ' ' (Windows strips them)"
|
||||
},
|
||||
{
|
||||
"code": "reserved_name",
|
||||
"rule": "no segment matches reserved_segment_regex"
|
||||
},
|
||||
{
|
||||
"code": "reserved_path_alias",
|
||||
"rule": "a first segment whose comparison key equals a canonical_spelling_roots entry is spelled exactly like it, and an allowed rule that matches the folded path also matches the path as spelled"
|
||||
},
|
||||
{
|
||||
"code": "excluded_path",
|
||||
"rule": "the first rule matching the folded path does not have server_action 'reject'"
|
||||
}
|
||||
],
|
||||
"check_order": "checks run in the order listed; the first failing check names the error"
|
||||
},
|
||||
"matching": {
|
||||
"syntax": "each rule's regex must match the whole path (Python re.fullmatch, JavaScript ^(?:regex)$ with the 's' and 'u' flags)",
|
||||
"subject": "the folded path: every segment replaced by its comparison key, joined with '/'",
|
||||
"first_match_wins": true,
|
||||
"case": "rule regexes are written in lowercase, because they are matched against the case-folded path",
|
||||
"client_actions": {
|
||||
"include": "list the file",
|
||||
"exclude": "leave the file out",
|
||||
"wait_then_exclude": "wait for the writer to finish (the reserved sibling may be incomplete), then leave the marker out"
|
||||
},
|
||||
"rule_order": "every rule that rejects at any depth comes before the first allow rule, so a credential, environment file, package-manager config, VCS or dependency directory, agent state, build cache or OS metadata inside an allowed .hyperframes subtree stays out"
|
||||
},
|
||||
"rules": [
|
||||
{
|
||||
"id": "media-lock",
|
||||
"regex": "\\.media/\\.lock",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Process lock held by the media writer, not project content"
|
||||
},
|
||||
{
|
||||
"id": "media-reservation",
|
||||
"regex": "\\.media/(?:audio/(?:bgm|sfx|voice)|images|video|luts|recipes)/[^/]+\\.hf[0-9a-f]{6}\\.tmp",
|
||||
"client_action": "wait_then_exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Media id reservation: the sibling asset may still be incomplete, so capture waits for the writer to finish"
|
||||
},
|
||||
{
|
||||
"id": "media-ledger",
|
||||
"regex": "\\.media/manifest\\.jsonl",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Generated media metadata that the project does not use for preview or rendering"
|
||||
},
|
||||
{
|
||||
"id": "unfinished-writes",
|
||||
"regex": "(?:[^/]+/)*(?:[^/]+\\.hf[0-9a-f]{6}\\.tmp|[^/]*\\.(?:tmp|part|swp)|[^/]+~)",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Unfinished or abandoned writes at any depth: core's atomic-write temp <file>.hfXXXXXX.tmp, *.tmp, *.part, editor *.swp and *~"
|
||||
},
|
||||
{
|
||||
"id": "render-temp-dirs",
|
||||
"regex": "(?:[^/]+/)*(?:(?:work-[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}|hf-render|\\.[^/]+\\.hf-transaction)-[A-Za-z0-9]{6}|[^/]+\\.assemble-work-[A-Za-z0-9]{6})(?:/.*)?",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Render work directories at any depth (frames, often GBs): core RENDER_TEMP_DIR (packages/core/src/projectFolders.ts) plus distributed assembly's <output>.assemble-work-XXXXXX"
|
||||
},
|
||||
{
|
||||
"id": "agent-client-state",
|
||||
"regex": "(?:[^/]+/)*(?:\\.claude|\\.codex|\\.agents|\\.cursor|\\.vscode|\\.idea)(?:/.*)?",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Agent and IDE state at any depth; Desktop vendors .claude/skills into every project and regenerates it"
|
||||
},
|
||||
{
|
||||
"id": "dependencies-and-git",
|
||||
"regex": "(?:[^/]+/)*(?:\\.git|\\.svn|\\.hg|node_modules)(?:/.*)?",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Version control and installed dependencies at any depth"
|
||||
},
|
||||
{
|
||||
"id": "build-caches",
|
||||
"regex": "(?:[^/]+/)*(?:\\.cache|\\.vite|\\.next|\\.turbo|\\.parcel-cache)(?:/.*)?",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Build tool caches at any depth"
|
||||
},
|
||||
{
|
||||
"id": "environment-secrets",
|
||||
"regex": "(?:[^/]+/)*\\.env(?:rc|\\.[^/]+)?",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Credentials: .env, every .env.* (templates included: real keys land there and no reader needs them) and .envrc, at any depth"
|
||||
},
|
||||
{
|
||||
"id": "package-manager-config",
|
||||
"regex": "(?:[^/]+/)*(?:\\.npmrc|\\.yarnrc|\\.yarnrc\\.yml|pip\\.conf)",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "May hold registry tokens, and nothing in a HyperFrames project reads them; stripping credentials would change the bytes"
|
||||
},
|
||||
{
|
||||
"id": "credential-files",
|
||||
"regex": "(?:[^/]+/)*(?:\\.netrc|\\.pypirc|\\.git-credentials|[^/]*\\.(?:pem|p12|pfx)|id_rsa[^/]*|id_ed25519[^/]*|(?:\\.ssh|\\.aws)(?:/.*)?)",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Credential files by name at any depth. *.key is deliberately absent: Keynote decks are plausible project sources"
|
||||
},
|
||||
{
|
||||
"id": "os-metadata",
|
||||
"regex": "(?:[^/]+/)*(?:\\.ds_store|thumbs\\.db|desktop\\.ini|\\._[^/]*|__macosx(?:/.*)?)",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Operating-system metadata: it changes when a folder is browsed and would show as false edits and conflicts"
|
||||
},
|
||||
{
|
||||
"id": "portable-transcripts",
|
||||
"regex": "\\.hyperframes/transcripts/(?:[^/]+/)*[^/]+\\.json",
|
||||
"client_action": "include",
|
||||
"server_action": "allow",
|
||||
"basis": "Completed transcripts, keyed by media hash; costly to regenerate"
|
||||
},
|
||||
{
|
||||
"id": "portable-transcription-settings",
|
||||
"regex": "\\.hyperframes/transcription\\.json",
|
||||
"client_action": "include",
|
||||
"server_action": "allow",
|
||||
"basis": "Transcription (ASR) preferences, not run state; Desktop's Duplicate copies it with the transcripts"
|
||||
},
|
||||
{
|
||||
"id": "portable-brand-kit",
|
||||
"regex": "\\.hyperframes/brand-kit\\.json",
|
||||
"client_action": "include",
|
||||
"server_action": "allow",
|
||||
"basis": "The project's brand kit selection is authored, portable project content"
|
||||
},
|
||||
{
|
||||
"id": "portable-studio-edits",
|
||||
"regex": "\\.hyperframes/(?:studio-manual-edits|studio-motion)\\.json",
|
||||
"client_action": "include",
|
||||
"server_action": "allow",
|
||||
"basis": "Studio edits read at render and preview time; without them a pulled project renders differently"
|
||||
},
|
||||
{
|
||||
"id": "portable-caption-skin",
|
||||
"regex": "\\.hyperframes/caption-skin\\.html",
|
||||
"client_action": "include",
|
||||
"server_action": "allow",
|
||||
"basis": "Authored input to the caption build; rebuilding captions needs it"
|
||||
},
|
||||
{
|
||||
"id": "portable-podcast-media",
|
||||
"regex": "\\.hyperframes/prepared-assets/podcast/(?:[^/]+/)*[^./][^/]*",
|
||||
"client_action": "include",
|
||||
"server_action": "allow",
|
||||
"basis": "Desktop's podcast camera and recorder transcodes (media/, proxies/), which the page plays by this path and a receiving client cannot regenerate without the original files; a dotfile there stays out"
|
||||
},
|
||||
{
|
||||
"id": "sync-and-share-bindings",
|
||||
"regex": "\\.hyperframes/(?:share\\.json|review\\.json|review-export\\.json|review-[^/]+\\.json|project\\.json|lambda-stack-[^/]+\\.json|(?:sync|receipts)/.*)",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "External bindings and sync receipts: share and review links (review files carry the owner's email), the CLI publish id and deploy stacks. A copy must start unbound, so these never travel"
|
||||
},
|
||||
{
|
||||
"id": "known-hyperframes-state",
|
||||
"regex": "\\.hyperframes/(?:comments\\.jsonl|agent-runs\\.jsonl|app-history\\.jsonl|chat-starts\\.json|agent-handoff\\.json|agent-handoff-read\\.json|history-id|video-state\\.json|film-snapshot\\.json|moved-from\\.jsonl|hf-ids-stamped\\.json|settled-signature|render-outputs\\.json|render-outputs-lock|project\\.lock|history-turn\\.json|history-turns\\.json|voices\\.json|asset-creation\\.json|lesson\\.json|element-map\\.txt|\\.gitignore|(?:live-runs|run-traces|agent-sessions|agent-steps|video-state-lock|backup|preview|history|undo|artifacts|heygen-recipes|heygen-reperform|fixes|heygen-jobs|heygen-paid|podcast|prepared-assets|requests|recordings|frames|refs|references|review|heygen-previews|frame-packets|anim-map|contrast|bin|diagnostics)(?:/.*)?)",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Known client state: comments, chat and run logs; machine pointers, locks and caches; personal data (photo and voice copies) and account-bound job state, podcast job state included; studio preview caches under prepared-assets/ other than the podcast media above"
|
||||
},
|
||||
{
|
||||
"id": "unknown-hyperframes-state",
|
||||
"regex": "\\.hyperframes(?:/.*)?",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Reserved client state area: only the named portable files above travel"
|
||||
},
|
||||
{
|
||||
"id": "root-generated-output",
|
||||
"regex": "(?:\\.thumbnails|\\.transcode-cache|\\.waveform-cache|renders|snapshots)(?:/.*)?",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Caches and render or snapshot output at the project root"
|
||||
},
|
||||
{
|
||||
"id": "root-build-output",
|
||||
"regex": "(?:dist|coverage)(?:/.*)?",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Web build and test-coverage output at the project root, which the CLI publish leaves out and render never reads (build/ and outputs/ travel, because the publish ships them)"
|
||||
},
|
||||
{
|
||||
"id": "authoring-scratch",
|
||||
"regex": "(?:remotion-src|moments|references/moments)(?:/.*)?",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Authoring scratch at the project root: a Remotion port's source and the moment working copies with their references, none of which preview or render reads"
|
||||
},
|
||||
{
|
||||
"id": "reference-video",
|
||||
"regex": "\\.viral-clone(?:/.*)?",
|
||||
"client_action": "exclude",
|
||||
"server_action": "reject",
|
||||
"basis": "Third-party reference video, large and re-downloadable"
|
||||
},
|
||||
{
|
||||
"id": "default-authored-file",
|
||||
"regex": ".+",
|
||||
"client_action": "include",
|
||||
"server_action": "allow",
|
||||
"basis": "Authored source and config (index.html, compositions/, scenes/, assets/, hyperframes.json, meta.json, project.json, package.json, lockfiles, .hyperframesignore, .gitignore), portable briefs, AGENTS.md and CLAUDE.md, completed .media assets with unused alternatives, .media/index.md, .media/preferences.json and .media/CREDITS.md, build/ and outputs/, and any unknown regular file outside a reserved area"
|
||||
}
|
||||
],
|
||||
"client_notes": {
|
||||
"project_json": "Root project.json is portable project content and is included. It is not sync identity: the owner workspace, composition, base snapshot, revision and receipts live outside the inventory, never in project.json.",
|
||||
"capture_preconditions": "Do not capture while .hyperframes/history-turn.json exists (an agent turn is open) or while .hyperframes/project.lock is held.",
|
||||
"render_outputs": "Leave out every path listed in .hyperframes/render-outputs.json: render outputs written outside renders/ are recorded there, and the server cannot know them.",
|
||||
"complete_capture": "A file the client cannot read fails the capture; it never becomes a deletion. The exclusions above and the symlinks client_symlinks skips are the only files left out.",
|
||||
"symlinks": "Symlinks and special files fail the capture, listing them; they are neither followed nor skipped, except the symlinks client_symlinks skips.",
|
||||
"empty_directories": "Empty directories are not represented. Pull never deletes a local empty directory."
|
||||
},
|
||||
"client_prefilter": {
|
||||
"names": ["Icon\r"],
|
||||
"rule": "skip a file whose last segment is one of these before running any path check; macOS writes 'Icon\\r' for a folder with a custom icon, and the carriage return would otherwise fail the whole capture with path_invalid_character"
|
||||
},
|
||||
"client_symlinks": {
|
||||
"default": "fail_capture",
|
||||
"skip_prefixes": ["assets/podcast/"],
|
||||
"match": "a symlink whose folded path (each segment replaced by its comparison key, as for rules) starts with a listed prefix, at any depth below it",
|
||||
"rule": "skip such a symlink instead of failing the capture, and on Pull leave such a local symlink in place as local state; a regular file under the prefix is captured like any other, and every other symlink or special file fails the capture",
|
||||
"basis": "Desktop's podcast intake links the original camera files, which live outside the project, under assets/podcast/, and the page plays the transcodes under .hyperframes/prepared-assets/podcast/; real files there, such as the joins under assets/podcast/joins/, travel normally"
|
||||
},
|
||||
"stored_manifest": {
|
||||
"legacy_agent_state_regex": "(?:[^/]+/)*(?:\\.claude|\\.agents|\\.cursor|\\.vscode|\\.idea)(?:/.*)?",
|
||||
"rule": "Only stored snapshot reads use the v3 agent-state rule to retain immutable manifest digests. New captures and uploads use the current rules. Clients never install excluded paths; the server never signs their download URLs."
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,77 @@
|
||||
// @ts-check
|
||||
// Map sync API error codes to stable client error kinds.
|
||||
|
||||
/** @typedef {import("./types.js").SyncErrorKind} SyncErrorKind */
|
||||
|
||||
/** @type {Record<string, SyncErrorKind>} */
|
||||
const KIND_BY_CODE = {
|
||||
hyperframes_composition_sync_not_enabled: "not-enabled",
|
||||
hf_subscription_required: "plan-required",
|
||||
hyperframes_composition_sync_limit_reached: "limit-reached",
|
||||
byok_not_supported: "byok",
|
||||
resource_access_denied: "no-write-role",
|
||||
forbidden: "forbidden",
|
||||
hyperframes_composition_not_found: "composition-not-found",
|
||||
resource_not_found: "not-found",
|
||||
hyperframes_composition_upload_too_large: "too-large",
|
||||
hyperframes_composition_upload_reference_invalid: "reference-invalid",
|
||||
hyperframes_composition_upload_not_pending: "upload-not-pending",
|
||||
hyperframes_composition_active_uploads_exceeded: "active-uploads-exceeded",
|
||||
hyperframes_composition_upload_file_not_writable: "file-not-writable",
|
||||
conflict: "idempotency-key-reused",
|
||||
request_in_progress: "request-in-progress",
|
||||
hyperframes_composition_snapshot_not_downloadable: "snapshot-not-downloadable",
|
||||
hyperframes_composition_conflict_not_open: "conflict-not-open",
|
||||
hyperframes_composition_download_expired: "download-expired",
|
||||
invalid_parameter: "invalid-parameter",
|
||||
rate_limit_exceeded: "rate-limited",
|
||||
service_unavailable: "unavailable",
|
||||
};
|
||||
|
||||
/** A sync route's refusal or failure. `status` is 0 when the server was never reached. */
|
||||
export class SyncApiError extends Error {
|
||||
/** @param {SyncErrorKind} kind @param {{ status?: number, code?: string | null, param?: string | null,
|
||||
* requestId?: string | null, retryAfterMs?: number | null, detail?: string }} [about] */
|
||||
constructor(kind, about = {}) {
|
||||
super(`composition sync: ${kind}${about.code ? ` (${about.code})` : ""}`);
|
||||
this.name = "SyncApiError";
|
||||
this.kind = kind;
|
||||
this.status = about.status ?? 0;
|
||||
this.code = about.code ?? null;
|
||||
this.param = about.param ?? null;
|
||||
this.requestId = about.requestId ?? null;
|
||||
this.retryAfterMs = about.retryAfterMs ?? null;
|
||||
this.detail = about.detail ?? null;
|
||||
}
|
||||
}
|
||||
|
||||
/** The kind for an HTTP status whose code this client does not know. @param {number} status
|
||||
* @returns {SyncErrorKind} */
|
||||
function kindOfStatus(status) {
|
||||
if (status === 404) return "not-found";
|
||||
if (status === 429) return "rate-limited";
|
||||
if (status === 503) return "unavailable";
|
||||
if (status >= 500) return "server-error";
|
||||
return "refused";
|
||||
}
|
||||
|
||||
/** The typed error for a refusal: the envelope's code when it is one of the sync routes', else the status's.
|
||||
* @param {Response} reply @returns {Promise<SyncApiError>} */
|
||||
export async function refusalOf(reply) {
|
||||
const payload = await reply.json().catch(() => null);
|
||||
/** @type {{ code?: unknown, param?: unknown, request_id?: unknown, message?: unknown } | undefined} */
|
||||
const error = payload?.error;
|
||||
const code = typeof error?.code === "string" ? error.code : null;
|
||||
const seconds = Number(reply.headers.get("retry-after") ?? Number.NaN);
|
||||
return new SyncApiError((code && KIND_BY_CODE[code]) || kindOfStatus(reply.status), {
|
||||
status: reply.status,
|
||||
code,
|
||||
param: typeof error?.param === "string" ? error.param : null,
|
||||
requestId:
|
||||
typeof error?.request_id === "string"
|
||||
? error.request_id
|
||||
: reply.headers.get("x-heygen-request-id"),
|
||||
retryAfterMs: seconds >= 0 ? seconds * 1000 : null,
|
||||
...(typeof error?.message === "string" ? { detail: error.message } : {}),
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
// @ts-check
|
||||
// Check the workspace upload limits before hashing or sending files.
|
||||
|
||||
const MIB = 1024 * 1024;
|
||||
const GIB = 1024 * MIB;
|
||||
|
||||
/** @typedef {import("./types.js").Limits} Limits @typedef {import("./types.js").LimitName} LimitName */
|
||||
|
||||
/** @type {Limits} */
|
||||
export const V1_LIMITS = Object.freeze({
|
||||
maxFiles: 10_000,
|
||||
maxFileBytes: 5 * GIB,
|
||||
maxProjectBytes: 10 * GIB,
|
||||
maxManifestBytes: 8 * MIB,
|
||||
});
|
||||
|
||||
/** How many of the biggest files an over-limit answer names. */
|
||||
const NAMED = 10;
|
||||
|
||||
/** A manifest's file count, largest file, total and canonical size; every listed file counts, reused ones too.
|
||||
* @param {readonly { size: number }[]} files @param {number} manifestBytes */
|
||||
const measure = (files, manifestBytes) => ({
|
||||
fileCount: files.length,
|
||||
largestFileBytes: files.reduce((most, file) => Math.max(most, file.size), 0),
|
||||
projectBytes: files.reduce((sum, file) => sum + file.size, 0),
|
||||
manifestBytes,
|
||||
});
|
||||
|
||||
/** The first limit broken, in the server's fixed order (file count, file size, project size, manifest size), with
|
||||
* the biggest files first; null when the push fits. @param {readonly { path: string, size: number }[]} files
|
||||
* @param {number} manifestBytes @param {Limits} [limits] @returns {import("./types.js").OverLimit | null} */
|
||||
export function overLimit(files, manifestBytes, limits = V1_LIMITS) {
|
||||
const size = measure(files, manifestBytes);
|
||||
/** @type {[LimitName, number, number][]} */
|
||||
const checks = [
|
||||
["file_count", size.fileCount, limits.maxFiles],
|
||||
["file_size", size.largestFileBytes, limits.maxFileBytes],
|
||||
["project_size", size.projectBytes, limits.maxProjectBytes],
|
||||
["manifest_size", size.manifestBytes, limits.maxManifestBytes],
|
||||
];
|
||||
const broken = checks.find(([, actual, maximum]) => actual > maximum);
|
||||
if (!broken) return null;
|
||||
const biggest = [...files]
|
||||
.sort((a, b) => b.size - a.size || (a.path < b.path ? -1 : 1))
|
||||
.slice(0, NAMED)
|
||||
.map(({ path, size: bytes }) => ({ path, size: bytes }));
|
||||
return { limit: broken[0], actual: broken[1], maximum: broken[2], biggest };
|
||||
}
|
||||
@@ -0,0 +1,154 @@
|
||||
// @ts-check
|
||||
// Serialize snapshots to canonical UTF-8 bytes so clients and the server compute the same digest.
|
||||
import { createHash } from "node:crypto";
|
||||
import { PathViolation, checkInventoryPaths, validatePath } from "./pathPolicy.mjs";
|
||||
|
||||
const FORMAT_VERSION = 1;
|
||||
export const ENTRYPOINT = "index.html";
|
||||
const MAX_SAFE_SIZE = Number.MAX_SAFE_INTEGER;
|
||||
const SHA256 = /^[0-9a-f]{64}$/;
|
||||
|
||||
/** @typedef {import("./types.js").ManifestFile} ManifestFile @typedef {import("./types.js").Violation} Violation
|
||||
* @typedef {import("./types.js").ManifestDocument} ManifestDocument @typedef {import("./types.js").Manifest} Manifest */
|
||||
|
||||
/** Why a manifest was refused; `code` is the first violation's, in the server's order. */
|
||||
export class ManifestInvalid extends Error {
|
||||
/** @param {Violation[]} violations */
|
||||
constructor(violations) {
|
||||
super(violations.map((violation) => violation.code).join(", "));
|
||||
this.name = "ManifestInvalid";
|
||||
this.violations = violations;
|
||||
this.code = violations[0]?.code ?? "invalid_schema";
|
||||
}
|
||||
}
|
||||
|
||||
/** @param {unknown} value @returns {value is Record<string, unknown>} */
|
||||
const isObject = (value) => typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
/** @param {unknown} error @returns {Violation} */
|
||||
const violationOf = (error) => {
|
||||
if (!(error instanceof PathViolation)) throw error;
|
||||
return { code: error.code, path: error.path, ...(error.rule ? { rule: error.rule } : {}) };
|
||||
};
|
||||
|
||||
/** A schema violation for each field `fields` does not name.
|
||||
* @param {Record<string, unknown>} value @param {readonly string[]} fields @param {Violation[]} out */
|
||||
function unknownFields(value, fields, out) {
|
||||
for (const key of Object.keys(value))
|
||||
if (!fields.includes(key)) out.push({ code: "invalid_schema" });
|
||||
}
|
||||
|
||||
/** @param {unknown} raw @param {Violation[]} out @param {boolean} stored @returns {ManifestFile | null} */
|
||||
function fileOf(raw, out, stored) {
|
||||
if (!isObject(raw)) return (out.push({ code: "invalid_schema" }), null);
|
||||
const before = out.length;
|
||||
const { path, sha256, size } = raw;
|
||||
if (typeof path !== "string") out.push({ code: "invalid_schema" });
|
||||
else {
|
||||
try {
|
||||
validatePath(path, stored);
|
||||
} catch (error) {
|
||||
out.push(violationOf(error));
|
||||
}
|
||||
}
|
||||
checkBlob(sha256, size, out);
|
||||
unknownFields(raw, ["path", "sha256", "size"], out);
|
||||
return out.length === before
|
||||
? { path: String(path), sha256: String(sha256), size: Number(size) }
|
||||
: null;
|
||||
}
|
||||
|
||||
/** @param {unknown} sha256 @param {unknown} size @param {Violation[]} out */
|
||||
function checkBlob(sha256, size, out) {
|
||||
if (typeof sha256 !== "string") out.push({ code: "invalid_schema" });
|
||||
else if (!SHA256.test(sha256)) out.push({ code: "invalid_sha256" });
|
||||
if (typeof size !== "number" || !Number.isInteger(size)) out.push({ code: "invalid_schema" });
|
||||
else if (size < 0 || size > MAX_SAFE_SIZE) out.push({ code: "invalid_size" });
|
||||
}
|
||||
|
||||
/** UTF-8 byte order, which is code point order; JavaScript's default sort compares UTF-16 units instead.
|
||||
* @param {ManifestFile} a @param {ManifestFile} b */
|
||||
const byUtf8 = (a, b) => Buffer.compare(Buffer.from(a.path, "utf8"), Buffer.from(b.path, "utf8"));
|
||||
|
||||
/** The files in canonical order, or the inventory's first violation. @param {ManifestFile[]} files
|
||||
* @param {Violation[]} out */
|
||||
function canonicalInventory(files, out) {
|
||||
const ordered = [...files].sort(byUtf8);
|
||||
/** @type {Map<string, number>} */
|
||||
const sizeByDigest = new Map();
|
||||
for (const file of ordered) {
|
||||
const size = sizeByDigest.get(file.sha256) ?? file.size;
|
||||
sizeByDigest.set(file.sha256, size);
|
||||
if (size !== file.size)
|
||||
return (out.push({ code: "inconsistent_digest_size", path: file.path }), null);
|
||||
}
|
||||
try {
|
||||
checkInventoryPaths(ordered.map((file) => file.path));
|
||||
} catch (error) {
|
||||
return (out.push(violationOf(error)), null);
|
||||
}
|
||||
return ordered;
|
||||
}
|
||||
|
||||
/** @param {unknown} raw @param {Violation[]} out @param {boolean} stored */
|
||||
function filesOf(raw, out, stored) {
|
||||
if (!Array.isArray(raw)) return (out.push({ code: "invalid_schema" }), null);
|
||||
const before = out.length;
|
||||
const files = raw.map((item) => fileOf(item, out, stored));
|
||||
if (out.length !== before) return null;
|
||||
return canonicalInventory(/** @type {ManifestFile[]} */ (files), out);
|
||||
}
|
||||
|
||||
/** The canonical bytes of a valid document. @param {ManifestDocument} document */
|
||||
export function canonicalBytes(document) {
|
||||
const files = document.files
|
||||
.map(
|
||||
({ path, sha256, size }) =>
|
||||
`{"path":${JSON.stringify(path)},"sha256":${JSON.stringify(sha256)},"size":${size}}`,
|
||||
)
|
||||
.join(",");
|
||||
const text = `{"entrypoint":${JSON.stringify(document.entrypoint)},"files":[${files}],"format_version":${document.format_version}}`;
|
||||
return Buffer.from(text, "utf8");
|
||||
}
|
||||
|
||||
/** Validates a manifest document as the server does; throws ManifestInvalid. @param {unknown} raw
|
||||
* @param {boolean} [stored] @returns {Manifest} */
|
||||
export function parseManifest(raw, stored = false) {
|
||||
if (!isObject(raw)) throw new ManifestInvalid([{ code: "invalid_schema" }]);
|
||||
/** @type {Violation[]} */
|
||||
const out = [];
|
||||
const { format_version: version, entrypoint } = raw;
|
||||
if (!("format_version" in raw) || typeof version !== "number" || !Number.isInteger(version))
|
||||
out.push({ code: "invalid_schema" });
|
||||
else if (version !== FORMAT_VERSION) out.push({ code: "unsupported_format_version" });
|
||||
if (typeof entrypoint !== "string") out.push({ code: "invalid_schema" });
|
||||
else if (entrypoint !== ENTRYPOINT) out.push({ code: "invalid_entrypoint" });
|
||||
const files = filesOf(raw.files, out, stored);
|
||||
unknownFields(raw, ["format_version", "entrypoint", "files"], out);
|
||||
if (out.length) throw new ManifestInvalid(out);
|
||||
const ordered = /** @type {ManifestFile[]} */ (files);
|
||||
if (!ordered.some((file) => file.path === ENTRYPOINT))
|
||||
throw new ManifestInvalid([{ code: "entrypoint_not_listed" }]);
|
||||
/** @type {ManifestDocument} */
|
||||
const document = { format_version: FORMAT_VERSION, entrypoint: ENTRYPOINT, files: ordered };
|
||||
const canonical = canonicalBytes(document);
|
||||
return { document, canonical, sha256: createHash("sha256").update(canonical).digest("hex") };
|
||||
}
|
||||
|
||||
/** Stored manifest bytes, which must already be canonical so their hash is the snapshot's manifest_sha256.
|
||||
* @param {Uint8Array} body @returns {Manifest} */
|
||||
export function parseCanonicalManifest(body) {
|
||||
let document;
|
||||
try {
|
||||
document = JSON.parse(new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(body));
|
||||
} catch {
|
||||
throw new ManifestInvalid([{ code: "invalid_json" }]);
|
||||
}
|
||||
const manifest = parseManifest(document);
|
||||
if (!manifest.canonical.equals(Buffer.from(body)))
|
||||
throw new ManifestInvalid([{ code: "not_canonical" }]);
|
||||
return manifest;
|
||||
}
|
||||
|
||||
/** A published snapshot or local base can retain v3 paths; its bytes and digest remain immutable.
|
||||
* @param {unknown} raw @returns {Manifest} */
|
||||
export const parseStoredManifest = (raw) => parseManifest(raw, true);
|
||||
@@ -0,0 +1,197 @@
|
||||
// @ts-check
|
||||
// Validate snapshot paths in the order defined by the bundled portable-source-v4 policy.
|
||||
import { readContract } from "./contract.mjs";
|
||||
import { UNICODE_VERSION, allAssigned, comparisonKey, nfc } from "./unicode.mjs";
|
||||
|
||||
const PATH_POLICY_VERSION = "portable-source-v4";
|
||||
|
||||
/** @typedef {import("./types.js").PathRule} PathRule @typedef {import("./types.js").PolicyDocument} PolicyDocument */
|
||||
|
||||
/** A path the policy refuses: `code` is the golden vectors' shared vocabulary, `rule` the excluding rule's id. */
|
||||
export class PathViolation extends Error {
|
||||
/** @param {string} code @param {string} path @param {string} [rule] */
|
||||
constructor(code, path, rule) {
|
||||
super(`${code}: ${JSON.stringify(path)}${rule ? ` (${rule})` : ""}`);
|
||||
this.name = "PathViolation";
|
||||
this.code = code;
|
||||
this.path = path;
|
||||
this.rule = rule ?? null;
|
||||
}
|
||||
}
|
||||
|
||||
const ASCII = /^\p{ASCII}*$/u;
|
||||
|
||||
/** @param {string} pattern */
|
||||
const whole = (pattern) => new RegExp(`^(?:${pattern})$`, "su");
|
||||
/** @param {string} text */
|
||||
const escaped = (text) => text.replace(/[\\\]^-]/g, "\\$&");
|
||||
|
||||
/** @param {PolicyDocument} document */
|
||||
function compile(document) {
|
||||
if (document.policy_version !== PATH_POLICY_VERSION)
|
||||
throw new Error(`path policy is ${document.policy_version}, expected ${PATH_POLICY_VERSION}`);
|
||||
if (document.unicode.version !== UNICODE_VERSION)
|
||||
throw new Error(
|
||||
`path policy pins Unicode ${document.unicode.version}, the table is ${UNICODE_VERSION}`,
|
||||
);
|
||||
if (document.rules.at(-1)?.regex !== ".+")
|
||||
throw new Error("path policy must end with a catch-all rule, so every path is classified");
|
||||
for (const rule of document.rules)
|
||||
if ((rule.server_action === "allow") !== (rule.client_action === "include"))
|
||||
throw new Error(`path rule ${rule.id}: the server allows exactly what clients include`);
|
||||
const limits = document.path;
|
||||
return {
|
||||
limits,
|
||||
invalidCharacter: new RegExp(
|
||||
`[\\x00-\\x1f\\x7f-\\x9f\\ud800-\\udfff${escaped(limits.forbidden_characters)}]`,
|
||||
"u",
|
||||
),
|
||||
reservedSegment: whole(limits.reserved_segment_regex),
|
||||
canonicalRoots: new Map(
|
||||
limits.canonical_spelling_roots.map((root) => [comparisonKey(root), root]),
|
||||
),
|
||||
rules: document.rules.map((rule) => ({ rule, pattern: whole(rule.regex) })),
|
||||
storedRules: document.rules.map((rule) => ({
|
||||
rule,
|
||||
pattern: whole(
|
||||
rule.id === "agent-client-state"
|
||||
? document.stored_manifest.legacy_agent_state_regex
|
||||
: rule.regex,
|
||||
),
|
||||
})),
|
||||
prefilter: new Set(document.client_prefilter.names),
|
||||
symlinkPrefixes: document.client_symlinks.skip_prefixes,
|
||||
};
|
||||
}
|
||||
|
||||
/** @type {ReturnType<typeof compile> | null} */
|
||||
let compiled = null;
|
||||
/** The policy, read and compiled when a path is first checked: an app that never syncs never reads it. */
|
||||
const policy = () => (compiled ??= compile(readContract("path-policy.json")));
|
||||
|
||||
/** Every segment replaced by its comparison key. @param {string} path */
|
||||
const foldedPath = (path) => path.split("/").map(comparisonKey).join("/");
|
||||
|
||||
/** The first rule matching a path's folded spelling. @param {string} path @returns {PathRule} */
|
||||
export function classifyPath(path) {
|
||||
const folded = foldedPath(path);
|
||||
const found = policy().rules.find(({ pattern }) => pattern.test(folded));
|
||||
if (!found) throw new Error("the path policy's catch-all rule did not match");
|
||||
return found.rule;
|
||||
}
|
||||
|
||||
/** Checks 1 to 10 of the registry's `path.checks`: the ones about the string itself. @param {string} path */
|
||||
function checkSpelling(path) {
|
||||
const { limits, invalidCharacter } = policy();
|
||||
if (!path) throw new PathViolation("path_empty", path);
|
||||
if (invalidCharacter.test(path)) throw new PathViolation("path_invalid_character", path);
|
||||
if (!ASCII.test(path) && !allAssigned(path))
|
||||
throw new PathViolation("path_unassigned_code_point", path);
|
||||
if (nfc(path) !== path) throw new PathViolation("path_not_nfc", path);
|
||||
if (Buffer.byteLength(path, "utf8") > limits.max_path_bytes)
|
||||
throw new PathViolation("path_too_long", path);
|
||||
if (path.startsWith("/")) throw new PathViolation("path_not_relative", path);
|
||||
return checkSegments(path, limits);
|
||||
}
|
||||
/** @param {string} path @param {PolicyDocument["path"]} limits */
|
||||
function checkSegments(path, limits) {
|
||||
const segments = path.split("/");
|
||||
if (segments.some((segment) => !segment)) throw new PathViolation("path_empty_segment", path);
|
||||
if (segments.some((segment) => segment === "." || segment === ".."))
|
||||
throw new PathViolation("path_dot_segment", path);
|
||||
if (segments.some((segment) => Buffer.byteLength(segment, "utf8") > limits.max_segment_bytes))
|
||||
throw new PathViolation("segment_too_long", path);
|
||||
if (segments.some((segment) => segment.endsWith(".") || segment.endsWith(" ")))
|
||||
throw new PathViolation("segment_trailing_dot_or_space", path);
|
||||
return segments;
|
||||
}
|
||||
|
||||
/** `path` unchanged when every client can store it and the policy lets it travel; throws PathViolation otherwise,
|
||||
* naming the first failing check in the registry's order. @param {string} path
|
||||
* @param {boolean} [stored] */
|
||||
export function validatePath(path, stored = false) {
|
||||
const segments = checkSpelling(path);
|
||||
const keys = segments.map(comparisonKey);
|
||||
const { reservedSegment, canonicalRoots } = policy();
|
||||
const rules = stored ? policy().storedRules : policy().rules;
|
||||
if (keys.some((key) => reservedSegment.test(key))) throw new PathViolation("reserved_name", path);
|
||||
const root = canonicalRoots.get(keys[0] ?? "");
|
||||
if (root !== undefined && segments[0] !== root)
|
||||
throw new PathViolation("reserved_path_alias", path);
|
||||
const folded = keys.join("/");
|
||||
const found = rules.find(({ pattern }) => pattern.test(folded));
|
||||
if (!found) throw new Error("the path policy's catch-all rule did not match");
|
||||
if (found.rule.server_action === "reject")
|
||||
throw new PathViolation("excluded_path", path, found.rule.id);
|
||||
if (!found.pattern.test(path))
|
||||
throw new PathViolation("reserved_path_alias", path, found.rule.id);
|
||||
return path;
|
||||
}
|
||||
|
||||
/** Refuses duplicates, and any two paths (or a path and a directory prefix) one filesystem would store as one name.
|
||||
* Each path must already pass validatePath. @param {readonly string[]} paths */
|
||||
export function checkInventoryPaths(paths) {
|
||||
const seen = { spellingByKey: new Map(), fileKeys: new Set(), directoryKeys: new Set() };
|
||||
for (const path of paths) {
|
||||
const segments = path.split("/");
|
||||
const keys = segments.map(comparisonKey);
|
||||
for (let depth = 1; depth <= segments.length; depth++)
|
||||
notePrefix(seen, path, {
|
||||
prefix: segments.slice(0, depth).join("/"),
|
||||
key: keys.slice(0, depth).join("/"),
|
||||
isFile: depth === segments.length,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/** One prefix of `path`, against every prefix before it, in the server's order of checks.
|
||||
* @param {{ spellingByKey: Map<string, string>, fileKeys: Set<string>, directoryKeys: Set<string> }} seen
|
||||
* @param {string} path @param {{ prefix: string, key: string, isFile: boolean }} at */
|
||||
function notePrefix(seen, path, { prefix, key, isFile }) {
|
||||
const spelling = seen.spellingByKey.get(key) ?? prefix;
|
||||
seen.spellingByKey.set(key, spelling);
|
||||
if (isFile && seen.fileKeys.has(key))
|
||||
throw new PathViolation(spelling === prefix ? "duplicate_path" : "path_collision", path);
|
||||
if (spelling !== prefix) throw new PathViolation("path_collision", path);
|
||||
if (isFile ? seen.directoryKeys.has(key) : seen.fileKeys.has(key))
|
||||
throw new PathViolation("file_directory_collision", path);
|
||||
(isFile ? seen.fileKeys : seen.directoryKeys).add(key);
|
||||
}
|
||||
|
||||
/** The registry's `client_prefilter`: a name (macOS's `Icon\r`) skipped before any path check. @param {string} name */
|
||||
export const prefiltered = (name) => policy().prefilter.has(name);
|
||||
|
||||
/** The registry's `client_symlinks`: a link under `assets/podcast/` is skipped instead of failing the capture.
|
||||
* @param {string} path */
|
||||
export const skippedSymlink = (path) =>
|
||||
policy().symlinkPrefixes.some((prefix) => foldedPath(path).startsWith(prefix));
|
||||
|
||||
/** The literal text a rule's regex starts with, up to its first metacharacter. @param {string} regex */
|
||||
function literalPrefix(regex) {
|
||||
let text = "";
|
||||
for (let i = 0; i < regex.length; i++) {
|
||||
const char = regex[i] ?? "";
|
||||
if (char === "\\" && /[./-]/.test(regex[i + 1] ?? "")) text += regex[++i];
|
||||
else if (/[\w/-]/.test(char)) text += char;
|
||||
else break;
|
||||
}
|
||||
return text;
|
||||
}
|
||||
|
||||
/** Whether no file under folder `dir` can be listed, so a capture need not open it: its first matching rule
|
||||
* excludes the folder and all below it, and no include rule before that one can reach inside. @param {string} dir */
|
||||
export function excludedFolder(dir) {
|
||||
const folded = foldedPath(dir);
|
||||
const at = policy().rules.findIndex(
|
||||
({ pattern, rule }) =>
|
||||
rule.client_action !== "include" && pattern.test(folded) && pattern.test(`${folded}/x`),
|
||||
);
|
||||
if (at < 0) return false;
|
||||
return policy()
|
||||
.rules.slice(0, at)
|
||||
.every(({ rule }) => {
|
||||
if (rule.client_action !== "include") return true;
|
||||
const prefix = literalPrefix(rule.regex);
|
||||
return !(prefix.startsWith(`${folded}/`) || `${folded}/`.startsWith(prefix));
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
// @ts-check
|
||||
// What the HyperFrames CLI keeps in a project's .hyperframes folder that a capture must honour: render outputs written
|
||||
// outside renders/ (left out of every manifest), an open agent turn and a held project lock (no capture until both
|
||||
// end). The formats are the pinned hyperframes CLI's, version 1 of render-outputs.json.
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
/** A predicate for the paths the project's render-outputs.json lists, and everything under them; none when the file
|
||||
* is missing or not version 1. @param {string} root @returns {Promise<(rel: string) => boolean>} */
|
||||
export async function readRenderOutputs(root) {
|
||||
const raw = await readFile(join(root, ".hyperframes", "render-outputs.json"), "utf8").catch(
|
||||
() => null,
|
||||
);
|
||||
/** @type {{ version?: unknown, paths?: unknown } | null} */
|
||||
let record = null;
|
||||
try {
|
||||
record = raw === null ? null : JSON.parse(raw);
|
||||
} catch {
|
||||
record = null;
|
||||
}
|
||||
const paths =
|
||||
record?.version === 1 && Array.isArray(record.paths)
|
||||
? record.paths
|
||||
.filter((path) => typeof path === "string")
|
||||
.map((path) => path.replace(/\\/g, "/"))
|
||||
: [];
|
||||
const listed = new Set(paths);
|
||||
return (rel) => listed.has(rel) || paths.some((path) => rel.startsWith(`${path}/`));
|
||||
}
|
||||
|
||||
/** Whether a pid still runs: EPERM is a process another user owns. @param {number} pid */
|
||||
function alive(pid) {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (error) {
|
||||
return /** @type {NodeJS.ErrnoException} */ (error).code === "EPERM";
|
||||
}
|
||||
}
|
||||
|
||||
/** An open agent turn (its history-turn.json exists) or a project lock whose holder still runs.
|
||||
* @param {string} root */
|
||||
export async function settledState(root) {
|
||||
const dir = join(root, ".hyperframes");
|
||||
const turn = await readFile(join(dir, "history-turn.json")).then(
|
||||
() => true,
|
||||
() => false,
|
||||
);
|
||||
const lock = await readFile(join(dir, "project.lock"), "utf8").catch(() => null);
|
||||
/** @type {{ pid?: unknown } | null} */
|
||||
let holder = null;
|
||||
try {
|
||||
holder = lock === null ? null : JSON.parse(lock);
|
||||
} catch {
|
||||
holder = null;
|
||||
}
|
||||
const locked = lock !== null && (typeof holder?.pid !== "number" || alive(holder.pid));
|
||||
return { agentTurnOpen: turn, projectLocked: locked };
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
// @ts-check
|
||||
import { createHash } from "node:crypto";
|
||||
import { constants } from "node:fs";
|
||||
import { lstat, open } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { errorCode, mkdirInside, writeJsonAtomic } from "../storage.mjs";
|
||||
import { parseStoredManifest } from "./manifest.mjs";
|
||||
|
||||
/** @param {unknown} value @returns {value is Record<string, unknown>} */
|
||||
const object = (value) => typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
/** @param {unknown} raw @param {import("./types.js").BindingKey} key @returns {import("./types.js").Binding} */
|
||||
function bindingOf(raw, key) {
|
||||
if (
|
||||
!object(raw) ||
|
||||
raw.version !== 1 ||
|
||||
raw.backend !== key.backend ||
|
||||
raw.account !== key.account ||
|
||||
raw.projectPath !== key.projectPath ||
|
||||
typeof raw.compositionId !== "string" ||
|
||||
typeof raw.ownerSpaceId !== "string"
|
||||
)
|
||||
throw new Error(
|
||||
"Source binding belongs to another account, backend or project. Reset explicitly before sharing this copy.",
|
||||
);
|
||||
checkSnapshots(raw);
|
||||
return /** @type {import("./types.js").Binding} */ (/** @type {unknown} */ (raw));
|
||||
}
|
||||
|
||||
/** @param {Record<string, unknown>} raw */
|
||||
function checkSnapshots(raw) {
|
||||
if (
|
||||
raw.base !== null &&
|
||||
(!object(raw.base) ||
|
||||
typeof raw.base.snapshotId !== "string" ||
|
||||
parseStoredManifest(raw.base.manifest).sha256 !== raw.base.manifestSha256)
|
||||
)
|
||||
throw new Error("Invalid source base snapshot.");
|
||||
if (
|
||||
raw.inflight !== null &&
|
||||
(!object(raw.inflight) ||
|
||||
typeof raw.inflight.idempotencyKey !== "string" ||
|
||||
parseStoredManifest(raw.inflight.manifest).sha256 !== raw.inflight.manifestSha256)
|
||||
)
|
||||
throw new Error("Invalid pending source upload.");
|
||||
}
|
||||
|
||||
/** Project-local state is excluded by the portable source policy. Account/root mismatches fail closed, including copied folders.
|
||||
* @returns {import("./adapterTypes.js").SourceStore} */
|
||||
export function createProjectShareSourceStore() {
|
||||
/** @param {import("./types.js").BindingKey} key */
|
||||
const location = async (key) => {
|
||||
const dir = await mkdirInside(key.projectPath, ".hyperframes");
|
||||
const environment = createHash("sha256").update(key.backend).digest("hex").slice(0, 16);
|
||||
return join(dir, `share-sync.${environment}.json`);
|
||||
};
|
||||
return {
|
||||
read: async (key) => {
|
||||
const file = await location(key);
|
||||
const handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW).catch((error) => {
|
||||
if (errorCode(error) === "ENOENT") return null;
|
||||
throw error;
|
||||
});
|
||||
if (!handle) return null;
|
||||
try {
|
||||
if (!(await handle.stat()).isFile())
|
||||
throw new Error("Source binding must be a regular file.");
|
||||
return bindingOf(JSON.parse(await handle.readFile("utf8")), key);
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
},
|
||||
write: async (binding) => {
|
||||
const file = await location(binding);
|
||||
const held = await lstat(file).catch((error) => {
|
||||
if (errorCode(error) === "ENOENT") return null;
|
||||
throw error;
|
||||
});
|
||||
if (held && !held.isFile()) throw new Error("Source binding must be a regular file.");
|
||||
await writeJsonAtomic(file, binding);
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
// @ts-check
|
||||
// Bytes to signed storage URLs: one file or multipart range, streamed and hashed as it is sent.
|
||||
// Large media costs one chunk at a time. Storage URLs carry their own signature, never an OAuth bearer.
|
||||
import { createHash } from "node:crypto";
|
||||
import { constants, createReadStream } from "node:fs";
|
||||
import { open, realpath } from "node:fs/promises";
|
||||
import { dirname } from "node:path";
|
||||
import { setTimeout as sleep } from "node:timers/promises";
|
||||
import { trustedUploadUrl } from "../reviewUpload.mjs";
|
||||
|
||||
/** Why a transfer stopped. `expired`: the signed URL no longer works (ask for a new one). `exists`: a create-only PUT
|
||||
* found the object already written. `changed`: the file's bytes are not the ones captured. `unreadable`: the file
|
||||
* could not be read (a drive unplugged, a file deleted). `corrupt`: downloaded bytes do not hash to what was asked.
|
||||
* `transient`: the network or the store failed; the same request may be sent again. */
|
||||
export class TransferError extends Error {
|
||||
/** @param {"expired" | "exists" | "changed" | "unreadable" | "corrupt" | "transient" | "refused"} kind
|
||||
* @param {{ status?: number, path?: string }} [about] */
|
||||
constructor(kind, about = {}) {
|
||||
super(`composition sync transfer: ${kind}${about.status ? ` (${about.status})` : ""}`);
|
||||
this.name = "TransferError";
|
||||
this.kind = kind;
|
||||
this.status = about.status ?? 0;
|
||||
this.path = about.path ?? null;
|
||||
}
|
||||
}
|
||||
|
||||
/** @param {unknown} error */
|
||||
export const isAbort = (error) =>
|
||||
error instanceof Error && (error.name === "AbortError" || error.name === "TimeoutError");
|
||||
|
||||
/** @param {number} ms @param {AbortSignal} [signal] */
|
||||
export const realWait = (ms, signal) => sleep(ms, undefined, { signal });
|
||||
|
||||
/** S3 answers 403 for a URL past its expiry; a 5xx or 429 is worth sending again. @param {number} status
|
||||
* @returns {TransferError} */
|
||||
function refusalOf(status) {
|
||||
if (status === 403) return new TransferError("expired", { status });
|
||||
if (status === 412) return new TransferError("exists", { status });
|
||||
if (status === 429 || status >= 500) return new TransferError("transient", { status });
|
||||
return new TransferError("refused", { status });
|
||||
}
|
||||
|
||||
/** @param {string} file */
|
||||
async function openSource(file) {
|
||||
const handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW);
|
||||
try {
|
||||
const stat = await handle.stat();
|
||||
if (!stat.isFile() || (await realpath(dirname(file))) !== dirname(file))
|
||||
throw new TransferError("unreadable", { path: file });
|
||||
return handle;
|
||||
} catch (error) {
|
||||
await handle.close();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/** @param {string} file @param {number} start @param {number} length @param {(bytes: number) => void} onBytes */
|
||||
async function hashedBody(file, start, length, onBytes) {
|
||||
const handle = await openSource(file);
|
||||
const hash = createHash("sha256");
|
||||
const seen = { bytes: 0, ended: false, readError: /** @type {unknown} */ (null) };
|
||||
const chunks =
|
||||
length === 0
|
||||
? null
|
||||
: handle
|
||||
.createReadStream({ start, end: start + length - 1, autoClose: false })
|
||||
[Symbol.asyncIterator]();
|
||||
const body = new ReadableStream(
|
||||
{
|
||||
async pull(controller) {
|
||||
if (!chunks) return controller.close();
|
||||
try {
|
||||
const { value, done } = await chunks.next();
|
||||
if (done) return ((seen.ended = true), controller.close());
|
||||
hash.update(value);
|
||||
seen.bytes += value.length;
|
||||
onBytes(value.length);
|
||||
controller.enqueue(value);
|
||||
} catch (error) {
|
||||
seen.readError = error;
|
||||
controller.error(error);
|
||||
}
|
||||
},
|
||||
cancel: () => void chunks?.return?.(),
|
||||
},
|
||||
{ highWaterMark: 0 },
|
||||
);
|
||||
return { body, seen, digest: () => hash.digest("hex"), close: () => handle.close() };
|
||||
}
|
||||
|
||||
/** What the bytes taken from the file say about it once a PUT ended: unreadable, changed (shorter than captured,
|
||||
* or another hash), or null when they are the captured bytes, all of them or a prefix the network cut short.
|
||||
* @param {Awaited<ReturnType<typeof hashedBody>>} sent @param {{ length: number, sha256: string, path: string }} expected */
|
||||
function fileProblem({ seen, digest }, { length, sha256, path }) {
|
||||
if (seen.readError) return new TransferError("unreadable", { path });
|
||||
const short = seen.ended && seen.bytes < length;
|
||||
if (short || (seen.bytes === length && digest() !== sha256))
|
||||
return new TransferError("changed", { path });
|
||||
return null;
|
||||
}
|
||||
|
||||
/** PUTs bytes of `file` to a signed URL with exactly the headers the grant signed. Resolves once storage keeps them;
|
||||
* the bytes sent must hash to `sha256`, or it throws `changed` whatever storage answered.
|
||||
* @param {{ fetchImpl: typeof fetch, url: string, headers: Record<string, string>, file: string, path: string,
|
||||
* start: number, length: number, sha256: string, signal: AbortSignal, onBytes?: (bytes: number) => void, stallMs?: number }} put */
|
||||
export async function putRange(put) {
|
||||
const { fetchImpl, url, headers, file, start, length, signal } = put;
|
||||
const target = trustedUploadUrl(url);
|
||||
if (!target) throw new TransferError("refused", { path: put.path });
|
||||
signal.throwIfAborted();
|
||||
const stall = new AbortController();
|
||||
const sent = await hashedBody(file, start, length, (bytes) => {
|
||||
timer.refresh();
|
||||
put.onBytes?.(bytes);
|
||||
});
|
||||
const timer = setTimeout(() => stall.abort(), put.stallMs ?? 60_000);
|
||||
// Node's fetch needs `duplex` for a streamed body; lib.dom's RequestInit does not name it.
|
||||
const init = {
|
||||
method: "PUT",
|
||||
headers,
|
||||
body: sent.body,
|
||||
duplex: "half",
|
||||
redirect: "error",
|
||||
signal: AbortSignal.any([signal, stall.signal]),
|
||||
};
|
||||
/** @type {Response} */
|
||||
let reply;
|
||||
try {
|
||||
reply = await fetchImpl(target.href, /** @type {RequestInit} */ (init));
|
||||
} catch (error) {
|
||||
signal.throwIfAborted();
|
||||
if (!stall.signal.aborted && isAbort(error)) throw error;
|
||||
throw fileProblem(sent, put) ?? new TransferError("transient", { path: put.path });
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
await sent.close();
|
||||
}
|
||||
await reply.body?.cancel().catch(() => {});
|
||||
const problem = fileProblem(sent, put);
|
||||
if (problem) throw problem;
|
||||
if (!reply.ok) throw refusalOf(reply.status);
|
||||
}
|
||||
|
||||
/** Each part's SHA-256 (parts of `partBytes`, the last one shorter) and the whole file's, in one read.
|
||||
* @param {string} file @param {number} size @param {number} partBytes @param {AbortSignal} signal */
|
||||
export async function partDigests(file, size, partBytes, signal) {
|
||||
const whole = createHash("sha256");
|
||||
/** @type {string[]} */
|
||||
const parts = [];
|
||||
let part = createHash("sha256");
|
||||
let inPart = 0;
|
||||
for await (const chunk of createReadStream(file, { highWaterMark: 1 << 20, signal })) {
|
||||
whole.update(chunk);
|
||||
for (let at = 0; at < chunk.length; ) {
|
||||
const take = Math.min(partBytes - inPart, chunk.length - at);
|
||||
part.update(chunk.subarray(at, at + take));
|
||||
inPart += take;
|
||||
at += take;
|
||||
if (inPart === partBytes) {
|
||||
parts.push(part.digest("hex"));
|
||||
part = createHash("sha256");
|
||||
inPart = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (inPart > 0 || parts.length === 0) parts.push(part.digest("hex"));
|
||||
return {
|
||||
parts,
|
||||
whole: whole.digest("hex"),
|
||||
complete: parts.length === Math.ceil(size / partBytes),
|
||||
};
|
||||
}
|
||||
|
||||
/** Runs `work` over `items`, `limit` at a time. The first failure stops new work; the rest finish, then it throws.
|
||||
* @template T @param {readonly T[]} items @param {number} limit @param {(item: T) => Promise<void>} work */
|
||||
export async function eachLimit(items, limit, work) {
|
||||
let next = 0;
|
||||
const failure = { failed: false, error: /** @type {unknown} */ (null) };
|
||||
const worker = async () => {
|
||||
while (!failure.failed && next < items.length) {
|
||||
const item = /** @type {T} */ (items[next++]);
|
||||
try {
|
||||
await work(item);
|
||||
} catch (error) {
|
||||
if (!failure.failed) Object.assign(failure, { failed: true, error });
|
||||
}
|
||||
}
|
||||
};
|
||||
await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
|
||||
if (failure.failed) throw failure.error;
|
||||
}
|
||||
|
||||
/** The wait before attempt `attempt` (from 1): doubling from `baseMs`, at most `maxMs`, or what the server asked.
|
||||
* @param {number} attempt @param {number | null} [askedMs] */
|
||||
export const backoffMs = (attempt, askedMs = null, baseMs = 1_000, maxMs = 30_000) =>
|
||||
askedMs ?? Math.min(maxMs, baseMs * 2 ** (attempt - 1));
|
||||
+353
@@ -0,0 +1,353 @@
|
||||
export interface SyncTooBig {
|
||||
limit: "file_count" | "file_size" | "project_size" | "manifest_size";
|
||||
actual: number;
|
||||
maximum: number;
|
||||
biggest: { path: string; size: number }[];
|
||||
}
|
||||
|
||||
/** A file that keeps a project from being captured; it is never left out silently. */
|
||||
export interface SyncCaptureProblem {
|
||||
path: string;
|
||||
problem: "symlink" | "special-file" | "unreadable" | "invalid-name" | "changed" | "no-entrypoint";
|
||||
/** For `invalid-name`, the path policy's code (`reserved_name`, `path_collision`, ...). */
|
||||
code?: string;
|
||||
}
|
||||
|
||||
export interface ManifestFile {
|
||||
path: string;
|
||||
sha256: string;
|
||||
size: number;
|
||||
}
|
||||
|
||||
export interface ManifestDocument {
|
||||
format_version: 1;
|
||||
entrypoint: "index.html";
|
||||
files: ManifestFile[];
|
||||
}
|
||||
|
||||
/** A valid manifest: its document in canonical order, the canonical bytes and their SHA-256. */
|
||||
export interface Manifest {
|
||||
document: ManifestDocument;
|
||||
canonical: Buffer;
|
||||
sha256: string;
|
||||
}
|
||||
|
||||
export interface Violation {
|
||||
code: string;
|
||||
path?: string;
|
||||
rule?: string;
|
||||
}
|
||||
|
||||
export type LimitName = SyncTooBig["limit"];
|
||||
|
||||
export interface Limits {
|
||||
maxFiles: number;
|
||||
maxFileBytes: number;
|
||||
maxProjectBytes: number;
|
||||
maxManifestBytes: number;
|
||||
}
|
||||
|
||||
export type OverLimit = SyncTooBig;
|
||||
|
||||
type ClientAction = "include" | "exclude" | "wait_then_exclude";
|
||||
|
||||
export interface PathRule {
|
||||
id: string;
|
||||
regex: string;
|
||||
client_action: ClientAction;
|
||||
server_action: "allow" | "reject";
|
||||
}
|
||||
|
||||
/** The fields of contract/path-policy.json this client reads. */
|
||||
export interface PolicyDocument {
|
||||
policy_version: string;
|
||||
stored_manifest: { legacy_agent_state_regex: string };
|
||||
unicode: { version: string };
|
||||
rules: PathRule[];
|
||||
path: {
|
||||
max_path_bytes: number;
|
||||
max_segment_bytes: number;
|
||||
forbidden_characters: string;
|
||||
reserved_segment_regex: string;
|
||||
canonical_spelling_roots: string[];
|
||||
};
|
||||
client_prefilter: { names: string[] };
|
||||
client_symlinks: { skip_prefixes: string[] };
|
||||
}
|
||||
|
||||
export type CaptureProblem = SyncCaptureProblem;
|
||||
|
||||
/** What an owner workspace's plan allows composition sync, as the limits route answered it. `unread` when it could not
|
||||
* be read: `not-deployed` is a server without the route (a 404), and nothing is refused on an unread answer. */
|
||||
export type SyncLimitsRead =
|
||||
| {
|
||||
at: "read";
|
||||
syncAllowed: boolean;
|
||||
maxCompositions: number;
|
||||
/** Live compositions in the workspace when it was read. */
|
||||
compositions: number;
|
||||
push: Limits;
|
||||
}
|
||||
| { at: "unread"; why: "not-deployed" | "offline" | "failed" };
|
||||
|
||||
/** A file a capture lists, before hashing. */
|
||||
export interface Found {
|
||||
path: string;
|
||||
full: string;
|
||||
size: number;
|
||||
mtimeMs: number;
|
||||
ino: number;
|
||||
dev: number;
|
||||
}
|
||||
|
||||
export interface Walk {
|
||||
files: Found[];
|
||||
problems: CaptureProblem[];
|
||||
waiting: string[];
|
||||
skippedLinks: string[];
|
||||
}
|
||||
|
||||
export type Capture =
|
||||
| {
|
||||
at: "captured";
|
||||
manifest: Manifest;
|
||||
settled: { agentTurnOpen: boolean; projectLocked: boolean };
|
||||
waiting: string[];
|
||||
skippedLinks: string[];
|
||||
}
|
||||
| { at: "problems"; problems: CaptureProblem[] }
|
||||
| { at: "too-big"; over: OverLimit };
|
||||
|
||||
export type SyncErrorKind =
|
||||
| "offline"
|
||||
| "signed-out"
|
||||
| "malformed"
|
||||
| "not-enabled"
|
||||
| "byok"
|
||||
| "no-write-role"
|
||||
| "forbidden"
|
||||
| "composition-not-found"
|
||||
| "not-found"
|
||||
| "too-large"
|
||||
| "reference-invalid"
|
||||
| "upload-not-pending"
|
||||
| "active-uploads-exceeded"
|
||||
| "file-not-writable"
|
||||
| "idempotency-key-reused"
|
||||
| "request-in-progress"
|
||||
| "snapshot-not-downloadable"
|
||||
| "conflict-not-open"
|
||||
| "download-expired"
|
||||
| "invalid-parameter"
|
||||
| "plan-required"
|
||||
| "limit-reached"
|
||||
| "rate-limited"
|
||||
| "unavailable"
|
||||
| "server-error"
|
||||
| "refused";
|
||||
|
||||
export interface Composition {
|
||||
id: string;
|
||||
spaceId: string;
|
||||
title: string;
|
||||
createdBy: string;
|
||||
/** Null before the first publish. */
|
||||
latestSnapshotId: string | null;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export interface Snapshot {
|
||||
id: string;
|
||||
compositionId: string;
|
||||
baseSnapshotId: string | null;
|
||||
publicationOutcome: "published" | "conflict_saved";
|
||||
conflictedWithSnapshotId: string | null;
|
||||
conflictState: "open" | "resolved" | "discarded" | null;
|
||||
resolvedBySnapshotId: string | null;
|
||||
createdBy: string;
|
||||
createdByName?: string | null;
|
||||
clientType: string;
|
||||
summary: string | null;
|
||||
summarySource: string;
|
||||
manifestSha256: string;
|
||||
createdAt: string;
|
||||
/** Only with `include=manifest`, and only once its bytes hash to manifestSha256. */
|
||||
manifest: Manifest | null;
|
||||
}
|
||||
|
||||
type UploadStatus = "pending" | "verifying" | "completed" | "failed" | "expired";
|
||||
|
||||
export interface UploadReceipt {
|
||||
uploadId: string;
|
||||
compositionId: string;
|
||||
snapshotId: string;
|
||||
status: UploadStatus;
|
||||
/** Why a failed upload failed: cancelled, content_incomplete, verification_timeout, access_revoked. */
|
||||
errorCode: string | null;
|
||||
baseSnapshotId: string | null;
|
||||
resolvesConflictSnapshotIds: string[];
|
||||
manifestSha256: string;
|
||||
uploadDeadline: string;
|
||||
verificationDeadline: string | null;
|
||||
completedAt: string | null;
|
||||
result: {
|
||||
snapshotId: string;
|
||||
publicationOutcome: "published" | "conflict_saved";
|
||||
conflictedWithSnapshotId: string | null;
|
||||
} | null;
|
||||
}
|
||||
|
||||
interface BlobRef {
|
||||
sha256: string;
|
||||
size: number;
|
||||
}
|
||||
|
||||
export interface BeginResult extends UploadReceipt {
|
||||
/** Already saved in the base or a named conflict: never uploaded. */
|
||||
present: BlobRef[];
|
||||
/** To upload through upload-urls before submitting. */
|
||||
needed: BlobRef[];
|
||||
/** False for a replay of the same push under the same Idempotency-Key. */
|
||||
created: boolean;
|
||||
}
|
||||
|
||||
export interface BeginRequest {
|
||||
idempotencyKey: string;
|
||||
baseSnapshotId: string | null;
|
||||
resolvesConflictSnapshotIds: string[];
|
||||
summary: { text: string; source: "user" | "agent" } | null;
|
||||
manifest: Manifest;
|
||||
}
|
||||
|
||||
interface PartGrant {
|
||||
partNumber: number;
|
||||
length: number;
|
||||
url: string;
|
||||
headers: Record<string, string>;
|
||||
}
|
||||
|
||||
export interface FileGrant {
|
||||
sha256: string;
|
||||
size: number;
|
||||
expiresAt: string;
|
||||
/** Files up to 64 MiB: one PUT. */
|
||||
url: string | null;
|
||||
headers: Record<string, string>;
|
||||
/** Files over 64 MiB: the file's one multipart upload, and the requested parts. */
|
||||
multipartUploadId: string | null;
|
||||
parts: PartGrant[];
|
||||
}
|
||||
|
||||
export interface GrantRequest {
|
||||
sha256: string;
|
||||
multipartUploadId?: string;
|
||||
parts?: { partNumber: number; sha256: string }[];
|
||||
}
|
||||
|
||||
export interface DownloadBatch {
|
||||
snapshotId: string;
|
||||
manifestSha256: string;
|
||||
continuationToken: string;
|
||||
continuationExpiresAt: string;
|
||||
urls: { sha256: string; size: number; url: string; expiresAt: string }[];
|
||||
}
|
||||
|
||||
export interface Page<T> {
|
||||
items: T[];
|
||||
nextToken: string | null;
|
||||
}
|
||||
|
||||
export type CompositionRead =
|
||||
| { at: "fresh"; composition: Composition; etag: string | null }
|
||||
| { at: "unchanged"; etag: string };
|
||||
|
||||
/** Whose binding: the backend's API host, the signed-in account (accountPrincipal) and the project's real path. */
|
||||
export interface BindingKey {
|
||||
backend: string;
|
||||
account: string;
|
||||
projectPath: string;
|
||||
}
|
||||
|
||||
/** A push recorded before its Begin is sent, so a lost answer is recovered by sending the same body under the same
|
||||
* Idempotency-Key; the upload's ids and deadlines land once an answer comes back. */
|
||||
export interface InflightUpload {
|
||||
idempotencyKey: string;
|
||||
baseSnapshotId: string | null;
|
||||
resolvesConflictSnapshotIds: string[];
|
||||
summary: { text: string; source: "user" | "agent" } | null;
|
||||
manifest: ManifestDocument;
|
||||
manifestSha256: string;
|
||||
uploadId: string | null;
|
||||
snapshotId: string | null;
|
||||
uploadDeadline: string | null;
|
||||
verificationDeadline: string | null;
|
||||
begunAt: number;
|
||||
/** Files whose bytes reached staging, by SHA-256, so a resumed send does not write them again. */
|
||||
sent?: string[];
|
||||
/** Each multipart file's pinned upload and the parts already written, by the file's SHA-256. */
|
||||
multipart?: Record<string, { uploadId: string; partsDone: number[] }>;
|
||||
}
|
||||
|
||||
export interface Binding extends BindingKey {
|
||||
version: 1;
|
||||
/** Client-generated, kept before the first request so a retried create names the same composition. */
|
||||
compositionId: string;
|
||||
ownerSpaceId: string;
|
||||
/** The owner workspace as the account's list last named it, so a project whose workspace the account left can
|
||||
* still be named. Absent from bindings made before it was kept, and until a list names the workspace. */
|
||||
owner?: { name: string; kind: "personal" | "team" };
|
||||
boundAt: number;
|
||||
/** The composition as last read, and the ETag it came with; null before the create was answered. */
|
||||
composition: Composition | null;
|
||||
etag: string | null;
|
||||
/** The snapshot this copy was last sent as or got from, with its manifest, so local changes need no network. */
|
||||
base: { snapshotId: string; manifest: ManifestDocument; manifestSha256: string } | null;
|
||||
inflight: InflightUpload | null;
|
||||
/** Set while Get from the cloud is filling a folder it made, until the folder is added to Home. Until then the
|
||||
* folder is not a project on this Mac, and the next Get of the composition finishes it. */
|
||||
gettingFromCloud?: true;
|
||||
/** What the marker that Get put in its folder holds: the only proof a file of that name there is not the person's. */
|
||||
getToken?: string;
|
||||
/** Set on the folder a Duplicate fills from the cloud: it is bound only while it downloads, so no Get ever opens
|
||||
* it, and its binding goes once latest is in (duplicate.mjs). */
|
||||
duplicate?: true;
|
||||
/** The conflicts this Mac's sends saved, newest last: Compare names them as this project's, not another
|
||||
* computer's. */
|
||||
savedHere?: string[];
|
||||
/** Set on a binding Work on {owner}'s project made from a share: the account reaches the owner's workspace through
|
||||
* that share's grant, not membership, so every request names the review project (sync-backed sharing C5). */
|
||||
sharedFrom?: SharedFrom;
|
||||
}
|
||||
|
||||
/** The share a binding came from: its review project, the owner's name and their workspace's, as the share named
|
||||
* them when it was opened. */
|
||||
export interface SharedFrom {
|
||||
reviewProjectId: string;
|
||||
owner: string;
|
||||
workspace: string | null;
|
||||
}
|
||||
|
||||
/** A workspace's last refusal of a new composition: not admitted to sync yet, or encrypted (BYOK). */
|
||||
export interface SpaceRefusal {
|
||||
kind: "not-enabled" | "byok";
|
||||
at: number;
|
||||
}
|
||||
|
||||
/** One send or get under way for a project: the binding it owns until it ends, and where its states go. */
|
||||
export interface UploadJob {
|
||||
api: import("./api.mjs").SourceApi;
|
||||
space: string;
|
||||
composition: string;
|
||||
uploadId: string;
|
||||
inflight: InflightUpload;
|
||||
local: Map<string, { path: string; full: string }>;
|
||||
needed: { sha256: string; size: number }[];
|
||||
fetchImpl: typeof fetch;
|
||||
wait: (ms: number, signal?: AbortSignal) => Promise<void>;
|
||||
signal: AbortSignal;
|
||||
now: () => number;
|
||||
persist: () => void;
|
||||
onBytes: (bytes: number) => void;
|
||||
onFile: () => void;
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
// @ts-check
|
||||
// Use the bundled Unicode 15.1 tables so path comparison is independent of the runtime Unicode version.
|
||||
import { readContract } from "./contract.mjs";
|
||||
|
||||
export const UNICODE_VERSION = "15.1.0";
|
||||
|
||||
/** @typedef {{ unicode_version: string, casefold: Record<string, string>, combining: Record<string, number>,
|
||||
* decomposition: Record<string, number[]>, composition: Record<string, number>,
|
||||
* assigned_ranges: [number, number][] }} UnicodeTableFile */
|
||||
|
||||
const HANGUL_BASE = 0xac00;
|
||||
const HANGUL_COUNT = 11172;
|
||||
const L_BASE = 0x1100;
|
||||
const V_BASE = 0x1161;
|
||||
const T_BASE = 0x11a7;
|
||||
const L_COUNT = 19;
|
||||
const V_COUNT = 21;
|
||||
const T_COUNT = 28;
|
||||
const N_COUNT = V_COUNT * T_COUNT;
|
||||
|
||||
/** The table re-keyed by code point, decompositions fully expanded. @param {UnicodeTableFile} raw */
|
||||
function tablesOf(raw) {
|
||||
if (raw.unicode_version !== UNICODE_VERSION)
|
||||
throw new Error(`unicode table is ${raw.unicode_version}, expected ${UNICODE_VERSION}`);
|
||||
const hex = (/** @type {string} */ code) => Number.parseInt(code, 16);
|
||||
const casefold = new Map(Object.entries(raw.casefold).map(([code, to]) => [hex(code), to]));
|
||||
const combining = new Map(Object.entries(raw.combining).map(([code, cc]) => [hex(code), cc]));
|
||||
const direct = new Map(Object.entries(raw.decomposition).map(([code, seq]) => [hex(code), seq]));
|
||||
/** @param {number} code @returns {number[]} */
|
||||
const expand = (code) => direct.get(code)?.flatMap(expand) ?? [code];
|
||||
const decomposition = new Map([...direct.keys()].map((code) => [code, expand(code)]));
|
||||
const composition = new Map(
|
||||
Object.entries(raw.composition).map(([pair, composite]) => {
|
||||
const [first = "", second = ""] = pair.split(",");
|
||||
return [`${hex(first)},${hex(second)}`, composite];
|
||||
}),
|
||||
);
|
||||
const starts = raw.assigned_ranges.map(([start]) => start);
|
||||
const ends = raw.assigned_ranges.map(([, end]) => end);
|
||||
return { casefold, combining, decomposition, composition, starts, ends };
|
||||
}
|
||||
|
||||
/** @type {ReturnType<typeof tablesOf> | null} */
|
||||
let loaded = null;
|
||||
/** The table, read and indexed when a path is first compared: an app that never syncs never reads it. */
|
||||
const tables = () => (loaded ??= tablesOf(readContract("unicode-15.1.0.json")));
|
||||
|
||||
/** @param {number} code */
|
||||
const classOf = (code) => tables().combining.get(code) ?? 0;
|
||||
|
||||
/** @param {number} code @returns {readonly number[]} */
|
||||
function decompose(code) {
|
||||
const syllable = code - HANGUL_BASE;
|
||||
if (syllable >= 0 && syllable < HANGUL_COUNT) {
|
||||
const trailing = syllable % T_COUNT;
|
||||
const leading = [
|
||||
L_BASE + Math.floor(syllable / N_COUNT),
|
||||
V_BASE + Math.floor((syllable % N_COUNT) / T_COUNT),
|
||||
];
|
||||
return trailing ? [...leading, T_BASE + trailing] : leading;
|
||||
}
|
||||
return tables().decomposition.get(code) ?? [code];
|
||||
}
|
||||
|
||||
/** @param {number} first @param {number} second @returns {number | undefined} */
|
||||
function composePair(first, second) {
|
||||
if (first >= L_BASE && first < L_BASE + L_COUNT && second >= V_BASE && second < V_BASE + V_COUNT)
|
||||
return HANGUL_BASE + ((first - L_BASE) * V_COUNT + second - V_BASE) * T_COUNT;
|
||||
return composeSyllable(first, second) ?? tables().composition.get(`${first},${second}`);
|
||||
}
|
||||
/** @param {number} first @param {number} second */
|
||||
function composeSyllable(first, second) {
|
||||
if (
|
||||
first >= HANGUL_BASE &&
|
||||
first < HANGUL_BASE + HANGUL_COUNT &&
|
||||
(first - HANGUL_BASE) % T_COUNT === 0 &&
|
||||
second > T_BASE &&
|
||||
second < T_BASE + T_COUNT
|
||||
)
|
||||
return first + second - T_BASE;
|
||||
}
|
||||
|
||||
/** Canonical decomposition then canonical ordering of combining marks. @param {string} value */
|
||||
function orderedCodes(value) {
|
||||
/** @type {number[]} */
|
||||
const ordered = [];
|
||||
for (const char of value) {
|
||||
for (const code of decompose(char.codePointAt(0) ?? 0)) {
|
||||
const current = classOf(code);
|
||||
let position = ordered.length;
|
||||
while (current && position && classOf(ordered[position - 1] ?? 0) > current) position--;
|
||||
ordered.splice(position, 0, code);
|
||||
}
|
||||
}
|
||||
return ordered;
|
||||
}
|
||||
|
||||
/** Canonical composition under the pinned Unicode table. @param {number[]} ordered */
|
||||
function composed(ordered) {
|
||||
/** @type {number[]} */
|
||||
const result = [];
|
||||
let starter = 0;
|
||||
let lastClass = 0;
|
||||
for (const code of ordered) {
|
||||
const current = classOf(code);
|
||||
const head = result[starter];
|
||||
const composite = head === undefined ? undefined : composePair(head, code);
|
||||
if (composite !== undefined && (lastClass < current || lastClass === 0)) {
|
||||
result[starter] = composite;
|
||||
continue;
|
||||
}
|
||||
if (current === 0) starter = result.length;
|
||||
result.push(code);
|
||||
lastClass = current;
|
||||
}
|
||||
return String.fromCodePoint(...result);
|
||||
}
|
||||
|
||||
const ASCII = /^\p{ASCII}*$/u;
|
||||
|
||||
/** NFC (UAX #15) under the pinned table. @param {string} value */
|
||||
export const nfc = (value) => (ASCII.test(value) ? value : composed(orderedCodes(value)));
|
||||
|
||||
/** Default full case folding (status C and F), so `ß` folds to `ss`. @param {string} value */
|
||||
function fullCasefold(value) {
|
||||
if (ASCII.test(value)) return value.toLowerCase();
|
||||
let folded = "";
|
||||
for (const char of value) folded += tables().casefold.get(char.codePointAt(0) ?? 0) ?? char;
|
||||
return folded;
|
||||
}
|
||||
|
||||
/** Whether every code point is assigned in Unicode 15.1. @param {string} value */
|
||||
export function allAssigned(value) {
|
||||
const { starts, ends } = tables();
|
||||
for (const char of value) {
|
||||
const code = char.codePointAt(0) ?? 0;
|
||||
let low = 0;
|
||||
let high = starts.length;
|
||||
while (low < high) {
|
||||
const middle = (low + high) >>> 1;
|
||||
if ((starts[middle] ?? Infinity) <= code) low = middle + 1;
|
||||
else high = middle;
|
||||
}
|
||||
if (low === 0 || code > (ends[low - 1] ?? -1)) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Directory names repeat across a manifest's paths, and each segment is keyed twice (policy, then collisions). */
|
||||
const KEYS_KEPT = 16384;
|
||||
/** @type {Map<string, string>} */
|
||||
const keys = new Map();
|
||||
|
||||
/** The spelling two names share on a case-insensitive or normalizing filesystem: NFC(casefold(NFC(value))).
|
||||
* @param {string} value */
|
||||
export function comparisonKey(value) {
|
||||
const known = keys.get(value);
|
||||
if (known !== undefined) return known;
|
||||
const key = nfc(fullCasefold(nfc(value)));
|
||||
if (keys.size >= KEYS_KEPT) keys.clear();
|
||||
keys.set(value, key);
|
||||
return key;
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
// @ts-check
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { join } from "node:path";
|
||||
import { NotWritable, uploadNeeded } from "./uploadFiles.mjs";
|
||||
import { parseStoredManifest } from "./manifest.mjs";
|
||||
import { SyncApiError } from "./errors.mjs";
|
||||
|
||||
/** @typedef {import("./adapterTypes.js").SourceRun} SourceRun
|
||||
* @typedef {Pick<SourceRun, "binding" | "api" | "store">} SourceUpload
|
||||
* @typedef {import("./types.js").UploadReceipt} UploadReceipt */
|
||||
/** @param {SourceUpload & Pick<SourceRun, "manifest">} run */
|
||||
async function begin(run) {
|
||||
const { binding, manifest, api, store } = run;
|
||||
if (!binding.inflight) {
|
||||
binding.inflight = {
|
||||
idempotencyKey: randomUUID(),
|
||||
baseSnapshotId: binding.base?.snapshotId ?? null,
|
||||
resolvesConflictSnapshotIds: [],
|
||||
summary: null,
|
||||
manifest: manifest.document,
|
||||
manifestSha256: manifest.sha256,
|
||||
uploadId: null,
|
||||
snapshotId: null,
|
||||
uploadDeadline: null,
|
||||
verificationDeadline: null,
|
||||
begunAt: Date.now(),
|
||||
sent: [],
|
||||
multipart: {},
|
||||
};
|
||||
await store.write(binding);
|
||||
}
|
||||
const inflight = binding.inflight;
|
||||
const receipt = await api.begin(binding.ownerSpaceId, binding.compositionId, {
|
||||
...inflight,
|
||||
manifest,
|
||||
});
|
||||
assertReceipt(binding, receipt);
|
||||
Object.assign(inflight, {
|
||||
uploadId: receipt.uploadId,
|
||||
snapshotId: receipt.snapshotId,
|
||||
uploadDeadline: receipt.uploadDeadline,
|
||||
verificationDeadline: receipt.verificationDeadline,
|
||||
});
|
||||
await store.write(binding);
|
||||
return receipt;
|
||||
}
|
||||
/** @param {SourceRun} run @param {import("./types.js").BeginResult} receipt */
|
||||
async function sendFiles(run, receipt) {
|
||||
const { binding, manifest, store } = run;
|
||||
if (!binding.inflight) throw new Error("Missing source upload intent.");
|
||||
const local = new Map(
|
||||
manifest.document.files.map((file) => [
|
||||
file.sha256,
|
||||
{ path: file.path, full: join(binding.projectPath, file.path) },
|
||||
]),
|
||||
);
|
||||
const known = new Map(manifest.document.files.map((file) => [file.sha256, file.size]));
|
||||
for (const blob of receipt.needed)
|
||||
if (known.get(blob.sha256) !== blob.size)
|
||||
throw new Error("Upload requested an unknown source file.");
|
||||
let persisted = Promise.resolve();
|
||||
await uploadNeeded({
|
||||
api: run.api,
|
||||
space: binding.ownerSpaceId,
|
||||
composition: binding.compositionId,
|
||||
uploadId: receipt.uploadId,
|
||||
inflight: binding.inflight,
|
||||
local,
|
||||
needed: receipt.needed,
|
||||
fetchImpl: run.fetchImpl,
|
||||
wait: run.wait,
|
||||
signal: run.signal,
|
||||
now: Date.now,
|
||||
persist: () => {
|
||||
persisted = persisted.then(() => store.write(binding));
|
||||
},
|
||||
onBytes: run.onProgress,
|
||||
onFile: () => {},
|
||||
}).finally(() => persisted);
|
||||
return run.api.submit(binding.ownerSpaceId, binding.compositionId, receipt.uploadId);
|
||||
}
|
||||
/** @param {SourceUpload["binding"]} binding @param {UploadReceipt} receipt */
|
||||
function assertReceipt(binding, receipt) {
|
||||
const saved = binding.inflight;
|
||||
if (!saved || receipt.manifestSha256 !== saved.manifestSha256)
|
||||
throw new Error("Source upload receipt does not match the captured project.");
|
||||
if (receipt.compositionId !== binding.compositionId)
|
||||
throw new Error("Source upload receipt does not match the composition.");
|
||||
if (saved.uploadId !== null && receipt.uploadId !== saved.uploadId)
|
||||
throw new Error("Source upload receipt identity changed.");
|
||||
if (saved.snapshotId !== null && receipt.snapshotId !== saved.snapshotId)
|
||||
throw new Error("Source snapshot identity changed.");
|
||||
}
|
||||
|
||||
/** @param {SourceUpload} job @param {UploadReceipt} receipt */
|
||||
async function storePublished({ binding, store }, receipt) {
|
||||
const saved = binding.inflight;
|
||||
const result = receipt.result;
|
||||
if (!saved || !result || result.snapshotId !== receipt.snapshotId)
|
||||
throw new Error("Source snapshot identity changed.");
|
||||
binding.base = {
|
||||
snapshotId: result.snapshotId,
|
||||
manifest: saved.manifest,
|
||||
manifestSha256: saved.manifestSha256,
|
||||
};
|
||||
await clearIntent({ binding, store });
|
||||
}
|
||||
|
||||
/** @param {Pick<SourceUpload, "binding" | "store">} job */
|
||||
async function clearIntent({ binding, store }) {
|
||||
binding.inflight = null;
|
||||
await store.write(binding);
|
||||
}
|
||||
|
||||
/** @param {SourceUpload} job @param {UploadReceipt} receipt */
|
||||
async function cancelPending({ binding, api }, receipt) {
|
||||
const { ownerSpaceId, compositionId } = binding;
|
||||
try {
|
||||
return await api.cancel(ownerSpaceId, compositionId, receipt.uploadId);
|
||||
} catch (error) {
|
||||
if (!(error instanceof SyncApiError) || error.kind !== "upload-not-pending") throw error;
|
||||
return api.status(ownerSpaceId, compositionId, receipt.uploadId);
|
||||
}
|
||||
}
|
||||
|
||||
/** Resolve an older capture before starting an upload for the current files.
|
||||
* @param {SourceUpload} job @returns {Promise<import("./adapterTypes.js").SourceResult | null>} */
|
||||
export async function recoverSourceUpload(job) {
|
||||
const { binding, api } = job;
|
||||
const saved = binding.inflight;
|
||||
if (!saved) return null;
|
||||
const manifest = parseStoredManifest(saved.manifest);
|
||||
if (manifest.sha256 !== saved.manifestSha256)
|
||||
throw new Error("Saved source upload manifest does not match its checksum.");
|
||||
let receipt = saved.uploadId
|
||||
? await api.status(binding.ownerSpaceId, binding.compositionId, saved.uploadId)
|
||||
: await begin({ ...job, manifest });
|
||||
assertReceipt(binding, receipt);
|
||||
if (receipt.status === "pending") receipt = await cancelPending(job, receipt);
|
||||
assertReceipt(binding, receipt);
|
||||
return recoverReceipt(job, receipt);
|
||||
}
|
||||
|
||||
/** @param {SourceUpload} job @param {UploadReceipt} receipt */
|
||||
async function recoverReceipt(job, receipt) {
|
||||
if (receipt.status === "completed") {
|
||||
if (receipt.result?.publicationOutcome !== "published")
|
||||
return { at: /** @type {const} */ ("unavailable"), reason: "source_conflict" };
|
||||
await storePublished(job, receipt);
|
||||
} else if (terminalFailure(receipt)) await clearIntent(job);
|
||||
else return { at: /** @type {const} */ ("unavailable"), reason: "source_verification_pending" };
|
||||
return null;
|
||||
}
|
||||
|
||||
/** @param {UploadReceipt} receipt */
|
||||
const terminalFailure = (receipt) => receipt.status === "failed" || receipt.status === "expired";
|
||||
|
||||
/** Only published verification results can become share source; conflict_saved is preserved for explicit recovery.
|
||||
* @param {SourceRun} run @returns {Promise<import("./adapterTypes.js").SourceResult>} */
|
||||
export async function uploadSource(run) {
|
||||
const receipt = await begin(run);
|
||||
let status;
|
||||
try {
|
||||
status = receipt.status === "pending" ? await sendFiles(run, receipt) : receipt;
|
||||
} catch (error) {
|
||||
if (!(error instanceof NotWritable)) throw error;
|
||||
return (
|
||||
(await recoverSourceUpload(run)) ?? {
|
||||
at: "unavailable",
|
||||
reason: "source_upload_not_writable",
|
||||
}
|
||||
);
|
||||
}
|
||||
const deadline = Date.now() + run.timeoutMs;
|
||||
while (status.status === "verifying") {
|
||||
run.signal.throwIfAborted();
|
||||
if (Date.now() >= deadline) return { at: "unavailable", reason: "source_verification_pending" };
|
||||
await run.wait(1000, run.signal);
|
||||
status = await run.api.status(
|
||||
run.binding.ownerSpaceId,
|
||||
run.binding.compositionId,
|
||||
receipt.uploadId,
|
||||
);
|
||||
}
|
||||
assertReceipt(run.binding, status);
|
||||
if (status.status !== "completed" || !status.result) {
|
||||
if (terminalFailure(status)) await clearIntent(run);
|
||||
return { at: "unavailable", reason: status.errorCode ?? `source_upload_${status.status}` };
|
||||
}
|
||||
if (status.result.publicationOutcome !== "published")
|
||||
return { at: "unavailable", reason: "source_conflict" };
|
||||
await storePublished(run, status);
|
||||
return {
|
||||
at: "ready",
|
||||
snapshot: {
|
||||
space_id: run.binding.ownerSpaceId,
|
||||
composition_id: run.binding.compositionId,
|
||||
snapshot_id: status.snapshotId,
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,263 @@
|
||||
// @ts-check
|
||||
// The files a Begin said are needed, written to the upload's staging through signed URLs: one PUT for a file up to
|
||||
// 64 MiB, parts of 64 MiB above that through the file's one pinned multipart upload. URLs live at most 15 minutes, so
|
||||
// one about to end is asked for again; a send resumed after a quit skips what the in-flight record says was written.
|
||||
import { SyncApiError } from "./errors.mjs";
|
||||
import {
|
||||
TransferError,
|
||||
backoffMs,
|
||||
eachLimit,
|
||||
isAbort,
|
||||
partDigests,
|
||||
putRange,
|
||||
} from "./transfer.mjs";
|
||||
import { lstat } from "node:fs/promises";
|
||||
|
||||
/** @typedef {import("./types.js").FileGrant} FileGrant @typedef {import("./types.js").GrantRequest} GrantRequest
|
||||
* @typedef {import("./types.js").InflightUpload} InflightUpload */
|
||||
|
||||
/** The server's single-PUT ceiling and part size. */
|
||||
const PART_BYTES = 64 * 1024 * 1024;
|
||||
const GRANT_FILES = 100;
|
||||
const PARTS_PER_GRANT = 8;
|
||||
const AT_ONCE = 4;
|
||||
const ATTEMPTS = 5;
|
||||
/** A URL with less than this left is asked for again rather than started. */
|
||||
const URL_MARGIN_MS = 60_000;
|
||||
|
||||
/** The file's multipart upload can take no more parts (`hyperframes_composition_upload_file_not_writable`): this
|
||||
* upload cannot finish, so its owner must resolve it before a retry can begin a new one. */
|
||||
export class NotWritable extends Error {
|
||||
constructor() {
|
||||
super("composition sync: a file's multipart upload can no longer take parts");
|
||||
this.name = "NotWritable";
|
||||
}
|
||||
}
|
||||
|
||||
/** @typedef {import("./types.js").UploadJob} UploadJob */
|
||||
|
||||
/** @param {UploadJob} job @param {unknown} error @param {number} attempt the attempt that just failed, from 1
|
||||
* @returns {Promise<boolean>} whether to try again */
|
||||
async function againAfter(job, error, attempt) {
|
||||
if (attempt >= ATTEMPTS || isAbort(error)) return false;
|
||||
const transient =
|
||||
(error instanceof TransferError && (error.kind === "transient" || error.kind === "corrupt")) ||
|
||||
(error instanceof SyncApiError &&
|
||||
["offline", "unavailable", "server-error", "rate-limited"].includes(error.kind));
|
||||
if (!transient) return false;
|
||||
const asked = error instanceof SyncApiError ? error.retryAfterMs : null;
|
||||
await job.wait(backoffMs(attempt, asked), job.signal);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** A refusal of a pinned multipart id, which its refusal says to drop and ask again without.
|
||||
* @param {unknown} error @param {GrantRequest[]} asked */
|
||||
const wrongPinnedId = (error, asked) =>
|
||||
error instanceof SyncApiError &&
|
||||
error.kind === "invalid-parameter" &&
|
||||
(error.param ?? "").endsWith("multipart_upload_id") &&
|
||||
asked.some((file) => file.multipartUploadId);
|
||||
|
||||
/** Write grants for `files`, sent again on a transient failure. A file that can take no more parts throws NotWritable.
|
||||
* @param {UploadJob} job @param {GrantRequest[]} files @returns {Promise<FileGrant[]>} */
|
||||
async function grant(job, files) {
|
||||
let asked = files;
|
||||
for (let attempt = 1; ; attempt++) {
|
||||
try {
|
||||
return await job.api.uploadUrls(job.space, job.composition, job.uploadId, asked);
|
||||
} catch (error) {
|
||||
if (error instanceof SyncApiError && error.kind === "file-not-writable")
|
||||
throw new NotWritable();
|
||||
if (wrongPinnedId(error, asked))
|
||||
asked = asked.map(({ multipartUploadId: _dropped, ...file }) => file);
|
||||
else if (!(await againAfter(job, error, attempt))) throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** @param {UploadJob} job @param {string} sha256 */
|
||||
function localFile(job, sha256) {
|
||||
const file = job.local.get(sha256);
|
||||
if (!file) throw new Error(`composition sync: needed file ${sha256} is not in the capture`);
|
||||
return file;
|
||||
}
|
||||
|
||||
/** @typedef {{ url: string, headers: Record<string, string>, expiresAt: string }} Signed
|
||||
* @typedef {{ number: number, sha256: string, start: number, length: number }} Part
|
||||
* @typedef {{ file: { path: string, full: string }, start: number, length: number, sha256: string }} Bytes */
|
||||
|
||||
/** One PUT attempt: `done` once stored (or found already stored), `expired` when storage says the URL has, else the
|
||||
* failure. Bytes counted for an attempt that failed are taken back.
|
||||
* @param {UploadJob} job @param {Signed} held @param {Bytes} bytes
|
||||
* @returns {Promise<"done" | "expired" | { failed: unknown }>} */
|
||||
async function attemptPut(job, held, { file, ...range }) {
|
||||
let sent = 0;
|
||||
try {
|
||||
await putRange({
|
||||
fetchImpl: job.fetchImpl,
|
||||
url: held.url,
|
||||
headers: held.headers,
|
||||
file: file.full,
|
||||
path: file.path,
|
||||
...range,
|
||||
signal: job.signal,
|
||||
onBytes: (count) => ((sent += count), job.onBytes(count)),
|
||||
});
|
||||
return "done";
|
||||
} catch (error) {
|
||||
job.onBytes(-sent);
|
||||
if (!(error instanceof TransferError)) return { failed: error };
|
||||
if (error.kind === "exists") return (job.onBytes(range.length), "done");
|
||||
return error.kind === "expired" ? "expired" : { failed: error };
|
||||
}
|
||||
}
|
||||
|
||||
/** Writes `bytes` through `given`, or a fresh grant when it is missing or about to end; asks again when storage says
|
||||
* the URL ended, and sends again after a transient failure.
|
||||
* @param {UploadJob} job @param {Signed | null} given @param {() => Promise<Signed>} fresh @param {Bytes} bytes */
|
||||
async function putWithGrant(job, given, fresh, bytes) {
|
||||
const usable = (/** @type {Signed | null} */ signed) =>
|
||||
signed && Date.parse(signed.expiresAt) - job.now() >= URL_MARGIN_MS ? signed : null;
|
||||
let held = usable(given);
|
||||
for (let attempt = 1; ; attempt++) {
|
||||
held ??= await fresh();
|
||||
const outcome = await attemptPut(job, held, bytes);
|
||||
if (outcome === "done") return;
|
||||
if (outcome === "expired" && attempt < ATTEMPTS) held = null;
|
||||
else {
|
||||
const failed = outcome === "expired" ? new TransferError("expired") : outcome.failed;
|
||||
if (!(await againAfter(job, failed, attempt))) throw failed;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** @param {FileGrant | undefined} held @returns {Signed} */
|
||||
function singleUrl(held) {
|
||||
if (!held?.url) throw new Error("composition sync: a single-PUT grant came without its URL");
|
||||
return { url: held.url, headers: held.headers, expiresAt: held.expiresAt };
|
||||
}
|
||||
|
||||
/** Files up to 64 MiB: grants a hundred at a time, written four at once. @param {UploadJob} job
|
||||
* @param {{ sha256: string, size: number }[]} blobs */
|
||||
async function sendSmall(job, blobs) {
|
||||
for (let at = 0; at < blobs.length; at += GRANT_FILES) {
|
||||
const batch = blobs.slice(at, at + GRANT_FILES);
|
||||
const grants = await grant(
|
||||
job,
|
||||
batch.map((blob) => ({ sha256: blob.sha256 })),
|
||||
);
|
||||
const bySha = new Map(grants.map((one) => [one.sha256, one]));
|
||||
await eachLimit(batch, AT_ONCE, async (blob) => {
|
||||
const given = bySha.has(blob.sha256) ? singleUrl(bySha.get(blob.sha256)) : null;
|
||||
const fresh = async () => singleUrl((await grant(job, [{ sha256: blob.sha256 }]))[0]);
|
||||
const bytes = {
|
||||
file: localFile(job, blob.sha256),
|
||||
start: 0,
|
||||
length: blob.size,
|
||||
sha256: blob.sha256,
|
||||
};
|
||||
await putWithGrant(job, given, fresh, bytes);
|
||||
(job.inflight.sent ??= []).push(blob.sha256);
|
||||
job.persist();
|
||||
job.onFile();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/** Grants for `parts` of a multipart file, naming its pinned upload once one is known, and keeping the id the server
|
||||
* pins. @param {UploadJob} job @param {{ sha256: string }} blob @param {Part[]} parts */
|
||||
async function partGrants(job, blob, parts) {
|
||||
const pinned = job.inflight.multipart?.[blob.sha256];
|
||||
const [held] = await grant(job, [
|
||||
{
|
||||
sha256: blob.sha256,
|
||||
...(pinned && { multipartUploadId: pinned.uploadId }),
|
||||
parts: parts.map((part) => ({ partNumber: part.number, sha256: part.sha256 })),
|
||||
},
|
||||
]);
|
||||
if (!held?.multipartUploadId)
|
||||
throw new Error("composition sync: a multipart grant came without its upload id");
|
||||
// Parts recorded as written belong to the pinned upload; another id would leave them unwritten.
|
||||
if (pinned && pinned.uploadId !== held.multipartUploadId) throw new NotWritable();
|
||||
if (!pinned) {
|
||||
job.inflight.multipart = {
|
||||
...job.inflight.multipart,
|
||||
[blob.sha256]: { uploadId: held.multipartUploadId, partsDone: [] },
|
||||
};
|
||||
job.persist();
|
||||
}
|
||||
return { expiresAt: held.expiresAt, parts: held.parts };
|
||||
}
|
||||
|
||||
/** The file's parts still to write, each with its SHA-256; throws `changed` when the file no longer hashes to the
|
||||
* captured SHA-256. @param {UploadJob} job @param {{ sha256: string, size: number }} blob
|
||||
* @returns {Promise<Part[]>} */
|
||||
async function partsToSend(job, blob) {
|
||||
const file = localFile(job, blob.sha256);
|
||||
const digests = await partDigests(file.full, blob.size, PART_BYTES, job.signal).catch((error) => {
|
||||
if (isAbort(error)) throw error;
|
||||
throw new TransferError("unreadable", { path: file.path });
|
||||
});
|
||||
if (digests.whole !== blob.sha256 || !digests.complete)
|
||||
throw new TransferError("changed", { path: file.path });
|
||||
const done = new Set(job.inflight.multipart?.[blob.sha256]?.partsDone ?? []);
|
||||
return digests.parts
|
||||
.map((sha256, index) => ({
|
||||
number: index + 1,
|
||||
sha256,
|
||||
start: index * PART_BYTES,
|
||||
length: Math.min(PART_BYTES, blob.size - index * PART_BYTES),
|
||||
}))
|
||||
.filter((part) => !done.has(part.number));
|
||||
}
|
||||
|
||||
/** A file over 64 MiB, eight parts a grant, four parts at once; its size and time are compared after, so a file that
|
||||
* changed while its parts were read fails as changed. @param {UploadJob} job
|
||||
* @param {{ sha256: string, size: number }} blob */
|
||||
async function sendLarge(job, blob) {
|
||||
const file = localFile(job, blob.sha256);
|
||||
const before = await lstat(file.full);
|
||||
const parts = await partsToSend(job, blob);
|
||||
job.onBytes(blob.size - parts.reduce((sum, part) => sum + part.length, 0));
|
||||
for (let at = 0; at < parts.length; at += PARTS_PER_GRANT) {
|
||||
const batch = parts.slice(at, at + PARTS_PER_GRANT);
|
||||
const granted = await partGrants(job, blob, batch);
|
||||
const byNumber = new Map(granted.parts.map((part) => [part.partNumber, part]));
|
||||
await eachLimit(batch, AT_ONCE, async (part) => {
|
||||
const signed = byNumber.get(part.number);
|
||||
const given = signed
|
||||
? { url: signed.url, headers: signed.headers, expiresAt: granted.expiresAt }
|
||||
: null;
|
||||
const fresh = async () => {
|
||||
const again = await partGrants(job, blob, [part]);
|
||||
const one = again.parts[0];
|
||||
if (!one) throw new Error("composition sync: a part grant came without its part");
|
||||
return { url: one.url, headers: one.headers, expiresAt: again.expiresAt };
|
||||
};
|
||||
await putWithGrant(job, given, fresh, { file, ...part });
|
||||
job.inflight.multipart?.[blob.sha256]?.partsDone.push(part.number);
|
||||
job.persist();
|
||||
});
|
||||
}
|
||||
const after = await lstat(file.full).catch(() => null);
|
||||
if (!after || after.size !== before.size || after.mtimeMs !== before.mtimeMs)
|
||||
throw new TransferError("changed", { path: file.path });
|
||||
(job.inflight.sent ??= []).push(blob.sha256);
|
||||
job.persist();
|
||||
job.onFile();
|
||||
}
|
||||
|
||||
/** Writes every needed file the in-flight record does not list as sent. @param {UploadJob} job */
|
||||
export async function uploadNeeded(job) {
|
||||
const sent = new Set(job.inflight.sent ?? []);
|
||||
const waiting = job.needed.filter((blob) => !sent.has(blob.sha256));
|
||||
for (const blob of job.needed.filter((one) => sent.has(one.sha256))) {
|
||||
job.onBytes(blob.size);
|
||||
job.onFile();
|
||||
}
|
||||
await sendSmall(
|
||||
job,
|
||||
waiting.filter((blob) => blob.size <= PART_BYTES),
|
||||
);
|
||||
for (const blob of waiting.filter((one) => one.size > PART_BYTES)) await sendLarge(job, blob);
|
||||
}
|
||||
@@ -0,0 +1,205 @@
|
||||
// @ts-check
|
||||
// Parse and validate sync API responses at the transport boundary.
|
||||
import { ManifestInvalid, parseStoredManifest } from "./manifest.mjs";
|
||||
import { SyncApiError } from "./errors.mjs";
|
||||
|
||||
/** @typedef {import("./types.js").SyncLimitsRead} SyncLimitsRead
|
||||
* @typedef {import("./types.js").Composition} Composition @typedef {import("./types.js").Snapshot} Snapshot
|
||||
* @typedef {import("./types.js").UploadReceipt} UploadReceipt @typedef {import("./types.js").BeginResult} BeginResult
|
||||
* @typedef {import("./types.js").FileGrant} FileGrant @typedef {import("./types.js").DownloadBatch} DownloadBatch */
|
||||
|
||||
/** @param {string} field */
|
||||
const malformed = (field) =>
|
||||
new SyncApiError("malformed", { detail: `missing or mistyped ${field}` });
|
||||
|
||||
/** @param {unknown} value @param {string} field @returns {Record<string, unknown>} */
|
||||
export function objectOf(value, field) {
|
||||
if (typeof value !== "object" || value === null || Array.isArray(value)) throw malformed(field);
|
||||
return /** @type {Record<string, unknown>} */ (value);
|
||||
}
|
||||
/** @param {Record<string, unknown>} o @param {string} key @returns {string} */
|
||||
const str = (o, key) => {
|
||||
const value = o[key];
|
||||
if (typeof value !== "string") throw malformed(key);
|
||||
return value;
|
||||
};
|
||||
/** Absent (the server drops nulls) reads as null. @param {Record<string, unknown>} o @param {string} key */
|
||||
const optStr = (o, key) => (o[key] === undefined || o[key] === null ? null : str(o, key));
|
||||
/** @param {Record<string, unknown>} o @param {string} key @returns {number} */
|
||||
const int = (o, key) => {
|
||||
const value = o[key];
|
||||
if (typeof value !== "number" || !Number.isSafeInteger(value)) throw malformed(key);
|
||||
return value;
|
||||
};
|
||||
/** @param {Record<string, unknown>} o @param {string} key @returns {unknown[]} */
|
||||
const list = (o, key) => {
|
||||
const value = o[key] ?? [];
|
||||
if (!Array.isArray(value)) throw malformed(key);
|
||||
return value;
|
||||
};
|
||||
/** @param {Record<string, unknown>} o @param {string} key @returns {Record<string, string>} */
|
||||
const headers = (o, key) => {
|
||||
const value = objectOf(o[key] ?? {}, key);
|
||||
return Object.fromEntries(Object.keys(value).map((name) => [name, str(value, name)]));
|
||||
};
|
||||
/** @template {string} T @param {Record<string, unknown>} o @param {string} key @param {readonly T[]} allowed
|
||||
* @returns {T} */
|
||||
const oneOf = (o, key, allowed) => {
|
||||
const value = str(o, key);
|
||||
if (!allowed.includes(/** @type {T} */ (value))) throw malformed(key);
|
||||
return /** @type {T} */ (value);
|
||||
};
|
||||
|
||||
/** @param {unknown} raw @returns {Composition} */
|
||||
export function compositionOf(raw) {
|
||||
const o = objectOf(raw, "composition");
|
||||
return {
|
||||
id: str(o, "id"),
|
||||
spaceId: str(o, "space_id"),
|
||||
title: str(o, "title"),
|
||||
createdBy: str(o, "created_by"),
|
||||
latestSnapshotId: optStr(o, "latest_snapshot_id"),
|
||||
createdAt: str(o, "created_at"),
|
||||
updatedAt: str(o, "updated_at"),
|
||||
};
|
||||
}
|
||||
|
||||
/** The snapshot's manifest, trusted only when its canonical bytes hash to the snapshot's manifest_sha256.
|
||||
* @param {unknown} raw @param {string} sha256 */
|
||||
function snapshotManifestOf(raw, sha256) {
|
||||
const o = objectOf(raw, "manifest");
|
||||
try {
|
||||
const manifest = parseStoredManifest(o.document);
|
||||
if (manifest.sha256 !== sha256) throw malformed("manifest.document (sha256)");
|
||||
return manifest;
|
||||
} catch (error) {
|
||||
if (error instanceof ManifestInvalid) throw malformed(`manifest.document (${error.code})`);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/** @param {unknown} raw @returns {Snapshot} */
|
||||
export function snapshotOf(raw) {
|
||||
const o = objectOf(raw, "snapshot");
|
||||
const manifestSha256 = str(o, "manifest_sha256");
|
||||
const state = optStr(o, "conflict_state");
|
||||
if (state !== null && !["open", "resolved", "discarded"].includes(state))
|
||||
throw malformed("conflict_state");
|
||||
return {
|
||||
id: str(o, "id"),
|
||||
compositionId: str(o, "composition_id"),
|
||||
baseSnapshotId: optStr(o, "base_snapshot_id"),
|
||||
publicationOutcome: oneOf(
|
||||
o,
|
||||
"publication_outcome",
|
||||
/** @type {const} */ (["published", "conflict_saved"]),
|
||||
),
|
||||
conflictedWithSnapshotId: optStr(o, "conflicted_with_snapshot_id"),
|
||||
conflictState: /** @type {Snapshot["conflictState"]} */ (state),
|
||||
resolvedBySnapshotId: optStr(o, "resolved_by_snapshot_id"),
|
||||
createdBy: str(o, "created_by"),
|
||||
createdByName: optStr(o, "created_by_name"),
|
||||
clientType: str(o, "client_type"),
|
||||
summary: optStr(o, "summary"),
|
||||
summarySource: str(o, "summary_source"),
|
||||
manifestSha256,
|
||||
createdAt: str(o, "created_at"),
|
||||
manifest: o.manifest === undefined ? null : snapshotManifestOf(o.manifest, manifestSha256),
|
||||
};
|
||||
}
|
||||
|
||||
const UPLOAD_STATUSES = /** @type {const} */ ([
|
||||
"pending",
|
||||
"verifying",
|
||||
"completed",
|
||||
"failed",
|
||||
"expired",
|
||||
]);
|
||||
|
||||
/** @param {unknown} raw @returns {UploadReceipt} */
|
||||
export function uploadOf(raw) {
|
||||
const o = objectOf(raw, "upload");
|
||||
const result = o.result === undefined || o.result === null ? null : objectOf(o.result, "result");
|
||||
return {
|
||||
uploadId: str(o, "upload_id"),
|
||||
compositionId: str(o, "composition_id"),
|
||||
snapshotId: str(o, "snapshot_id"),
|
||||
status: oneOf(o, "status", UPLOAD_STATUSES),
|
||||
errorCode: optStr(o, "error_code"),
|
||||
baseSnapshotId: optStr(o, "base_snapshot_id"),
|
||||
resolvesConflictSnapshotIds: list(o, "resolves_conflict_snapshot_ids").map((id) => {
|
||||
if (typeof id !== "string") throw malformed("resolves_conflict_snapshot_ids");
|
||||
return id;
|
||||
}),
|
||||
manifestSha256: str(o, "manifest_sha256"),
|
||||
uploadDeadline: str(o, "upload_deadline"),
|
||||
verificationDeadline: optStr(o, "verification_deadline"),
|
||||
completedAt: optStr(o, "completed_at"),
|
||||
result: result && {
|
||||
snapshotId: str(result, "snapshot_id"),
|
||||
publicationOutcome: oneOf(
|
||||
result,
|
||||
"publication_outcome",
|
||||
/** @type {const} */ (["published", "conflict_saved"]),
|
||||
),
|
||||
conflictedWithSnapshotId: optStr(result, "conflicted_with_snapshot_id"),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** @param {unknown} raw @param {string} field */
|
||||
const blobsOf = (raw, field) =>
|
||||
list(objectOf(raw, field), field).map((item) => {
|
||||
const blob = objectOf(item, field);
|
||||
return { sha256: str(blob, "sha256"), size: int(blob, "size") };
|
||||
});
|
||||
|
||||
/** @param {unknown} raw @param {boolean} created @returns {BeginResult} */
|
||||
export const beginOf = (raw, created) => ({
|
||||
...uploadOf(raw),
|
||||
present: blobsOf(raw, "present"),
|
||||
needed: blobsOf(raw, "needed"),
|
||||
created,
|
||||
});
|
||||
|
||||
/** @param {unknown} raw @returns {FileGrant[]} */
|
||||
export const grantsOf = (raw) =>
|
||||
list(objectOf(raw, "grants"), "files").map((item) => {
|
||||
const o = objectOf(item, "files");
|
||||
return {
|
||||
sha256: str(o, "sha256"),
|
||||
size: int(o, "size"),
|
||||
expiresAt: str(o, "expires_at"),
|
||||
url: optStr(o, "url"),
|
||||
headers: headers(o, "headers"),
|
||||
multipartUploadId: optStr(o, "multipart_upload_id"),
|
||||
parts: list(o, "parts").map((part) => {
|
||||
const p = objectOf(part, "parts");
|
||||
return {
|
||||
partNumber: int(p, "part_number"),
|
||||
length: int(p, "length"),
|
||||
url: str(p, "url"),
|
||||
headers: headers(p, "headers"),
|
||||
};
|
||||
}),
|
||||
};
|
||||
});
|
||||
|
||||
/** @param {unknown} raw @returns {Extract<SyncLimitsRead, { at: "read" }>} */
|
||||
export function syncLimitsOf(raw) {
|
||||
const o = objectOf(raw, "limits answer");
|
||||
const limits = objectOf(o.limits, "limits");
|
||||
if (typeof o.sync_allowed !== "boolean") throw malformed("sync_allowed");
|
||||
return {
|
||||
at: "read",
|
||||
syncAllowed: o.sync_allowed,
|
||||
maxCompositions: int(limits, "max_compositions"),
|
||||
compositions: int(objectOf(o.usage, "usage"), "compositions"),
|
||||
push: {
|
||||
maxFiles: int(limits, "max_files"),
|
||||
maxFileBytes: int(limits, "max_file_bytes"),
|
||||
maxProjectBytes: int(limits, "max_project_bytes"),
|
||||
maxManifestBytes: int(limits, "max_manifest_bytes"),
|
||||
},
|
||||
};
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,6 +1,6 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs, { mkdtemp, mkdir, readFile, writeFile, rm, symlink } from "node:fs/promises";
|
||||
import fs, { mkdtemp, mkdir, readFile, writeFile, rm, symlink, utimes } from "node:fs/promises";
|
||||
import { tmpdir, hostname } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { realpathSync } from "node:fs";
|
||||
@@ -437,3 +437,88 @@ for (const name of ["review.json", "review-export.json", "review-switch.json"])
|
||||
assert.equal(await readFile(file, "utf8"), "");
|
||||
});
|
||||
}
|
||||
test("required source never silently degrades after preparation or a server mismatch", async (t) => {
|
||||
const dir = await folder(t);
|
||||
await mkdir(join(dir, ".hyperframes"));
|
||||
await writeFile(join(dir, ".hyperframes/video.mp4"), "video");
|
||||
bindShareHost({ projectFingerprint: async () => "rendered" });
|
||||
t.after(() => bindShareHost({ projectFingerprint }));
|
||||
let creates = 0;
|
||||
const api = {
|
||||
createExport: async () => {
|
||||
creates++;
|
||||
throw new Error("should not send");
|
||||
},
|
||||
};
|
||||
const request = {
|
||||
projectPath: dir,
|
||||
title: "Video",
|
||||
client: api,
|
||||
projectId: "project",
|
||||
requireSource: true,
|
||||
probe: async () => ({ width: 100, height: 100, duration: 1 }),
|
||||
filesForExport: async () => null,
|
||||
};
|
||||
assert.match(await addVersion(request), /Source changed/);
|
||||
assert.equal(creates, 0);
|
||||
const { ReviewFailed } = await import("../src/reviewClient.mjs");
|
||||
api.createExport = async () => {
|
||||
creates++;
|
||||
throw new ReviewFailed(422, "snapshot_mismatch");
|
||||
};
|
||||
await addVersion({
|
||||
...request,
|
||||
filesForExport: async () => ({
|
||||
space_id: "space",
|
||||
composition_id: "comp",
|
||||
snapshot_id: "snap",
|
||||
}),
|
||||
});
|
||||
assert.equal(creates, 1);
|
||||
});
|
||||
|
||||
test("content fingerprints detect same-size changes with restored timestamps", async (t) => {
|
||||
const dir = await folder(t);
|
||||
const file = join(dir, "asset.txt");
|
||||
const time = new Date("2026-11-01T00:00:00Z");
|
||||
await writeFile(file, "first");
|
||||
await utimes(file, time, time);
|
||||
const metadata = await projectFingerprint(dir);
|
||||
const content = await projectFingerprint(dir, { byContent: true });
|
||||
await writeFile(file, "other");
|
||||
await utimes(file, time, time);
|
||||
assert.equal(await projectFingerprint(dir), metadata);
|
||||
assert.notEqual(await projectFingerprint(dir, { byContent: true }), content);
|
||||
});
|
||||
|
||||
test("a video-only retry never replays an unanswered source-bearing export", async (t) => {
|
||||
const dir = await folder(t);
|
||||
await mkdir(join(dir, ".hyperframes"));
|
||||
await writeFile(join(dir, ".hyperframes/video.mp4"), "video bytes");
|
||||
bindShareHost({ projectFingerprint: async () => "rendered" });
|
||||
t.after(() => bindShareHost({ projectFingerprint }));
|
||||
let creates = 0;
|
||||
const request = {
|
||||
projectPath: dir,
|
||||
title: "Video",
|
||||
client: {
|
||||
createExport: async () => {
|
||||
creates++;
|
||||
throw new Error("lost response");
|
||||
},
|
||||
},
|
||||
projectId: "project",
|
||||
probe: async () => ({ width: 100, height: 100, duration: 1 }),
|
||||
filesForExport: async () => ({
|
||||
space_id: "space",
|
||||
composition_id: "composition",
|
||||
snapshot_id: "snapshot",
|
||||
}),
|
||||
};
|
||||
await addVersion({ ...request, requireSource: true });
|
||||
assert.match(
|
||||
await addVersion({ ...request, omitSource: true }),
|
||||
/pending upload includes source/,
|
||||
);
|
||||
assert.equal(creates, 1);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,410 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, writeFile, rm, readFile, symlink, mkdir, realpath } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join } from "node:path";
|
||||
import { createHash } from "node:crypto";
|
||||
import {
|
||||
shareSources,
|
||||
shareSourceFingerprint,
|
||||
createProjectShareSourceStore,
|
||||
} from "../src/shareSources.mjs";
|
||||
import { projectFingerprint } from "../src/projectSnapshot.mjs";
|
||||
import { captureProject } from "../src/sources/capture.mjs";
|
||||
import { sourceApi } from "../src/sources/api.mjs";
|
||||
import { SyncApiError } from "../src/sources/errors.mjs";
|
||||
import { V1_LIMITS } from "../src/sources/limits.mjs";
|
||||
import { bindShareHost } from "../src/host.mjs";
|
||||
|
||||
bindShareHost({ reviewUploadOrigins: ["https://storage.test"] });
|
||||
|
||||
const signed = {
|
||||
token: "test-token",
|
||||
owner: "owner",
|
||||
heygen: "https://app.test",
|
||||
heygenApi: "https://api.test",
|
||||
};
|
||||
async function fixture(t, files = {}) {
|
||||
const root = await realpath(await mkdtemp(join(tmpdir(), "hf-source-test-")));
|
||||
t.after(() => rm(root, { recursive: true, force: true }));
|
||||
await writeFile(join(root, "index.html"), "<html>video</html>");
|
||||
for (const [file, content] of Object.entries(files)) await writeFile(join(root, file), content);
|
||||
const rendered = await projectFingerprint(root);
|
||||
const store = createProjectShareSourceStore();
|
||||
return {
|
||||
root,
|
||||
rendered,
|
||||
store,
|
||||
key: { backend: signed.heygenApi, account: signed.owner, projectPath: root },
|
||||
};
|
||||
}
|
||||
function fakeServer(overrides = {}) {
|
||||
const calls = { create: 0, begin: [], puts: [], submit: 0, snapshot: 0 };
|
||||
let held;
|
||||
let composition;
|
||||
const receipt = (status = "pending") => ({
|
||||
uploadId: "upload",
|
||||
compositionId: composition,
|
||||
snapshotId: "snapshot",
|
||||
status,
|
||||
errorCode: null,
|
||||
baseSnapshotId: null,
|
||||
resolvesConflictSnapshotIds: [],
|
||||
manifestSha256: held.manifest.sha256,
|
||||
uploadDeadline: "2099-01-01T00:00:00Z",
|
||||
verificationDeadline: null,
|
||||
completedAt: null,
|
||||
result:
|
||||
status === "completed"
|
||||
? {
|
||||
snapshotId: "snapshot",
|
||||
publicationOutcome: "published",
|
||||
conflictedWithSnapshotId: null,
|
||||
}
|
||||
: null,
|
||||
});
|
||||
const api = {
|
||||
limits: async () => ({
|
||||
at: "read",
|
||||
syncAllowed: true,
|
||||
maxCompositions: 5,
|
||||
compositions: 0,
|
||||
push: V1_LIMITS,
|
||||
}),
|
||||
create: async (_space, id) => {
|
||||
calls.create++;
|
||||
composition = id;
|
||||
return { id };
|
||||
},
|
||||
begin: async (_space, id, request) => {
|
||||
composition = id;
|
||||
held = request;
|
||||
calls.begin.push(request);
|
||||
return { ...receipt(), created: true, present: [], needed: request.manifest.document.files };
|
||||
},
|
||||
uploadUrls: async (_space, _comp, _upload, files) =>
|
||||
files.map((file) => ({
|
||||
...file,
|
||||
url: `https://storage.test/${file.sha256}`,
|
||||
headers: {},
|
||||
expiresAt: "2099-01-01T00:00:00Z",
|
||||
parts: [],
|
||||
})),
|
||||
submit: async () => {
|
||||
calls.submit++;
|
||||
return receipt("completed");
|
||||
},
|
||||
status: async () => receipt("completed"),
|
||||
snapshot: async (_space, id, snapshot) => {
|
||||
calls.snapshot++;
|
||||
return {
|
||||
id: snapshot,
|
||||
compositionId: id,
|
||||
manifestSha256: held.manifest.sha256,
|
||||
manifest: held.manifest,
|
||||
publicationOutcome: "published",
|
||||
};
|
||||
},
|
||||
...overrides,
|
||||
};
|
||||
const fetchImpl = async (url, init) => {
|
||||
assert.equal(init.headers.authorization, undefined);
|
||||
const bytes = Buffer.from(await new Response(init.body).arrayBuffer());
|
||||
assert.equal(createHash("sha256").update(bytes).digest("hex"), url.split("/").at(-1));
|
||||
calls.puts.push(bytes);
|
||||
return new Response(null, { status: 200 });
|
||||
};
|
||||
return { api, calls, fetchImpl, receipt };
|
||||
}
|
||||
|
||||
for (const path of [".media/voice.mp3", ".hyperframes/studio-manual-edits.json"]) {
|
||||
test(`changing ${path} refuses source preparation and final attachment to the old render`, async (t) => {
|
||||
const f = await fixture(t);
|
||||
await writeFile(join(f.root, "index.html"), '<audio src=".media/voice.mp3"></audio>');
|
||||
await mkdir(dirname(join(f.root, path)), { recursive: true });
|
||||
await writeFile(join(f.root, path), "original");
|
||||
const rendered = await projectFingerprint(f.root);
|
||||
const content = await shareSourceFingerprint(f.root);
|
||||
const server = fakeServer();
|
||||
const source = shareSources({
|
||||
access: async () => signed,
|
||||
spaceId: "personal",
|
||||
...server,
|
||||
fingerprint: async (root) =>
|
||||
content === (await shareSourceFingerprint(root)) ? projectFingerprint(root) : null,
|
||||
});
|
||||
await writeFile(join(f.root, path), "changed");
|
||||
assert.equal(await projectFingerprint(f.root), rendered);
|
||||
assert.equal((await source.prepare(f.root, rendered)).reason, "source_render_changed");
|
||||
assert.equal(server.calls.begin.length, 0);
|
||||
assert.equal(await source.filesForExport(f.root, rendered), null);
|
||||
await writeFile(join(f.root, path), "original");
|
||||
assert.equal((await source.prepare(f.root, rendered)).at, "ready");
|
||||
assert.ok(await source.filesForExport(f.root, rendered));
|
||||
await writeFile(join(f.root, path), "changed after preparation");
|
||||
assert.equal(await source.filesForExport(f.root, rendered), null);
|
||||
});
|
||||
}
|
||||
|
||||
test("source upload verifies before attachment and a fresh adapter reuses the exact published snapshot", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const server = fakeServer();
|
||||
const options = { access: async () => signed, spaceId: "personal", ...server };
|
||||
const source = shareSources(options);
|
||||
assert.equal(await source.filesForExport(f.root, f.rendered), null);
|
||||
const result = await source.prepare(f.root, f.rendered);
|
||||
assert.equal(result.at, "ready");
|
||||
assert.equal(server.calls.puts.length, 1);
|
||||
assert.equal(server.calls.submit, 1);
|
||||
assert.deepEqual(await source.filesForExport(f.root, f.rendered), result.snapshot);
|
||||
assert.equal((await shareSources(options).prepare(f.root, f.rendered)).at, "ready");
|
||||
assert.equal(server.calls.begin.length, 1);
|
||||
assert.equal(server.calls.snapshot, 1);
|
||||
const binding = await f.store.read(f.key);
|
||||
assert.equal(binding.inflight, null);
|
||||
assert.equal(binding.base.snapshotId, "snapshot");
|
||||
});
|
||||
|
||||
test("changed assets invalidate prepared source even when the HTML render fingerprint stays unchanged", async (t) => {
|
||||
const f = await fixture(t, { "media.mp4": "first" });
|
||||
const source = shareSources({
|
||||
access: async () => signed,
|
||||
spaceId: "personal",
|
||||
...fakeServer(),
|
||||
fingerprint: async () => f.rendered,
|
||||
});
|
||||
assert.equal((await source.prepare(f.root, f.rendered)).at, "ready");
|
||||
await writeFile(join(f.root, "media.mp4"), "other");
|
||||
assert.equal(await source.filesForExport(f.root, f.rendered), null);
|
||||
});
|
||||
|
||||
test("eligibility and busy edits refuse before creating cloud compositions", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const server = fakeServer({ limits: async () => ({ syncAllowed: false }) });
|
||||
assert.deepEqual(
|
||||
await shareSources({ access: async () => signed, spaceId: "personal", ...server }).prepare(
|
||||
f.root,
|
||||
f.rendered,
|
||||
),
|
||||
{ at: "unavailable", reason: "source_plan_required" },
|
||||
);
|
||||
assert.equal(server.calls.create, 0);
|
||||
await mkdir(join(f.root, ".hyperframes"), { recursive: true });
|
||||
await writeFile(join(f.root, ".hyperframes/history-turn.json"), "{}");
|
||||
const next = fakeServer();
|
||||
assert.equal(
|
||||
(
|
||||
await shareSources({ access: async () => signed, spaceId: "personal", ...next }).prepare(
|
||||
f.root,
|
||||
f.rendered,
|
||||
)
|
||||
).reason,
|
||||
"source_edit_in_progress",
|
||||
);
|
||||
assert.equal(next.calls.create, 0);
|
||||
});
|
||||
|
||||
test("lost Begin response retries the durable idempotency key, without creating a second composition", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const server = fakeServer();
|
||||
const begin = server.api.begin;
|
||||
let fail = true;
|
||||
server.api.begin = async (...args) => {
|
||||
const answer = await begin(...args);
|
||||
if (fail) {
|
||||
fail = false;
|
||||
throw new Error("network lost");
|
||||
}
|
||||
return answer;
|
||||
};
|
||||
const opts = { access: async () => signed, spaceId: "personal", ...server };
|
||||
await assert.rejects(shareSources(opts).prepare(f.root, f.rendered), /network lost/);
|
||||
assert.equal((await shareSources(opts).prepare(f.root, f.rendered)).at, "ready");
|
||||
assert.equal(server.calls.begin[0].idempotencyKey, server.calls.begin[1].idempotencyKey);
|
||||
assert.deepEqual(server.calls.begin[0].manifest, server.calls.begin[1].manifest);
|
||||
});
|
||||
|
||||
test("confirmed expired upload clears its intent so a retry starts with a new key", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const server = fakeServer();
|
||||
const begin = server.api.begin;
|
||||
server.api.begin = async (...args) => {
|
||||
const answer = await begin(...args);
|
||||
return server.calls.begin.length === 1 ? { ...answer, status: "expired" } : answer;
|
||||
};
|
||||
const source = shareSources({ access: async () => signed, spaceId: "personal", ...server });
|
||||
|
||||
assert.equal((await source.prepare(f.root, f.rendered)).reason, "source_upload_expired");
|
||||
assert.equal((await f.store.read(f.key)).inflight, null);
|
||||
assert.equal((await source.prepare(f.root, f.rendered)).at, "ready");
|
||||
assert.notEqual(server.calls.begin[0].idempotencyKey, server.calls.begin[1].idempotencyKey);
|
||||
});
|
||||
|
||||
test("an unwritable pending upload is cancelled so unchanged files can retry with a new key", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const server = fakeServer();
|
||||
const grants = server.api.uploadUrls;
|
||||
server.api.uploadUrls = async () => {
|
||||
throw new SyncApiError("file-not-writable");
|
||||
};
|
||||
server.api.status = async () => server.receipt("pending");
|
||||
let cancelled = 0;
|
||||
server.api.cancel = async () => {
|
||||
cancelled++;
|
||||
return { ...server.receipt("failed"), errorCode: "cancelled" };
|
||||
};
|
||||
const source = shareSources({ access: async () => signed, spaceId: "personal", ...server });
|
||||
assert.equal((await source.prepare(f.root, f.rendered)).reason, "source_upload_not_writable");
|
||||
assert.equal(cancelled, 1);
|
||||
assert.equal((await f.store.read(f.key)).inflight, null);
|
||||
server.api.uploadUrls = grants;
|
||||
assert.equal((await source.prepare(f.root, f.rendered)).at, "ready");
|
||||
assert.notEqual(server.calls.begin[0].idempotencyKey, server.calls.begin[1].idempotencyKey);
|
||||
});
|
||||
|
||||
for (const status of ["verifying", "completed"]) {
|
||||
test(`unwritable recovery retains an upload that raced cancellation into ${status}`, async (t) => {
|
||||
const f = await fixture(t);
|
||||
const server = fakeServer();
|
||||
server.api.uploadUrls = async () => {
|
||||
throw new SyncApiError("file-not-writable");
|
||||
};
|
||||
let reads = 0;
|
||||
server.api.status = async () => {
|
||||
const receipt = server.receipt(++reads === 1 ? "pending" : status);
|
||||
if (receipt.result) receipt.result.publicationOutcome = "conflict_saved";
|
||||
return receipt;
|
||||
};
|
||||
server.api.cancel = async () => {
|
||||
throw new SyncApiError("upload-not-pending");
|
||||
};
|
||||
const source = shareSources({ access: async () => signed, spaceId: "personal", ...server });
|
||||
assert.equal(
|
||||
(await source.prepare(f.root, f.rendered)).reason,
|
||||
status === "verifying" ? "source_verification_pending" : "source_conflict",
|
||||
);
|
||||
assert.equal((await f.store.read(f.key)).inflight.uploadId, "upload");
|
||||
assert.equal(server.calls.begin.length, 1);
|
||||
});
|
||||
}
|
||||
|
||||
test("changed source cancels the previous pending upload before using a new key", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const server = fakeServer();
|
||||
const submit = server.api.submit;
|
||||
server.api.submit = async () => {
|
||||
throw new Error("lost connection");
|
||||
};
|
||||
server.api.status = async () => server.receipt("pending");
|
||||
let cancelled = 0;
|
||||
server.api.cancel = async () => {
|
||||
cancelled++;
|
||||
return { ...server.receipt("failed"), errorCode: "cancelled" };
|
||||
};
|
||||
const source = shareSources({ access: async () => signed, spaceId: "personal", ...server });
|
||||
|
||||
await assert.rejects(source.prepare(f.root, f.rendered), /lost connection/);
|
||||
await writeFile(join(f.root, "index.html"), "<html>updated</html>");
|
||||
server.api.submit = submit;
|
||||
assert.equal((await source.prepare(f.root, await projectFingerprint(f.root))).at, "ready");
|
||||
assert.equal(cancelled, 1);
|
||||
assert.notEqual(server.calls.begin[0].idempotencyKey, server.calls.begin[1].idempotencyKey);
|
||||
});
|
||||
|
||||
test("conflict and pending verification cannot be attached and keep recovery intent", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const server = fakeServer();
|
||||
server.api.submit = async () => ({
|
||||
...server.receipt("completed"),
|
||||
result: { snapshotId: "snapshot", publicationOutcome: "conflict_saved" },
|
||||
});
|
||||
const source = shareSources({ access: async () => signed, spaceId: "personal", ...server });
|
||||
assert.equal((await source.prepare(f.root, f.rendered)).reason, "source_conflict");
|
||||
assert.equal(await source.filesForExport(f.root, f.rendered), null);
|
||||
assert.ok((await f.store.read(f.key)).inflight);
|
||||
server.api.submit = async () => server.receipt("verifying");
|
||||
const pending = shareSources({
|
||||
access: async () => signed,
|
||||
spaceId: "personal",
|
||||
...server,
|
||||
verificationTimeoutMs: 0,
|
||||
});
|
||||
assert.equal((await pending.prepare(f.root, f.rendered)).reason, "source_verification_pending");
|
||||
});
|
||||
|
||||
test("Desktop-owned source is read-only and never re-shared into a different workspace", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const server = fakeServer();
|
||||
const capture = await captureProject(f.root);
|
||||
const binding = {
|
||||
...f.key,
|
||||
version: 1,
|
||||
compositionId: "desktop",
|
||||
ownerSpaceId: "team",
|
||||
base: { manifestSha256: capture.manifest.sha256, snapshotId: "desktop-snapshot" },
|
||||
inflight: null,
|
||||
};
|
||||
const opts = {
|
||||
access: async () => signed,
|
||||
spaceId: "personal",
|
||||
existingBinding: async () => binding,
|
||||
...server,
|
||||
};
|
||||
assert.equal(
|
||||
(await shareSources(opts).prepare(f.root, f.rendered)).reason,
|
||||
"source_other_workspace",
|
||||
);
|
||||
binding.ownerSpaceId = "personal";
|
||||
binding.sharedFrom = { reviewProjectId: "someone-elses-share" };
|
||||
assert.equal(
|
||||
(await shareSources(opts).prepare(f.root, f.rendered)).reason,
|
||||
"source_owned_by_someone_else",
|
||||
);
|
||||
assert.equal(server.calls.create, 0);
|
||||
});
|
||||
|
||||
test("binding reads reject copied folders, account changes, and symlink state files", async (t) => {
|
||||
const f = await fixture(t);
|
||||
await shareSources({ access: async () => signed, spaceId: "personal", ...fakeServer() }).prepare(
|
||||
f.root,
|
||||
f.rendered,
|
||||
);
|
||||
await assert.rejects(f.store.read({ ...f.key, account: "other" }), /another account/);
|
||||
const g = await fixture(t);
|
||||
const hash = createHash("sha256").update(signed.heygenApi).digest("hex").slice(0, 16);
|
||||
const name = `.hyperframes/share-sync.${hash}.json`;
|
||||
await mkdir(join(g.root, ".hyperframes"));
|
||||
await writeFile(join(g.root, name), await readFile(join(f.root, name)));
|
||||
await assert.rejects(g.store.read(g.key), /another account/);
|
||||
await rm(join(g.root, name));
|
||||
await symlink(join(f.root, name), join(g.root, name));
|
||||
await assert.rejects(g.store.read(g.key));
|
||||
});
|
||||
|
||||
test("HTTP adapter identifies CLI and sends credentials only to the API host", async () => {
|
||||
const requests = [];
|
||||
const api = sourceApi({
|
||||
access: async () => signed,
|
||||
fetchImpl: async (url, init) => {
|
||||
requests.push({ url, init });
|
||||
return Response.json({
|
||||
data: {
|
||||
upload_id: "up",
|
||||
composition_id: "comp",
|
||||
snapshot_id: "snap",
|
||||
status: "pending",
|
||||
manifest_sha256: "hash",
|
||||
upload_deadline: "2099-01-01T00:00:00Z",
|
||||
},
|
||||
});
|
||||
},
|
||||
});
|
||||
await api.begin("space", "comp", {
|
||||
idempotencyKey: "key",
|
||||
baseSnapshotId: null,
|
||||
manifest: { sha256: "hash", document: {} },
|
||||
});
|
||||
assert.equal(JSON.parse(requests[0].init.body).client_type, "cli");
|
||||
assert.equal(requests[0].init.headers.authorization, "Bearer test-token");
|
||||
assert.match(requests[0].url, /^https:\/\/api.test\/v3\//);
|
||||
});
|
||||
@@ -0,0 +1,245 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import {
|
||||
lstat,
|
||||
mkdir,
|
||||
readdir,
|
||||
mkdtemp,
|
||||
realpath,
|
||||
rename,
|
||||
rm,
|
||||
symlink,
|
||||
utimes,
|
||||
writeFile,
|
||||
} from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join } from "node:path";
|
||||
import { captureProject, hashListedFile, hashMemo } from "../src/sources/capture.mjs";
|
||||
|
||||
/** A project folder holding `files` (path to contents). @param {Record<string, string>} files */
|
||||
async function project(files) {
|
||||
const root = await mkdtemp(join(tmpdir(), "hf-sync-capture-"));
|
||||
for (const [path, text] of Object.entries(files)) {
|
||||
await mkdir(dirname(join(root, path)), { recursive: true });
|
||||
await writeFile(join(root, path), text);
|
||||
}
|
||||
return root;
|
||||
}
|
||||
|
||||
const sha = (/** @type {string} */ text) => createHash("sha256").update(text).digest("hex");
|
||||
|
||||
/** @param {Awaited<ReturnType<typeof captureProject>>} capture */
|
||||
function captured(capture) {
|
||||
assert.equal(capture.at, "captured", JSON.stringify(capture));
|
||||
return /** @type {Extract<typeof capture, { at: "captured" }>} */ (capture);
|
||||
}
|
||||
|
||||
test("a capture lists portable files with their hashes and leaves out what the policy excludes", async () => {
|
||||
const root = await project({
|
||||
"index.html": "<html></html>",
|
||||
"assets/logo.png": "png",
|
||||
".hyperframes/brand-kit.json": "{}",
|
||||
".hyperframes/comments.jsonl": "{}",
|
||||
".hyperframes/share.json": "{}",
|
||||
"node_modules/x/index.js": "x",
|
||||
".env": "SECRET=1",
|
||||
".DS_Store": "",
|
||||
"renders/out.mp4": "mp4",
|
||||
"Icon\r": "",
|
||||
});
|
||||
const { manifest, waiting, skippedLinks } = captured(await captureProject(root));
|
||||
assert.deepEqual(manifest.document.files, [
|
||||
{ path: ".hyperframes/brand-kit.json", sha256: sha("{}"), size: 2 },
|
||||
{ path: "assets/logo.png", sha256: sha("png"), size: 3 },
|
||||
{ path: "index.html", sha256: sha("<html></html>"), size: 13 },
|
||||
]);
|
||||
assert.match(manifest.sha256, /^[0-9a-f]{64}$/);
|
||||
assert.deepEqual([waiting, skippedLinks], [[], []]);
|
||||
});
|
||||
|
||||
test("a render output the CLI recorded outside renders/ stays out", async () => {
|
||||
const root = await project({
|
||||
"index.html": "x",
|
||||
"exports/cut.mp4": "mp4",
|
||||
"exports/keep.txt": "keep",
|
||||
"stills/a/b.png": "png",
|
||||
".hyperframes/render-outputs.json": JSON.stringify({
|
||||
version: 1,
|
||||
paths: ["exports/cut.mp4", "stills"],
|
||||
}),
|
||||
});
|
||||
const { manifest } = captured(await captureProject(root));
|
||||
assert.deepEqual(
|
||||
manifest.document.files.map((file) => file.path),
|
||||
["exports/keep.txt", "index.html"],
|
||||
);
|
||||
});
|
||||
|
||||
test("a link fails the capture, except one under assets/podcast/, which is skipped", async () => {
|
||||
const root = await project({ "index.html": "x", "outside.mov": "mov" });
|
||||
await mkdir(join(root, "assets/podcast"), { recursive: true });
|
||||
await symlink(join(root, "outside.mov"), join(root, "assets/podcast/cam.mov"));
|
||||
const skipped = captured(await captureProject(root));
|
||||
assert.deepEqual(skipped.skippedLinks, ["assets/podcast/cam.mov"]);
|
||||
await symlink(join(root, "outside.mov"), join(root, "assets/cam.mov"));
|
||||
assert.deepEqual(await captureProject(root), {
|
||||
at: "problems",
|
||||
problems: [{ path: "assets/cam.mov", problem: "symlink" }],
|
||||
});
|
||||
});
|
||||
|
||||
test("a name no client can store fails the capture with its code", async () => {
|
||||
const root = await project({ "index.html": "x", "scenes/con.html": "x", "a?.txt": "x" });
|
||||
const capture = await captureProject(root);
|
||||
assert.equal(capture.at, "problems");
|
||||
assert.deepEqual(
|
||||
capture.at === "problems" ? capture.problems.map((p) => [p.path, p.code]).sort() : [],
|
||||
[
|
||||
["a?.txt", "path_invalid_character"],
|
||||
["scenes/con.html", "reserved_name"],
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test("a project without index.html cannot be captured", async () => {
|
||||
const root = await project({ "scene.html": "x" });
|
||||
assert.deepEqual(await captureProject(root), {
|
||||
at: "problems",
|
||||
problems: [{ path: "index.html", problem: "no-entrypoint" }],
|
||||
});
|
||||
});
|
||||
|
||||
test("over a limit, the capture says which and names the biggest files before hashing any", async () => {
|
||||
const root = await project({ "index.html": "x", "big.bin": "0123456789", "mid.bin": "01234" });
|
||||
const limits = { maxFiles: 10, maxFileBytes: 6, maxProjectBytes: 100, maxManifestBytes: 10_000 };
|
||||
const capture = await captureProject(root, { limits });
|
||||
assert.equal(capture.at, "too-big");
|
||||
if (capture.at !== "too-big") return;
|
||||
assert.deepEqual(
|
||||
{ limit: capture.over.limit, actual: capture.over.actual, maximum: capture.over.maximum },
|
||||
{ limit: "file_size", actual: 10, maximum: 6 },
|
||||
);
|
||||
assert.deepEqual(
|
||||
capture.over.biggest.map((file) => file.path),
|
||||
["big.bin", "mid.bin", "index.html"],
|
||||
);
|
||||
});
|
||||
|
||||
test("a media reservation is waited on and left out, and an open agent turn is reported", async () => {
|
||||
const root = await project({
|
||||
"index.html": "x",
|
||||
".media/images/a.hf0a1b2c.tmp": "",
|
||||
".hyperframes/history-turn.json": "{}",
|
||||
});
|
||||
const capture = captured(await captureProject(root));
|
||||
assert.deepEqual(capture.waiting, [".media/images/a.hf0a1b2c.tmp"]);
|
||||
assert.deepEqual(capture.settled, { agentTurnOpen: true, projectLocked: false });
|
||||
});
|
||||
|
||||
test("a project lock held by a running process is reported; a dead holder's is not", async () => {
|
||||
const root = await project({ "index.html": "x" });
|
||||
const lock = join(root, ".hyperframes/project.lock");
|
||||
await mkdir(dirname(lock), { recursive: true });
|
||||
await writeFile(lock, JSON.stringify({ pid: process.pid, startedAt: 1, command: "render" }));
|
||||
assert.equal(captured(await captureProject(root)).settled.projectLocked, true);
|
||||
await writeFile(lock, JSON.stringify({ pid: 2 ** 30, startedAt: 1, command: "render" }));
|
||||
assert.equal(captured(await captureProject(root)).settled.projectLocked, false);
|
||||
});
|
||||
|
||||
test("a file over 4 MiB is hashed as a bounded stream, to the same digest", async () => {
|
||||
const big = "x".repeat(5 * 1024 * 1024);
|
||||
const root = await project({ "index.html": "x", "assets/big.bin": big });
|
||||
const { manifest } = captured(await captureProject(root));
|
||||
assert.deepEqual(manifest.document.files[0], {
|
||||
path: "assets/big.bin",
|
||||
sha256: sha(big),
|
||||
size: big.length,
|
||||
});
|
||||
});
|
||||
|
||||
test("a kept hash is reused only while the file's size and time are unchanged", async () => {
|
||||
const root = await project({ "index.html": "one" });
|
||||
const memo = hashMemo();
|
||||
const first = captured(await captureProject(root, { memo }));
|
||||
await writeFile(join(root, "index.html"), "two");
|
||||
await utimes(join(root, "index.html"), new Date(2030, 0, 1), new Date(2030, 0, 1));
|
||||
const second = captured(await captureProject(root, { memo }));
|
||||
assert.equal(first.manifest.document.files[0]?.sha256, sha("one"));
|
||||
assert.equal(second.manifest.document.files[0]?.sha256, sha("two"));
|
||||
});
|
||||
|
||||
test("a same-size edit inside the file time's granularity is hashed again, never read from a kept hash", async () => {
|
||||
const root = await project({ "index.html": "one" });
|
||||
const file = join(root, "index.html");
|
||||
const tick = new Date();
|
||||
await utimes(file, tick, tick);
|
||||
const memo = hashMemo();
|
||||
const first = captured(await captureProject(root, { memo }));
|
||||
await writeFile(file, "two");
|
||||
await utimes(file, tick, tick);
|
||||
const second = captured(await captureProject(root, { memo }));
|
||||
assert.equal(first.manifest.document.files[0]?.sha256, sha("one"));
|
||||
assert.equal(second.manifest.document.files[0]?.sha256, sha("two"));
|
||||
});
|
||||
|
||||
test("a settled file's hash is kept, so a check after a focus reads no bytes", async () => {
|
||||
const root = await project({ "index.html": "one" });
|
||||
const memo = hashMemo();
|
||||
captured(await captureProject(root, { memo }));
|
||||
await writeFile(join(root, "index.html"), "two");
|
||||
const settled = new Date(2020, 0, 1);
|
||||
await utimes(join(root, "index.html"), settled, settled);
|
||||
captured(await captureProject(root, { memo }));
|
||||
await writeFile(join(root, "index.html"), "six");
|
||||
await utimes(join(root, "index.html"), settled, settled);
|
||||
const kept = captured(await captureProject(root, { memo }));
|
||||
assert.equal(kept.manifest.document.files[0]?.sha256, sha("two"));
|
||||
});
|
||||
|
||||
/** A file as the walk lists it, read from disk now. @param {string} root @param {string} path */
|
||||
async function listed(root, path) {
|
||||
const full = join(root, path);
|
||||
const { size, mtimeMs, ino, dev } = await lstat(full);
|
||||
return { path, full, size, mtimeMs, ino, dev };
|
||||
}
|
||||
|
||||
test("a file swapped for a link of the same size after the walk is refused, never followed", async () => {
|
||||
const root = await realpath(await mkdtemp(join(tmpdir(), "hf-sync-swap-")));
|
||||
const outside = await realpath(await mkdtemp(join(tmpdir(), "hf-sync-outside-")));
|
||||
await writeFile(join(root, "a.txt"), "inside");
|
||||
await writeFile(join(outside, "secret"), "SECRET");
|
||||
// The same size and time as the file listed: only the file's identity tells them apart.
|
||||
await utimes(join(root, "a.txt"), 1_700_000_000, 1_700_000_000);
|
||||
await utimes(join(outside, "secret"), 1_700_000_000, 1_700_000_000);
|
||||
const file = await listed(root, "a.txt");
|
||||
assert.match(String(await hashListedFile(root, file)), /^[0-9a-f]{64}$/);
|
||||
await rm(join(root, "a.txt"));
|
||||
await symlink(join(outside, "secret"), join(root, "a.txt"));
|
||||
assert.equal(await hashListedFile(root, file), null);
|
||||
});
|
||||
|
||||
test("a folder swapped for a link to elsewhere after the walk is refused, even holding the same file", async () => {
|
||||
const root = await realpath(await mkdtemp(join(tmpdir(), "hf-sync-swap-")));
|
||||
const outside = await realpath(await mkdtemp(join(tmpdir(), "hf-sync-outside-")));
|
||||
await mkdir(join(root, "assets"));
|
||||
await writeFile(join(root, "assets", "logo.txt"), "inside");
|
||||
const file = await listed(root, "assets/logo.txt");
|
||||
await rename(join(root, "assets"), join(outside, "assets"));
|
||||
await symlink(join(outside, "assets"), join(root, "assets"));
|
||||
assert.equal(await hashListedFile(root, file), null, "the moved folder's own file was followed");
|
||||
});
|
||||
|
||||
test("a file whose folder cannot be resolved once opened is refused, and its handle is closed", async () => {
|
||||
const root = await realpath(await mkdtemp(join(tmpdir(), "hf-sync-close-")));
|
||||
await mkdir(join(root, "assets"));
|
||||
await writeFile(join(root, "assets", "logo.txt"), "inside");
|
||||
const file = await listed(root, "assets/logo.txt");
|
||||
const openFiles = async () => (await readdir("/dev/fd")).length;
|
||||
const before = await openFiles();
|
||||
const gone = async () => {
|
||||
throw Object.assign(new Error("ENOENT: the folder was renamed"), { code: "ENOENT" });
|
||||
};
|
||||
await assert.rejects(hashListedFile(root, file, gone), /ENOENT/);
|
||||
assert.equal(await openFiles(), before, "the opened file was left open");
|
||||
});
|
||||
@@ -0,0 +1,129 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import {
|
||||
ManifestInvalid,
|
||||
parseCanonicalManifest,
|
||||
parseManifest,
|
||||
} from "../src/sources/manifest.mjs";
|
||||
import { overLimit, V1_LIMITS } from "../src/sources/limits.mjs";
|
||||
import { excludedFolder, skippedSymlink } from "../src/sources/pathPolicy.mjs";
|
||||
|
||||
/** Reference vectors for the source manifest format. */
|
||||
const vectors = JSON.parse(
|
||||
readFileSync(
|
||||
new URL("./fixtures/compositionSync/manifest-vectors.json", import.meta.url),
|
||||
"utf8",
|
||||
),
|
||||
);
|
||||
|
||||
/** @param {() => unknown} run */
|
||||
function refusal(run) {
|
||||
try {
|
||||
run();
|
||||
} catch (error) {
|
||||
if (error instanceof ManifestInvalid) return error;
|
||||
throw error;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
test("the vectors are portable-source-v4's, and there are vectors of every kind", () => {
|
||||
assert.equal(vectors.policy_version, "portable-source-v4");
|
||||
assert.deepEqual(
|
||||
[vectors.valid.length, vectors.invalid.length, vectors.limits.length, vectors.stored.length],
|
||||
[11, 147, 5, 8],
|
||||
);
|
||||
});
|
||||
|
||||
test("every valid vector builds the server's canonical bytes and manifest_sha256", () => {
|
||||
for (const vector of vectors.valid) {
|
||||
const manifest = parseManifest(vector.manifest);
|
||||
assert.equal(manifest.canonical.toString("utf8"), vector.canonical, vector.name);
|
||||
assert.equal(manifest.sha256, vector.manifest_sha256, vector.name);
|
||||
assert.equal(
|
||||
createHash("sha256").update(Buffer.from(vector.canonical, "utf8")).digest("hex"),
|
||||
vector.manifest_sha256,
|
||||
);
|
||||
assert.equal(manifest.document.files.length, vector.file_count, vector.name);
|
||||
const bytes = manifest.document.files.reduce((sum, file) => sum + file.size, 0);
|
||||
assert.equal(bytes, vector.project_bytes, vector.name);
|
||||
}
|
||||
});
|
||||
|
||||
test("every invalid vector is refused with the server's code and rule", () => {
|
||||
for (const vector of vectors.invalid) {
|
||||
const error = refusal(() => parseManifest(vector.manifest));
|
||||
assert.ok(error, `${vector.name} was accepted`);
|
||||
assert.equal(error.code, vector.error.code, vector.name);
|
||||
if (vector.error.rule) assert.equal(error.violations[0]?.rule, vector.error.rule, vector.name);
|
||||
}
|
||||
});
|
||||
|
||||
test("every limits vector breaks the server's first limit, or none", () => {
|
||||
for (const vector of vectors.limits) {
|
||||
const manifest = parseManifest(vector.manifest);
|
||||
const limits = {
|
||||
maxFiles: vector.limits.max_files,
|
||||
maxFileBytes: vector.limits.max_file_bytes,
|
||||
maxProjectBytes: vector.limits.max_project_bytes,
|
||||
maxManifestBytes: vector.limits.max_manifest_bytes,
|
||||
};
|
||||
const over = overLimit(manifest.document.files, manifest.canonical.length, limits);
|
||||
assert.equal(over?.limit ?? null, vector.limit, vector.name);
|
||||
}
|
||||
});
|
||||
|
||||
test("every stored vector reads only canonical bytes", () => {
|
||||
for (const vector of vectors.stored) {
|
||||
const body = Buffer.from(vector.body, "base64");
|
||||
const error = refusal(() => parseCanonicalManifest(body));
|
||||
assert.equal(error?.code ?? null, vector.error?.code ?? null, vector.name);
|
||||
}
|
||||
});
|
||||
|
||||
test("V1 limits are 10,000 files, 5 GiB a file, 10 GiB a project and an 8 MiB manifest", () => {
|
||||
assert.deepEqual(V1_LIMITS, {
|
||||
maxFiles: 10000,
|
||||
maxFileBytes: 5368709120,
|
||||
maxProjectBytes: 10737418240,
|
||||
maxManifestBytes: 8388608,
|
||||
});
|
||||
});
|
||||
|
||||
test("a project over a limit names its biggest files first", () => {
|
||||
const files = [
|
||||
{ path: "index.html", size: 10 },
|
||||
{ path: "assets/b.mp4", size: 6 * 1024 ** 3 },
|
||||
{ path: "assets/a.mp4", size: 7 * 1024 ** 3 },
|
||||
];
|
||||
const over = overLimit(files, 300);
|
||||
assert.equal(over?.limit, "file_size");
|
||||
assert.equal(over?.actual, 7 * 1024 ** 3);
|
||||
assert.deepEqual(
|
||||
over?.biggest.map((file) => file.path),
|
||||
["assets/a.mp4", "assets/b.mp4", "index.html"],
|
||||
);
|
||||
});
|
||||
|
||||
test("a capture skips only folders no included file can be under", () => {
|
||||
assert.equal(excludedFolder("node_modules"), true);
|
||||
assert.equal(excludedFolder("assets/node_modules"), true);
|
||||
assert.equal(excludedFolder(".git"), true);
|
||||
assert.equal(excludedFolder(".claude"), true);
|
||||
assert.equal(excludedFolder("dist"), true);
|
||||
assert.equal(excludedFolder("assets/dist"), false);
|
||||
assert.equal(excludedFolder("renders"), true);
|
||||
assert.equal(excludedFolder(".hyperframes"), false);
|
||||
assert.equal(excludedFolder(".hyperframes/prepared-assets"), false);
|
||||
assert.equal(excludedFolder("assets"), false);
|
||||
assert.equal(excludedFolder("assets/renders"), false);
|
||||
});
|
||||
|
||||
test("only links under assets/podcast/, in any spelling, are skipped", () => {
|
||||
assert.equal(skippedSymlink("assets/podcast/cam-a.mov"), true);
|
||||
assert.equal(skippedSymlink("Assets/Podcast/raw/cam-a.mov"), true);
|
||||
assert.equal(skippedSymlink("assets/cam-a.mov"), false);
|
||||
assert.equal(skippedSymlink("other/assets/podcast/cam-a.mov"), false);
|
||||
});
|
||||
@@ -0,0 +1,271 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, open, rm, realpath, symlink, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { createHash } from "node:crypto";
|
||||
import { uploadNeeded } from "../src/sources/uploadFiles.mjs";
|
||||
import { partDigests, putRange } from "../src/sources/transfer.mjs";
|
||||
import fsPromises from "node:fs/promises";
|
||||
import { syncBuiltinESMExports } from "node:module";
|
||||
import { bindShareHost } from "../src/host.mjs";
|
||||
|
||||
bindShareHost({ reviewUploadOrigins: ["https://store.test"] });
|
||||
|
||||
async function folder(t) {
|
||||
const root = await realpath(await mkdtemp(join(tmpdir(), "hf-source-upload-")));
|
||||
t.after(() => rm(root, { recursive: true, force: true }));
|
||||
return root;
|
||||
}
|
||||
const signal = () => new AbortController().signal;
|
||||
|
||||
test("multipart retries retain the pinned upload and skip completed parts after restart", async (t) => {
|
||||
const root = await folder(t);
|
||||
const file = join(root, "media.mp4");
|
||||
const size = 65 * 1024 * 1024;
|
||||
const handle = await open(file, "w");
|
||||
await handle.truncate(size);
|
||||
await handle.close();
|
||||
const digests = await partDigests(file, size, 64 * 1024 * 1024, signal());
|
||||
const inflight = { sent: [], multipart: {} };
|
||||
const requests = [];
|
||||
let fail = true;
|
||||
let persisted = 0;
|
||||
const job = {
|
||||
api: {
|
||||
uploadUrls: async (_space, _comp, _upload, files) => {
|
||||
requests.push(...files);
|
||||
return files.map((item) => ({
|
||||
sha256: item.sha256,
|
||||
multipartUploadId: "pinned",
|
||||
expiresAt: "2099-01-01T00:00:00Z",
|
||||
parts: item.parts.map((part) => ({
|
||||
...part,
|
||||
url: `https://store.test/${part.partNumber}`,
|
||||
headers: { checksum: part.sha256 },
|
||||
})),
|
||||
}));
|
||||
},
|
||||
},
|
||||
space: "space",
|
||||
composition: "comp",
|
||||
uploadId: "upload",
|
||||
inflight,
|
||||
local: new Map([[digests.whole, { path: "media.mp4", full: file }]]),
|
||||
needed: [{ sha256: digests.whole, size }],
|
||||
fetchImpl: async (url, init) => {
|
||||
if (url.endsWith("/2") && fail) {
|
||||
await init.body.cancel();
|
||||
return new Response(null, { status: 400 });
|
||||
}
|
||||
const hash = createHash("sha256");
|
||||
for await (const chunk of init.body) hash.update(chunk);
|
||||
assert.equal(hash.digest("hex"), init.headers.checksum);
|
||||
return new Response(null, { status: 200 });
|
||||
},
|
||||
signal: signal(),
|
||||
wait: async () => {},
|
||||
now: Date.now,
|
||||
persist: () => persisted++,
|
||||
onBytes: () => {},
|
||||
onFile: () => {},
|
||||
};
|
||||
await assert.rejects(uploadNeeded(job), /refused/);
|
||||
assert.deepEqual(inflight.multipart[digests.whole].partsDone, [1]);
|
||||
fail = false;
|
||||
await uploadNeeded(job);
|
||||
assert.equal(requests[1].multipartUploadId, "pinned");
|
||||
assert.deepEqual(
|
||||
requests[1].parts.map((p) => p.partNumber),
|
||||
[2],
|
||||
);
|
||||
assert.deepEqual(inflight.sent, [digests.whole]);
|
||||
assert.ok(persisted >= 4);
|
||||
});
|
||||
|
||||
test("signed URL expiry refreshes the grant without changing the uploaded content", async (t) => {
|
||||
const root = await folder(t);
|
||||
const file = join(root, "index.html");
|
||||
await writeFile(file, "source");
|
||||
const digest = createHash("sha256").update("source").digest("hex");
|
||||
let grants = 0;
|
||||
let puts = 0;
|
||||
await uploadNeeded({
|
||||
api: {
|
||||
uploadUrls: async () => {
|
||||
grants++;
|
||||
return [
|
||||
{
|
||||
sha256: digest,
|
||||
url: "https://store.test/file",
|
||||
headers: {},
|
||||
expiresAt: "2099-01-01T00:00:00Z",
|
||||
},
|
||||
];
|
||||
},
|
||||
},
|
||||
space: "s",
|
||||
composition: "c",
|
||||
uploadId: "u",
|
||||
inflight: { sent: [] },
|
||||
local: new Map([[digest, { path: "index.html", full: file }]]),
|
||||
needed: [{ sha256: digest, size: 6 }],
|
||||
fetchImpl: async (_url, init) => {
|
||||
puts++;
|
||||
await new Response(init.body).arrayBuffer();
|
||||
return new Response(null, { status: puts === 1 ? 403 : 200 });
|
||||
},
|
||||
signal: signal(),
|
||||
wait: async () => {},
|
||||
now: Date.now,
|
||||
persist: () => {},
|
||||
onBytes: () => {},
|
||||
onFile: () => {},
|
||||
});
|
||||
assert.equal(grants, 2);
|
||||
assert.equal(puts, 2);
|
||||
});
|
||||
|
||||
test("storage PUT refuses a file swapped for a symlink before reading bytes", async (t) => {
|
||||
const root = await folder(t);
|
||||
const target = join(root, "private.txt");
|
||||
const file = join(root, "asset.txt");
|
||||
await writeFile(target, "secret");
|
||||
await symlink(target, file);
|
||||
let sent = false;
|
||||
await assert.rejects(
|
||||
putRange({
|
||||
file,
|
||||
path: "asset.txt",
|
||||
start: 0,
|
||||
length: 6,
|
||||
sha256: "wrong",
|
||||
signal: signal(),
|
||||
url: "https://store.test",
|
||||
headers: {},
|
||||
fetchImpl: async () => {
|
||||
sent = true;
|
||||
return new Response();
|
||||
},
|
||||
}),
|
||||
);
|
||||
assert.equal(sent, false);
|
||||
});
|
||||
|
||||
test("source PUT refuses untrusted or redirected targets before it opens a file", async () => {
|
||||
for (const url of [
|
||||
"https://untrusted.test/source",
|
||||
"http://127.0.0.1:18080/source",
|
||||
"https://store.test.untrusted.test/source",
|
||||
"https://user:secret@store.test/source",
|
||||
]) {
|
||||
await assert.rejects(
|
||||
putRange({
|
||||
file: "/missing-source",
|
||||
path: "index.html",
|
||||
start: 0,
|
||||
length: 6,
|
||||
sha256: "unused",
|
||||
signal: signal(),
|
||||
url,
|
||||
headers: {},
|
||||
fetchImpl: async () => assert.fail("Must not send bytes"),
|
||||
}),
|
||||
{ kind: "refused" },
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
async function sourcePut(t, overrides = {}) {
|
||||
const root = await folder(t);
|
||||
const file = join(root, "index.html");
|
||||
await writeFile(file, "source");
|
||||
return {
|
||||
file,
|
||||
path: "index.html",
|
||||
start: 0,
|
||||
length: 6,
|
||||
sha256: createHash("sha256").update("source").digest("hex"),
|
||||
signal: signal(),
|
||||
url: "https://store.test/file",
|
||||
headers: {},
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
for (const consume of [false, true]) {
|
||||
test(`source PUT times out when ${consume ? "the storage response" : "body consumption"} stalls`, async (t) => {
|
||||
let requestSignal;
|
||||
const put = await sourcePut(t, {
|
||||
stallMs: 20,
|
||||
signal: AbortSignal.timeout(2_000),
|
||||
fetchImpl: async (_url, init) => {
|
||||
requestSignal = init.signal;
|
||||
if (consume) await new Response(init.body).arrayBuffer();
|
||||
init.signal.throwIfAborted();
|
||||
return new Promise((_resolve, reject) => {
|
||||
init.signal.addEventListener("abort", () => reject(init.signal.reason), { once: true });
|
||||
});
|
||||
},
|
||||
});
|
||||
await assert.rejects(putRange(put), { kind: "transient" });
|
||||
assert.equal(requestSignal.aborted, true);
|
||||
});
|
||||
}
|
||||
|
||||
test("source PUT preserves caller cancellation instead of retrying it", async (t) => {
|
||||
const controller = new AbortController();
|
||||
const cancelled = new DOMException("Sharing cancelled", "AbortError");
|
||||
await assert.rejects(
|
||||
putRange(
|
||||
await sourcePut(t, {
|
||||
signal: controller.signal,
|
||||
fetchImpl: async (_url, init) => {
|
||||
assert.equal(init.redirect, "error");
|
||||
await new Response(init.body).arrayBuffer();
|
||||
controller.abort(cancelled);
|
||||
throw init.signal.reason;
|
||||
},
|
||||
}),
|
||||
),
|
||||
(error) => error === cancelled,
|
||||
);
|
||||
});
|
||||
|
||||
test("successful source PUT clears its stall timer", async (t) => {
|
||||
let requestSignal;
|
||||
await putRange(
|
||||
await sourcePut(t, {
|
||||
stallMs: 200,
|
||||
fetchImpl: async (_url, init) => {
|
||||
requestSignal = init.signal;
|
||||
await new Response(init.body).arrayBuffer();
|
||||
return new Response(null, { status: 200 });
|
||||
},
|
||||
}),
|
||||
);
|
||||
await new Promise((resolve) => setTimeout(resolve, 250));
|
||||
assert.equal(requestSignal.aborted, false);
|
||||
});
|
||||
|
||||
test("source PUT closes its file when resolving its parent fails", async (t) => {
|
||||
const put = await sourcePut(t, { fetchImpl: async () => assert.fail("Must not send bytes") });
|
||||
const originalOpen = fsPromises.open;
|
||||
let opened;
|
||||
t.mock.method(fsPromises, "open", async (...args) => {
|
||||
opened = await originalOpen(...args);
|
||||
return opened;
|
||||
});
|
||||
t.mock.method(fsPromises, "realpath", async () => {
|
||||
throw Object.assign(new Error("Parent directory disappeared"), { code: "ENOENT" });
|
||||
});
|
||||
syncBuiltinESMExports();
|
||||
try {
|
||||
await assert.rejects(putRange(put), { code: "ENOENT" });
|
||||
assert.equal(opened.fd, -1, "source file handle was left open");
|
||||
} finally {
|
||||
await opened?.close();
|
||||
t.mock.restoreAll();
|
||||
syncBuiltinESMExports();
|
||||
}
|
||||
});
|
||||
@@ -48,8 +48,8 @@
|
||||
"unpacked": 825937
|
||||
},
|
||||
"@hyperframes/share": {
|
||||
"packed": 60820,
|
||||
"unpacked": 240537
|
||||
"packed": 140130,
|
||||
"unpacked": 542222
|
||||
},
|
||||
"@hyperframes/studio": {
|
||||
"packed": 5998328,
|
||||
|
||||
Reference in New Issue
Block a user