feat(share): attach editable project sources to reviews (#7137)

This commit is contained in:
terencecho
2026-11-09 16:10:31 -08:00
committed by Terence Cho
parent b72970cb4a
commit 9cc2bcdfa2
45 changed files with 14993 additions and 63 deletions
+91
View File
@@ -0,0 +1,91 @@
---
title: "Editable source in a share"
description: "Choose whether a CLI review includes the project files used by its rendered video."
---
`hyperframes share` can attach editable project files to the same review page
used by Desktop. Source is available when the workspace's sharing and
Composition Sync permissions allow it. The recipient's access still depends on
the share's settings and their plan.
```bash
hyperframes share --public # Video and source when eligible
hyperframes share --update # Update the existing review
hyperframes share --update --video-only # Explicitly omit editable source
```
The five-free-project sharing limit is unchanged. A workspace that cannot use
Composition Sync can still share its rendered video. The command reports
`source.at: "video_only"` and a reason in JSON; this is a successful video share.
A source upload does not grant anyone editing access by itself. Manage access
through `hyperframes share --manage`.
## What travels
Source uses the same portable project policy as Desktop: authored HTML, scripts,
styles, assets, and supported Studio edit metadata. Known credential filenames
(such as `.env`, `.npmrc`, and `*.pem`), dependency directories, chat
history, local sharing state, and recorded render outputs stay out.
These are filename rules, not a secret scanner. `.gitignore` and
`.hyperframesignore` do not control source capture. Other regular files can be
included even if Git ignores them. Keep private material outside the project,
or use `--video-only` when you do not want to upload project files. Unsupported symlinks and invalid file names stop source
preparation rather than silently producing an incomplete editable project.
The upload uses the workspace's file-count, per-file, project-size and manifest
limits. Large media uploads in parts. Only a verified published snapshot that
still matches the rendered project can be attached to the video version. The CLI
checks the contents of rendered assets, including changes that preserve file
sizes and timestamps. Deployments must include the exact source storage origins
in `HYPERFRAMES_REVIEW_UPLOAD_ORIGINS` alongside the video storage origins.
## Interrupted uploads and unavailable source
Retry the same command after a network interruption. Its local record preserves
the composition identity, upload idempotency key and completed file parts.
A storage upload that makes no progress for 60 seconds is stopped and retried
within the retry limit. The source record lives at `.hyperframes/share-sync.<backend-hash>.json` and
contains no sign-in token. Do not copy that record into another project or edit
its account, workspace or path: mismatches stop sharing to avoid overwriting
another project's source.
If source changed, conflicts with another snapshot, or remains under verification,
the command explains why it stopped. Retry after resolving the problem, or pass
`--video-only` to share just the rendered video. An unexpected source upload
failure never silently turns an eligible source share into video-only. If the
server confirms an upload expired or failed, a retry starts a new upload. If
the files changed while an earlier upload was still pending, the CLI cancels
that upload before starting a new one. It waits for uploads already under
verification and preserves conflicts for explicit resolution.
A video-only retry does not resume an interrupted video upload that already
includes source. Check its status before retrying. To publish the same render as
a new video-only version after the earlier upload finishes, use
`hyperframes share --update --video-only`. Previous versions retain their files.
`--status` reads the review without uploading anything. Its source state is
`not_checked`; it does not promise that local files match the shared version.
The upload result uses `ready` only after source verification and attachment
preparation; video readiness remains the separate top-level `ready` field.
## Integrating another local host
`@hyperframes/share` exports `shareSources` and
`createProjectShareSourceStore`. Call `prepare(projectPath, renderedFingerprint)`
before `addVersion`, then pass `filesForExport` and `requireSource: true` for a
ready source result. `requireSource` prevents a later mismatch from degrading to
a video-only version. Pass `omitSource: true` for a video-only operation that
must not reuse a source-bearing version or pending upload.
A host may inject its existing binding store, or supply a read-only
`existingBinding` resolver for Desktop bindings. Exact snapshots can be reused
only in the same account and owner workspace, without a pending sync operation.
Unsynced changes in a Desktop-owned binding require a Desktop sync first. The
standalone CLI keeps its own project-local sync record; it does not discover or
modify Desktop's application data directory.
## Related topics
- [CLI reference](/packages/cli#share) — audiences, updates, and processing status.
- [CLI guide](/developers/cli) — choose the command for your task.
+1
View File
@@ -196,6 +196,7 @@
"pages": [
"developers/cli",
"packages/cli",
"developers/share-source",
"developers/catalog-search",
"developers/harness-tool-context",
"packages/lint"
+8 -1
View File
@@ -38,6 +38,10 @@ export default defineCommand({
type: "boolean",
description: "Render and add a new video version to the existing share",
},
"video-only": {
type: "boolean",
description: "Share only the rendered video, without editable source",
},
status: { type: "boolean", description: "Read status without rendering or uploading" },
manage: { type: "boolean", description: "Open this share's web management page" },
space: { type: "string", description: "Owning workspace for a new share" },
@@ -87,6 +91,7 @@ interface ShareArgs {
json?: boolean;
"wait-seconds"?: string;
space?: string;
"video-only"?: boolean;
}
function parseInput(args: ShareArgs): ShareAction {
const waitSeconds = Number(args["wait-seconds"]);
@@ -97,7 +102,7 @@ function parseInput(args: ShareArgs): ShareAction {
const audience = requestedAudience(args);
const action = args.status ? "status" : args.manage ? "manage" : args.update ? "update" : "share";
return { action, audience, space: args.space, waitSeconds };
return { action, audience, space: args.space, waitSeconds, videoOnly: args["video-only"] };
}
function requestedAudience(args: ShareArgs) {
@@ -137,6 +142,8 @@ async function chooseAudience(input: ShareAction, args: ShareArgs, service: Shar
}
async function presentResult(result: ShareState, input: ShareAction, json: boolean) {
if (!json && result.source.at === "video_only")
console.error(`Sharing video only (${result.source.reason}).`);
if (json) console.log(JSON.stringify({ ok: true, ...result }));
else if (input.action === "manage") {
console.log(result.manageUrl);
+3
View File
@@ -1,4 +1,5 @@
import { ReviewFailed, reviewSentence } from "@hyperframes/share";
import { ShareSourceError } from "./sources.js";
import { isAuthError } from "../auth/errors.js";
import { CliUsageError } from "../utils/commandResult.js";
@@ -20,6 +21,8 @@ const HINTS: Record<string, string> = {
};
export function shareError(error: unknown): { code: string; message: string } {
if (error instanceof CliUsageError) return { code: error.code, message: error.message };
if (error instanceof ShareSourceError)
return { code: "source_unavailable", message: error.message };
if (isAuthError(error))
return {
code: error.code.toLowerCase(),
@@ -5,7 +5,10 @@ import { join } from "node:path";
import type { RenderExecutionDependencies } from "../commands/render/execute.js";
import type { RenderOptions } from "../commands/render.js";
const mocks = vi.hoisted(() => ({ render: vi.fn(), plan: vi.fn(), execute: vi.fn() }));
vi.mock("@hyperframes/share", () => ({ projectFingerprint: async () => "unchanged" }));
vi.mock("@hyperframes/share", () => ({
projectFingerprint: async () => "unchanged",
shareSourceFingerprint: async () => "unchanged-content",
}));
vi.mock("../commands/render/plan.js", () => ({ createRenderPlan: mocks.plan }));
vi.mock("../commands/render/execute.js", () => ({ executeRenderPlan: mocks.execute }));
vi.mock("../commands/render.js", () => ({
+33 -3
View File
@@ -1,8 +1,16 @@
import { mkdtempSync, mkdirSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { latestShareRender, prepareShareRender } from "./render.js";
import { dirname, join } from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { projectFingerprint } from "@hyperframes/share";
import { latestShareRender, prepareShareRender, shareContentFingerprint } from "./render.js";
const rendering = vi.hoisted(() => ({ execute: vi.fn() }));
vi.mock("../commands/render/plan.js", () => ({ createRenderPlan: (plan: unknown) => plan }));
vi.mock("../commands/render/execute.js", () => ({ executeRenderPlan: rendering.execute }));
vi.mock("../commands/render.js", () => ({
renderLocal: vi.fn(),
checkRenderResolutionPreflight: vi.fn(),
}));
const dirs: string[] = [];
function folder() {
const dir = realpathSync(mkdtempSync(join(tmpdir(), "hf-share-")));
@@ -41,4 +49,26 @@ describe("local sharing render boundary", () => {
);
expect(latestShareRender(dir)).toEqual({ id, path: output, fingerprint: "f" });
});
it.each([".media/voice.mp3", ".hyperframes/studio-manual-edits.json"])(
"invalidates a cached render when %s changes",
async (path) => {
const dir = folder();
writeFileSync(join(dir, "index.html"), '<audio src=".media/voice.mp3"></audio>');
mkdirSync(dirname(join(dir, path)), { recursive: true });
writeFileSync(join(dir, path), "original");
rendering.execute.mockImplementation(async ({ output }: { output: string }) => {
writeFileSync(output, "rendered video");
});
await prepareShareRender(dir);
const original = latestShareRender(dir);
const watcher = await projectFingerprint(dir);
await prepareShareRender(dir);
expect(latestShareRender(dir)?.id).toBe(original?.id);
writeFileSync(join(dir, path), "changed");
expect(await projectFingerprint(dir)).toBe(watcher);
expect(await shareContentFingerprint(dir)).not.toBe(original?.contentFingerprint);
await prepareShareRender(dir);
expect(latestShareRender(dir)?.id).not.toBe(original?.id);
},
);
});
+29 -6
View File
@@ -2,12 +2,15 @@ import { randomUUID } from "node:crypto";
import { existsSync, readFileSync, realpathSync } from "node:fs";
import { mkdir, writeFile, rename, rm } from "node:fs/promises";
import { join } from "node:path";
import { projectFingerprint } from "@hyperframes/share";
import { projectFingerprint, shareSourceFingerprint } from "@hyperframes/share";
export const shareContentFingerprint = shareSourceFingerprint;
export interface ShareRender {
path: string;
id: string;
fingerprint: string;
contentFingerprint?: string;
}
export function latestShareRender(dir: string): ShareRender | null {
@@ -33,16 +36,29 @@ export function latestShareRender(dir: string): ShareRender | null {
const expected = join(realpathSync(dir), ".hyperframes", "share-renders", `${value.id}.mp4`);
if (value.path !== expected || !existsSync(expected) || realpathSync(expected) !== expected)
return null;
return { path: expected, id: value.id, fingerprint: value.fingerprint };
const contentFingerprint =
"contentFingerprint" in value && typeof value.contentFingerprint === "string"
? value.contentFingerprint
: undefined;
return {
path: expected,
id: value.id,
fingerprint: value.fingerprint,
...(contentFingerprint && { contentFingerprint }),
};
} catch {
return null;
}
}
export async function prepareShareRender(dir: string): Promise<void> {
const before = await projectFingerprint(dir);
const [before, content] = await Promise.all([
projectFingerprint(dir),
shareContentFingerprint(dir),
]);
const previous = latestShareRender(dir);
if (previous?.fingerprint === before) return;
if (content && previous?.fingerprint === before && previous.contentFingerprint === content)
return;
const stateFolder = join(dir, ".hyperframes");
if (existsSync(stateFolder) && realpathSync(stateFolder) !== stateFolder)
throw new Error("Sharing state must stay inside this project, without symlinks.");
@@ -79,11 +95,18 @@ export async function prepareShareRender(dir: string): Promise<void> {
});
if (!existsSync(output))
throw new Error("The render did not produce a video. Run hyperframes render to diagnose it.");
if ((await projectFingerprint(dir)) !== before)
const [after, afterContent] = await Promise.all([
projectFingerprint(dir),
shareContentFingerprint(dir),
]);
if (after !== before || afterContent !== content)
throw new Error(
"The project changed while rendering. Share again to capture the current project.",
);
await writeFile(temporary, JSON.stringify({ id, path: output, fingerprint: before }));
await writeFile(
temporary,
JSON.stringify({ id, path: output, fingerprint: before, contentFingerprint: content }),
);
await rename(temporary, target);
} catch (error) {
await Promise.allSettled([rm(output, { force: true }), rm(temporary, { force: true })]);
+12
View File
@@ -214,6 +214,18 @@ describe("share operation consent and identity", () => {
"Video rejected.",
);
});
it("keeps the kernel's default source callback for a video-only update", async () => {
mocks.status.mockResolvedValue(shared);
mocks.flags.mockResolvedValue({ versionSharing: true, shareFiles: true });
mocks.add.mockResolvedValue(null);
const result = await createShareService("/project").run({ action: "update", videoOnly: true });
expect(result.source).toEqual({ at: "video_only", reason: "requested" });
expect(mocks.add).toHaveBeenCalledOnce();
expect(mocks.add.mock.calls[0]?.[0]).not.toHaveProperty("filesForExport");
expect(mocks.add).toHaveBeenCalledWith(expect.objectContaining({ omitSource: true }));
});
it("does not silently upload changed edits while resuming first share", async () => {
mocks.status.mockResolvedValue(shared);
mocks.project.mockResolvedValue({ latest_ready_version_id: null });
+34 -10
View File
@@ -20,12 +20,14 @@ import {
import { createShareAccess, shareSite } from "./auth.js";
import { assertReviewUploadStorage, bindCliShareProject } from "./host.js";
import { prepareShareRender } from "./render.js";
import { prepareCliShareSource, uncheckedSource, type PreparedShareSource } from "./sources.js";
export interface ShareAction {
action: "share" | "update" | "status" | "manage";
audience?: AccessMode;
space?: string;
waitSeconds?: number;
videoOnly?: boolean;
}
function linkState(share: ReviewShare | null, ready: boolean): string {
if (!share) return "not_shared";
@@ -49,6 +51,7 @@ async function readStatus(dir: string, client: ReviewClient, expectedProjectId?:
pending,
state: linkState(share, ready),
ready,
source: uncheckedSource(),
...statusLinks(activeUrl, ready, processing, project),
};
}
@@ -77,11 +80,13 @@ function selectSpace(spaces: Awaited<ReturnType<ReviewClient["spaces"]>>, id?: s
if (id) return spaces.find((space) => space.space_id === id);
return spaces.find((space) => space.bound) ?? spaces.find((space) => space.kind === "personal");
}
function assertUploadProject(share: ReviewShare | null, expectedId?: string): string {
function assertUploadProject(
share: ReviewShare | null,
expectedId?: string,
): asserts share is ReviewShare & { projectId: string } {
const id = share?.projectId;
if (!id || (expectedId && id !== expectedId))
throw new ReviewFailed(409, "review_binding_changed");
return id;
}
function assertExistingIntent(share: ReviewShare | null, input: ShareAction) {
if (!share) {
@@ -145,7 +150,7 @@ export function createShareService(
await auth.access();
return readStatus(dir, client, expectedProjectId);
};
const upload = async (input: ShareAction, projectId: string) => {
const upload = async (input: ShareAction, projectId: string, sources: PreparedShareSource) => {
const end = Date.now() + (input.waitSeconds ?? 300) * 1000;
let added = false;
const failure = await addVersion({
@@ -155,13 +160,16 @@ export function createShareService(
client,
onProgress: progress,
allowPending: true,
...(sources.filesForExport && { filesForExport: sources.filesForExport }),
requireSource: sources.requireSource,
omitSource: sources.source.at === "video_only",
onAdded: () => {
added = true;
},
open: () => Date.now() < end,
});
if (failure) throw new Error(failure);
const result = await status(projectId);
const result = { ...(await status(projectId)), source: sources.source };
if (result.version.failure) throw new Error(result.version.failure);
if (!added && !result.version.last?.current)
return { ...result, state: "processing", ready: false, url: null };
@@ -177,29 +185,45 @@ export function createShareService(
spaceId: space,
audience: input.audience,
});
const projectId = assertUploadProject(share, current?.projectId);
if (share?.takedown) throw new ReviewFailed(409, "moderation_hold");
assertUploadProject(share, current?.projectId);
if (share.takedown) throw new ReviewFailed(409, "moderation_hold");
assertExistingIntent(share, {
...input,
space: input.space ?? space,
audience: input.audience ?? current?.access,
});
return projectId;
return { projectId: share.projectId, spaceId: share.space?.id, audience: share.access };
};
const checkPreparedTarget = async (target: Awaited<ReturnType<typeof uploadTarget>>) => {
const share = await reviewStatus({ projectPath: dir, client });
assertUploadProject(share, target.projectId);
if (share.takedown) throw new ReviewFailed(409, "moderation_hold");
assertExistingIntent(share, {
action: "update",
space: target.spaceId,
audience: target.audience,
});
};
const run = async (input: ShareAction) => {
const current = await status();
if (input.action === "status" || input.action === "manage") return current;
assertExistingIntent(current.share, input);
if (retrievesExisting(current, input)) return current;
if (!(await client.reviewFlags()).versionSharing)
throw new ReviewFailed(403, "feature_unavailable");
const flags = await client.reviewFlags();
if (!flags.versionSharing) throw new ReviewFailed(403, "feature_unavailable");
const space = current.share
? current.share.space?.id
: await preflightFirst(dir, client, input);
assertResumeSource(current, input);
assertReviewUploadStorage();
await prepareShareRender(dir);
return upload(input, await uploadTarget(current.share, input, space));
const target = await uploadTarget(current.share, input, space);
const sources = await prepareCliShareSource(dir, auth.access, target.spaceId, {
videoOnly: input.videoOnly,
shareFiles: flags.shareFiles,
});
if (sources.requireSource) await checkPreparedTarget(target);
return upload(input, target.projectId, sources);
};
return {
status,
+78
View File
@@ -0,0 +1,78 @@
import { beforeEach, expect, it, vi } from "vitest";
const mocks = vi.hoisted(() => ({
prepare: vi.fn(),
filesForExport: vi.fn(),
factory: vi.fn(),
content: vi.fn(),
}));
vi.mock("@hyperframes/share", () => ({
shareSources: mocks.factory,
projectFingerprint: async () => "rendered",
}));
vi.mock("./render.js", () => ({
latestShareRender: () => ({ fingerprint: "rendered", contentFingerprint: "original-content" }),
shareContentFingerprint: mocks.content,
}));
import { prepareCliShareSource, ShareSourceError } from "./sources.js";
import { shareError } from "./errors.js";
const access = async () => ({
token: "test",
heygen: "https://app.test",
heygenApi: "https://api.test",
owner: "owner",
});
beforeEach(() => {
vi.resetAllMocks();
mocks.content.mockResolvedValue("original-content");
mocks.factory.mockReturnValue({ prepare: mocks.prepare, filesForExport: mocks.filesForExport });
});
it("eligible source requires attachment on the version", async () => {
const snapshot = { space_id: "space", composition_id: "composition", snapshot_id: "snapshot" };
mocks.prepare.mockResolvedValue({ at: "ready", snapshot });
expect(await prepareCliShareSource("/project", access, "space", { shareFiles: true })).toEqual({
source: { at: "ready", snapshot },
requireSource: true,
filesForExport: mocks.filesForExport,
});
expect(mocks.prepare).toHaveBeenCalledWith("/project", "rendered");
});
it("refuses source bytes that differ from the rendered assets even when metadata matches", async () => {
mocks.prepare.mockImplementation(async (dir: string) => {
const fingerprint = mocks.factory.mock.calls[0]?.[0].fingerprint;
expect(await fingerprint(dir)).toBe("rendered");
mocks.content.mockResolvedValue("changed-content");
expect(await fingerprint(dir)).toBeNull();
return { at: "unavailable", reason: "source_render_changed" };
});
await expect(
prepareCliShareSource("/project", access, "space", { shareFiles: true }),
).rejects.toThrow("source_render_changed");
});
it("free plan ineligibility preserves video sharing and reports source unavailable", async () => {
mocks.prepare.mockResolvedValue({ at: "unavailable", reason: "source_plan_required" });
expect(await prepareCliShareSource("/project", access, "space", { shareFiles: true })).toEqual({
source: { at: "video_only", reason: "source_plan_required" },
requireSource: false,
});
});
it("explicit video-only skips source network requests", async () => {
await prepareCliShareSource("/project", access, "space", { shareFiles: true, videoOnly: true });
expect(mocks.factory).not.toHaveBeenCalled();
});
it("unexpected upload errors and stale source block with an explicit video-only retry", async () => {
mocks.prepare.mockRejectedValueOnce(new Error("network failed"));
await expect(
prepareCliShareSource("/project", access, "space", { shareFiles: true }),
).rejects.toThrow("--video-only");
mocks.prepare.mockResolvedValue({ at: "unavailable", reason: "source_render_changed" });
await expect(
prepareCliShareSource("/project", access, "space", { shareFiles: true }),
).rejects.toThrow("--video-only");
});
it("source failure has a stable bridge and JSON error code", () => {
expect(shareError(new ShareSourceError("Retry or choose video-only"))).toEqual({
code: "source_unavailable",
message: "Retry or choose video-only",
});
});
+60
View File
@@ -0,0 +1,60 @@
import { projectFingerprint, shareSources, type VersionRequest } from "@hyperframes/share";
import type { createShareAccess } from "./auth.js";
import { latestShareRender, shareContentFingerprint } from "./render.js";
export class ShareSourceError extends Error {}
export type ShareSourceState =
| { at: "not_checked" }
| { at: "video_only"; reason: string }
| { at: "ready"; snapshot: { space_id: string; composition_id: string; snapshot_id: string } };
export interface PreparedShareSource {
source: ShareSourceState;
filesForExport?: VersionRequest["filesForExport"];
requireSource: boolean;
}
const unavailablePlans = new Set([
"source_disabled",
"source_plan_required",
"source_composition_limit",
]);
export function uncheckedSource(): ShareSourceState {
return { at: "not_checked" };
}
export async function prepareCliShareSource(
dir: string,
access: ReturnType<typeof createShareAccess>["access"],
spaceId: string | undefined,
options: { videoOnly?: boolean; shareFiles: boolean },
): Promise<PreparedShareSource> {
if (options.videoOnly) return videoOnly("requested");
if (!options.shareFiles) return videoOnly("source_disabled");
if (!spaceId) throw new Error("The owning workspace could not be verified. Retry sharing.");
try {
const render = latestShareRender(dir);
const source = shareSources({
access,
spaceId,
fingerprint: async (path) =>
render?.contentFingerprint === (await shareContentFingerprint(path))
? projectFingerprint(path)
: null,
});
const prepared = await source.prepare(dir, render?.fingerprint ?? null);
if (prepared.at === "ready")
return { source: prepared, filesForExport: source.filesForExport, requireSource: true };
if (unavailablePlans.has(prepared.reason)) return videoOnly(prepared.reason);
throw new Error(`Source could not be attached (${prepared.reason}).`);
} catch (error) {
const reason = error instanceof Error ? error.message : "Source upload failed.";
throw new ShareSourceError(
`${reason} Retry sharing, or use --video-only to share just the video.`,
{
cause: error,
},
);
}
}
function videoOnly(reason: string): PreparedShareSource {
return { source: { at: "video_only", reason }, requireSource: false };
}
+58 -1
View File
@@ -35,7 +35,7 @@ const [current, version] = await Promise.all([
`access(refusedToken?)` returns `{ token, heygen, heygenApi, owner }` or null. It is read for every request. A refused bearer is passed back so the host can refresh it. `heygen` is the review API base. A host can inject its own `signedInRequest` to retain network tracing. `execFile` can likewise be injected to retain native process policy.
Video uploads require exact storage origins from trusted host configuration. Set `reviewUploadOrigins` on the host or supply a comma-separated `HYPERFRAMES_REVIEW_UPLOAD_ORIGINS` environment variable. Configure the bucket-specific HTTPS origins used by the signer. Use URL origins without paths, trailing slashes, or wildcards. Local tests can explicitly allow an HTTP origin on `127.0.0.1`. An empty list refuses every video upload.
Video and source uploads require exact storage origins from trusted host configuration. Set `reviewUploadOrigins` on the host or supply a comma-separated `HYPERFRAMES_REVIEW_UPLOAD_ORIGINS` environment variable. Configure the bucket-specific HTTPS origins used by the signer. Use URL origins without paths, trailing slashes, or wildcards. Local tests can explicitly allow an HTTP origin on `127.0.0.1`. An empty list refuses every upload.
Before first registration the host must obtain explicit audience consent and prepare a current MP4. `shareForReview` registers/enables the link; it does not imply playable media. Await `addVersion`, then check the server project/latest ready version before reporting success. `addFirstVersion` starts background work and is for long-lived hosts only. `open: () => boolean` on `addVersion` bounds the polling lifetime without discarding the saved operation. `filesForExport(projectPath, renderedFingerprint)` optionally supplies an already-synced matching snapshot. Omit it for ordinary video sharing.
@@ -50,3 +50,60 @@ Metadata mutations and version uploads use separate cross-process locks under `.
`snapshotProject` skips symlinks by default, retaining Desktop watcher, change-set and saved-render compatibility; `snapshotFiles` also skips symlink entries. Set `followSymlinks: true` for a link-aware full snapshot. The CLI-facing `projectFingerprint` opts into following linked media (with cycle detection), so edited linked files invalidate its cached render. Both comparisons ignore HTML ID stamps. Export/history metadata, caches, hidden folders and installed packages are excluded. Hosts must record the fingerprint of the actual rendered source and must not relabel an old render with a newer fingerprint.
Public type exports include `ShareHost`, `ReviewClient`, `ReviewShare`, `VersionRequest`, and `Wire`, plus `@hyperframes/share/types/reviewWire` and `types/share`. Per-module entry points preserve Desktop imports during adoption.
## Source snapshots
`shareSources({access, spaceId})` prepares editable source for a rendered version:
```js
const sources = shareSources({ access, spaceId });
const result = await sources.prepare(projectPath, renderedFingerprint);
if (result.at === "ready") {
await addVersion({
projectPath,
title,
client,
filesForExport: sources.filesForExport,
requireSource: true,
});
}
```
Handle `unavailable.reason` before creating the video version. Plan ineligibility
can use video-only; changed source, conflicts and unexpected upload failures need
an explicit choice to omit source. The CLI exposes `--video-only` for that choice.
`filesForExport` only validates already-prepared source; it does not upload.
The adapter uses the portable-source-v4 capture/manifest rules and
multipart transport. The pinned policy, Unicode 15.1 registry (with its license),
and manifest test vectors are checked into this repository. Source is captured through
verified file handles; upload bytes are streamed and checked against their hash.
Workspace limits, upload receipts and published snapshot identity are checked.
Pending uploads retain their original idempotency key across retries. A conflict
is never auto-resolved. When local source changes, the adapter cancels an older
pending upload before starting another; uploads under verification are preserved.
A stalled storage PUT stops after 60 seconds without progress and can retry with
the same upload identity.
Default state uses `.hyperframes/share-sync.<backend-hash>.json`, with strict
account/backend/realpath binding, private atomic writes, and a process lock.
Hosts can inject a compatible `store` or read-only `existingBinding` resolver.
An external Desktop binding is reused only when exact and settled; this adapter
never writes Desktop's binding or starts a second upload on its behalf.
Source sharing requires an authorized account and a workspace that supports editable project files.
Source-plan/feature eligibility remains server controlled. Automated fixtures do
not establish live entitlement, moderation, source download or outside-editor
acceptance. See [the source guide](https://hyperframes.heygen.com/developers/share-source).
Tests run locally without an account or service credentials:
```bash
node --test packages/share/tests/*.test.mjs
bun run --cwd packages/share build
```
The bundled Unicode data records SHA-256 hashes for its inputs in `sources`.
Those files are published in the [Unicode 15.1 database](https://www.unicode.org/Public/15.1.0/ucd/).
Keep the included Unicode license with redistributed data. The pinned version
keeps path comparison consistent across Node versions and operating systems.
+3
View File
@@ -1,3 +1,6 @@
import { cp } from "node:fs/promises";
await cp("src/types", "dist/types", { recursive: true });
await cp("src/hosts.d.ts", "dist/hosts.d.ts");
await cp("src/sources/types.d.ts", "dist/sources/types.d.ts");
await cp("src/sources/adapterTypes.d.ts", "dist/sources/adapterTypes.d.ts");
+6
View File
@@ -80,3 +80,9 @@ export { bindShareHost } from "./host.mjs";
/** @typedef {import("./types/share.js").AccessMode} AccessMode */
/** @typedef {import("./types/share.js").ReviewExportProgress} ReviewExportProgress */
export { pendingReviewExport } from "./reviewExport.mjs";
export {
shareSources,
shareSourceFingerprint,
createProjectShareSourceStore,
} from "./shareSources.mjs";
+9 -2
View File
@@ -80,7 +80,7 @@ async function claim(candidate, lock, owner) {
}
}
}
/** @template T @param {string} projectPath @param {"review" | "export"} lane @param {() => Promise<T>} run
/** @template T @param {string} projectPath @param {"review" | "export" | "source"} lane @param {() => Promise<T>} run
* @param {{ timeoutMs?: number, signal?: AbortSignal }} [options] @returns {Promise<T>} */
export async function withProjectShareLock(
projectPath,
@@ -90,7 +90,14 @@ export async function withProjectShareLock(
) {
const root = await realpath(projectPath);
const folder = await mkdirInside(root, ".hyperframes");
const lock = join(folder, lane === "review" ? "review.lock" : "review-export.lock");
const lock = join(
folder,
lane === "review"
? "review.lock"
: lane === "source"
? "share-source.lock"
: "review-export.lock",
);
const owner = { token: randomUUID(), pid: process.pid, hostname: hostname() };
const candidate = `${lock}.${owner.token}.tmp`;
await writeFile(candidate, JSON.stringify(owner), { flag: "wx", mode: 0o600 });
+11 -4
View File
@@ -31,7 +31,7 @@ async function snapshotKey(full, byContent) {
const stat = await handle.stat();
if (!stat.isFile()) return undefined;
if (full.endsWith(".html")) return contentKey(await handle.readFile("utf8"));
if (byContent) return sha256(await handle.readFile());
if (byContent) return await hashFile(handle);
return `${stat.mtimeMs}:${stat.size}`;
} catch {
return undefined;
@@ -40,6 +40,13 @@ async function snapshotKey(full, byContent) {
}
}
/** @param {import("node:fs/promises").FileHandle} handle */
async function hashFile(handle) {
const hash = createHash("sha256");
for await (const chunk of handle.createReadStream({ autoClose: false })) hash.update(chunk);
return hash.digest("hex");
}
/** @param {import("node:fs").Dirent} entry @param {string} full @param {boolean} followSymlinks */
async function entryKind(entry, full, followSymlinks) {
if (!entry.isSymbolicLink()) return entry;
@@ -115,6 +122,6 @@ export function diffChangedFiles(before, after) {
return changed;
}
/** @param {string} projectPath */
export const projectFingerprint = async (projectPath) =>
snapshotFingerprint(await snapshotProject(projectPath, { followSymlinks: true }));
/** @param {string} projectPath @param {{ byContent?: boolean }} [options] */
export const projectFingerprint = async (projectPath, options) =>
snapshotFingerprint(await snapshotProject(projectPath, { ...options, followSymlinks: true }));
+76 -16
View File
@@ -40,7 +40,8 @@ import { versionSentence } from "./reviewVersionSentences.mjs";
* known; `last`: the newest version.
* @typedef {{ pending: { body: ExportRequest, videoId: string | null, projectId?: string | null,
* checkingSince?: number, snapshotRefused?: boolean, unanswered?: boolean, reviewSent?: ReviewSent } | null,
* last: { fingerprint: string, versionId: string, exportedAt: string, projectId: string, files?: boolean, checksum?: string }
* last: { fingerprint: string, versionId: string, exportedAt: string, projectId: string, files?: boolean,
* snapshot?: ExportRequest["snapshot"], checksum?: string }
* | null }} VersionRecord */
/** `onAdded`: a version went ready, with whether it carries files and whether it is the page's first from here.
@@ -53,7 +54,7 @@ import { versionSentence } from "./reviewVersionSentences.mjs";
* @typedef {VersionHooks & { projectPath: string, title: string, client: ReviewClient, put?: typeof putFile,
* sleep?: (ms: number) => Promise<void>, probe?: (file: string) => Promise<VideoFacts | null>, now?: () => number,
* projectId?: string | null, filesForExport?: FilesForRender, fromReview?: FromReview | null,
* allowPending?: boolean }} VersionRequest */
* allowPending?: boolean, requireSource?: boolean, omitSource?: boolean }} VersionRequest */
/** The snapshot of `rendered`, the project as the export was rendered; null when the folder is not that render.
* @typedef {(projectPath: string, rendered: string | null) => Promise<import("./types/reviewWire.js").SnapshotRef | null>}
@@ -178,7 +179,15 @@ function isLast(value) {
isSavedId(value.versionId) &&
isSavedId(value.exportedAt) &&
isSavedId(value.projectId) &&
isLastMetadata(value)
);
}
/** @param {Record<string, unknown>} value */
function isLastMetadata(value) {
return (
isOptionalBoolean(value.files) &&
isOptionalSnapshot(value.snapshot) &&
(value.checksum === undefined || isSavedId(value.checksum))
);
}
@@ -280,8 +289,8 @@ export async function reviewVersion(projectPath, server = {}) {
/** The newest MP4 and the body that adds it; a stale, non-MP4, too big or too long export never sends anything.
* @param {string} dir @param {string} title @param {(file: string) => Promise<VideoFacts | null>} probe
* @param {FilesForRender} filesForExport */
async function exportSource(dir, title, probe, filesForExport) {
* @param {FilesForRender} filesForExport @param {boolean} requireSource */
async function exportSource(dir, title, probe, filesForExport, requireSource) {
const latest = latestExportFile(dir);
if (!latest || sourceOf(dir, await projectFingerprint(dir)) !== "ready")
return { failure: local("Export an MP4 of the project as it is now first.") };
@@ -308,18 +317,34 @@ async function exportSource(dir, title, probe, filesForExport) {
content_type: "video/mp4",
},
};
const snapshot = await filesForExport(dir, body.source_fingerprint).catch((error) => {
// expected: the version still goes up, and Share's files line says it has no files and offers Sync now.
console.error(
"[review files] couldn't read the synced snapshot; the version goes up without files:",
error,
);
return null;
});
const snapshot = await sourceSnapshot(
dir,
body.source_fingerprint,
filesForExport,
requireSource,
);
if (missingRequiredSource(requireSource, snapshot))
return {
failure: local(
"Source changed after preparation. Retry sharing, or choose video-only explicitly.",
),
};
if (snapshot) body.snapshot = snapshot;
return { file: latest.path, body };
}
/** @param {boolean} required @param {import("./types/reviewWire.js").SnapshotRef | null} snapshot */
const missingRequiredSource = (required, snapshot) => required && !snapshot;
/** @param {string} dir @param {string | null} rendered @param {FilesForRender} filesForExport @param {boolean} requireSource */
async function sourceSnapshot(dir, rendered, filesForExport, requireSource) {
return filesForExport(dir, rendered).catch((error) => {
if (requireSource) throw error;
console.error("[review files] could not read source; adding video only:", error);
return null;
});
}
/** @param {ExportRequest["snapshot"]} a @param {ExportRequest["snapshot"]} b */
const sameSnapshot = (a, b) =>
a?.space_id === b?.space_id &&
@@ -369,7 +394,7 @@ async function createdExport(r, id, pending, kept) {
return await sendCreate(r, id, pending, kept);
} catch (error) {
const refused = error instanceof ReviewFailed && error.reason === "snapshot_mismatch";
if (!refused || !pending.body.snapshot || pending.unanswered) throw error;
if (!refused || !pending.body.snapshot || pending.unanswered || r.requireSource) throw error;
console.error("[review] snapshot refused; the version goes up without files:", error);
const body = { ...pending.body, client_operation_id: `export-${randomUUID()}` };
delete body.snapshot;
@@ -571,6 +596,7 @@ async function finish(dir, state, pending, id, done, fromReview) {
exportedAt: done.exported_at,
projectId: id,
files: Boolean(pending.body.snapshot),
...(pending.body.snapshot && { snapshot: pending.body.snapshot }),
checksum: pending.body.local_upload.checksum_sha256,
};
await writeState(dir, { pending: null, last: ready ? last : state.last });
@@ -583,13 +609,32 @@ async function finish(dir, state, pending, id, done, fromReview) {
async function run(r, report) {
report({ step: "preparing", sent: 0, total: 0 });
const state = await readState(r.projectPath);
const source = await exportSource(r.projectPath, r.title, r.probe, r.filesForExport);
const source = await exportSource(
r.projectPath,
r.title,
r.probe,
r.omitSource ? noSource : r.filesForExport,
r.requireSource,
);
if ("failure" in source) return source.failure ?? null;
const id = r.projectId ?? (await reviewProjectId(r.projectPath));
if (!id) return local("Share the project first, then add a version.");
if (await alreadyUploaded(r.client, id, state, source.body)) return null;
if (
sameSourceMode(r, state.last, source.body.snapshot) &&
(await alreadyUploaded(r.client, id, state, source.body))
)
return null;
return sendVersion(r, report, source, id, state);
}
const noSource = async () => null;
/** @param {Required<VersionRequest>} r @param {VersionRecord["last"]} last @param {ExportRequest["snapshot"]} snapshot */
function sameSourceMode(r, last, snapshot) {
if (r.requireSource) return sameSnapshot(last?.snapshot, snapshot);
if (r.omitSource) return last?.files !== true;
return true;
}
/** @param {ReviewClient} client @param {string} id @param {VersionRecord} state @param {ExportRequest} body */
async function alreadyUploaded(client, id, state, body) {
const last = state.last;
@@ -606,6 +651,8 @@ async function alreadyUploaded(client, id, state, body) {
async function sendVersion(r, report, source, id, state) {
const first = state.last?.projectId !== id;
let pending = bodyToSend(state, source.body, id);
const sourceConflict = pendingSourceConflict(r, pending, source.body);
if (sourceConflict) return local(sourceConflict);
await writeState(r.projectPath, (state = { ...state, pending }));
try {
const answer = await createdExport(r, id, pending, (next) =>
@@ -638,6 +685,17 @@ async function sendVersion(r, report, source, id, state) {
}
}
/** @param {Required<VersionRequest>} r @param {NonNullable<VersionRecord["pending"]>} pending @param {ExportRequest} fresh */
function pendingSourceConflict(r, pending, fresh) {
if (r.requireSource && !pending.body.snapshot)
return "A pending video-only upload must finish before attaching source. Retry with --video-only, then share source again.";
if (r.omitSource && pending.body.snapshot)
return "A pending upload includes source. Check its status or retry with source before creating a video-only version.";
if (r.requireSource && !sameSnapshot(pending.body.snapshot, fresh.snapshot))
return "A pending upload includes a different source snapshot. Check its status before sharing again.";
return null;
}
/** @param {Required<VersionRequest>} r @param {VersionRecord} state
* @param {NonNullable<VersionRecord["pending"]>} pending @param {string} id @param {ExportDTO} done @param {boolean} first */
async function finishRun(r, state, pending, id, done, first) {
@@ -665,8 +723,10 @@ export async function addVersion(request) {
onProgress: () => {},
open: () => true,
allowPending: false,
filesForExport: async () => null,
filesForExport: noSource,
fromReview: null,
requireSource: false,
omitSource: false,
onAdded: () => {},
...request,
projectId: request.projectId ?? null,
+18 -16
View File
@@ -18,6 +18,22 @@ export function fileSha256Base64(file) {
/** @typedef {"sent" | "expired" | "refused" | "offline"} PutOutcome */
/** @param {string} url */
export function trustedUploadUrl(url) {
if (!URL.canParse(url)) return null;
const target = new URL(url);
const loopback = target.protocol === "http:" && target.hostname === "127.0.0.1";
if (
(target.protocol !== "https:" && !loopback) ||
target.username ||
target.password ||
target.hash ||
!reviewUploadOrigins().includes(target.origin)
)
return null;
return target;
}
/** Streams the file to the signed URL through main's fetch (the app's proxy), the server's headers as given plus
* the length, and no bearer. Nothing sent or answered for `stallMs` ends it as offline, never "Uploading" forever.
* @param {{ put_url: string, headers: Record<string, string> }} upload @param {string} file @param {number} size
@@ -29,22 +45,8 @@ export async function putFile(
onSent,
{ stallMs = 60_000, fetchImpl = fetch } = {},
) {
let target;
try {
target = new URL(upload.put_url);
} catch {
return "refused";
}
const loopback = target.protocol === "http:" && target.hostname === "127.0.0.1";
if (
(target.protocol !== "https:" && !loopback) ||
target.username ||
target.password ||
target.hash ||
!reviewUploadOrigins().includes(target.origin)
) {
return "refused";
}
const target = trustedUploadUrl(upload.put_url);
if (!target) return "refused";
const stall = new AbortController();
/** @type {ReturnType<typeof setTimeout> | undefined} */
let timer;
+236
View File
@@ -0,0 +1,236 @@
// @ts-check
import { realpath } from "node:fs/promises";
import { randomUUID } from "node:crypto";
import { basename } from "node:path";
import { projectFingerprint } from "./projectSnapshot.mjs";
import { withProjectShareLock } from "./projectLock.mjs";
import { captureProject } from "./sources/capture.mjs";
import { sourceApi } from "./sources/api.mjs";
import { createProjectShareSourceStore } from "./sources/store.mjs";
import { realWait } from "./sources/transfer.mjs";
import { recoverSourceUpload, uploadSource } from "./sources/upload.mjs";
/** @typedef {import("./sources/adapterTypes.js").SourceResult} SourceResult
* @typedef {import("./sources/types.js").Binding} Binding */
/** @param {string} reason @returns {SourceResult} */
const unavailable = (reason) => ({ at: "unavailable", reason });
/** Content identity for sharing, including portable hidden render inputs. The Desktop watch inventory stays separate.
* @param {string} projectPath @returns {Promise<string | null>} */
export async function shareSourceFingerprint(projectPath) {
const capture = await captureProject(await realpath(projectPath));
return capture.at === "captured" && !captureRefusal(capture) ? capture.manifest.sha256 : null;
}
/** @param {Binding} binding @param {string} space */
function bindingRefusal(binding, space) {
if (binding.ownerSpaceId !== space) return "source_other_workspace";
if (binding.sharedFrom) return "source_owned_by_someone_else";
if (binding.gettingFromCloud || binding.duplicate) return "source_operation_in_progress";
return null;
}
/** @param {import("./sources/types.js").Capture} capture */
function captureRefusal(capture) {
if (capture.at === "too-big") return `source_${capture.over.limit}_exceeded`;
if (capture.at === "problems") return `source_${capture.problems[0]?.problem ?? "invalid"}`;
if (capture.settled.agentTurnOpen || capture.settled.projectLocked || capture.waiting.length)
return "source_edit_in_progress";
return null;
}
/** Prepare source explicitly before adding the video version. No unexpected network or filesystem failure is
* converted to video-only. `filesForExport` revalidates the prepared bytes and never starts another upload.
* @param {import("./sources/adapterTypes.js").SourceOptions} options */
export function shareSources(options) {
const store = options.store ?? createProjectShareSourceStore();
const fingerprint = options.fingerprint ?? projectFingerprint;
const signal = options.signal ?? new AbortController().signal;
/** @type {Map<string, { rendered: string, digest: string, snapshot: import("./types/reviewWire.js").SnapshotRef, account: string, backend: string }>} */
const prepared = new Map();
/** @param {string} projectPath @param {string | null} rendered @returns {Promise<SourceResult>} */
const prepare = async (projectPath, rendered) => {
const root = await realpath(projectPath);
prepared.delete(root);
if (!enabled(options)) return unavailable("source_disabled");
if (!rendered || (await fingerprint(root)) !== rendered)
return unavailable("source_render_changed");
return withProjectShareLock(
root,
"source",
async () => {
const signed = await options.access();
if (!signed?.owner) return unavailable("source_signed_out");
const key = { backend: signed.heygenApi, account: signed.owner, projectPath: root };
const guardedAccess = async (/** @type {string | undefined} */ refused) => {
const next = await options.access(refused);
if (!next || next.owner !== key.account || next.heygenApi !== key.backend)
throw new Error("Source account or backend changed while sharing.");
return next;
};
const api =
options.api ?? sourceApi({ access: guardedAccess, fetchImpl: options.fetchImpl });
const { own, existing, binding } = await selectBinding(key, options, store);
const refused = bindingRefusal(binding, options.spaceId);
if (refused) return unavailable(refused);
const result = await prepareBinding({
binding,
own: Boolean(own),
external: Boolean(existing),
api,
store,
options,
signal,
fingerprint,
rendered,
});
const manifest = result.manifest;
await guardedAccess(undefined);
if (result.at !== "ready" || !manifest) return result;
if (!(await stillMatches(root, rendered, manifest.sha256, fingerprint)))
return unavailable("source_render_changed");
prepared.set(root, {
rendered,
digest: manifest.sha256,
snapshot: result.snapshot,
account: key.account,
backend: key.backend,
});
return { at: "ready", snapshot: result.snapshot };
},
{ signal },
);
};
/** @param {string} projectPath @param {string | null} rendered */
const filesForExport = async (projectPath, rendered) => {
const root = await realpath(projectPath);
const held = prepared.get(root);
const signed = await options.access();
if (
!held ||
!rendered ||
held.rendered !== rendered ||
!sameOwner(signed, held) ||
!enabled(options)
)
return null;
return (await stillMatches(root, rendered, held.digest, fingerprint)) ? held.snapshot : null;
};
return { prepare, filesForExport };
}
/** @param {import("./sources/adapterTypes.js").SourceOptions} options */
const enabled = (options) => options.enabled?.() ?? true;
/** @param {Awaited<ReturnType<import("./signedInRequest.mjs").SignedInAccess>>} signed
* @param {{account: string, backend: string}} held */
const sameOwner = (signed, held) =>
signed?.owner === held.account && signed?.heygenApi === held.backend;
/** @typedef {{ binding: Binding, own: boolean, external: boolean, api: import("./sources/api.mjs").SourceApi,
* store: import("./sources/adapterTypes.js").SourceStore, options: import("./sources/adapterTypes.js").SourceOptions,
* signal: AbortSignal, fingerprint: (path: string) => Promise<string | null>, rendered: string }} Preparation */
/** @param {Preparation} job @returns {Promise<SourceResult & {manifest?: import("./sources/types.js").Manifest}>} */
async function prepareBinding(job) {
const { binding, api, fingerprint, rendered } = job;
const limits = await api.limits(binding.ownerSpaceId);
if (!limits.syncAllowed) return unavailable("source_plan_required");
const capture = await captureProject(binding.projectPath, { limits: limits.push });
const bad = captureRefusal(capture);
if (bad) return unavailable(bad);
if (capture.at !== "captured") return unavailable("source_invalid");
if ((await fingerprint(binding.projectPath)) !== rendered)
return unavailable("source_render_changed");
const result = await snapshotFor(job, capture.manifest, limits);
return { ...result, manifest: capture.manifest };
}
/** @param {Preparation} job @param {import("./sources/types.js").Manifest} manifest
* @param {Extract<import("./sources/types.js").SyncLimitsRead, {at: "read"}>} limits @returns {Promise<SourceResult>} */
async function snapshotFor(job, manifest, limits) {
const { binding, api, store, options, signal } = job;
const blocked = await recoverEarlierCapture(job, manifest.sha256);
if (blocked) return blocked;
if (binding.base?.manifestSha256 === manifest.sha256 && !binding.inflight)
return reuseSnapshot(api, binding, manifest.sha256);
if (job.external) return unavailable("source_sync_required");
if (!job.own && limits.compositions >= limits.maxCompositions)
return unavailable("source_composition_limit");
await store.write(binding);
binding.composition = await api.create(
binding.ownerSpaceId,
binding.compositionId,
basename(binding.projectPath),
);
await store.write(binding);
return uploadSource({ binding, manifest, api, store, signal, ...uploadOptions(options) });
}
/** @param {Preparation} job @param {string} digest */
function recoverEarlierCapture({ external, binding, api, store }, digest) {
const inflight = binding.inflight;
if (external || !inflight || inflight.manifestSha256 === digest) return null;
return recoverSourceUpload({ binding, api, store });
}
/** @param {import("./sources/adapterTypes.js").SourceOptions} options */
function uploadOptions(options) {
return {
fetchImpl: options.fetchImpl ?? fetch,
wait: options.wait ?? realWait,
timeoutMs: options.verificationTimeoutMs ?? 120_000,
onProgress: options.onProgress ?? (() => {}),
};
}
/** @param {import("./sources/types.js").BindingKey} key @param {import("./sources/adapterTypes.js").SourceOptions} options
* @param {import("./sources/adapterTypes.js").SourceStore} store */
async function selectBinding(key, options, store) {
const own = await store.read(key);
const existing = own ? null : await options.existingBinding?.(key);
return { own, existing, binding: own ?? existing ?? newBinding(key, options.spaceId) };
}
/** @param {import("./sources/types.js").BindingKey} key @param {string} space @returns {Binding} */
function newBinding(key, space) {
return {
...key,
version: 1,
compositionId: randomUUID(),
ownerSpaceId: space,
boundAt: Date.now(),
composition: null,
etag: null,
base: null,
inflight: null,
};
}
/** @param {import("./sources/api.mjs").SourceApi} api @param {Binding} binding @param {string} digest @returns {Promise<SourceResult>} */
async function reuseSnapshot(api, binding, digest) {
if (!binding.base) return unavailable("source_sync_required");
const remote = await api.snapshot(
binding.ownerSpaceId,
binding.compositionId,
binding.base.snapshotId,
);
if (
remote.id !== binding.base.snapshotId ||
remote.compositionId !== binding.compositionId ||
remote.manifestSha256 !== digest ||
remote.manifest?.sha256 !== digest ||
remote.publicationOutcome !== "published"
)
return unavailable("source_snapshot_mismatch");
return {
at: "ready",
snapshot: {
space_id: binding.ownerSpaceId,
composition_id: binding.compositionId,
snapshot_id: remote.id,
},
};
}
/** @param {string} root @param {string} rendered @param {string} digest @param {(path: string) => Promise<string | null>} fingerprint */
async function stillMatches(root, rendered, digest, fingerprint) {
if ((await fingerprint(root)) !== rendered) return false;
const capture = await captureProject(root);
return (
!captureRefusal(capture) &&
capture.at === "captured" &&
capture.manifest.sha256 === digest &&
(await fingerprint(root)) === rendered
);
}
export { createProjectShareSourceStore } from "./sources/store.mjs";
+39
View File
@@ -0,0 +1,39 @@
import type { Binding, BindingKey, Manifest } from "./types.js";
import type { SnapshotRef } from "../types/reviewWire.js";
import type { SignedInAccess } from "../signedInRequest.mjs";
import type { SourceApi } from "./api.mjs";
export interface SourceStore {
read(key: BindingKey): Promise<Binding | null>;
write(binding: Binding): Promise<void>;
}
export type SourceResult =
| { at: "ready"; snapshot: SnapshotRef }
| { at: "unavailable"; reason: string };
export interface SourceOptions {
access: SignedInAccess;
spaceId: string;
store?: SourceStore;
enabled?: () => boolean;
fingerprint?: (path: string) => Promise<string | null>;
/** Host-owned Desktop bindings may be read for exact reuse, never overwritten. */
existingBinding?: (key: BindingKey) => Promise<Binding | null>;
fetchImpl?: typeof fetch;
signal?: AbortSignal;
verificationTimeoutMs?: number;
onProgress?: (bytes: number) => void;
/** Test/host seams; the default is the authenticated v3 client. */
api?: SourceApi;
wait?: (ms: number, signal?: AbortSignal) => Promise<void>;
}
export interface SourceRun {
binding: Binding;
manifest: Manifest;
api: SourceApi;
store: SourceStore;
fetchImpl: typeof fetch;
signal: AbortSignal;
wait: (ms: number, signal?: AbortSignal) => Promise<void>;
timeoutMs: number;
onProgress: (bytes: number) => void;
}
+114
View File
@@ -0,0 +1,114 @@
// @ts-check
import { signedInRequest } from "../signedInRequest.mjs";
import { SyncApiError, refusalOf } from "./errors.mjs";
import {
beginOf,
grantsOf,
objectOf,
uploadOf,
syncLimitsOf,
compositionOf,
snapshotOf,
} from "./wire.mjs";
/** @param {{ access: import("../signedInRequest.mjs").SignedInAccess, fetchImpl?: typeof fetch }} deps */
export function sourceApi({ access, fetchImpl = fetch }) {
const request = signedInRequest({
access,
fetchImpl,
timeoutMs: 120_000,
tag: "share-source",
host: "heygenApi",
offline: () => new SyncApiError("offline"),
signedOut: () => new SyncApiError("signed-out"),
});
/** @param {string} method @param {string} route @param {unknown} [body] @param {Record<string,string>} [headers] */
const send = async (method, route, body, headers) => {
const { reply } = await request(method, route, body, headers);
if (!reply.ok) throw await refusalOf(reply);
return objectOf(await reply.json(), "envelope").data;
};
/** @param {string} space @param {string} composition */
const path = (space, composition) =>
`/v3/hyperframes/workspaces/${encodeURIComponent(space)}/compositions/${encodeURIComponent(composition)}`;
return {
/** @param {string} space */
limits: async (space) =>
syncLimitsOf(
await send(
"GET",
`/v3/hyperframes/workspaces/${encodeURIComponent(space)}/composition-sync/limits`,
),
),
/** @param {string} space @param {string} composition @param {string} title */
create: async (space, composition, title) =>
compositionOf(await send("PUT", path(space, composition), { title })),
/** @param {string} space @param {string} composition @param {string} snapshot */
snapshot: async (space, composition, snapshot) =>
snapshotOf(
await send(
"GET",
`${path(space, composition)}/snapshots/${encodeURIComponent(snapshot)}?include=manifest`,
),
),
/** @param {string} space @param {string} composition @param {import("./types.js").BeginRequest} body */
begin: async (space, composition, body) =>
beginOf(
await send(
"POST",
`${path(space, composition)}/uploads`,
{
base_snapshot_id: body.baseSnapshotId,
resolves_conflict_snapshot_ids: [],
client_type: "cli",
manifest_sha256: body.manifest.sha256,
manifest: body.manifest.document,
},
{ "Idempotency-Key": body.idempotencyKey },
),
false,
),
/** @param {string} space @param {string} composition @param {string} upload @param {import("./types.js").GrantRequest[]} files */
uploadUrls: async (space, composition, upload, files) =>
grantsOf(
await send(
"POST",
`${path(space, composition)}/uploads/${encodeURIComponent(upload)}/upload-urls`,
{
files: files.map((file) => ({
sha256: file.sha256,
...(file.multipartUploadId && { multipart_upload_id: file.multipartUploadId }),
...(file.parts && {
parts: file.parts.map((part) => ({
part_number: part.partNumber,
sha256: part.sha256,
})),
}),
})),
},
),
),
/** @param {string} space @param {string} composition @param {string} upload */
status: async (space, composition, upload) =>
uploadOf(
await send("GET", `${path(space, composition)}/uploads/${encodeURIComponent(upload)}`),
),
/** @param {string} space @param {string} composition @param {string} upload */
cancel: async (space, composition, upload) =>
uploadOf(
await send(
"POST",
`${path(space, composition)}/uploads/${encodeURIComponent(upload)}/cancel`,
),
),
/** @param {string} space @param {string} composition @param {string} upload */
submit: async (space, composition, upload) =>
uploadOf(
await send(
"POST",
`${path(space, composition)}/uploads/${encodeURIComponent(upload)}/verification`,
),
),
};
}
/** @typedef {ReturnType<typeof sourceApi>} SourceApi */
+262
View File
@@ -0,0 +1,262 @@
// @ts-check
// Captures a project folder as the canonical manifest the server accepts: every portable regular file with its
// SHA-256 and length. A file that cannot be read, a link, a special file or a name no client can store fails the
// capture with a list of them; it never becomes a deletion. Limits are checked from sizes alone, before any hashing.
import { createHash } from "node:crypto";
import { constants } from "node:fs";
import { lstat, open, readdir, realpath } from "node:fs/promises";
import { dirname, join } from "node:path";
import { ENTRYPOINT, ManifestInvalid, canonicalBytes, parseManifest } from "./manifest.mjs";
import { V1_LIMITS, overLimit } from "./limits.mjs";
import {
PathViolation,
checkInventoryPaths,
classifyPath,
excludedFolder,
prefiltered,
skippedSymlink,
validatePath,
} from "./pathPolicy.mjs";
import { readRenderOutputs, settledState } from "./projectState.mjs";
/** @typedef {import("./types.js").CaptureProblem} CaptureProblem @typedef {import("./types.js").Found} Found
* @typedef {import("./types.js").Walk} Walk @typedef {import("./types.js").Capture} Capture */
/** Files up to this size use one bounded buffer; larger files stream from the verified handle. */
const INLINE_HASH_BYTES = 4 * 1024 * 1024;
const HASHING_AT_ONCE = 4;
/** @param {string} root @param {string} rel @param {import("node:fs").Dirent} entry @param {Walk} walk
* @param {string[]} dirs */
async function visit(root, rel, entry, walk, dirs) {
if (entry.isDirectory()) {
if (!excludedFolder(rel)) dirs.push(rel);
return;
}
const action = classifyPath(rel).client_action;
if (action === "exclude") return;
if (action === "wait_then_exclude") return void walk.waiting.push(rel);
if (entry.isSymbolicLink()) {
if (skippedSymlink(rel)) walk.skippedLinks.push(rel);
else walk.problems.push({ path: rel, problem: "symlink" });
return;
}
if (!entry.isFile()) return void walk.problems.push({ path: rel, problem: "special-file" });
await visitFile(root, rel, walk);
}
/** @param {string} root @param {string} rel @param {Walk} walk */
async function visitFile(root, rel, walk) {
try {
validatePath(rel);
} catch (error) {
if (!(error instanceof PathViolation)) throw error;
return void walk.problems.push({ path: rel, problem: "invalid-name", code: error.code });
}
const full = join(root, rel);
try {
const { size, mtimeMs, ino, dev } = await lstat(full);
walk.files.push({ path: rel, full, size, mtimeMs, ino, dev });
} catch {
walk.problems.push({ path: rel, problem: "unreadable" });
}
}
/** A folder that was a link, or another folder, by the time it was read. */
class FolderMoved extends Error {}
/** A folder's entries, read only while the same real folder stands at its path before and after: one swapped for a
* link to somewhere else is never listed. @param {string} root @param {string} dir */
async function listFolder(root, dir) {
const full = join(root, dir);
const before = await lstat(full);
if (!before.isDirectory()) throw new FolderMoved();
const entries = await readdir(full, { withFileTypes: true });
const after = await lstat(full);
if (!after.isDirectory() || after.ino !== before.ino || after.dev !== before.dev)
throw new FolderMoved();
return entries;
}
/** Every file a capture would list, and what keeps one from being listed. @param {string} root
* @param {(rel: string) => boolean} renderOutput @returns {Promise<Walk>} */
async function walkProject(root, renderOutput) {
/** @type {Walk} */
const walk = { files: [], problems: [], waiting: [], skippedLinks: [] };
const dirs = [""];
for (let dir = dirs.pop(); dir !== undefined; dir = dirs.pop()) {
let entries;
try {
entries = await listFolder(root, dir);
} catch (error) {
const changed = error instanceof FolderMoved;
walk.problems.push({ path: dir, problem: changed ? "changed" : "unreadable" });
continue;
}
for (const entry of entries) {
const rel = dir ? `${dir}/${entry.name}` : entry.name;
if (prefiltered(entry.name) || renderOutput(rel)) continue;
await visit(root, rel, entry, walk, dirs);
}
}
return walk;
}
/** The manifest's canonical length, from paths and sizes alone: every SHA-256 is 64 characters.
* @param {Found[]} files */
const manifestBytes = (files) =>
canonicalBytes({
format_version: 1,
entrypoint: ENTRYPOINT,
files: files.map(({ path, size }) => ({ path, sha256: "0".repeat(64), size })),
}).length;
/** A file changed this recently can be written again within its time's granularity, keeping its size and time. */
const SETTLED_MS = 2_000;
/** A file's SHA-256 by path, size, time and inode, kept for the launch so a check after a focus reads no bytes.
* Only a settled file's hash is kept. */
export function hashMemo(now = Date.now) {
/** @type {Map<string, { size: number, mtimeMs: number, ino: number, sha256: string }>} */
const known = new Map();
return {
/** @param {Found} file */
get: (file) => {
const kept = known.get(file.full);
const same =
kept?.size === file.size && kept.mtimeMs === file.mtimeMs && kept.ino === file.ino;
return same ? kept.sha256 : null;
},
/** @param {Found} file @param {string} sha256 */
set: (file, sha256) => {
if (now() - file.mtimeMs > SETTLED_MS) known.set(file.full, { ...file, sha256 });
},
};
}
/** The file the walk listed, as an open handle: never through a link at its own name (O_NOFOLLOW), and only when the
* opened file is the same regular file (inode, device, size) found inside the project, so a file or folder swapped
* for a link after the walk is refused rather than followed. Null when it is not. @param {string} root
* @param {Found} file @param {typeof realpath} [resolve] */
async function openListed(root, file, resolve = realpath) {
const handle = await open(file.full, constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0)).catch(
(/** @type {NodeJS.ErrnoException} */ error) => {
if (error.code === "ELOOP" || error.code === "EMLINK") return null;
throw error;
},
);
if (!handle) return null;
try {
const opened = await handle.stat();
const same =
opened.isFile() &&
opened.ino === file.ino &&
opened.dev === file.dev &&
opened.size === file.size &&
(await resolve(dirname(file.full))) === join(root, dirname(file.path));
if (same) return handle;
} catch (error) {
await handle.close();
throw error;
}
await handle.close();
return null;
}
/** The file's SHA-256, read through the handle `openListed` checked; null when the file is not the one listed, or its
* bytes changed while being read. `root` is the project's real path; `resolve` finds a folder's real path.
* @param {string} root @param {Found} file @param {typeof realpath} [resolve] */
export async function hashListedFile(root, file, resolve = realpath) {
const handle = await openListed(root, file, resolve);
if (!handle) return null;
try {
const sha256 =
file.size <= INLINE_HASH_BYTES
? await handle
.readFile()
.then((bytes) =>
bytes.length === file.size ? createHash("sha256").update(bytes).digest("hex") : null,
)
: await hashHandle(handle);
const after = await handle.stat();
return after.size === file.size && after.mtimeMs === file.mtimeMs ? sha256 : null;
} finally {
await handle.close();
}
}
/** @param {string} root @param {Found[]} files @param {ReturnType<typeof hashMemo>} memo
* @returns {Promise<{ hashed: import("./types.js").ManifestFile[], problems: CaptureProblem[] }>} */
async function hashAll(root, files, memo) {
/** @type {import("./types.js").ManifestFile[]} */
const hashed = [];
/** @type {CaptureProblem[]} */
const problems = [];
const queue = [...files];
const worker = async () => {
for (let file = queue.shift(); file; file = queue.shift()) {
const sha256 = memo.get(file) ?? (await hashListedFile(root, file).catch(() => undefined));
if (sha256 === undefined) problems.push({ path: file.path, problem: "unreadable" });
else if (sha256 === null) problems.push({ path: file.path, problem: "changed" });
else {
memo.set(file, sha256);
hashed.push({ path: file.path, sha256, size: file.size });
}
}
};
await Promise.all(Array.from({ length: HASHING_AT_ONCE }, worker));
return { hashed, problems };
}
/** Problems that keep the files found from being one manifest: two names one disk stores as one, no index.html.
* @param {Found[]} files @returns {CaptureProblem[]} */
function inventoryProblems(files) {
if (!files.some((file) => file.path === ENTRYPOINT))
return [{ path: ENTRYPOINT, problem: "no-entrypoint" }];
try {
checkInventoryPaths(files.map((file) => file.path));
return [];
} catch (error) {
if (!(error instanceof PathViolation)) throw error;
return [{ path: error.path, problem: "invalid-name", code: error.code }];
}
}
/** Captures `root` (a project's real path). `memo` reuses hashes of unchanged files; a send should pass a fresh one.
* @param {string} root @param {{ memo?: ReturnType<typeof hashMemo>, limits?: import("./types.js").Limits }} [options]
* @returns {Promise<Capture>} */
export async function captureProject(root, { memo = hashMemo(), limits = V1_LIMITS } = {}) {
const renderOutput = await readRenderOutputs(root);
const walk = await walkProject(root, renderOutput);
const problems = walk.problems.length ? walk.problems : inventoryProblems(walk.files);
if (problems.length) return { at: "problems", problems };
const over = overLimit(walk.files, manifestBytes(walk.files), limits);
if (over) return { at: "too-big", over };
const { hashed, problems: unhashed } = await hashAll(await realpath(root), walk.files, memo);
if (unhashed.length) return { at: "problems", problems: unhashed };
try {
const manifest = parseManifest({ format_version: 1, entrypoint: ENTRYPOINT, files: hashed });
const settled = await settledState(root);
return {
at: "captured",
manifest,
settled,
waiting: walk.waiting,
skippedLinks: walk.skippedLinks,
};
} catch (error) {
if (!(error instanceof ManifestInvalid)) throw error;
const first = error.violations[0];
return {
at: "problems",
problems: [{ path: first?.path ?? "", problem: "invalid-name", code: error.code }],
};
}
}
/** @param {import("node:fs/promises").FileHandle} handle */
async function hashHandle(handle) {
const hash = createHash("sha256");
for await (const chunk of handle.createReadStream({ autoClose: false })) hash.update(chunk);
return hash.digest("hex");
}
+13
View File
@@ -0,0 +1,13 @@
// @ts-check
// Load the bundled snapshot policy and Unicode registries from JSON or Brotli.
import { existsSync, readFileSync } from "node:fs";
import { brotliDecompressSync } from "node:zlib";
/** A registry's parsed JSON. @param {string} name a file in contract/, such as "path-policy.json" @returns {any} */
export function readContract(name) {
const packed = new URL(`./contract/${name}.br`, import.meta.url);
const text = existsSync(packed)
? brotliDecompressSync(readFileSync(packed)).toString("utf8")
: readFileSync(new URL(`./contract/${name}`, import.meta.url), "utf8");
return JSON.parse(text);
}
@@ -0,0 +1,39 @@
UNICODE LICENSE V3
COPYRIGHT AND PERMISSION NOTICE
Copyright © 1991-2026 Unicode, Inc.
NOTICE TO USER: Carefully read the following legal agreement. BY
DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR
SOFTWARE, YOU UNEQUIVOCALLY ACCEPT, AND AGREE TO BE BOUND BY, ALL OF THE
TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE, DO NOT
DOWNLOAD, INSTALL, COPY, DISTRIBUTE OR USE THE DATA FILES OR SOFTWARE.
Permission is hereby granted, free of charge, to any person obtaining a
copy of data files and any associated documentation (the "Data Files") or
software and any associated documentation (the "Software") to deal in the
Data Files or Software without restriction, including without limitation
the rights to use, copy, modify, merge, publish, distribute, and/or sell
copies of the Data Files or Software, and to permit persons to whom the
Data Files or Software are furnished to do so, provided that either (a)
this copyright and permission notice appear with all copies of the Data
Files or Software, or (b) this copyright and permission notice appear in
associated Documentation.
THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
THIRD PARTY RIGHTS.
IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE
BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES,
OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA
FILES OR SOFTWARE.
Except as contained in this notice, the name of a copyright holder shall
not be used in advertising or otherwise to promote the sale, use or other
dealings in these Data Files or Software without prior written
authorization of the copyright holder.
@@ -0,0 +1,311 @@
{
"policy_version": "portable-source-v4",
"manifest": {
"format_version": 1,
"entrypoint": "index.html",
"fields": ["format_version", "entrypoint", "files"],
"file_fields": ["path", "sha256", "size"],
"sha256": "64 lowercase hex characters: the SHA-256 of the file's exact bytes",
"size": "the file's byte length, an integer from 0 to 9007199254740991 (2^53 - 1, the largest integer every JSON client reads exactly)",
"same_digest_same_size": "two entries with the same sha256 must have the same size",
"not_in_manifest": "upload state, latest pointer, authors, comments, timestamps, URLs, S3 keys and sync binding metadata"
},
"canonical_encoding": {
"files_order": "ascending by the UTF-8 bytes of path (not UTF-16 code units: JavaScript's default sort differs above U+FFFF)",
"object_keys": "sorted; every key is ASCII, so byte, code-point and UTF-16 order agree",
"whitespace": "none outside strings: separators are ',' and ':'",
"strings": "UTF-8; only '\"' and '\\' are escaped, because policy-valid paths contain no control characters or lone surrogates",
"numbers": "integers in decimal with no sign, fraction or exponent",
"manifest_sha256": "lowercase hex SHA-256 of the canonical bytes"
},
"unicode": {
"version": "15.1.0",
"table": "unicode-15.1.0.json",
"assigned_only": "every code point must be assigned in Unicode 15.1; unassigned and future code points are refused",
"normalization": "every path must already be NFC",
"comparison_key": "NFC(full_casefold(NFC(segment))), with default full case folding (CaseFolding.txt status C and F)"
},
"path": {
"max_path_bytes": 1024,
"max_segment_bytes": 240,
"separator": "/",
"forbidden_characters": "<>:\"|?*\\",
"reserved_segment_regex": "(?:con|prn|aux|nul|com[1-9\u00b9\u00b2\u00b3]|lpt[1-9\u00b9\u00b2\u00b3])(?:\\..*)?",
"reserved_segment_match": "full match against each segment's comparison key, so CON, Con.txt and nul.tar.gz are all reserved",
"canonical_spelling_roots": [".media", ".hyperframes"],
"checks": [
{
"code": "path_empty",
"rule": "the path is not the empty string"
},
{
"code": "path_invalid_character",
"rule": "no C0 or C1 control character, DEL, lone surrogate or forbidden character"
},
{
"code": "path_unassigned_code_point",
"rule": "every code point is assigned in Unicode 15.1"
},
{
"code": "path_not_nfc",
"rule": "the path equals its NFC form"
},
{
"code": "path_too_long",
"rule": "at most max_path_bytes UTF-8 bytes"
},
{
"code": "path_not_relative",
"rule": "does not start with '/'"
},
{
"code": "path_empty_segment",
"rule": "no empty segment ('a//b', a trailing '/')"
},
{
"code": "path_dot_segment",
"rule": "no '.' or '..' segment"
},
{
"code": "segment_too_long",
"rule": "each segment is at most max_segment_bytes UTF-8 bytes"
},
{
"code": "segment_trailing_dot_or_space",
"rule": "no segment ends in '.' or ' ' (Windows strips them)"
},
{
"code": "reserved_name",
"rule": "no segment matches reserved_segment_regex"
},
{
"code": "reserved_path_alias",
"rule": "a first segment whose comparison key equals a canonical_spelling_roots entry is spelled exactly like it, and an allowed rule that matches the folded path also matches the path as spelled"
},
{
"code": "excluded_path",
"rule": "the first rule matching the folded path does not have server_action 'reject'"
}
],
"check_order": "checks run in the order listed; the first failing check names the error"
},
"matching": {
"syntax": "each rule's regex must match the whole path (Python re.fullmatch, JavaScript ^(?:regex)$ with the 's' and 'u' flags)",
"subject": "the folded path: every segment replaced by its comparison key, joined with '/'",
"first_match_wins": true,
"case": "rule regexes are written in lowercase, because they are matched against the case-folded path",
"client_actions": {
"include": "list the file",
"exclude": "leave the file out",
"wait_then_exclude": "wait for the writer to finish (the reserved sibling may be incomplete), then leave the marker out"
},
"rule_order": "every rule that rejects at any depth comes before the first allow rule, so a credential, environment file, package-manager config, VCS or dependency directory, agent state, build cache or OS metadata inside an allowed .hyperframes subtree stays out"
},
"rules": [
{
"id": "media-lock",
"regex": "\\.media/\\.lock",
"client_action": "exclude",
"server_action": "reject",
"basis": "Process lock held by the media writer, not project content"
},
{
"id": "media-reservation",
"regex": "\\.media/(?:audio/(?:bgm|sfx|voice)|images|video|luts|recipes)/[^/]+\\.hf[0-9a-f]{6}\\.tmp",
"client_action": "wait_then_exclude",
"server_action": "reject",
"basis": "Media id reservation: the sibling asset may still be incomplete, so capture waits for the writer to finish"
},
{
"id": "media-ledger",
"regex": "\\.media/manifest\\.jsonl",
"client_action": "exclude",
"server_action": "reject",
"basis": "Generated media metadata that the project does not use for preview or rendering"
},
{
"id": "unfinished-writes",
"regex": "(?:[^/]+/)*(?:[^/]+\\.hf[0-9a-f]{6}\\.tmp|[^/]*\\.(?:tmp|part|swp)|[^/]+~)",
"client_action": "exclude",
"server_action": "reject",
"basis": "Unfinished or abandoned writes at any depth: core's atomic-write temp <file>.hfXXXXXX.tmp, *.tmp, *.part, editor *.swp and *~"
},
{
"id": "render-temp-dirs",
"regex": "(?:[^/]+/)*(?:(?:work-[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}|hf-render|\\.[^/]+\\.hf-transaction)-[A-Za-z0-9]{6}|[^/]+\\.assemble-work-[A-Za-z0-9]{6})(?:/.*)?",
"client_action": "exclude",
"server_action": "reject",
"basis": "Render work directories at any depth (frames, often GBs): core RENDER_TEMP_DIR (packages/core/src/projectFolders.ts) plus distributed assembly's <output>.assemble-work-XXXXXX"
},
{
"id": "agent-client-state",
"regex": "(?:[^/]+/)*(?:\\.claude|\\.codex|\\.agents|\\.cursor|\\.vscode|\\.idea)(?:/.*)?",
"client_action": "exclude",
"server_action": "reject",
"basis": "Agent and IDE state at any depth; Desktop vendors .claude/skills into every project and regenerates it"
},
{
"id": "dependencies-and-git",
"regex": "(?:[^/]+/)*(?:\\.git|\\.svn|\\.hg|node_modules)(?:/.*)?",
"client_action": "exclude",
"server_action": "reject",
"basis": "Version control and installed dependencies at any depth"
},
{
"id": "build-caches",
"regex": "(?:[^/]+/)*(?:\\.cache|\\.vite|\\.next|\\.turbo|\\.parcel-cache)(?:/.*)?",
"client_action": "exclude",
"server_action": "reject",
"basis": "Build tool caches at any depth"
},
{
"id": "environment-secrets",
"regex": "(?:[^/]+/)*\\.env(?:rc|\\.[^/]+)?",
"client_action": "exclude",
"server_action": "reject",
"basis": "Credentials: .env, every .env.* (templates included: real keys land there and no reader needs them) and .envrc, at any depth"
},
{
"id": "package-manager-config",
"regex": "(?:[^/]+/)*(?:\\.npmrc|\\.yarnrc|\\.yarnrc\\.yml|pip\\.conf)",
"client_action": "exclude",
"server_action": "reject",
"basis": "May hold registry tokens, and nothing in a HyperFrames project reads them; stripping credentials would change the bytes"
},
{
"id": "credential-files",
"regex": "(?:[^/]+/)*(?:\\.netrc|\\.pypirc|\\.git-credentials|[^/]*\\.(?:pem|p12|pfx)|id_rsa[^/]*|id_ed25519[^/]*|(?:\\.ssh|\\.aws)(?:/.*)?)",
"client_action": "exclude",
"server_action": "reject",
"basis": "Credential files by name at any depth. *.key is deliberately absent: Keynote decks are plausible project sources"
},
{
"id": "os-metadata",
"regex": "(?:[^/]+/)*(?:\\.ds_store|thumbs\\.db|desktop\\.ini|\\._[^/]*|__macosx(?:/.*)?)",
"client_action": "exclude",
"server_action": "reject",
"basis": "Operating-system metadata: it changes when a folder is browsed and would show as false edits and conflicts"
},
{
"id": "portable-transcripts",
"regex": "\\.hyperframes/transcripts/(?:[^/]+/)*[^/]+\\.json",
"client_action": "include",
"server_action": "allow",
"basis": "Completed transcripts, keyed by media hash; costly to regenerate"
},
{
"id": "portable-transcription-settings",
"regex": "\\.hyperframes/transcription\\.json",
"client_action": "include",
"server_action": "allow",
"basis": "Transcription (ASR) preferences, not run state; Desktop's Duplicate copies it with the transcripts"
},
{
"id": "portable-brand-kit",
"regex": "\\.hyperframes/brand-kit\\.json",
"client_action": "include",
"server_action": "allow",
"basis": "The project's brand kit selection is authored, portable project content"
},
{
"id": "portable-studio-edits",
"regex": "\\.hyperframes/(?:studio-manual-edits|studio-motion)\\.json",
"client_action": "include",
"server_action": "allow",
"basis": "Studio edits read at render and preview time; without them a pulled project renders differently"
},
{
"id": "portable-caption-skin",
"regex": "\\.hyperframes/caption-skin\\.html",
"client_action": "include",
"server_action": "allow",
"basis": "Authored input to the caption build; rebuilding captions needs it"
},
{
"id": "portable-podcast-media",
"regex": "\\.hyperframes/prepared-assets/podcast/(?:[^/]+/)*[^./][^/]*",
"client_action": "include",
"server_action": "allow",
"basis": "Desktop's podcast camera and recorder transcodes (media/, proxies/), which the page plays by this path and a receiving client cannot regenerate without the original files; a dotfile there stays out"
},
{
"id": "sync-and-share-bindings",
"regex": "\\.hyperframes/(?:share\\.json|review\\.json|review-export\\.json|review-[^/]+\\.json|project\\.json|lambda-stack-[^/]+\\.json|(?:sync|receipts)/.*)",
"client_action": "exclude",
"server_action": "reject",
"basis": "External bindings and sync receipts: share and review links (review files carry the owner's email), the CLI publish id and deploy stacks. A copy must start unbound, so these never travel"
},
{
"id": "known-hyperframes-state",
"regex": "\\.hyperframes/(?:comments\\.jsonl|agent-runs\\.jsonl|app-history\\.jsonl|chat-starts\\.json|agent-handoff\\.json|agent-handoff-read\\.json|history-id|video-state\\.json|film-snapshot\\.json|moved-from\\.jsonl|hf-ids-stamped\\.json|settled-signature|render-outputs\\.json|render-outputs-lock|project\\.lock|history-turn\\.json|history-turns\\.json|voices\\.json|asset-creation\\.json|lesson\\.json|element-map\\.txt|\\.gitignore|(?:live-runs|run-traces|agent-sessions|agent-steps|video-state-lock|backup|preview|history|undo|artifacts|heygen-recipes|heygen-reperform|fixes|heygen-jobs|heygen-paid|podcast|prepared-assets|requests|recordings|frames|refs|references|review|heygen-previews|frame-packets|anim-map|contrast|bin|diagnostics)(?:/.*)?)",
"client_action": "exclude",
"server_action": "reject",
"basis": "Known client state: comments, chat and run logs; machine pointers, locks and caches; personal data (photo and voice copies) and account-bound job state, podcast job state included; studio preview caches under prepared-assets/ other than the podcast media above"
},
{
"id": "unknown-hyperframes-state",
"regex": "\\.hyperframes(?:/.*)?",
"client_action": "exclude",
"server_action": "reject",
"basis": "Reserved client state area: only the named portable files above travel"
},
{
"id": "root-generated-output",
"regex": "(?:\\.thumbnails|\\.transcode-cache|\\.waveform-cache|renders|snapshots)(?:/.*)?",
"client_action": "exclude",
"server_action": "reject",
"basis": "Caches and render or snapshot output at the project root"
},
{
"id": "root-build-output",
"regex": "(?:dist|coverage)(?:/.*)?",
"client_action": "exclude",
"server_action": "reject",
"basis": "Web build and test-coverage output at the project root, which the CLI publish leaves out and render never reads (build/ and outputs/ travel, because the publish ships them)"
},
{
"id": "authoring-scratch",
"regex": "(?:remotion-src|moments|references/moments)(?:/.*)?",
"client_action": "exclude",
"server_action": "reject",
"basis": "Authoring scratch at the project root: a Remotion port's source and the moment working copies with their references, none of which preview or render reads"
},
{
"id": "reference-video",
"regex": "\\.viral-clone(?:/.*)?",
"client_action": "exclude",
"server_action": "reject",
"basis": "Third-party reference video, large and re-downloadable"
},
{
"id": "default-authored-file",
"regex": ".+",
"client_action": "include",
"server_action": "allow",
"basis": "Authored source and config (index.html, compositions/, scenes/, assets/, hyperframes.json, meta.json, project.json, package.json, lockfiles, .hyperframesignore, .gitignore), portable briefs, AGENTS.md and CLAUDE.md, completed .media assets with unused alternatives, .media/index.md, .media/preferences.json and .media/CREDITS.md, build/ and outputs/, and any unknown regular file outside a reserved area"
}
],
"client_notes": {
"project_json": "Root project.json is portable project content and is included. It is not sync identity: the owner workspace, composition, base snapshot, revision and receipts live outside the inventory, never in project.json.",
"capture_preconditions": "Do not capture while .hyperframes/history-turn.json exists (an agent turn is open) or while .hyperframes/project.lock is held.",
"render_outputs": "Leave out every path listed in .hyperframes/render-outputs.json: render outputs written outside renders/ are recorded there, and the server cannot know them.",
"complete_capture": "A file the client cannot read fails the capture; it never becomes a deletion. The exclusions above and the symlinks client_symlinks skips are the only files left out.",
"symlinks": "Symlinks and special files fail the capture, listing them; they are neither followed nor skipped, except the symlinks client_symlinks skips.",
"empty_directories": "Empty directories are not represented. Pull never deletes a local empty directory."
},
"client_prefilter": {
"names": ["Icon\r"],
"rule": "skip a file whose last segment is one of these before running any path check; macOS writes 'Icon\\r' for a folder with a custom icon, and the carriage return would otherwise fail the whole capture with path_invalid_character"
},
"client_symlinks": {
"default": "fail_capture",
"skip_prefixes": ["assets/podcast/"],
"match": "a symlink whose folded path (each segment replaced by its comparison key, as for rules) starts with a listed prefix, at any depth below it",
"rule": "skip such a symlink instead of failing the capture, and on Pull leave such a local symlink in place as local state; a regular file under the prefix is captured like any other, and every other symlink or special file fails the capture",
"basis": "Desktop's podcast intake links the original camera files, which live outside the project, under assets/podcast/, and the page plays the transcodes under .hyperframes/prepared-assets/podcast/; real files there, such as the joins under assets/podcast/joins/, travel normally"
},
"stored_manifest": {
"legacy_agent_state_regex": "(?:[^/]+/)*(?:\\.claude|\\.agents|\\.cursor|\\.vscode|\\.idea)(?:/.*)?",
"rule": "Only stored snapshot reads use the v3 agent-state rule to retain immutable manifest digests. New captures and uploads use the current rules. Clients never install excluded paths; the server never signs their download URLs."
}
}
File diff suppressed because it is too large Load Diff
+77
View File
@@ -0,0 +1,77 @@
// @ts-check
// Map sync API error codes to stable client error kinds.
/** @typedef {import("./types.js").SyncErrorKind} SyncErrorKind */
/** @type {Record<string, SyncErrorKind>} */
const KIND_BY_CODE = {
hyperframes_composition_sync_not_enabled: "not-enabled",
hf_subscription_required: "plan-required",
hyperframes_composition_sync_limit_reached: "limit-reached",
byok_not_supported: "byok",
resource_access_denied: "no-write-role",
forbidden: "forbidden",
hyperframes_composition_not_found: "composition-not-found",
resource_not_found: "not-found",
hyperframes_composition_upload_too_large: "too-large",
hyperframes_composition_upload_reference_invalid: "reference-invalid",
hyperframes_composition_upload_not_pending: "upload-not-pending",
hyperframes_composition_active_uploads_exceeded: "active-uploads-exceeded",
hyperframes_composition_upload_file_not_writable: "file-not-writable",
conflict: "idempotency-key-reused",
request_in_progress: "request-in-progress",
hyperframes_composition_snapshot_not_downloadable: "snapshot-not-downloadable",
hyperframes_composition_conflict_not_open: "conflict-not-open",
hyperframes_composition_download_expired: "download-expired",
invalid_parameter: "invalid-parameter",
rate_limit_exceeded: "rate-limited",
service_unavailable: "unavailable",
};
/** A sync route's refusal or failure. `status` is 0 when the server was never reached. */
export class SyncApiError extends Error {
/** @param {SyncErrorKind} kind @param {{ status?: number, code?: string | null, param?: string | null,
* requestId?: string | null, retryAfterMs?: number | null, detail?: string }} [about] */
constructor(kind, about = {}) {
super(`composition sync: ${kind}${about.code ? ` (${about.code})` : ""}`);
this.name = "SyncApiError";
this.kind = kind;
this.status = about.status ?? 0;
this.code = about.code ?? null;
this.param = about.param ?? null;
this.requestId = about.requestId ?? null;
this.retryAfterMs = about.retryAfterMs ?? null;
this.detail = about.detail ?? null;
}
}
/** The kind for an HTTP status whose code this client does not know. @param {number} status
* @returns {SyncErrorKind} */
function kindOfStatus(status) {
if (status === 404) return "not-found";
if (status === 429) return "rate-limited";
if (status === 503) return "unavailable";
if (status >= 500) return "server-error";
return "refused";
}
/** The typed error for a refusal: the envelope's code when it is one of the sync routes', else the status's.
* @param {Response} reply @returns {Promise<SyncApiError>} */
export async function refusalOf(reply) {
const payload = await reply.json().catch(() => null);
/** @type {{ code?: unknown, param?: unknown, request_id?: unknown, message?: unknown } | undefined} */
const error = payload?.error;
const code = typeof error?.code === "string" ? error.code : null;
const seconds = Number(reply.headers.get("retry-after") ?? Number.NaN);
return new SyncApiError((code && KIND_BY_CODE[code]) || kindOfStatus(reply.status), {
status: reply.status,
code,
param: typeof error?.param === "string" ? error.param : null,
requestId:
typeof error?.request_id === "string"
? error.request_id
: reply.headers.get("x-heygen-request-id"),
retryAfterMs: seconds >= 0 ? seconds * 1000 : null,
...(typeof error?.message === "string" ? { detail: error.message } : {}),
});
}
+48
View File
@@ -0,0 +1,48 @@
// @ts-check
// Check the workspace upload limits before hashing or sending files.
const MIB = 1024 * 1024;
const GIB = 1024 * MIB;
/** @typedef {import("./types.js").Limits} Limits @typedef {import("./types.js").LimitName} LimitName */
/** @type {Limits} */
export const V1_LIMITS = Object.freeze({
maxFiles: 10_000,
maxFileBytes: 5 * GIB,
maxProjectBytes: 10 * GIB,
maxManifestBytes: 8 * MIB,
});
/** How many of the biggest files an over-limit answer names. */
const NAMED = 10;
/** A manifest's file count, largest file, total and canonical size; every listed file counts, reused ones too.
* @param {readonly { size: number }[]} files @param {number} manifestBytes */
const measure = (files, manifestBytes) => ({
fileCount: files.length,
largestFileBytes: files.reduce((most, file) => Math.max(most, file.size), 0),
projectBytes: files.reduce((sum, file) => sum + file.size, 0),
manifestBytes,
});
/** The first limit broken, in the server's fixed order (file count, file size, project size, manifest size), with
* the biggest files first; null when the push fits. @param {readonly { path: string, size: number }[]} files
* @param {number} manifestBytes @param {Limits} [limits] @returns {import("./types.js").OverLimit | null} */
export function overLimit(files, manifestBytes, limits = V1_LIMITS) {
const size = measure(files, manifestBytes);
/** @type {[LimitName, number, number][]} */
const checks = [
["file_count", size.fileCount, limits.maxFiles],
["file_size", size.largestFileBytes, limits.maxFileBytes],
["project_size", size.projectBytes, limits.maxProjectBytes],
["manifest_size", size.manifestBytes, limits.maxManifestBytes],
];
const broken = checks.find(([, actual, maximum]) => actual > maximum);
if (!broken) return null;
const biggest = [...files]
.sort((a, b) => b.size - a.size || (a.path < b.path ? -1 : 1))
.slice(0, NAMED)
.map(({ path, size: bytes }) => ({ path, size: bytes }));
return { limit: broken[0], actual: broken[1], maximum: broken[2], biggest };
}
+154
View File
@@ -0,0 +1,154 @@
// @ts-check
// Serialize snapshots to canonical UTF-8 bytes so clients and the server compute the same digest.
import { createHash } from "node:crypto";
import { PathViolation, checkInventoryPaths, validatePath } from "./pathPolicy.mjs";
const FORMAT_VERSION = 1;
export const ENTRYPOINT = "index.html";
const MAX_SAFE_SIZE = Number.MAX_SAFE_INTEGER;
const SHA256 = /^[0-9a-f]{64}$/;
/** @typedef {import("./types.js").ManifestFile} ManifestFile @typedef {import("./types.js").Violation} Violation
* @typedef {import("./types.js").ManifestDocument} ManifestDocument @typedef {import("./types.js").Manifest} Manifest */
/** Why a manifest was refused; `code` is the first violation's, in the server's order. */
export class ManifestInvalid extends Error {
/** @param {Violation[]} violations */
constructor(violations) {
super(violations.map((violation) => violation.code).join(", "));
this.name = "ManifestInvalid";
this.violations = violations;
this.code = violations[0]?.code ?? "invalid_schema";
}
}
/** @param {unknown} value @returns {value is Record<string, unknown>} */
const isObject = (value) => typeof value === "object" && value !== null && !Array.isArray(value);
/** @param {unknown} error @returns {Violation} */
const violationOf = (error) => {
if (!(error instanceof PathViolation)) throw error;
return { code: error.code, path: error.path, ...(error.rule ? { rule: error.rule } : {}) };
};
/** A schema violation for each field `fields` does not name.
* @param {Record<string, unknown>} value @param {readonly string[]} fields @param {Violation[]} out */
function unknownFields(value, fields, out) {
for (const key of Object.keys(value))
if (!fields.includes(key)) out.push({ code: "invalid_schema" });
}
/** @param {unknown} raw @param {Violation[]} out @param {boolean} stored @returns {ManifestFile | null} */
function fileOf(raw, out, stored) {
if (!isObject(raw)) return (out.push({ code: "invalid_schema" }), null);
const before = out.length;
const { path, sha256, size } = raw;
if (typeof path !== "string") out.push({ code: "invalid_schema" });
else {
try {
validatePath(path, stored);
} catch (error) {
out.push(violationOf(error));
}
}
checkBlob(sha256, size, out);
unknownFields(raw, ["path", "sha256", "size"], out);
return out.length === before
? { path: String(path), sha256: String(sha256), size: Number(size) }
: null;
}
/** @param {unknown} sha256 @param {unknown} size @param {Violation[]} out */
function checkBlob(sha256, size, out) {
if (typeof sha256 !== "string") out.push({ code: "invalid_schema" });
else if (!SHA256.test(sha256)) out.push({ code: "invalid_sha256" });
if (typeof size !== "number" || !Number.isInteger(size)) out.push({ code: "invalid_schema" });
else if (size < 0 || size > MAX_SAFE_SIZE) out.push({ code: "invalid_size" });
}
/** UTF-8 byte order, which is code point order; JavaScript's default sort compares UTF-16 units instead.
* @param {ManifestFile} a @param {ManifestFile} b */
const byUtf8 = (a, b) => Buffer.compare(Buffer.from(a.path, "utf8"), Buffer.from(b.path, "utf8"));
/** The files in canonical order, or the inventory's first violation. @param {ManifestFile[]} files
* @param {Violation[]} out */
function canonicalInventory(files, out) {
const ordered = [...files].sort(byUtf8);
/** @type {Map<string, number>} */
const sizeByDigest = new Map();
for (const file of ordered) {
const size = sizeByDigest.get(file.sha256) ?? file.size;
sizeByDigest.set(file.sha256, size);
if (size !== file.size)
return (out.push({ code: "inconsistent_digest_size", path: file.path }), null);
}
try {
checkInventoryPaths(ordered.map((file) => file.path));
} catch (error) {
return (out.push(violationOf(error)), null);
}
return ordered;
}
/** @param {unknown} raw @param {Violation[]} out @param {boolean} stored */
function filesOf(raw, out, stored) {
if (!Array.isArray(raw)) return (out.push({ code: "invalid_schema" }), null);
const before = out.length;
const files = raw.map((item) => fileOf(item, out, stored));
if (out.length !== before) return null;
return canonicalInventory(/** @type {ManifestFile[]} */ (files), out);
}
/** The canonical bytes of a valid document. @param {ManifestDocument} document */
export function canonicalBytes(document) {
const files = document.files
.map(
({ path, sha256, size }) =>
`{"path":${JSON.stringify(path)},"sha256":${JSON.stringify(sha256)},"size":${size}}`,
)
.join(",");
const text = `{"entrypoint":${JSON.stringify(document.entrypoint)},"files":[${files}],"format_version":${document.format_version}}`;
return Buffer.from(text, "utf8");
}
/** Validates a manifest document as the server does; throws ManifestInvalid. @param {unknown} raw
* @param {boolean} [stored] @returns {Manifest} */
export function parseManifest(raw, stored = false) {
if (!isObject(raw)) throw new ManifestInvalid([{ code: "invalid_schema" }]);
/** @type {Violation[]} */
const out = [];
const { format_version: version, entrypoint } = raw;
if (!("format_version" in raw) || typeof version !== "number" || !Number.isInteger(version))
out.push({ code: "invalid_schema" });
else if (version !== FORMAT_VERSION) out.push({ code: "unsupported_format_version" });
if (typeof entrypoint !== "string") out.push({ code: "invalid_schema" });
else if (entrypoint !== ENTRYPOINT) out.push({ code: "invalid_entrypoint" });
const files = filesOf(raw.files, out, stored);
unknownFields(raw, ["format_version", "entrypoint", "files"], out);
if (out.length) throw new ManifestInvalid(out);
const ordered = /** @type {ManifestFile[]} */ (files);
if (!ordered.some((file) => file.path === ENTRYPOINT))
throw new ManifestInvalid([{ code: "entrypoint_not_listed" }]);
/** @type {ManifestDocument} */
const document = { format_version: FORMAT_VERSION, entrypoint: ENTRYPOINT, files: ordered };
const canonical = canonicalBytes(document);
return { document, canonical, sha256: createHash("sha256").update(canonical).digest("hex") };
}
/** Stored manifest bytes, which must already be canonical so their hash is the snapshot's manifest_sha256.
* @param {Uint8Array} body @returns {Manifest} */
export function parseCanonicalManifest(body) {
let document;
try {
document = JSON.parse(new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(body));
} catch {
throw new ManifestInvalid([{ code: "invalid_json" }]);
}
const manifest = parseManifest(document);
if (!manifest.canonical.equals(Buffer.from(body)))
throw new ManifestInvalid([{ code: "not_canonical" }]);
return manifest;
}
/** A published snapshot or local base can retain v3 paths; its bytes and digest remain immutable.
* @param {unknown} raw @returns {Manifest} */
export const parseStoredManifest = (raw) => parseManifest(raw, true);
+197
View File
@@ -0,0 +1,197 @@
// @ts-check
// Validate snapshot paths in the order defined by the bundled portable-source-v4 policy.
import { readContract } from "./contract.mjs";
import { UNICODE_VERSION, allAssigned, comparisonKey, nfc } from "./unicode.mjs";
const PATH_POLICY_VERSION = "portable-source-v4";
/** @typedef {import("./types.js").PathRule} PathRule @typedef {import("./types.js").PolicyDocument} PolicyDocument */
/** A path the policy refuses: `code` is the golden vectors' shared vocabulary, `rule` the excluding rule's id. */
export class PathViolation extends Error {
/** @param {string} code @param {string} path @param {string} [rule] */
constructor(code, path, rule) {
super(`${code}: ${JSON.stringify(path)}${rule ? ` (${rule})` : ""}`);
this.name = "PathViolation";
this.code = code;
this.path = path;
this.rule = rule ?? null;
}
}
const ASCII = /^\p{ASCII}*$/u;
/** @param {string} pattern */
const whole = (pattern) => new RegExp(`^(?:${pattern})$`, "su");
/** @param {string} text */
const escaped = (text) => text.replace(/[\\\]^-]/g, "\\$&");
/** @param {PolicyDocument} document */
function compile(document) {
if (document.policy_version !== PATH_POLICY_VERSION)
throw new Error(`path policy is ${document.policy_version}, expected ${PATH_POLICY_VERSION}`);
if (document.unicode.version !== UNICODE_VERSION)
throw new Error(
`path policy pins Unicode ${document.unicode.version}, the table is ${UNICODE_VERSION}`,
);
if (document.rules.at(-1)?.regex !== ".+")
throw new Error("path policy must end with a catch-all rule, so every path is classified");
for (const rule of document.rules)
if ((rule.server_action === "allow") !== (rule.client_action === "include"))
throw new Error(`path rule ${rule.id}: the server allows exactly what clients include`);
const limits = document.path;
return {
limits,
invalidCharacter: new RegExp(
`[\\x00-\\x1f\\x7f-\\x9f\\ud800-\\udfff${escaped(limits.forbidden_characters)}]`,
"u",
),
reservedSegment: whole(limits.reserved_segment_regex),
canonicalRoots: new Map(
limits.canonical_spelling_roots.map((root) => [comparisonKey(root), root]),
),
rules: document.rules.map((rule) => ({ rule, pattern: whole(rule.regex) })),
storedRules: document.rules.map((rule) => ({
rule,
pattern: whole(
rule.id === "agent-client-state"
? document.stored_manifest.legacy_agent_state_regex
: rule.regex,
),
})),
prefilter: new Set(document.client_prefilter.names),
symlinkPrefixes: document.client_symlinks.skip_prefixes,
};
}
/** @type {ReturnType<typeof compile> | null} */
let compiled = null;
/** The policy, read and compiled when a path is first checked: an app that never syncs never reads it. */
const policy = () => (compiled ??= compile(readContract("path-policy.json")));
/** Every segment replaced by its comparison key. @param {string} path */
const foldedPath = (path) => path.split("/").map(comparisonKey).join("/");
/** The first rule matching a path's folded spelling. @param {string} path @returns {PathRule} */
export function classifyPath(path) {
const folded = foldedPath(path);
const found = policy().rules.find(({ pattern }) => pattern.test(folded));
if (!found) throw new Error("the path policy's catch-all rule did not match");
return found.rule;
}
/** Checks 1 to 10 of the registry's `path.checks`: the ones about the string itself. @param {string} path */
function checkSpelling(path) {
const { limits, invalidCharacter } = policy();
if (!path) throw new PathViolation("path_empty", path);
if (invalidCharacter.test(path)) throw new PathViolation("path_invalid_character", path);
if (!ASCII.test(path) && !allAssigned(path))
throw new PathViolation("path_unassigned_code_point", path);
if (nfc(path) !== path) throw new PathViolation("path_not_nfc", path);
if (Buffer.byteLength(path, "utf8") > limits.max_path_bytes)
throw new PathViolation("path_too_long", path);
if (path.startsWith("/")) throw new PathViolation("path_not_relative", path);
return checkSegments(path, limits);
}
/** @param {string} path @param {PolicyDocument["path"]} limits */
function checkSegments(path, limits) {
const segments = path.split("/");
if (segments.some((segment) => !segment)) throw new PathViolation("path_empty_segment", path);
if (segments.some((segment) => segment === "." || segment === ".."))
throw new PathViolation("path_dot_segment", path);
if (segments.some((segment) => Buffer.byteLength(segment, "utf8") > limits.max_segment_bytes))
throw new PathViolation("segment_too_long", path);
if (segments.some((segment) => segment.endsWith(".") || segment.endsWith(" ")))
throw new PathViolation("segment_trailing_dot_or_space", path);
return segments;
}
/** `path` unchanged when every client can store it and the policy lets it travel; throws PathViolation otherwise,
* naming the first failing check in the registry's order. @param {string} path
* @param {boolean} [stored] */
export function validatePath(path, stored = false) {
const segments = checkSpelling(path);
const keys = segments.map(comparisonKey);
const { reservedSegment, canonicalRoots } = policy();
const rules = stored ? policy().storedRules : policy().rules;
if (keys.some((key) => reservedSegment.test(key))) throw new PathViolation("reserved_name", path);
const root = canonicalRoots.get(keys[0] ?? "");
if (root !== undefined && segments[0] !== root)
throw new PathViolation("reserved_path_alias", path);
const folded = keys.join("/");
const found = rules.find(({ pattern }) => pattern.test(folded));
if (!found) throw new Error("the path policy's catch-all rule did not match");
if (found.rule.server_action === "reject")
throw new PathViolation("excluded_path", path, found.rule.id);
if (!found.pattern.test(path))
throw new PathViolation("reserved_path_alias", path, found.rule.id);
return path;
}
/** Refuses duplicates, and any two paths (or a path and a directory prefix) one filesystem would store as one name.
* Each path must already pass validatePath. @param {readonly string[]} paths */
export function checkInventoryPaths(paths) {
const seen = { spellingByKey: new Map(), fileKeys: new Set(), directoryKeys: new Set() };
for (const path of paths) {
const segments = path.split("/");
const keys = segments.map(comparisonKey);
for (let depth = 1; depth <= segments.length; depth++)
notePrefix(seen, path, {
prefix: segments.slice(0, depth).join("/"),
key: keys.slice(0, depth).join("/"),
isFile: depth === segments.length,
});
}
}
/** One prefix of `path`, against every prefix before it, in the server's order of checks.
* @param {{ spellingByKey: Map<string, string>, fileKeys: Set<string>, directoryKeys: Set<string> }} seen
* @param {string} path @param {{ prefix: string, key: string, isFile: boolean }} at */
function notePrefix(seen, path, { prefix, key, isFile }) {
const spelling = seen.spellingByKey.get(key) ?? prefix;
seen.spellingByKey.set(key, spelling);
if (isFile && seen.fileKeys.has(key))
throw new PathViolation(spelling === prefix ? "duplicate_path" : "path_collision", path);
if (spelling !== prefix) throw new PathViolation("path_collision", path);
if (isFile ? seen.directoryKeys.has(key) : seen.fileKeys.has(key))
throw new PathViolation("file_directory_collision", path);
(isFile ? seen.fileKeys : seen.directoryKeys).add(key);
}
/** The registry's `client_prefilter`: a name (macOS's `Icon\r`) skipped before any path check. @param {string} name */
export const prefiltered = (name) => policy().prefilter.has(name);
/** The registry's `client_symlinks`: a link under `assets/podcast/` is skipped instead of failing the capture.
* @param {string} path */
export const skippedSymlink = (path) =>
policy().symlinkPrefixes.some((prefix) => foldedPath(path).startsWith(prefix));
/** The literal text a rule's regex starts with, up to its first metacharacter. @param {string} regex */
function literalPrefix(regex) {
let text = "";
for (let i = 0; i < regex.length; i++) {
const char = regex[i] ?? "";
if (char === "\\" && /[./-]/.test(regex[i + 1] ?? "")) text += regex[++i];
else if (/[\w/-]/.test(char)) text += char;
else break;
}
return text;
}
/** Whether no file under folder `dir` can be listed, so a capture need not open it: its first matching rule
* excludes the folder and all below it, and no include rule before that one can reach inside. @param {string} dir */
export function excludedFolder(dir) {
const folded = foldedPath(dir);
const at = policy().rules.findIndex(
({ pattern, rule }) =>
rule.client_action !== "include" && pattern.test(folded) && pattern.test(`${folded}/x`),
);
if (at < 0) return false;
return policy()
.rules.slice(0, at)
.every(({ rule }) => {
if (rule.client_action !== "include") return true;
const prefix = literalPrefix(rule.regex);
return !(prefix.startsWith(`${folded}/`) || `${folded}/`.startsWith(prefix));
});
}
@@ -0,0 +1,59 @@
// @ts-check
// What the HyperFrames CLI keeps in a project's .hyperframes folder that a capture must honour: render outputs written
// outside renders/ (left out of every manifest), an open agent turn and a held project lock (no capture until both
// end). The formats are the pinned hyperframes CLI's, version 1 of render-outputs.json.
import { readFile } from "node:fs/promises";
import { join } from "node:path";
/** A predicate for the paths the project's render-outputs.json lists, and everything under them; none when the file
* is missing or not version 1. @param {string} root @returns {Promise<(rel: string) => boolean>} */
export async function readRenderOutputs(root) {
const raw = await readFile(join(root, ".hyperframes", "render-outputs.json"), "utf8").catch(
() => null,
);
/** @type {{ version?: unknown, paths?: unknown } | null} */
let record = null;
try {
record = raw === null ? null : JSON.parse(raw);
} catch {
record = null;
}
const paths =
record?.version === 1 && Array.isArray(record.paths)
? record.paths
.filter((path) => typeof path === "string")
.map((path) => path.replace(/\\/g, "/"))
: [];
const listed = new Set(paths);
return (rel) => listed.has(rel) || paths.some((path) => rel.startsWith(`${path}/`));
}
/** Whether a pid still runs: EPERM is a process another user owns. @param {number} pid */
function alive(pid) {
try {
process.kill(pid, 0);
return true;
} catch (error) {
return /** @type {NodeJS.ErrnoException} */ (error).code === "EPERM";
}
}
/** An open agent turn (its history-turn.json exists) or a project lock whose holder still runs.
* @param {string} root */
export async function settledState(root) {
const dir = join(root, ".hyperframes");
const turn = await readFile(join(dir, "history-turn.json")).then(
() => true,
() => false,
);
const lock = await readFile(join(dir, "project.lock"), "utf8").catch(() => null);
/** @type {{ pid?: unknown } | null} */
let holder = null;
try {
holder = lock === null ? null : JSON.parse(lock);
} catch {
holder = null;
}
const locked = lock !== null && (typeof holder?.pid !== "number" || alive(holder.pid));
return { agentTurnOpen: turn, projectLocked: locked };
}
+82
View File
@@ -0,0 +1,82 @@
// @ts-check
import { createHash } from "node:crypto";
import { constants } from "node:fs";
import { lstat, open } from "node:fs/promises";
import { join } from "node:path";
import { errorCode, mkdirInside, writeJsonAtomic } from "../storage.mjs";
import { parseStoredManifest } from "./manifest.mjs";
/** @param {unknown} value @returns {value is Record<string, unknown>} */
const object = (value) => typeof value === "object" && value !== null && !Array.isArray(value);
/** @param {unknown} raw @param {import("./types.js").BindingKey} key @returns {import("./types.js").Binding} */
function bindingOf(raw, key) {
if (
!object(raw) ||
raw.version !== 1 ||
raw.backend !== key.backend ||
raw.account !== key.account ||
raw.projectPath !== key.projectPath ||
typeof raw.compositionId !== "string" ||
typeof raw.ownerSpaceId !== "string"
)
throw new Error(
"Source binding belongs to another account, backend or project. Reset explicitly before sharing this copy.",
);
checkSnapshots(raw);
return /** @type {import("./types.js").Binding} */ (/** @type {unknown} */ (raw));
}
/** @param {Record<string, unknown>} raw */
function checkSnapshots(raw) {
if (
raw.base !== null &&
(!object(raw.base) ||
typeof raw.base.snapshotId !== "string" ||
parseStoredManifest(raw.base.manifest).sha256 !== raw.base.manifestSha256)
)
throw new Error("Invalid source base snapshot.");
if (
raw.inflight !== null &&
(!object(raw.inflight) ||
typeof raw.inflight.idempotencyKey !== "string" ||
parseStoredManifest(raw.inflight.manifest).sha256 !== raw.inflight.manifestSha256)
)
throw new Error("Invalid pending source upload.");
}
/** Project-local state is excluded by the portable source policy. Account/root mismatches fail closed, including copied folders.
* @returns {import("./adapterTypes.js").SourceStore} */
export function createProjectShareSourceStore() {
/** @param {import("./types.js").BindingKey} key */
const location = async (key) => {
const dir = await mkdirInside(key.projectPath, ".hyperframes");
const environment = createHash("sha256").update(key.backend).digest("hex").slice(0, 16);
return join(dir, `share-sync.${environment}.json`);
};
return {
read: async (key) => {
const file = await location(key);
const handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW).catch((error) => {
if (errorCode(error) === "ENOENT") return null;
throw error;
});
if (!handle) return null;
try {
if (!(await handle.stat()).isFile())
throw new Error("Source binding must be a regular file.");
return bindingOf(JSON.parse(await handle.readFile("utf8")), key);
} finally {
await handle.close();
}
},
write: async (binding) => {
const file = await location(binding);
const held = await lstat(file).catch((error) => {
if (errorCode(error) === "ENOENT") return null;
throw error;
});
if (held && !held.isFile()) throw new Error("Source binding must be a regular file.");
await writeJsonAtomic(file, binding);
},
};
}
+196
View File
@@ -0,0 +1,196 @@
// @ts-check
// Bytes to signed storage URLs: one file or multipart range, streamed and hashed as it is sent.
// Large media costs one chunk at a time. Storage URLs carry their own signature, never an OAuth bearer.
import { createHash } from "node:crypto";
import { constants, createReadStream } from "node:fs";
import { open, realpath } from "node:fs/promises";
import { dirname } from "node:path";
import { setTimeout as sleep } from "node:timers/promises";
import { trustedUploadUrl } from "../reviewUpload.mjs";
/** Why a transfer stopped. `expired`: the signed URL no longer works (ask for a new one). `exists`: a create-only PUT
* found the object already written. `changed`: the file's bytes are not the ones captured. `unreadable`: the file
* could not be read (a drive unplugged, a file deleted). `corrupt`: downloaded bytes do not hash to what was asked.
* `transient`: the network or the store failed; the same request may be sent again. */
export class TransferError extends Error {
/** @param {"expired" | "exists" | "changed" | "unreadable" | "corrupt" | "transient" | "refused"} kind
* @param {{ status?: number, path?: string }} [about] */
constructor(kind, about = {}) {
super(`composition sync transfer: ${kind}${about.status ? ` (${about.status})` : ""}`);
this.name = "TransferError";
this.kind = kind;
this.status = about.status ?? 0;
this.path = about.path ?? null;
}
}
/** @param {unknown} error */
export const isAbort = (error) =>
error instanceof Error && (error.name === "AbortError" || error.name === "TimeoutError");
/** @param {number} ms @param {AbortSignal} [signal] */
export const realWait = (ms, signal) => sleep(ms, undefined, { signal });
/** S3 answers 403 for a URL past its expiry; a 5xx or 429 is worth sending again. @param {number} status
* @returns {TransferError} */
function refusalOf(status) {
if (status === 403) return new TransferError("expired", { status });
if (status === 412) return new TransferError("exists", { status });
if (status === 429 || status >= 500) return new TransferError("transient", { status });
return new TransferError("refused", { status });
}
/** @param {string} file */
async function openSource(file) {
const handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW);
try {
const stat = await handle.stat();
if (!stat.isFile() || (await realpath(dirname(file))) !== dirname(file))
throw new TransferError("unreadable", { path: file });
return handle;
} catch (error) {
await handle.close();
throw error;
}
}
/** @param {string} file @param {number} start @param {number} length @param {(bytes: number) => void} onBytes */
async function hashedBody(file, start, length, onBytes) {
const handle = await openSource(file);
const hash = createHash("sha256");
const seen = { bytes: 0, ended: false, readError: /** @type {unknown} */ (null) };
const chunks =
length === 0
? null
: handle
.createReadStream({ start, end: start + length - 1, autoClose: false })
[Symbol.asyncIterator]();
const body = new ReadableStream(
{
async pull(controller) {
if (!chunks) return controller.close();
try {
const { value, done } = await chunks.next();
if (done) return ((seen.ended = true), controller.close());
hash.update(value);
seen.bytes += value.length;
onBytes(value.length);
controller.enqueue(value);
} catch (error) {
seen.readError = error;
controller.error(error);
}
},
cancel: () => void chunks?.return?.(),
},
{ highWaterMark: 0 },
);
return { body, seen, digest: () => hash.digest("hex"), close: () => handle.close() };
}
/** What the bytes taken from the file say about it once a PUT ended: unreadable, changed (shorter than captured,
* or another hash), or null when they are the captured bytes, all of them or a prefix the network cut short.
* @param {Awaited<ReturnType<typeof hashedBody>>} sent @param {{ length: number, sha256: string, path: string }} expected */
function fileProblem({ seen, digest }, { length, sha256, path }) {
if (seen.readError) return new TransferError("unreadable", { path });
const short = seen.ended && seen.bytes < length;
if (short || (seen.bytes === length && digest() !== sha256))
return new TransferError("changed", { path });
return null;
}
/** PUTs bytes of `file` to a signed URL with exactly the headers the grant signed. Resolves once storage keeps them;
* the bytes sent must hash to `sha256`, or it throws `changed` whatever storage answered.
* @param {{ fetchImpl: typeof fetch, url: string, headers: Record<string, string>, file: string, path: string,
* start: number, length: number, sha256: string, signal: AbortSignal, onBytes?: (bytes: number) => void, stallMs?: number }} put */
export async function putRange(put) {
const { fetchImpl, url, headers, file, start, length, signal } = put;
const target = trustedUploadUrl(url);
if (!target) throw new TransferError("refused", { path: put.path });
signal.throwIfAborted();
const stall = new AbortController();
const sent = await hashedBody(file, start, length, (bytes) => {
timer.refresh();
put.onBytes?.(bytes);
});
const timer = setTimeout(() => stall.abort(), put.stallMs ?? 60_000);
// Node's fetch needs `duplex` for a streamed body; lib.dom's RequestInit does not name it.
const init = {
method: "PUT",
headers,
body: sent.body,
duplex: "half",
redirect: "error",
signal: AbortSignal.any([signal, stall.signal]),
};
/** @type {Response} */
let reply;
try {
reply = await fetchImpl(target.href, /** @type {RequestInit} */ (init));
} catch (error) {
signal.throwIfAborted();
if (!stall.signal.aborted && isAbort(error)) throw error;
throw fileProblem(sent, put) ?? new TransferError("transient", { path: put.path });
} finally {
clearTimeout(timer);
await sent.close();
}
await reply.body?.cancel().catch(() => {});
const problem = fileProblem(sent, put);
if (problem) throw problem;
if (!reply.ok) throw refusalOf(reply.status);
}
/** Each part's SHA-256 (parts of `partBytes`, the last one shorter) and the whole file's, in one read.
* @param {string} file @param {number} size @param {number} partBytes @param {AbortSignal} signal */
export async function partDigests(file, size, partBytes, signal) {
const whole = createHash("sha256");
/** @type {string[]} */
const parts = [];
let part = createHash("sha256");
let inPart = 0;
for await (const chunk of createReadStream(file, { highWaterMark: 1 << 20, signal })) {
whole.update(chunk);
for (let at = 0; at < chunk.length; ) {
const take = Math.min(partBytes - inPart, chunk.length - at);
part.update(chunk.subarray(at, at + take));
inPart += take;
at += take;
if (inPart === partBytes) {
parts.push(part.digest("hex"));
part = createHash("sha256");
inPart = 0;
}
}
}
if (inPart > 0 || parts.length === 0) parts.push(part.digest("hex"));
return {
parts,
whole: whole.digest("hex"),
complete: parts.length === Math.ceil(size / partBytes),
};
}
/** Runs `work` over `items`, `limit` at a time. The first failure stops new work; the rest finish, then it throws.
* @template T @param {readonly T[]} items @param {number} limit @param {(item: T) => Promise<void>} work */
export async function eachLimit(items, limit, work) {
let next = 0;
const failure = { failed: false, error: /** @type {unknown} */ (null) };
const worker = async () => {
while (!failure.failed && next < items.length) {
const item = /** @type {T} */ (items[next++]);
try {
await work(item);
} catch (error) {
if (!failure.failed) Object.assign(failure, { failed: true, error });
}
}
};
await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
if (failure.failed) throw failure.error;
}
/** The wait before attempt `attempt` (from 1): doubling from `baseMs`, at most `maxMs`, or what the server asked.
* @param {number} attempt @param {number | null} [askedMs] */
export const backoffMs = (attempt, askedMs = null, baseMs = 1_000, maxMs = 30_000) =>
askedMs ?? Math.min(maxMs, baseMs * 2 ** (attempt - 1));
+353
View File
@@ -0,0 +1,353 @@
export interface SyncTooBig {
limit: "file_count" | "file_size" | "project_size" | "manifest_size";
actual: number;
maximum: number;
biggest: { path: string; size: number }[];
}
/** A file that keeps a project from being captured; it is never left out silently. */
export interface SyncCaptureProblem {
path: string;
problem: "symlink" | "special-file" | "unreadable" | "invalid-name" | "changed" | "no-entrypoint";
/** For `invalid-name`, the path policy's code (`reserved_name`, `path_collision`, ...). */
code?: string;
}
export interface ManifestFile {
path: string;
sha256: string;
size: number;
}
export interface ManifestDocument {
format_version: 1;
entrypoint: "index.html";
files: ManifestFile[];
}
/** A valid manifest: its document in canonical order, the canonical bytes and their SHA-256. */
export interface Manifest {
document: ManifestDocument;
canonical: Buffer;
sha256: string;
}
export interface Violation {
code: string;
path?: string;
rule?: string;
}
export type LimitName = SyncTooBig["limit"];
export interface Limits {
maxFiles: number;
maxFileBytes: number;
maxProjectBytes: number;
maxManifestBytes: number;
}
export type OverLimit = SyncTooBig;
type ClientAction = "include" | "exclude" | "wait_then_exclude";
export interface PathRule {
id: string;
regex: string;
client_action: ClientAction;
server_action: "allow" | "reject";
}
/** The fields of contract/path-policy.json this client reads. */
export interface PolicyDocument {
policy_version: string;
stored_manifest: { legacy_agent_state_regex: string };
unicode: { version: string };
rules: PathRule[];
path: {
max_path_bytes: number;
max_segment_bytes: number;
forbidden_characters: string;
reserved_segment_regex: string;
canonical_spelling_roots: string[];
};
client_prefilter: { names: string[] };
client_symlinks: { skip_prefixes: string[] };
}
export type CaptureProblem = SyncCaptureProblem;
/** What an owner workspace's plan allows composition sync, as the limits route answered it. `unread` when it could not
* be read: `not-deployed` is a server without the route (a 404), and nothing is refused on an unread answer. */
export type SyncLimitsRead =
| {
at: "read";
syncAllowed: boolean;
maxCompositions: number;
/** Live compositions in the workspace when it was read. */
compositions: number;
push: Limits;
}
| { at: "unread"; why: "not-deployed" | "offline" | "failed" };
/** A file a capture lists, before hashing. */
export interface Found {
path: string;
full: string;
size: number;
mtimeMs: number;
ino: number;
dev: number;
}
export interface Walk {
files: Found[];
problems: CaptureProblem[];
waiting: string[];
skippedLinks: string[];
}
export type Capture =
| {
at: "captured";
manifest: Manifest;
settled: { agentTurnOpen: boolean; projectLocked: boolean };
waiting: string[];
skippedLinks: string[];
}
| { at: "problems"; problems: CaptureProblem[] }
| { at: "too-big"; over: OverLimit };
export type SyncErrorKind =
| "offline"
| "signed-out"
| "malformed"
| "not-enabled"
| "byok"
| "no-write-role"
| "forbidden"
| "composition-not-found"
| "not-found"
| "too-large"
| "reference-invalid"
| "upload-not-pending"
| "active-uploads-exceeded"
| "file-not-writable"
| "idempotency-key-reused"
| "request-in-progress"
| "snapshot-not-downloadable"
| "conflict-not-open"
| "download-expired"
| "invalid-parameter"
| "plan-required"
| "limit-reached"
| "rate-limited"
| "unavailable"
| "server-error"
| "refused";
export interface Composition {
id: string;
spaceId: string;
title: string;
createdBy: string;
/** Null before the first publish. */
latestSnapshotId: string | null;
createdAt: string;
updatedAt: string;
}
export interface Snapshot {
id: string;
compositionId: string;
baseSnapshotId: string | null;
publicationOutcome: "published" | "conflict_saved";
conflictedWithSnapshotId: string | null;
conflictState: "open" | "resolved" | "discarded" | null;
resolvedBySnapshotId: string | null;
createdBy: string;
createdByName?: string | null;
clientType: string;
summary: string | null;
summarySource: string;
manifestSha256: string;
createdAt: string;
/** Only with `include=manifest`, and only once its bytes hash to manifestSha256. */
manifest: Manifest | null;
}
type UploadStatus = "pending" | "verifying" | "completed" | "failed" | "expired";
export interface UploadReceipt {
uploadId: string;
compositionId: string;
snapshotId: string;
status: UploadStatus;
/** Why a failed upload failed: cancelled, content_incomplete, verification_timeout, access_revoked. */
errorCode: string | null;
baseSnapshotId: string | null;
resolvesConflictSnapshotIds: string[];
manifestSha256: string;
uploadDeadline: string;
verificationDeadline: string | null;
completedAt: string | null;
result: {
snapshotId: string;
publicationOutcome: "published" | "conflict_saved";
conflictedWithSnapshotId: string | null;
} | null;
}
interface BlobRef {
sha256: string;
size: number;
}
export interface BeginResult extends UploadReceipt {
/** Already saved in the base or a named conflict: never uploaded. */
present: BlobRef[];
/** To upload through upload-urls before submitting. */
needed: BlobRef[];
/** False for a replay of the same push under the same Idempotency-Key. */
created: boolean;
}
export interface BeginRequest {
idempotencyKey: string;
baseSnapshotId: string | null;
resolvesConflictSnapshotIds: string[];
summary: { text: string; source: "user" | "agent" } | null;
manifest: Manifest;
}
interface PartGrant {
partNumber: number;
length: number;
url: string;
headers: Record<string, string>;
}
export interface FileGrant {
sha256: string;
size: number;
expiresAt: string;
/** Files up to 64 MiB: one PUT. */
url: string | null;
headers: Record<string, string>;
/** Files over 64 MiB: the file's one multipart upload, and the requested parts. */
multipartUploadId: string | null;
parts: PartGrant[];
}
export interface GrantRequest {
sha256: string;
multipartUploadId?: string;
parts?: { partNumber: number; sha256: string }[];
}
export interface DownloadBatch {
snapshotId: string;
manifestSha256: string;
continuationToken: string;
continuationExpiresAt: string;
urls: { sha256: string; size: number; url: string; expiresAt: string }[];
}
export interface Page<T> {
items: T[];
nextToken: string | null;
}
export type CompositionRead =
| { at: "fresh"; composition: Composition; etag: string | null }
| { at: "unchanged"; etag: string };
/** Whose binding: the backend's API host, the signed-in account (accountPrincipal) and the project's real path. */
export interface BindingKey {
backend: string;
account: string;
projectPath: string;
}
/** A push recorded before its Begin is sent, so a lost answer is recovered by sending the same body under the same
* Idempotency-Key; the upload's ids and deadlines land once an answer comes back. */
export interface InflightUpload {
idempotencyKey: string;
baseSnapshotId: string | null;
resolvesConflictSnapshotIds: string[];
summary: { text: string; source: "user" | "agent" } | null;
manifest: ManifestDocument;
manifestSha256: string;
uploadId: string | null;
snapshotId: string | null;
uploadDeadline: string | null;
verificationDeadline: string | null;
begunAt: number;
/** Files whose bytes reached staging, by SHA-256, so a resumed send does not write them again. */
sent?: string[];
/** Each multipart file's pinned upload and the parts already written, by the file's SHA-256. */
multipart?: Record<string, { uploadId: string; partsDone: number[] }>;
}
export interface Binding extends BindingKey {
version: 1;
/** Client-generated, kept before the first request so a retried create names the same composition. */
compositionId: string;
ownerSpaceId: string;
/** The owner workspace as the account's list last named it, so a project whose workspace the account left can
* still be named. Absent from bindings made before it was kept, and until a list names the workspace. */
owner?: { name: string; kind: "personal" | "team" };
boundAt: number;
/** The composition as last read, and the ETag it came with; null before the create was answered. */
composition: Composition | null;
etag: string | null;
/** The snapshot this copy was last sent as or got from, with its manifest, so local changes need no network. */
base: { snapshotId: string; manifest: ManifestDocument; manifestSha256: string } | null;
inflight: InflightUpload | null;
/** Set while Get from the cloud is filling a folder it made, until the folder is added to Home. Until then the
* folder is not a project on this Mac, and the next Get of the composition finishes it. */
gettingFromCloud?: true;
/** What the marker that Get put in its folder holds: the only proof a file of that name there is not the person's. */
getToken?: string;
/** Set on the folder a Duplicate fills from the cloud: it is bound only while it downloads, so no Get ever opens
* it, and its binding goes once latest is in (duplicate.mjs). */
duplicate?: true;
/** The conflicts this Mac's sends saved, newest last: Compare names them as this project's, not another
* computer's. */
savedHere?: string[];
/** Set on a binding Work on {owner}'s project made from a share: the account reaches the owner's workspace through
* that share's grant, not membership, so every request names the review project (sync-backed sharing C5). */
sharedFrom?: SharedFrom;
}
/** The share a binding came from: its review project, the owner's name and their workspace's, as the share named
* them when it was opened. */
export interface SharedFrom {
reviewProjectId: string;
owner: string;
workspace: string | null;
}
/** A workspace's last refusal of a new composition: not admitted to sync yet, or encrypted (BYOK). */
export interface SpaceRefusal {
kind: "not-enabled" | "byok";
at: number;
}
/** One send or get under way for a project: the binding it owns until it ends, and where its states go. */
export interface UploadJob {
api: import("./api.mjs").SourceApi;
space: string;
composition: string;
uploadId: string;
inflight: InflightUpload;
local: Map<string, { path: string; full: string }>;
needed: { sha256: string; size: number }[];
fetchImpl: typeof fetch;
wait: (ms: number, signal?: AbortSignal) => Promise<void>;
signal: AbortSignal;
now: () => number;
persist: () => void;
onBytes: (bytes: number) => void;
onFile: () => void;
}
+163
View File
@@ -0,0 +1,163 @@
// @ts-check
// Use the bundled Unicode 15.1 tables so path comparison is independent of the runtime Unicode version.
import { readContract } from "./contract.mjs";
export const UNICODE_VERSION = "15.1.0";
/** @typedef {{ unicode_version: string, casefold: Record<string, string>, combining: Record<string, number>,
* decomposition: Record<string, number[]>, composition: Record<string, number>,
* assigned_ranges: [number, number][] }} UnicodeTableFile */
const HANGUL_BASE = 0xac00;
const HANGUL_COUNT = 11172;
const L_BASE = 0x1100;
const V_BASE = 0x1161;
const T_BASE = 0x11a7;
const L_COUNT = 19;
const V_COUNT = 21;
const T_COUNT = 28;
const N_COUNT = V_COUNT * T_COUNT;
/** The table re-keyed by code point, decompositions fully expanded. @param {UnicodeTableFile} raw */
function tablesOf(raw) {
if (raw.unicode_version !== UNICODE_VERSION)
throw new Error(`unicode table is ${raw.unicode_version}, expected ${UNICODE_VERSION}`);
const hex = (/** @type {string} */ code) => Number.parseInt(code, 16);
const casefold = new Map(Object.entries(raw.casefold).map(([code, to]) => [hex(code), to]));
const combining = new Map(Object.entries(raw.combining).map(([code, cc]) => [hex(code), cc]));
const direct = new Map(Object.entries(raw.decomposition).map(([code, seq]) => [hex(code), seq]));
/** @param {number} code @returns {number[]} */
const expand = (code) => direct.get(code)?.flatMap(expand) ?? [code];
const decomposition = new Map([...direct.keys()].map((code) => [code, expand(code)]));
const composition = new Map(
Object.entries(raw.composition).map(([pair, composite]) => {
const [first = "", second = ""] = pair.split(",");
return [`${hex(first)},${hex(second)}`, composite];
}),
);
const starts = raw.assigned_ranges.map(([start]) => start);
const ends = raw.assigned_ranges.map(([, end]) => end);
return { casefold, combining, decomposition, composition, starts, ends };
}
/** @type {ReturnType<typeof tablesOf> | null} */
let loaded = null;
/** The table, read and indexed when a path is first compared: an app that never syncs never reads it. */
const tables = () => (loaded ??= tablesOf(readContract("unicode-15.1.0.json")));
/** @param {number} code */
const classOf = (code) => tables().combining.get(code) ?? 0;
/** @param {number} code @returns {readonly number[]} */
function decompose(code) {
const syllable = code - HANGUL_BASE;
if (syllable >= 0 && syllable < HANGUL_COUNT) {
const trailing = syllable % T_COUNT;
const leading = [
L_BASE + Math.floor(syllable / N_COUNT),
V_BASE + Math.floor((syllable % N_COUNT) / T_COUNT),
];
return trailing ? [...leading, T_BASE + trailing] : leading;
}
return tables().decomposition.get(code) ?? [code];
}
/** @param {number} first @param {number} second @returns {number | undefined} */
function composePair(first, second) {
if (first >= L_BASE && first < L_BASE + L_COUNT && second >= V_BASE && second < V_BASE + V_COUNT)
return HANGUL_BASE + ((first - L_BASE) * V_COUNT + second - V_BASE) * T_COUNT;
return composeSyllable(first, second) ?? tables().composition.get(`${first},${second}`);
}
/** @param {number} first @param {number} second */
function composeSyllable(first, second) {
if (
first >= HANGUL_BASE &&
first < HANGUL_BASE + HANGUL_COUNT &&
(first - HANGUL_BASE) % T_COUNT === 0 &&
second > T_BASE &&
second < T_BASE + T_COUNT
)
return first + second - T_BASE;
}
/** Canonical decomposition then canonical ordering of combining marks. @param {string} value */
function orderedCodes(value) {
/** @type {number[]} */
const ordered = [];
for (const char of value) {
for (const code of decompose(char.codePointAt(0) ?? 0)) {
const current = classOf(code);
let position = ordered.length;
while (current && position && classOf(ordered[position - 1] ?? 0) > current) position--;
ordered.splice(position, 0, code);
}
}
return ordered;
}
/** Canonical composition under the pinned Unicode table. @param {number[]} ordered */
function composed(ordered) {
/** @type {number[]} */
const result = [];
let starter = 0;
let lastClass = 0;
for (const code of ordered) {
const current = classOf(code);
const head = result[starter];
const composite = head === undefined ? undefined : composePair(head, code);
if (composite !== undefined && (lastClass < current || lastClass === 0)) {
result[starter] = composite;
continue;
}
if (current === 0) starter = result.length;
result.push(code);
lastClass = current;
}
return String.fromCodePoint(...result);
}
const ASCII = /^\p{ASCII}*$/u;
/** NFC (UAX #15) under the pinned table. @param {string} value */
export const nfc = (value) => (ASCII.test(value) ? value : composed(orderedCodes(value)));
/** Default full case folding (status C and F), so `ß` folds to `ss`. @param {string} value */
function fullCasefold(value) {
if (ASCII.test(value)) return value.toLowerCase();
let folded = "";
for (const char of value) folded += tables().casefold.get(char.codePointAt(0) ?? 0) ?? char;
return folded;
}
/** Whether every code point is assigned in Unicode 15.1. @param {string} value */
export function allAssigned(value) {
const { starts, ends } = tables();
for (const char of value) {
const code = char.codePointAt(0) ?? 0;
let low = 0;
let high = starts.length;
while (low < high) {
const middle = (low + high) >>> 1;
if ((starts[middle] ?? Infinity) <= code) low = middle + 1;
else high = middle;
}
if (low === 0 || code > (ends[low - 1] ?? -1)) return false;
}
return true;
}
/** Directory names repeat across a manifest's paths, and each segment is keyed twice (policy, then collisions). */
const KEYS_KEPT = 16384;
/** @type {Map<string, string>} */
const keys = new Map();
/** The spelling two names share on a case-insensitive or normalizing filesystem: NFC(casefold(NFC(value))).
* @param {string} value */
export function comparisonKey(value) {
const known = keys.get(value);
if (known !== undefined) return known;
const key = nfc(fullCasefold(nfc(value)));
if (keys.size >= KEYS_KEPT) keys.clear();
keys.set(value, key);
return key;
}
+201
View File
@@ -0,0 +1,201 @@
// @ts-check
import { randomUUID } from "node:crypto";
import { join } from "node:path";
import { NotWritable, uploadNeeded } from "./uploadFiles.mjs";
import { parseStoredManifest } from "./manifest.mjs";
import { SyncApiError } from "./errors.mjs";
/** @typedef {import("./adapterTypes.js").SourceRun} SourceRun
* @typedef {Pick<SourceRun, "binding" | "api" | "store">} SourceUpload
* @typedef {import("./types.js").UploadReceipt} UploadReceipt */
/** @param {SourceUpload & Pick<SourceRun, "manifest">} run */
async function begin(run) {
const { binding, manifest, api, store } = run;
if (!binding.inflight) {
binding.inflight = {
idempotencyKey: randomUUID(),
baseSnapshotId: binding.base?.snapshotId ?? null,
resolvesConflictSnapshotIds: [],
summary: null,
manifest: manifest.document,
manifestSha256: manifest.sha256,
uploadId: null,
snapshotId: null,
uploadDeadline: null,
verificationDeadline: null,
begunAt: Date.now(),
sent: [],
multipart: {},
};
await store.write(binding);
}
const inflight = binding.inflight;
const receipt = await api.begin(binding.ownerSpaceId, binding.compositionId, {
...inflight,
manifest,
});
assertReceipt(binding, receipt);
Object.assign(inflight, {
uploadId: receipt.uploadId,
snapshotId: receipt.snapshotId,
uploadDeadline: receipt.uploadDeadline,
verificationDeadline: receipt.verificationDeadline,
});
await store.write(binding);
return receipt;
}
/** @param {SourceRun} run @param {import("./types.js").BeginResult} receipt */
async function sendFiles(run, receipt) {
const { binding, manifest, store } = run;
if (!binding.inflight) throw new Error("Missing source upload intent.");
const local = new Map(
manifest.document.files.map((file) => [
file.sha256,
{ path: file.path, full: join(binding.projectPath, file.path) },
]),
);
const known = new Map(manifest.document.files.map((file) => [file.sha256, file.size]));
for (const blob of receipt.needed)
if (known.get(blob.sha256) !== blob.size)
throw new Error("Upload requested an unknown source file.");
let persisted = Promise.resolve();
await uploadNeeded({
api: run.api,
space: binding.ownerSpaceId,
composition: binding.compositionId,
uploadId: receipt.uploadId,
inflight: binding.inflight,
local,
needed: receipt.needed,
fetchImpl: run.fetchImpl,
wait: run.wait,
signal: run.signal,
now: Date.now,
persist: () => {
persisted = persisted.then(() => store.write(binding));
},
onBytes: run.onProgress,
onFile: () => {},
}).finally(() => persisted);
return run.api.submit(binding.ownerSpaceId, binding.compositionId, receipt.uploadId);
}
/** @param {SourceUpload["binding"]} binding @param {UploadReceipt} receipt */
function assertReceipt(binding, receipt) {
const saved = binding.inflight;
if (!saved || receipt.manifestSha256 !== saved.manifestSha256)
throw new Error("Source upload receipt does not match the captured project.");
if (receipt.compositionId !== binding.compositionId)
throw new Error("Source upload receipt does not match the composition.");
if (saved.uploadId !== null && receipt.uploadId !== saved.uploadId)
throw new Error("Source upload receipt identity changed.");
if (saved.snapshotId !== null && receipt.snapshotId !== saved.snapshotId)
throw new Error("Source snapshot identity changed.");
}
/** @param {SourceUpload} job @param {UploadReceipt} receipt */
async function storePublished({ binding, store }, receipt) {
const saved = binding.inflight;
const result = receipt.result;
if (!saved || !result || result.snapshotId !== receipt.snapshotId)
throw new Error("Source snapshot identity changed.");
binding.base = {
snapshotId: result.snapshotId,
manifest: saved.manifest,
manifestSha256: saved.manifestSha256,
};
await clearIntent({ binding, store });
}
/** @param {Pick<SourceUpload, "binding" | "store">} job */
async function clearIntent({ binding, store }) {
binding.inflight = null;
await store.write(binding);
}
/** @param {SourceUpload} job @param {UploadReceipt} receipt */
async function cancelPending({ binding, api }, receipt) {
const { ownerSpaceId, compositionId } = binding;
try {
return await api.cancel(ownerSpaceId, compositionId, receipt.uploadId);
} catch (error) {
if (!(error instanceof SyncApiError) || error.kind !== "upload-not-pending") throw error;
return api.status(ownerSpaceId, compositionId, receipt.uploadId);
}
}
/** Resolve an older capture before starting an upload for the current files.
* @param {SourceUpload} job @returns {Promise<import("./adapterTypes.js").SourceResult | null>} */
export async function recoverSourceUpload(job) {
const { binding, api } = job;
const saved = binding.inflight;
if (!saved) return null;
const manifest = parseStoredManifest(saved.manifest);
if (manifest.sha256 !== saved.manifestSha256)
throw new Error("Saved source upload manifest does not match its checksum.");
let receipt = saved.uploadId
? await api.status(binding.ownerSpaceId, binding.compositionId, saved.uploadId)
: await begin({ ...job, manifest });
assertReceipt(binding, receipt);
if (receipt.status === "pending") receipt = await cancelPending(job, receipt);
assertReceipt(binding, receipt);
return recoverReceipt(job, receipt);
}
/** @param {SourceUpload} job @param {UploadReceipt} receipt */
async function recoverReceipt(job, receipt) {
if (receipt.status === "completed") {
if (receipt.result?.publicationOutcome !== "published")
return { at: /** @type {const} */ ("unavailable"), reason: "source_conflict" };
await storePublished(job, receipt);
} else if (terminalFailure(receipt)) await clearIntent(job);
else return { at: /** @type {const} */ ("unavailable"), reason: "source_verification_pending" };
return null;
}
/** @param {UploadReceipt} receipt */
const terminalFailure = (receipt) => receipt.status === "failed" || receipt.status === "expired";
/** Only published verification results can become share source; conflict_saved is preserved for explicit recovery.
* @param {SourceRun} run @returns {Promise<import("./adapterTypes.js").SourceResult>} */
export async function uploadSource(run) {
const receipt = await begin(run);
let status;
try {
status = receipt.status === "pending" ? await sendFiles(run, receipt) : receipt;
} catch (error) {
if (!(error instanceof NotWritable)) throw error;
return (
(await recoverSourceUpload(run)) ?? {
at: "unavailable",
reason: "source_upload_not_writable",
}
);
}
const deadline = Date.now() + run.timeoutMs;
while (status.status === "verifying") {
run.signal.throwIfAborted();
if (Date.now() >= deadline) return { at: "unavailable", reason: "source_verification_pending" };
await run.wait(1000, run.signal);
status = await run.api.status(
run.binding.ownerSpaceId,
run.binding.compositionId,
receipt.uploadId,
);
}
assertReceipt(run.binding, status);
if (status.status !== "completed" || !status.result) {
if (terminalFailure(status)) await clearIntent(run);
return { at: "unavailable", reason: status.errorCode ?? `source_upload_${status.status}` };
}
if (status.result.publicationOutcome !== "published")
return { at: "unavailable", reason: "source_conflict" };
await storePublished(run, status);
return {
at: "ready",
snapshot: {
space_id: run.binding.ownerSpaceId,
composition_id: run.binding.compositionId,
snapshot_id: status.snapshotId,
},
};
}
+263
View File
@@ -0,0 +1,263 @@
// @ts-check
// The files a Begin said are needed, written to the upload's staging through signed URLs: one PUT for a file up to
// 64 MiB, parts of 64 MiB above that through the file's one pinned multipart upload. URLs live at most 15 minutes, so
// one about to end is asked for again; a send resumed after a quit skips what the in-flight record says was written.
import { SyncApiError } from "./errors.mjs";
import {
TransferError,
backoffMs,
eachLimit,
isAbort,
partDigests,
putRange,
} from "./transfer.mjs";
import { lstat } from "node:fs/promises";
/** @typedef {import("./types.js").FileGrant} FileGrant @typedef {import("./types.js").GrantRequest} GrantRequest
* @typedef {import("./types.js").InflightUpload} InflightUpload */
/** The server's single-PUT ceiling and part size. */
const PART_BYTES = 64 * 1024 * 1024;
const GRANT_FILES = 100;
const PARTS_PER_GRANT = 8;
const AT_ONCE = 4;
const ATTEMPTS = 5;
/** A URL with less than this left is asked for again rather than started. */
const URL_MARGIN_MS = 60_000;
/** The file's multipart upload can take no more parts (`hyperframes_composition_upload_file_not_writable`): this
* upload cannot finish, so its owner must resolve it before a retry can begin a new one. */
export class NotWritable extends Error {
constructor() {
super("composition sync: a file's multipart upload can no longer take parts");
this.name = "NotWritable";
}
}
/** @typedef {import("./types.js").UploadJob} UploadJob */
/** @param {UploadJob} job @param {unknown} error @param {number} attempt the attempt that just failed, from 1
* @returns {Promise<boolean>} whether to try again */
async function againAfter(job, error, attempt) {
if (attempt >= ATTEMPTS || isAbort(error)) return false;
const transient =
(error instanceof TransferError && (error.kind === "transient" || error.kind === "corrupt")) ||
(error instanceof SyncApiError &&
["offline", "unavailable", "server-error", "rate-limited"].includes(error.kind));
if (!transient) return false;
const asked = error instanceof SyncApiError ? error.retryAfterMs : null;
await job.wait(backoffMs(attempt, asked), job.signal);
return true;
}
/** A refusal of a pinned multipart id, which its refusal says to drop and ask again without.
* @param {unknown} error @param {GrantRequest[]} asked */
const wrongPinnedId = (error, asked) =>
error instanceof SyncApiError &&
error.kind === "invalid-parameter" &&
(error.param ?? "").endsWith("multipart_upload_id") &&
asked.some((file) => file.multipartUploadId);
/** Write grants for `files`, sent again on a transient failure. A file that can take no more parts throws NotWritable.
* @param {UploadJob} job @param {GrantRequest[]} files @returns {Promise<FileGrant[]>} */
async function grant(job, files) {
let asked = files;
for (let attempt = 1; ; attempt++) {
try {
return await job.api.uploadUrls(job.space, job.composition, job.uploadId, asked);
} catch (error) {
if (error instanceof SyncApiError && error.kind === "file-not-writable")
throw new NotWritable();
if (wrongPinnedId(error, asked))
asked = asked.map(({ multipartUploadId: _dropped, ...file }) => file);
else if (!(await againAfter(job, error, attempt))) throw error;
}
}
}
/** @param {UploadJob} job @param {string} sha256 */
function localFile(job, sha256) {
const file = job.local.get(sha256);
if (!file) throw new Error(`composition sync: needed file ${sha256} is not in the capture`);
return file;
}
/** @typedef {{ url: string, headers: Record<string, string>, expiresAt: string }} Signed
* @typedef {{ number: number, sha256: string, start: number, length: number }} Part
* @typedef {{ file: { path: string, full: string }, start: number, length: number, sha256: string }} Bytes */
/** One PUT attempt: `done` once stored (or found already stored), `expired` when storage says the URL has, else the
* failure. Bytes counted for an attempt that failed are taken back.
* @param {UploadJob} job @param {Signed} held @param {Bytes} bytes
* @returns {Promise<"done" | "expired" | { failed: unknown }>} */
async function attemptPut(job, held, { file, ...range }) {
let sent = 0;
try {
await putRange({
fetchImpl: job.fetchImpl,
url: held.url,
headers: held.headers,
file: file.full,
path: file.path,
...range,
signal: job.signal,
onBytes: (count) => ((sent += count), job.onBytes(count)),
});
return "done";
} catch (error) {
job.onBytes(-sent);
if (!(error instanceof TransferError)) return { failed: error };
if (error.kind === "exists") return (job.onBytes(range.length), "done");
return error.kind === "expired" ? "expired" : { failed: error };
}
}
/** Writes `bytes` through `given`, or a fresh grant when it is missing or about to end; asks again when storage says
* the URL ended, and sends again after a transient failure.
* @param {UploadJob} job @param {Signed | null} given @param {() => Promise<Signed>} fresh @param {Bytes} bytes */
async function putWithGrant(job, given, fresh, bytes) {
const usable = (/** @type {Signed | null} */ signed) =>
signed && Date.parse(signed.expiresAt) - job.now() >= URL_MARGIN_MS ? signed : null;
let held = usable(given);
for (let attempt = 1; ; attempt++) {
held ??= await fresh();
const outcome = await attemptPut(job, held, bytes);
if (outcome === "done") return;
if (outcome === "expired" && attempt < ATTEMPTS) held = null;
else {
const failed = outcome === "expired" ? new TransferError("expired") : outcome.failed;
if (!(await againAfter(job, failed, attempt))) throw failed;
}
}
}
/** @param {FileGrant | undefined} held @returns {Signed} */
function singleUrl(held) {
if (!held?.url) throw new Error("composition sync: a single-PUT grant came without its URL");
return { url: held.url, headers: held.headers, expiresAt: held.expiresAt };
}
/** Files up to 64 MiB: grants a hundred at a time, written four at once. @param {UploadJob} job
* @param {{ sha256: string, size: number }[]} blobs */
async function sendSmall(job, blobs) {
for (let at = 0; at < blobs.length; at += GRANT_FILES) {
const batch = blobs.slice(at, at + GRANT_FILES);
const grants = await grant(
job,
batch.map((blob) => ({ sha256: blob.sha256 })),
);
const bySha = new Map(grants.map((one) => [one.sha256, one]));
await eachLimit(batch, AT_ONCE, async (blob) => {
const given = bySha.has(blob.sha256) ? singleUrl(bySha.get(blob.sha256)) : null;
const fresh = async () => singleUrl((await grant(job, [{ sha256: blob.sha256 }]))[0]);
const bytes = {
file: localFile(job, blob.sha256),
start: 0,
length: blob.size,
sha256: blob.sha256,
};
await putWithGrant(job, given, fresh, bytes);
(job.inflight.sent ??= []).push(blob.sha256);
job.persist();
job.onFile();
});
}
}
/** Grants for `parts` of a multipart file, naming its pinned upload once one is known, and keeping the id the server
* pins. @param {UploadJob} job @param {{ sha256: string }} blob @param {Part[]} parts */
async function partGrants(job, blob, parts) {
const pinned = job.inflight.multipart?.[blob.sha256];
const [held] = await grant(job, [
{
sha256: blob.sha256,
...(pinned && { multipartUploadId: pinned.uploadId }),
parts: parts.map((part) => ({ partNumber: part.number, sha256: part.sha256 })),
},
]);
if (!held?.multipartUploadId)
throw new Error("composition sync: a multipart grant came without its upload id");
// Parts recorded as written belong to the pinned upload; another id would leave them unwritten.
if (pinned && pinned.uploadId !== held.multipartUploadId) throw new NotWritable();
if (!pinned) {
job.inflight.multipart = {
...job.inflight.multipart,
[blob.sha256]: { uploadId: held.multipartUploadId, partsDone: [] },
};
job.persist();
}
return { expiresAt: held.expiresAt, parts: held.parts };
}
/** The file's parts still to write, each with its SHA-256; throws `changed` when the file no longer hashes to the
* captured SHA-256. @param {UploadJob} job @param {{ sha256: string, size: number }} blob
* @returns {Promise<Part[]>} */
async function partsToSend(job, blob) {
const file = localFile(job, blob.sha256);
const digests = await partDigests(file.full, blob.size, PART_BYTES, job.signal).catch((error) => {
if (isAbort(error)) throw error;
throw new TransferError("unreadable", { path: file.path });
});
if (digests.whole !== blob.sha256 || !digests.complete)
throw new TransferError("changed", { path: file.path });
const done = new Set(job.inflight.multipart?.[blob.sha256]?.partsDone ?? []);
return digests.parts
.map((sha256, index) => ({
number: index + 1,
sha256,
start: index * PART_BYTES,
length: Math.min(PART_BYTES, blob.size - index * PART_BYTES),
}))
.filter((part) => !done.has(part.number));
}
/** A file over 64 MiB, eight parts a grant, four parts at once; its size and time are compared after, so a file that
* changed while its parts were read fails as changed. @param {UploadJob} job
* @param {{ sha256: string, size: number }} blob */
async function sendLarge(job, blob) {
const file = localFile(job, blob.sha256);
const before = await lstat(file.full);
const parts = await partsToSend(job, blob);
job.onBytes(blob.size - parts.reduce((sum, part) => sum + part.length, 0));
for (let at = 0; at < parts.length; at += PARTS_PER_GRANT) {
const batch = parts.slice(at, at + PARTS_PER_GRANT);
const granted = await partGrants(job, blob, batch);
const byNumber = new Map(granted.parts.map((part) => [part.partNumber, part]));
await eachLimit(batch, AT_ONCE, async (part) => {
const signed = byNumber.get(part.number);
const given = signed
? { url: signed.url, headers: signed.headers, expiresAt: granted.expiresAt }
: null;
const fresh = async () => {
const again = await partGrants(job, blob, [part]);
const one = again.parts[0];
if (!one) throw new Error("composition sync: a part grant came without its part");
return { url: one.url, headers: one.headers, expiresAt: again.expiresAt };
};
await putWithGrant(job, given, fresh, { file, ...part });
job.inflight.multipart?.[blob.sha256]?.partsDone.push(part.number);
job.persist();
});
}
const after = await lstat(file.full).catch(() => null);
if (!after || after.size !== before.size || after.mtimeMs !== before.mtimeMs)
throw new TransferError("changed", { path: file.path });
(job.inflight.sent ??= []).push(blob.sha256);
job.persist();
job.onFile();
}
/** Writes every needed file the in-flight record does not list as sent. @param {UploadJob} job */
export async function uploadNeeded(job) {
const sent = new Set(job.inflight.sent ?? []);
const waiting = job.needed.filter((blob) => !sent.has(blob.sha256));
for (const blob of job.needed.filter((one) => sent.has(one.sha256))) {
job.onBytes(blob.size);
job.onFile();
}
await sendSmall(
job,
waiting.filter((blob) => blob.size <= PART_BYTES),
);
for (const blob of waiting.filter((one) => one.size > PART_BYTES)) await sendLarge(job, blob);
}
+205
View File
@@ -0,0 +1,205 @@
// @ts-check
// Parse and validate sync API responses at the transport boundary.
import { ManifestInvalid, parseStoredManifest } from "./manifest.mjs";
import { SyncApiError } from "./errors.mjs";
/** @typedef {import("./types.js").SyncLimitsRead} SyncLimitsRead
* @typedef {import("./types.js").Composition} Composition @typedef {import("./types.js").Snapshot} Snapshot
* @typedef {import("./types.js").UploadReceipt} UploadReceipt @typedef {import("./types.js").BeginResult} BeginResult
* @typedef {import("./types.js").FileGrant} FileGrant @typedef {import("./types.js").DownloadBatch} DownloadBatch */
/** @param {string} field */
const malformed = (field) =>
new SyncApiError("malformed", { detail: `missing or mistyped ${field}` });
/** @param {unknown} value @param {string} field @returns {Record<string, unknown>} */
export function objectOf(value, field) {
if (typeof value !== "object" || value === null || Array.isArray(value)) throw malformed(field);
return /** @type {Record<string, unknown>} */ (value);
}
/** @param {Record<string, unknown>} o @param {string} key @returns {string} */
const str = (o, key) => {
const value = o[key];
if (typeof value !== "string") throw malformed(key);
return value;
};
/** Absent (the server drops nulls) reads as null. @param {Record<string, unknown>} o @param {string} key */
const optStr = (o, key) => (o[key] === undefined || o[key] === null ? null : str(o, key));
/** @param {Record<string, unknown>} o @param {string} key @returns {number} */
const int = (o, key) => {
const value = o[key];
if (typeof value !== "number" || !Number.isSafeInteger(value)) throw malformed(key);
return value;
};
/** @param {Record<string, unknown>} o @param {string} key @returns {unknown[]} */
const list = (o, key) => {
const value = o[key] ?? [];
if (!Array.isArray(value)) throw malformed(key);
return value;
};
/** @param {Record<string, unknown>} o @param {string} key @returns {Record<string, string>} */
const headers = (o, key) => {
const value = objectOf(o[key] ?? {}, key);
return Object.fromEntries(Object.keys(value).map((name) => [name, str(value, name)]));
};
/** @template {string} T @param {Record<string, unknown>} o @param {string} key @param {readonly T[]} allowed
* @returns {T} */
const oneOf = (o, key, allowed) => {
const value = str(o, key);
if (!allowed.includes(/** @type {T} */ (value))) throw malformed(key);
return /** @type {T} */ (value);
};
/** @param {unknown} raw @returns {Composition} */
export function compositionOf(raw) {
const o = objectOf(raw, "composition");
return {
id: str(o, "id"),
spaceId: str(o, "space_id"),
title: str(o, "title"),
createdBy: str(o, "created_by"),
latestSnapshotId: optStr(o, "latest_snapshot_id"),
createdAt: str(o, "created_at"),
updatedAt: str(o, "updated_at"),
};
}
/** The snapshot's manifest, trusted only when its canonical bytes hash to the snapshot's manifest_sha256.
* @param {unknown} raw @param {string} sha256 */
function snapshotManifestOf(raw, sha256) {
const o = objectOf(raw, "manifest");
try {
const manifest = parseStoredManifest(o.document);
if (manifest.sha256 !== sha256) throw malformed("manifest.document (sha256)");
return manifest;
} catch (error) {
if (error instanceof ManifestInvalid) throw malformed(`manifest.document (${error.code})`);
throw error;
}
}
/** @param {unknown} raw @returns {Snapshot} */
export function snapshotOf(raw) {
const o = objectOf(raw, "snapshot");
const manifestSha256 = str(o, "manifest_sha256");
const state = optStr(o, "conflict_state");
if (state !== null && !["open", "resolved", "discarded"].includes(state))
throw malformed("conflict_state");
return {
id: str(o, "id"),
compositionId: str(o, "composition_id"),
baseSnapshotId: optStr(o, "base_snapshot_id"),
publicationOutcome: oneOf(
o,
"publication_outcome",
/** @type {const} */ (["published", "conflict_saved"]),
),
conflictedWithSnapshotId: optStr(o, "conflicted_with_snapshot_id"),
conflictState: /** @type {Snapshot["conflictState"]} */ (state),
resolvedBySnapshotId: optStr(o, "resolved_by_snapshot_id"),
createdBy: str(o, "created_by"),
createdByName: optStr(o, "created_by_name"),
clientType: str(o, "client_type"),
summary: optStr(o, "summary"),
summarySource: str(o, "summary_source"),
manifestSha256,
createdAt: str(o, "created_at"),
manifest: o.manifest === undefined ? null : snapshotManifestOf(o.manifest, manifestSha256),
};
}
const UPLOAD_STATUSES = /** @type {const} */ ([
"pending",
"verifying",
"completed",
"failed",
"expired",
]);
/** @param {unknown} raw @returns {UploadReceipt} */
export function uploadOf(raw) {
const o = objectOf(raw, "upload");
const result = o.result === undefined || o.result === null ? null : objectOf(o.result, "result");
return {
uploadId: str(o, "upload_id"),
compositionId: str(o, "composition_id"),
snapshotId: str(o, "snapshot_id"),
status: oneOf(o, "status", UPLOAD_STATUSES),
errorCode: optStr(o, "error_code"),
baseSnapshotId: optStr(o, "base_snapshot_id"),
resolvesConflictSnapshotIds: list(o, "resolves_conflict_snapshot_ids").map((id) => {
if (typeof id !== "string") throw malformed("resolves_conflict_snapshot_ids");
return id;
}),
manifestSha256: str(o, "manifest_sha256"),
uploadDeadline: str(o, "upload_deadline"),
verificationDeadline: optStr(o, "verification_deadline"),
completedAt: optStr(o, "completed_at"),
result: result && {
snapshotId: str(result, "snapshot_id"),
publicationOutcome: oneOf(
result,
"publication_outcome",
/** @type {const} */ (["published", "conflict_saved"]),
),
conflictedWithSnapshotId: optStr(result, "conflicted_with_snapshot_id"),
},
};
}
/** @param {unknown} raw @param {string} field */
const blobsOf = (raw, field) =>
list(objectOf(raw, field), field).map((item) => {
const blob = objectOf(item, field);
return { sha256: str(blob, "sha256"), size: int(blob, "size") };
});
/** @param {unknown} raw @param {boolean} created @returns {BeginResult} */
export const beginOf = (raw, created) => ({
...uploadOf(raw),
present: blobsOf(raw, "present"),
needed: blobsOf(raw, "needed"),
created,
});
/** @param {unknown} raw @returns {FileGrant[]} */
export const grantsOf = (raw) =>
list(objectOf(raw, "grants"), "files").map((item) => {
const o = objectOf(item, "files");
return {
sha256: str(o, "sha256"),
size: int(o, "size"),
expiresAt: str(o, "expires_at"),
url: optStr(o, "url"),
headers: headers(o, "headers"),
multipartUploadId: optStr(o, "multipart_upload_id"),
parts: list(o, "parts").map((part) => {
const p = objectOf(part, "parts");
return {
partNumber: int(p, "part_number"),
length: int(p, "length"),
url: str(p, "url"),
headers: headers(p, "headers"),
};
}),
};
});
/** @param {unknown} raw @returns {Extract<SyncLimitsRead, { at: "read" }>} */
export function syncLimitsOf(raw) {
const o = objectOf(raw, "limits answer");
const limits = objectOf(o.limits, "limits");
if (typeof o.sync_allowed !== "boolean") throw malformed("sync_allowed");
return {
at: "read",
syncAllowed: o.sync_allowed,
maxCompositions: int(limits, "max_compositions"),
compositions: int(objectOf(o.usage, "usage"), "compositions"),
push: {
maxFiles: int(limits, "max_files"),
maxFileBytes: int(limits, "max_file_bytes"),
maxProjectBytes: int(limits, "max_project_bytes"),
maxManifestBytes: int(limits, "max_manifest_bytes"),
},
};
}
File diff suppressed because it is too large Load Diff
+86 -1
View File
@@ -1,6 +1,6 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import fs, { mkdtemp, mkdir, readFile, writeFile, rm, symlink } from "node:fs/promises";
import fs, { mkdtemp, mkdir, readFile, writeFile, rm, symlink, utimes } from "node:fs/promises";
import { tmpdir, hostname } from "node:os";
import { join } from "node:path";
import { realpathSync } from "node:fs";
@@ -437,3 +437,88 @@ for (const name of ["review.json", "review-export.json", "review-switch.json"])
assert.equal(await readFile(file, "utf8"), "");
});
}
test("required source never silently degrades after preparation or a server mismatch", async (t) => {
const dir = await folder(t);
await mkdir(join(dir, ".hyperframes"));
await writeFile(join(dir, ".hyperframes/video.mp4"), "video");
bindShareHost({ projectFingerprint: async () => "rendered" });
t.after(() => bindShareHost({ projectFingerprint }));
let creates = 0;
const api = {
createExport: async () => {
creates++;
throw new Error("should not send");
},
};
const request = {
projectPath: dir,
title: "Video",
client: api,
projectId: "project",
requireSource: true,
probe: async () => ({ width: 100, height: 100, duration: 1 }),
filesForExport: async () => null,
};
assert.match(await addVersion(request), /Source changed/);
assert.equal(creates, 0);
const { ReviewFailed } = await import("../src/reviewClient.mjs");
api.createExport = async () => {
creates++;
throw new ReviewFailed(422, "snapshot_mismatch");
};
await addVersion({
...request,
filesForExport: async () => ({
space_id: "space",
composition_id: "comp",
snapshot_id: "snap",
}),
});
assert.equal(creates, 1);
});
test("content fingerprints detect same-size changes with restored timestamps", async (t) => {
const dir = await folder(t);
const file = join(dir, "asset.txt");
const time = new Date("2026-11-01T00:00:00Z");
await writeFile(file, "first");
await utimes(file, time, time);
const metadata = await projectFingerprint(dir);
const content = await projectFingerprint(dir, { byContent: true });
await writeFile(file, "other");
await utimes(file, time, time);
assert.equal(await projectFingerprint(dir), metadata);
assert.notEqual(await projectFingerprint(dir, { byContent: true }), content);
});
test("a video-only retry never replays an unanswered source-bearing export", async (t) => {
const dir = await folder(t);
await mkdir(join(dir, ".hyperframes"));
await writeFile(join(dir, ".hyperframes/video.mp4"), "video bytes");
bindShareHost({ projectFingerprint: async () => "rendered" });
t.after(() => bindShareHost({ projectFingerprint }));
let creates = 0;
const request = {
projectPath: dir,
title: "Video",
client: {
createExport: async () => {
creates++;
throw new Error("lost response");
},
},
projectId: "project",
probe: async () => ({ width: 100, height: 100, duration: 1 }),
filesForExport: async () => ({
space_id: "space",
composition_id: "composition",
snapshot_id: "snapshot",
}),
};
await addVersion({ ...request, requireSource: true });
assert.match(
await addVersion({ ...request, omitSource: true }),
/pending upload includes source/,
);
assert.equal(creates, 1);
});
+410
View File
@@ -0,0 +1,410 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtemp, writeFile, rm, readFile, symlink, mkdir, realpath } from "node:fs/promises";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { createHash } from "node:crypto";
import {
shareSources,
shareSourceFingerprint,
createProjectShareSourceStore,
} from "../src/shareSources.mjs";
import { projectFingerprint } from "../src/projectSnapshot.mjs";
import { captureProject } from "../src/sources/capture.mjs";
import { sourceApi } from "../src/sources/api.mjs";
import { SyncApiError } from "../src/sources/errors.mjs";
import { V1_LIMITS } from "../src/sources/limits.mjs";
import { bindShareHost } from "../src/host.mjs";
bindShareHost({ reviewUploadOrigins: ["https://storage.test"] });
const signed = {
token: "test-token",
owner: "owner",
heygen: "https://app.test",
heygenApi: "https://api.test",
};
async function fixture(t, files = {}) {
const root = await realpath(await mkdtemp(join(tmpdir(), "hf-source-test-")));
t.after(() => rm(root, { recursive: true, force: true }));
await writeFile(join(root, "index.html"), "<html>video</html>");
for (const [file, content] of Object.entries(files)) await writeFile(join(root, file), content);
const rendered = await projectFingerprint(root);
const store = createProjectShareSourceStore();
return {
root,
rendered,
store,
key: { backend: signed.heygenApi, account: signed.owner, projectPath: root },
};
}
function fakeServer(overrides = {}) {
const calls = { create: 0, begin: [], puts: [], submit: 0, snapshot: 0 };
let held;
let composition;
const receipt = (status = "pending") => ({
uploadId: "upload",
compositionId: composition,
snapshotId: "snapshot",
status,
errorCode: null,
baseSnapshotId: null,
resolvesConflictSnapshotIds: [],
manifestSha256: held.manifest.sha256,
uploadDeadline: "2099-01-01T00:00:00Z",
verificationDeadline: null,
completedAt: null,
result:
status === "completed"
? {
snapshotId: "snapshot",
publicationOutcome: "published",
conflictedWithSnapshotId: null,
}
: null,
});
const api = {
limits: async () => ({
at: "read",
syncAllowed: true,
maxCompositions: 5,
compositions: 0,
push: V1_LIMITS,
}),
create: async (_space, id) => {
calls.create++;
composition = id;
return { id };
},
begin: async (_space, id, request) => {
composition = id;
held = request;
calls.begin.push(request);
return { ...receipt(), created: true, present: [], needed: request.manifest.document.files };
},
uploadUrls: async (_space, _comp, _upload, files) =>
files.map((file) => ({
...file,
url: `https://storage.test/${file.sha256}`,
headers: {},
expiresAt: "2099-01-01T00:00:00Z",
parts: [],
})),
submit: async () => {
calls.submit++;
return receipt("completed");
},
status: async () => receipt("completed"),
snapshot: async (_space, id, snapshot) => {
calls.snapshot++;
return {
id: snapshot,
compositionId: id,
manifestSha256: held.manifest.sha256,
manifest: held.manifest,
publicationOutcome: "published",
};
},
...overrides,
};
const fetchImpl = async (url, init) => {
assert.equal(init.headers.authorization, undefined);
const bytes = Buffer.from(await new Response(init.body).arrayBuffer());
assert.equal(createHash("sha256").update(bytes).digest("hex"), url.split("/").at(-1));
calls.puts.push(bytes);
return new Response(null, { status: 200 });
};
return { api, calls, fetchImpl, receipt };
}
for (const path of [".media/voice.mp3", ".hyperframes/studio-manual-edits.json"]) {
test(`changing ${path} refuses source preparation and final attachment to the old render`, async (t) => {
const f = await fixture(t);
await writeFile(join(f.root, "index.html"), '<audio src=".media/voice.mp3"></audio>');
await mkdir(dirname(join(f.root, path)), { recursive: true });
await writeFile(join(f.root, path), "original");
const rendered = await projectFingerprint(f.root);
const content = await shareSourceFingerprint(f.root);
const server = fakeServer();
const source = shareSources({
access: async () => signed,
spaceId: "personal",
...server,
fingerprint: async (root) =>
content === (await shareSourceFingerprint(root)) ? projectFingerprint(root) : null,
});
await writeFile(join(f.root, path), "changed");
assert.equal(await projectFingerprint(f.root), rendered);
assert.equal((await source.prepare(f.root, rendered)).reason, "source_render_changed");
assert.equal(server.calls.begin.length, 0);
assert.equal(await source.filesForExport(f.root, rendered), null);
await writeFile(join(f.root, path), "original");
assert.equal((await source.prepare(f.root, rendered)).at, "ready");
assert.ok(await source.filesForExport(f.root, rendered));
await writeFile(join(f.root, path), "changed after preparation");
assert.equal(await source.filesForExport(f.root, rendered), null);
});
}
test("source upload verifies before attachment and a fresh adapter reuses the exact published snapshot", async (t) => {
const f = await fixture(t);
const server = fakeServer();
const options = { access: async () => signed, spaceId: "personal", ...server };
const source = shareSources(options);
assert.equal(await source.filesForExport(f.root, f.rendered), null);
const result = await source.prepare(f.root, f.rendered);
assert.equal(result.at, "ready");
assert.equal(server.calls.puts.length, 1);
assert.equal(server.calls.submit, 1);
assert.deepEqual(await source.filesForExport(f.root, f.rendered), result.snapshot);
assert.equal((await shareSources(options).prepare(f.root, f.rendered)).at, "ready");
assert.equal(server.calls.begin.length, 1);
assert.equal(server.calls.snapshot, 1);
const binding = await f.store.read(f.key);
assert.equal(binding.inflight, null);
assert.equal(binding.base.snapshotId, "snapshot");
});
test("changed assets invalidate prepared source even when the HTML render fingerprint stays unchanged", async (t) => {
const f = await fixture(t, { "media.mp4": "first" });
const source = shareSources({
access: async () => signed,
spaceId: "personal",
...fakeServer(),
fingerprint: async () => f.rendered,
});
assert.equal((await source.prepare(f.root, f.rendered)).at, "ready");
await writeFile(join(f.root, "media.mp4"), "other");
assert.equal(await source.filesForExport(f.root, f.rendered), null);
});
test("eligibility and busy edits refuse before creating cloud compositions", async (t) => {
const f = await fixture(t);
const server = fakeServer({ limits: async () => ({ syncAllowed: false }) });
assert.deepEqual(
await shareSources({ access: async () => signed, spaceId: "personal", ...server }).prepare(
f.root,
f.rendered,
),
{ at: "unavailable", reason: "source_plan_required" },
);
assert.equal(server.calls.create, 0);
await mkdir(join(f.root, ".hyperframes"), { recursive: true });
await writeFile(join(f.root, ".hyperframes/history-turn.json"), "{}");
const next = fakeServer();
assert.equal(
(
await shareSources({ access: async () => signed, spaceId: "personal", ...next }).prepare(
f.root,
f.rendered,
)
).reason,
"source_edit_in_progress",
);
assert.equal(next.calls.create, 0);
});
test("lost Begin response retries the durable idempotency key, without creating a second composition", async (t) => {
const f = await fixture(t);
const server = fakeServer();
const begin = server.api.begin;
let fail = true;
server.api.begin = async (...args) => {
const answer = await begin(...args);
if (fail) {
fail = false;
throw new Error("network lost");
}
return answer;
};
const opts = { access: async () => signed, spaceId: "personal", ...server };
await assert.rejects(shareSources(opts).prepare(f.root, f.rendered), /network lost/);
assert.equal((await shareSources(opts).prepare(f.root, f.rendered)).at, "ready");
assert.equal(server.calls.begin[0].idempotencyKey, server.calls.begin[1].idempotencyKey);
assert.deepEqual(server.calls.begin[0].manifest, server.calls.begin[1].manifest);
});
test("confirmed expired upload clears its intent so a retry starts with a new key", async (t) => {
const f = await fixture(t);
const server = fakeServer();
const begin = server.api.begin;
server.api.begin = async (...args) => {
const answer = await begin(...args);
return server.calls.begin.length === 1 ? { ...answer, status: "expired" } : answer;
};
const source = shareSources({ access: async () => signed, spaceId: "personal", ...server });
assert.equal((await source.prepare(f.root, f.rendered)).reason, "source_upload_expired");
assert.equal((await f.store.read(f.key)).inflight, null);
assert.equal((await source.prepare(f.root, f.rendered)).at, "ready");
assert.notEqual(server.calls.begin[0].idempotencyKey, server.calls.begin[1].idempotencyKey);
});
test("an unwritable pending upload is cancelled so unchanged files can retry with a new key", async (t) => {
const f = await fixture(t);
const server = fakeServer();
const grants = server.api.uploadUrls;
server.api.uploadUrls = async () => {
throw new SyncApiError("file-not-writable");
};
server.api.status = async () => server.receipt("pending");
let cancelled = 0;
server.api.cancel = async () => {
cancelled++;
return { ...server.receipt("failed"), errorCode: "cancelled" };
};
const source = shareSources({ access: async () => signed, spaceId: "personal", ...server });
assert.equal((await source.prepare(f.root, f.rendered)).reason, "source_upload_not_writable");
assert.equal(cancelled, 1);
assert.equal((await f.store.read(f.key)).inflight, null);
server.api.uploadUrls = grants;
assert.equal((await source.prepare(f.root, f.rendered)).at, "ready");
assert.notEqual(server.calls.begin[0].idempotencyKey, server.calls.begin[1].idempotencyKey);
});
for (const status of ["verifying", "completed"]) {
test(`unwritable recovery retains an upload that raced cancellation into ${status}`, async (t) => {
const f = await fixture(t);
const server = fakeServer();
server.api.uploadUrls = async () => {
throw new SyncApiError("file-not-writable");
};
let reads = 0;
server.api.status = async () => {
const receipt = server.receipt(++reads === 1 ? "pending" : status);
if (receipt.result) receipt.result.publicationOutcome = "conflict_saved";
return receipt;
};
server.api.cancel = async () => {
throw new SyncApiError("upload-not-pending");
};
const source = shareSources({ access: async () => signed, spaceId: "personal", ...server });
assert.equal(
(await source.prepare(f.root, f.rendered)).reason,
status === "verifying" ? "source_verification_pending" : "source_conflict",
);
assert.equal((await f.store.read(f.key)).inflight.uploadId, "upload");
assert.equal(server.calls.begin.length, 1);
});
}
test("changed source cancels the previous pending upload before using a new key", async (t) => {
const f = await fixture(t);
const server = fakeServer();
const submit = server.api.submit;
server.api.submit = async () => {
throw new Error("lost connection");
};
server.api.status = async () => server.receipt("pending");
let cancelled = 0;
server.api.cancel = async () => {
cancelled++;
return { ...server.receipt("failed"), errorCode: "cancelled" };
};
const source = shareSources({ access: async () => signed, spaceId: "personal", ...server });
await assert.rejects(source.prepare(f.root, f.rendered), /lost connection/);
await writeFile(join(f.root, "index.html"), "<html>updated</html>");
server.api.submit = submit;
assert.equal((await source.prepare(f.root, await projectFingerprint(f.root))).at, "ready");
assert.equal(cancelled, 1);
assert.notEqual(server.calls.begin[0].idempotencyKey, server.calls.begin[1].idempotencyKey);
});
test("conflict and pending verification cannot be attached and keep recovery intent", async (t) => {
const f = await fixture(t);
const server = fakeServer();
server.api.submit = async () => ({
...server.receipt("completed"),
result: { snapshotId: "snapshot", publicationOutcome: "conflict_saved" },
});
const source = shareSources({ access: async () => signed, spaceId: "personal", ...server });
assert.equal((await source.prepare(f.root, f.rendered)).reason, "source_conflict");
assert.equal(await source.filesForExport(f.root, f.rendered), null);
assert.ok((await f.store.read(f.key)).inflight);
server.api.submit = async () => server.receipt("verifying");
const pending = shareSources({
access: async () => signed,
spaceId: "personal",
...server,
verificationTimeoutMs: 0,
});
assert.equal((await pending.prepare(f.root, f.rendered)).reason, "source_verification_pending");
});
test("Desktop-owned source is read-only and never re-shared into a different workspace", async (t) => {
const f = await fixture(t);
const server = fakeServer();
const capture = await captureProject(f.root);
const binding = {
...f.key,
version: 1,
compositionId: "desktop",
ownerSpaceId: "team",
base: { manifestSha256: capture.manifest.sha256, snapshotId: "desktop-snapshot" },
inflight: null,
};
const opts = {
access: async () => signed,
spaceId: "personal",
existingBinding: async () => binding,
...server,
};
assert.equal(
(await shareSources(opts).prepare(f.root, f.rendered)).reason,
"source_other_workspace",
);
binding.ownerSpaceId = "personal";
binding.sharedFrom = { reviewProjectId: "someone-elses-share" };
assert.equal(
(await shareSources(opts).prepare(f.root, f.rendered)).reason,
"source_owned_by_someone_else",
);
assert.equal(server.calls.create, 0);
});
test("binding reads reject copied folders, account changes, and symlink state files", async (t) => {
const f = await fixture(t);
await shareSources({ access: async () => signed, spaceId: "personal", ...fakeServer() }).prepare(
f.root,
f.rendered,
);
await assert.rejects(f.store.read({ ...f.key, account: "other" }), /another account/);
const g = await fixture(t);
const hash = createHash("sha256").update(signed.heygenApi).digest("hex").slice(0, 16);
const name = `.hyperframes/share-sync.${hash}.json`;
await mkdir(join(g.root, ".hyperframes"));
await writeFile(join(g.root, name), await readFile(join(f.root, name)));
await assert.rejects(g.store.read(g.key), /another account/);
await rm(join(g.root, name));
await symlink(join(f.root, name), join(g.root, name));
await assert.rejects(g.store.read(g.key));
});
test("HTTP adapter identifies CLI and sends credentials only to the API host", async () => {
const requests = [];
const api = sourceApi({
access: async () => signed,
fetchImpl: async (url, init) => {
requests.push({ url, init });
return Response.json({
data: {
upload_id: "up",
composition_id: "comp",
snapshot_id: "snap",
status: "pending",
manifest_sha256: "hash",
upload_deadline: "2099-01-01T00:00:00Z",
},
});
},
});
await api.begin("space", "comp", {
idempotencyKey: "key",
baseSnapshotId: null,
manifest: { sha256: "hash", document: {} },
});
assert.equal(JSON.parse(requests[0].init.body).client_type, "cli");
assert.equal(requests[0].init.headers.authorization, "Bearer test-token");
assert.match(requests[0].url, /^https:\/\/api.test\/v3\//);
});
+245
View File
@@ -0,0 +1,245 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import {
lstat,
mkdir,
readdir,
mkdtemp,
realpath,
rename,
rm,
symlink,
utimes,
writeFile,
} from "node:fs/promises";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { captureProject, hashListedFile, hashMemo } from "../src/sources/capture.mjs";
/** A project folder holding `files` (path to contents). @param {Record<string, string>} files */
async function project(files) {
const root = await mkdtemp(join(tmpdir(), "hf-sync-capture-"));
for (const [path, text] of Object.entries(files)) {
await mkdir(dirname(join(root, path)), { recursive: true });
await writeFile(join(root, path), text);
}
return root;
}
const sha = (/** @type {string} */ text) => createHash("sha256").update(text).digest("hex");
/** @param {Awaited<ReturnType<typeof captureProject>>} capture */
function captured(capture) {
assert.equal(capture.at, "captured", JSON.stringify(capture));
return /** @type {Extract<typeof capture, { at: "captured" }>} */ (capture);
}
test("a capture lists portable files with their hashes and leaves out what the policy excludes", async () => {
const root = await project({
"index.html": "<html></html>",
"assets/logo.png": "png",
".hyperframes/brand-kit.json": "{}",
".hyperframes/comments.jsonl": "{}",
".hyperframes/share.json": "{}",
"node_modules/x/index.js": "x",
".env": "SECRET=1",
".DS_Store": "",
"renders/out.mp4": "mp4",
"Icon\r": "",
});
const { manifest, waiting, skippedLinks } = captured(await captureProject(root));
assert.deepEqual(manifest.document.files, [
{ path: ".hyperframes/brand-kit.json", sha256: sha("{}"), size: 2 },
{ path: "assets/logo.png", sha256: sha("png"), size: 3 },
{ path: "index.html", sha256: sha("<html></html>"), size: 13 },
]);
assert.match(manifest.sha256, /^[0-9a-f]{64}$/);
assert.deepEqual([waiting, skippedLinks], [[], []]);
});
test("a render output the CLI recorded outside renders/ stays out", async () => {
const root = await project({
"index.html": "x",
"exports/cut.mp4": "mp4",
"exports/keep.txt": "keep",
"stills/a/b.png": "png",
".hyperframes/render-outputs.json": JSON.stringify({
version: 1,
paths: ["exports/cut.mp4", "stills"],
}),
});
const { manifest } = captured(await captureProject(root));
assert.deepEqual(
manifest.document.files.map((file) => file.path),
["exports/keep.txt", "index.html"],
);
});
test("a link fails the capture, except one under assets/podcast/, which is skipped", async () => {
const root = await project({ "index.html": "x", "outside.mov": "mov" });
await mkdir(join(root, "assets/podcast"), { recursive: true });
await symlink(join(root, "outside.mov"), join(root, "assets/podcast/cam.mov"));
const skipped = captured(await captureProject(root));
assert.deepEqual(skipped.skippedLinks, ["assets/podcast/cam.mov"]);
await symlink(join(root, "outside.mov"), join(root, "assets/cam.mov"));
assert.deepEqual(await captureProject(root), {
at: "problems",
problems: [{ path: "assets/cam.mov", problem: "symlink" }],
});
});
test("a name no client can store fails the capture with its code", async () => {
const root = await project({ "index.html": "x", "scenes/con.html": "x", "a?.txt": "x" });
const capture = await captureProject(root);
assert.equal(capture.at, "problems");
assert.deepEqual(
capture.at === "problems" ? capture.problems.map((p) => [p.path, p.code]).sort() : [],
[
["a?.txt", "path_invalid_character"],
["scenes/con.html", "reserved_name"],
],
);
});
test("a project without index.html cannot be captured", async () => {
const root = await project({ "scene.html": "x" });
assert.deepEqual(await captureProject(root), {
at: "problems",
problems: [{ path: "index.html", problem: "no-entrypoint" }],
});
});
test("over a limit, the capture says which and names the biggest files before hashing any", async () => {
const root = await project({ "index.html": "x", "big.bin": "0123456789", "mid.bin": "01234" });
const limits = { maxFiles: 10, maxFileBytes: 6, maxProjectBytes: 100, maxManifestBytes: 10_000 };
const capture = await captureProject(root, { limits });
assert.equal(capture.at, "too-big");
if (capture.at !== "too-big") return;
assert.deepEqual(
{ limit: capture.over.limit, actual: capture.over.actual, maximum: capture.over.maximum },
{ limit: "file_size", actual: 10, maximum: 6 },
);
assert.deepEqual(
capture.over.biggest.map((file) => file.path),
["big.bin", "mid.bin", "index.html"],
);
});
test("a media reservation is waited on and left out, and an open agent turn is reported", async () => {
const root = await project({
"index.html": "x",
".media/images/a.hf0a1b2c.tmp": "",
".hyperframes/history-turn.json": "{}",
});
const capture = captured(await captureProject(root));
assert.deepEqual(capture.waiting, [".media/images/a.hf0a1b2c.tmp"]);
assert.deepEqual(capture.settled, { agentTurnOpen: true, projectLocked: false });
});
test("a project lock held by a running process is reported; a dead holder's is not", async () => {
const root = await project({ "index.html": "x" });
const lock = join(root, ".hyperframes/project.lock");
await mkdir(dirname(lock), { recursive: true });
await writeFile(lock, JSON.stringify({ pid: process.pid, startedAt: 1, command: "render" }));
assert.equal(captured(await captureProject(root)).settled.projectLocked, true);
await writeFile(lock, JSON.stringify({ pid: 2 ** 30, startedAt: 1, command: "render" }));
assert.equal(captured(await captureProject(root)).settled.projectLocked, false);
});
test("a file over 4 MiB is hashed as a bounded stream, to the same digest", async () => {
const big = "x".repeat(5 * 1024 * 1024);
const root = await project({ "index.html": "x", "assets/big.bin": big });
const { manifest } = captured(await captureProject(root));
assert.deepEqual(manifest.document.files[0], {
path: "assets/big.bin",
sha256: sha(big),
size: big.length,
});
});
test("a kept hash is reused only while the file's size and time are unchanged", async () => {
const root = await project({ "index.html": "one" });
const memo = hashMemo();
const first = captured(await captureProject(root, { memo }));
await writeFile(join(root, "index.html"), "two");
await utimes(join(root, "index.html"), new Date(2030, 0, 1), new Date(2030, 0, 1));
const second = captured(await captureProject(root, { memo }));
assert.equal(first.manifest.document.files[0]?.sha256, sha("one"));
assert.equal(second.manifest.document.files[0]?.sha256, sha("two"));
});
test("a same-size edit inside the file time's granularity is hashed again, never read from a kept hash", async () => {
const root = await project({ "index.html": "one" });
const file = join(root, "index.html");
const tick = new Date();
await utimes(file, tick, tick);
const memo = hashMemo();
const first = captured(await captureProject(root, { memo }));
await writeFile(file, "two");
await utimes(file, tick, tick);
const second = captured(await captureProject(root, { memo }));
assert.equal(first.manifest.document.files[0]?.sha256, sha("one"));
assert.equal(second.manifest.document.files[0]?.sha256, sha("two"));
});
test("a settled file's hash is kept, so a check after a focus reads no bytes", async () => {
const root = await project({ "index.html": "one" });
const memo = hashMemo();
captured(await captureProject(root, { memo }));
await writeFile(join(root, "index.html"), "two");
const settled = new Date(2020, 0, 1);
await utimes(join(root, "index.html"), settled, settled);
captured(await captureProject(root, { memo }));
await writeFile(join(root, "index.html"), "six");
await utimes(join(root, "index.html"), settled, settled);
const kept = captured(await captureProject(root, { memo }));
assert.equal(kept.manifest.document.files[0]?.sha256, sha("two"));
});
/** A file as the walk lists it, read from disk now. @param {string} root @param {string} path */
async function listed(root, path) {
const full = join(root, path);
const { size, mtimeMs, ino, dev } = await lstat(full);
return { path, full, size, mtimeMs, ino, dev };
}
test("a file swapped for a link of the same size after the walk is refused, never followed", async () => {
const root = await realpath(await mkdtemp(join(tmpdir(), "hf-sync-swap-")));
const outside = await realpath(await mkdtemp(join(tmpdir(), "hf-sync-outside-")));
await writeFile(join(root, "a.txt"), "inside");
await writeFile(join(outside, "secret"), "SECRET");
// The same size and time as the file listed: only the file's identity tells them apart.
await utimes(join(root, "a.txt"), 1_700_000_000, 1_700_000_000);
await utimes(join(outside, "secret"), 1_700_000_000, 1_700_000_000);
const file = await listed(root, "a.txt");
assert.match(String(await hashListedFile(root, file)), /^[0-9a-f]{64}$/);
await rm(join(root, "a.txt"));
await symlink(join(outside, "secret"), join(root, "a.txt"));
assert.equal(await hashListedFile(root, file), null);
});
test("a folder swapped for a link to elsewhere after the walk is refused, even holding the same file", async () => {
const root = await realpath(await mkdtemp(join(tmpdir(), "hf-sync-swap-")));
const outside = await realpath(await mkdtemp(join(tmpdir(), "hf-sync-outside-")));
await mkdir(join(root, "assets"));
await writeFile(join(root, "assets", "logo.txt"), "inside");
const file = await listed(root, "assets/logo.txt");
await rename(join(root, "assets"), join(outside, "assets"));
await symlink(join(outside, "assets"), join(root, "assets"));
assert.equal(await hashListedFile(root, file), null, "the moved folder's own file was followed");
});
test("a file whose folder cannot be resolved once opened is refused, and its handle is closed", async () => {
const root = await realpath(await mkdtemp(join(tmpdir(), "hf-sync-close-")));
await mkdir(join(root, "assets"));
await writeFile(join(root, "assets", "logo.txt"), "inside");
const file = await listed(root, "assets/logo.txt");
const openFiles = async () => (await readdir("/dev/fd")).length;
const before = await openFiles();
const gone = async () => {
throw Object.assign(new Error("ENOENT: the folder was renamed"), { code: "ENOENT" });
};
await assert.rejects(hashListedFile(root, file, gone), /ENOENT/);
assert.equal(await openFiles(), before, "the opened file was left open");
});
@@ -0,0 +1,129 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import {
ManifestInvalid,
parseCanonicalManifest,
parseManifest,
} from "../src/sources/manifest.mjs";
import { overLimit, V1_LIMITS } from "../src/sources/limits.mjs";
import { excludedFolder, skippedSymlink } from "../src/sources/pathPolicy.mjs";
/** Reference vectors for the source manifest format. */
const vectors = JSON.parse(
readFileSync(
new URL("./fixtures/compositionSync/manifest-vectors.json", import.meta.url),
"utf8",
),
);
/** @param {() => unknown} run */
function refusal(run) {
try {
run();
} catch (error) {
if (error instanceof ManifestInvalid) return error;
throw error;
}
return null;
}
test("the vectors are portable-source-v4's, and there are vectors of every kind", () => {
assert.equal(vectors.policy_version, "portable-source-v4");
assert.deepEqual(
[vectors.valid.length, vectors.invalid.length, vectors.limits.length, vectors.stored.length],
[11, 147, 5, 8],
);
});
test("every valid vector builds the server's canonical bytes and manifest_sha256", () => {
for (const vector of vectors.valid) {
const manifest = parseManifest(vector.manifest);
assert.equal(manifest.canonical.toString("utf8"), vector.canonical, vector.name);
assert.equal(manifest.sha256, vector.manifest_sha256, vector.name);
assert.equal(
createHash("sha256").update(Buffer.from(vector.canonical, "utf8")).digest("hex"),
vector.manifest_sha256,
);
assert.equal(manifest.document.files.length, vector.file_count, vector.name);
const bytes = manifest.document.files.reduce((sum, file) => sum + file.size, 0);
assert.equal(bytes, vector.project_bytes, vector.name);
}
});
test("every invalid vector is refused with the server's code and rule", () => {
for (const vector of vectors.invalid) {
const error = refusal(() => parseManifest(vector.manifest));
assert.ok(error, `${vector.name} was accepted`);
assert.equal(error.code, vector.error.code, vector.name);
if (vector.error.rule) assert.equal(error.violations[0]?.rule, vector.error.rule, vector.name);
}
});
test("every limits vector breaks the server's first limit, or none", () => {
for (const vector of vectors.limits) {
const manifest = parseManifest(vector.manifest);
const limits = {
maxFiles: vector.limits.max_files,
maxFileBytes: vector.limits.max_file_bytes,
maxProjectBytes: vector.limits.max_project_bytes,
maxManifestBytes: vector.limits.max_manifest_bytes,
};
const over = overLimit(manifest.document.files, manifest.canonical.length, limits);
assert.equal(over?.limit ?? null, vector.limit, vector.name);
}
});
test("every stored vector reads only canonical bytes", () => {
for (const vector of vectors.stored) {
const body = Buffer.from(vector.body, "base64");
const error = refusal(() => parseCanonicalManifest(body));
assert.equal(error?.code ?? null, vector.error?.code ?? null, vector.name);
}
});
test("V1 limits are 10,000 files, 5 GiB a file, 10 GiB a project and an 8 MiB manifest", () => {
assert.deepEqual(V1_LIMITS, {
maxFiles: 10000,
maxFileBytes: 5368709120,
maxProjectBytes: 10737418240,
maxManifestBytes: 8388608,
});
});
test("a project over a limit names its biggest files first", () => {
const files = [
{ path: "index.html", size: 10 },
{ path: "assets/b.mp4", size: 6 * 1024 ** 3 },
{ path: "assets/a.mp4", size: 7 * 1024 ** 3 },
];
const over = overLimit(files, 300);
assert.equal(over?.limit, "file_size");
assert.equal(over?.actual, 7 * 1024 ** 3);
assert.deepEqual(
over?.biggest.map((file) => file.path),
["assets/a.mp4", "assets/b.mp4", "index.html"],
);
});
test("a capture skips only folders no included file can be under", () => {
assert.equal(excludedFolder("node_modules"), true);
assert.equal(excludedFolder("assets/node_modules"), true);
assert.equal(excludedFolder(".git"), true);
assert.equal(excludedFolder(".claude"), true);
assert.equal(excludedFolder("dist"), true);
assert.equal(excludedFolder("assets/dist"), false);
assert.equal(excludedFolder("renders"), true);
assert.equal(excludedFolder(".hyperframes"), false);
assert.equal(excludedFolder(".hyperframes/prepared-assets"), false);
assert.equal(excludedFolder("assets"), false);
assert.equal(excludedFolder("assets/renders"), false);
});
test("only links under assets/podcast/, in any spelling, are skipped", () => {
assert.equal(skippedSymlink("assets/podcast/cam-a.mov"), true);
assert.equal(skippedSymlink("Assets/Podcast/raw/cam-a.mov"), true);
assert.equal(skippedSymlink("assets/cam-a.mov"), false);
assert.equal(skippedSymlink("other/assets/podcast/cam-a.mov"), false);
});
+271
View File
@@ -0,0 +1,271 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtemp, open, rm, realpath, symlink, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createHash } from "node:crypto";
import { uploadNeeded } from "../src/sources/uploadFiles.mjs";
import { partDigests, putRange } from "../src/sources/transfer.mjs";
import fsPromises from "node:fs/promises";
import { syncBuiltinESMExports } from "node:module";
import { bindShareHost } from "../src/host.mjs";
bindShareHost({ reviewUploadOrigins: ["https://store.test"] });
async function folder(t) {
const root = await realpath(await mkdtemp(join(tmpdir(), "hf-source-upload-")));
t.after(() => rm(root, { recursive: true, force: true }));
return root;
}
const signal = () => new AbortController().signal;
test("multipart retries retain the pinned upload and skip completed parts after restart", async (t) => {
const root = await folder(t);
const file = join(root, "media.mp4");
const size = 65 * 1024 * 1024;
const handle = await open(file, "w");
await handle.truncate(size);
await handle.close();
const digests = await partDigests(file, size, 64 * 1024 * 1024, signal());
const inflight = { sent: [], multipart: {} };
const requests = [];
let fail = true;
let persisted = 0;
const job = {
api: {
uploadUrls: async (_space, _comp, _upload, files) => {
requests.push(...files);
return files.map((item) => ({
sha256: item.sha256,
multipartUploadId: "pinned",
expiresAt: "2099-01-01T00:00:00Z",
parts: item.parts.map((part) => ({
...part,
url: `https://store.test/${part.partNumber}`,
headers: { checksum: part.sha256 },
})),
}));
},
},
space: "space",
composition: "comp",
uploadId: "upload",
inflight,
local: new Map([[digests.whole, { path: "media.mp4", full: file }]]),
needed: [{ sha256: digests.whole, size }],
fetchImpl: async (url, init) => {
if (url.endsWith("/2") && fail) {
await init.body.cancel();
return new Response(null, { status: 400 });
}
const hash = createHash("sha256");
for await (const chunk of init.body) hash.update(chunk);
assert.equal(hash.digest("hex"), init.headers.checksum);
return new Response(null, { status: 200 });
},
signal: signal(),
wait: async () => {},
now: Date.now,
persist: () => persisted++,
onBytes: () => {},
onFile: () => {},
};
await assert.rejects(uploadNeeded(job), /refused/);
assert.deepEqual(inflight.multipart[digests.whole].partsDone, [1]);
fail = false;
await uploadNeeded(job);
assert.equal(requests[1].multipartUploadId, "pinned");
assert.deepEqual(
requests[1].parts.map((p) => p.partNumber),
[2],
);
assert.deepEqual(inflight.sent, [digests.whole]);
assert.ok(persisted >= 4);
});
test("signed URL expiry refreshes the grant without changing the uploaded content", async (t) => {
const root = await folder(t);
const file = join(root, "index.html");
await writeFile(file, "source");
const digest = createHash("sha256").update("source").digest("hex");
let grants = 0;
let puts = 0;
await uploadNeeded({
api: {
uploadUrls: async () => {
grants++;
return [
{
sha256: digest,
url: "https://store.test/file",
headers: {},
expiresAt: "2099-01-01T00:00:00Z",
},
];
},
},
space: "s",
composition: "c",
uploadId: "u",
inflight: { sent: [] },
local: new Map([[digest, { path: "index.html", full: file }]]),
needed: [{ sha256: digest, size: 6 }],
fetchImpl: async (_url, init) => {
puts++;
await new Response(init.body).arrayBuffer();
return new Response(null, { status: puts === 1 ? 403 : 200 });
},
signal: signal(),
wait: async () => {},
now: Date.now,
persist: () => {},
onBytes: () => {},
onFile: () => {},
});
assert.equal(grants, 2);
assert.equal(puts, 2);
});
test("storage PUT refuses a file swapped for a symlink before reading bytes", async (t) => {
const root = await folder(t);
const target = join(root, "private.txt");
const file = join(root, "asset.txt");
await writeFile(target, "secret");
await symlink(target, file);
let sent = false;
await assert.rejects(
putRange({
file,
path: "asset.txt",
start: 0,
length: 6,
sha256: "wrong",
signal: signal(),
url: "https://store.test",
headers: {},
fetchImpl: async () => {
sent = true;
return new Response();
},
}),
);
assert.equal(sent, false);
});
test("source PUT refuses untrusted or redirected targets before it opens a file", async () => {
for (const url of [
"https://untrusted.test/source",
"http://127.0.0.1:18080/source",
"https://store.test.untrusted.test/source",
"https://user:secret@store.test/source",
]) {
await assert.rejects(
putRange({
file: "/missing-source",
path: "index.html",
start: 0,
length: 6,
sha256: "unused",
signal: signal(),
url,
headers: {},
fetchImpl: async () => assert.fail("Must not send bytes"),
}),
{ kind: "refused" },
);
}
});
async function sourcePut(t, overrides = {}) {
const root = await folder(t);
const file = join(root, "index.html");
await writeFile(file, "source");
return {
file,
path: "index.html",
start: 0,
length: 6,
sha256: createHash("sha256").update("source").digest("hex"),
signal: signal(),
url: "https://store.test/file",
headers: {},
...overrides,
};
}
for (const consume of [false, true]) {
test(`source PUT times out when ${consume ? "the storage response" : "body consumption"} stalls`, async (t) => {
let requestSignal;
const put = await sourcePut(t, {
stallMs: 20,
signal: AbortSignal.timeout(2_000),
fetchImpl: async (_url, init) => {
requestSignal = init.signal;
if (consume) await new Response(init.body).arrayBuffer();
init.signal.throwIfAborted();
return new Promise((_resolve, reject) => {
init.signal.addEventListener("abort", () => reject(init.signal.reason), { once: true });
});
},
});
await assert.rejects(putRange(put), { kind: "transient" });
assert.equal(requestSignal.aborted, true);
});
}
test("source PUT preserves caller cancellation instead of retrying it", async (t) => {
const controller = new AbortController();
const cancelled = new DOMException("Sharing cancelled", "AbortError");
await assert.rejects(
putRange(
await sourcePut(t, {
signal: controller.signal,
fetchImpl: async (_url, init) => {
assert.equal(init.redirect, "error");
await new Response(init.body).arrayBuffer();
controller.abort(cancelled);
throw init.signal.reason;
},
}),
),
(error) => error === cancelled,
);
});
test("successful source PUT clears its stall timer", async (t) => {
let requestSignal;
await putRange(
await sourcePut(t, {
stallMs: 200,
fetchImpl: async (_url, init) => {
requestSignal = init.signal;
await new Response(init.body).arrayBuffer();
return new Response(null, { status: 200 });
},
}),
);
await new Promise((resolve) => setTimeout(resolve, 250));
assert.equal(requestSignal.aborted, false);
});
test("source PUT closes its file when resolving its parent fails", async (t) => {
const put = await sourcePut(t, { fetchImpl: async () => assert.fail("Must not send bytes") });
const originalOpen = fsPromises.open;
let opened;
t.mock.method(fsPromises, "open", async (...args) => {
opened = await originalOpen(...args);
return opened;
});
t.mock.method(fsPromises, "realpath", async () => {
throw Object.assign(new Error("Parent directory disappeared"), { code: "ENOENT" });
});
syncBuiltinESMExports();
try {
await assert.rejects(putRange(put), { code: "ENOENT" });
assert.equal(opened.fd, -1, "source file handle was left open");
} finally {
await opened?.close();
t.mock.restoreAll();
syncBuiltinESMExports();
}
});
+2 -2
View File
@@ -48,8 +48,8 @@
"unpacked": 825937
},
"@hyperframes/share": {
"packed": 60820,
"unpacked": 240537
"packed": 140130,
"unpacked": 542222
},
"@hyperframes/studio": {
"packed": 5998328,