mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-08-31 02:41:44 +00:00
feat(cli): search the catalog by meaning, on this machine (#3089)
* feat(cli): search the catalog by meaning, in three named tiers Browsing the registry means matching names and tags, which fails whenever the author's wording differs from yours. "make the pace feel faster" finds nothing when the move is described as "velocity-driven blur". This ranks by meaning instead. Three tiers, and the command always says which one answered: words shared vocabulary, free, offline, no account on-device bge-small, free, offline, one opt-in download hosted Gemini, free for signed-in HeyGen users The tier is stated because a quietly worse answer looks exactly like a good one. --json carries it as a token alongside dropped, shown, total and top_score, so an agent reads provenance as data rather than matching English that is written to be reworded. Two consents, asked once each, and never conflated. Sending a query is a privacy question, so the prompt says the query is sent. Downloading a model is a disk and bandwidth question, so that prompt talks about size. Neither fires without a terminal: an unattended run sends nothing and downloads nothing unless a flag records that a person agreed. The catalog is derived from registry-item.json rather than from a separate document, so the set that is ranked and the set that can be installed are the same object by construction. Only the on-device vectors are committed; the hosted vectors are nine megabytes and belong on the server. top_score is reported and never acted on. A "nothing matched" threshold looked clean on long briefs and collapsed on the short queries people type: "a logo appears" scores 0.6181 and keyboard mash scores 0.6417, so any cut that catches the noise rejects the real query. The measurement is in the evals directory rather than in this branch. Not covered here. The published recall figures were measured against a separate hand-written document, not against registry text, so they should not be quoted for this catalog until re-measured. The offline tier needs a normal install: a single-file build cannot load the native ONNX runtime, which the command now reports instead of silently degrading. And the drop-detection path has never been observed firing outside its author's tests. * fix(cli): make this branch pass the repo's own gates Three things `bun run lint` and `fallow audit --base origin/main` rejected. CI runs both, so none of this branch would have gone green. Found by running them, not by reading the diff. process.exit in catalog.ts, twice: an invalid --type and a cancelled picker. check:cli-process-ownership reserves that for cli.ts, and the rule is not cosmetic — process.exit tears the process down where it stands, so anything cli.ts has queued to run on the way out is dropped. finishCommand throws a CliResultSignal that cli.ts turns into the exit code, which is what init.ts already does for a cancelled prompt. Three exports with no consumers. normalize keeps its body and loses its export; localEmbedder is the only caller. modelsDirectory goes entirely, having no caller inside its file or out. The WordPieceConfig re-export goes, and with it the import it existed to forward: the type is exported from wordpiece.ts, where its consumers already take it from. Complexity. prepareOnDeviceTier is lifted out of run(), which took run from 64 cyclomatic and CRAP 948 to 54 and 684. That block is one decision — can the offline tier run, and if not, why not — and its only product is a list of warnings, so it reads and tests as a unit, which it could not do inline. The rest is suppressed rather than refactored, each with its reason on the line above. Finishing run() means extracting its three output paths, and that is a refactor of a command this branch already changes for other reasons: a separate initiative, not something to absorb here. Every suppression says what shape the function has and why; a bare marker on a function nobody can justify is how a threshold stops meaning anything. Verified: `bun run lint` exits 0, fallow reports no issues across 27 changed files, and 2540 CLI tests pass. * feat(cli): ship the local search tiers only, drop the hosted one Search now has two tiers, both local: shared-vocabulary word matching, and the opt-in on-device model. The hosted tier, which sent the query to a HeyGen endpoint and ranked it with a hosted model, is removed. This is a scope decision, not a defect. The endpoint works and its own change is reviewed and green; it is simply not what we want to ship first. Landing local only means the feature has no backend dependency, no auth requirement, and nothing leaves the machine unless someone opts into downloading a model. Gone: registry/smartSearch.ts and its test, the --smart and --no-smart flags, the outcome plumbing through the command, the remote branch of applySearch, the remote tier, and the hosted-only JSON fields (ranking, catalog_version, top_score). Also the smartSearchEnabled consent field in telemetry config, which was the persisted storage behind the hosted consent and would otherwise have been left as dead configuration surface. Kept exactly as they were: both local tiers, the --on-device and --yes flags, the download consent prompt, and the runtime check that happens before the download rather than after it. The --json envelope still reports query, tier, tier_detail, shown, total, dropped, warnings and results, so an agent can still tell which tier answered and why. tierToken now distinguishes on-device from words. Verified: lint exits 0, fallow reports no issues, 2522 CLI tests pass, and the command was exercised directly. A query answers on the on-device tier where the model is installed and falls back to word matching where it is not, reporting that fallback in warnings rather than silently. An unknown --type still exits 1 with a readable message, and --smart is now rejected as an unknown flag. * fix(cli): count only moves this registry cannot install as dropped The dropped count was computed against the list left after the user's own --type and --tag filters, so every move the user excluded was reported as one the registry is missing. Filtering made the number go up: the same query reported 277 unfiltered and 302 with --type block. The count exists so a caller can tell "nothing matched your words" apart from "the ranker suggested things this project cannot install". Conflating it with user filtering destroys exactly that signal, and worse, genuine index skew and a self-inflicted filter printed a byte-identical line with opposite remedies -- one means refresh the shelf, the other means drop a flag, and refreshing does nothing. Now counted against the registry rather than the filtered view. The manifest is already fetched whole and narrowed in memory, so keeping the unnarrowed name set costs no extra request, and item loading still runs only on the filtered subset. Verified against ground truth rather than by eye: the vector artifact holds 411 names, the registry holds 168 installable items, and 134 of those names exist in both, so 277 are genuinely uninstallable. The count now reads 277 unfiltered, 277 under --type block, 277 under --type component and 277 under --tag, and the skew it reports is real -- the artifact predates dropping the UI primitives and still ranks moves that are no longer on the shelf. Reported by Vance Ingalls, who also noted this closes an item the status doc listed as unverified. Two earlier sweeps could not make the count fire because neither combined a filter with a query. Tests pin the three cases: a genuinely absent name counts, a filter-excluded name does not, and a fully installable ranking reports zero. * fix(cli): tell the user when meaning search cannot see the catalog The on-device index was fetched once and never revalidated: the only freshness check was two existsSync calls. A move added after that fetch was invisible to meaning search permanently, not down-ranked but absent from the candidate set. The registry manifest on the same command carries a 24h TTL, so the two halves of one feature disagreed about staleness. The dropped count reported over-coverage only, names the index has that the registry lacks. Under-coverage was never computed, so the harmless direction was instrumented and the costly one was silent. Reproduced with an index truncated to 120 of 168 moves: dropped read 0, perfect health, while 48 moves were unreachable. Counts under-coverage from the name list the artifact already carries, so no extra request. Warns only when non-zero, and names the remedy. The remedy had to be made true: --on-device could not refresh a stale index because hasLocalVectors short-circuited the fetch. That flag now refetches when the index is absent or no longer covering. Two defects the reproduction surfaced. A failed refresh reported the tier unavailable while the old vectors were still on disk and still ranking. And the fetch wrote its two files one at a time, so failing between them paired a new name list with an old matrix, a hard load error rather than stale data. It now writes both or neither, which matters more once refresh runs on staleness. top_score returns, scoped to the on-device tier and set to the score of the best result actually shown rather than the ranking head, which can describe a row the caller never received. Also: scripts/ is now typechecked. It never was, which is how a build script that crashes after the paid embedding call, and two scripts whose imports do not resolve at all, went unnoticed. 43 errors fixed, no suppressions. And the docs stop describing a --smart hosted tier that was deleted, an item that does not exist, and a registry refresh that cannot fix a stale vector index. * ci: fail when the search index stops covering the registry The catalog vector artifact is regenerated by hand. Nothing in CI, in package.json or in a hook rebuilds it, because embedding needs the 32 MB model. So adding a registry item silently makes it invisible to meaning search until someone remembers to regenerate. The failure is asymmetric, which is what makes it easy to miss. Removing an item is self-healing: the ranker still scores the dead vector, then filters the name before display, so a user is never offered something they cannot install. Adding one is not: the item is absent from the candidate set entirely, not ranked low. Comparing the two name lists needs neither the model nor a network call, so the gate runs in seconds. CI checks rather than fixes, for the same reason it cannot regenerate. Scoped to blocks and components. Examples are starter projects a user scaffolds, never something catalog ranks, and the artifact carries no vector for them, so demanding one would keep this gate permanently red and it would be ignored within a week. Verified in both directions rather than assumed: adding an unindexed item exits 1 and names it, restoring the registry exits 0. * fix(catalog): rebuild the search index from the registry build-local-vectors.ts read registry/catalog-artifact/catalog.json, a file no script in this repo writes and which is not committed, so the documented regeneration command failed on a missing path. That is why the index could drift from the registry with nothing to run to fix it. It now reads registry/blocks/* and registry/components/* through catalogFromRegistry, the existing helper that already produced the right shape but had no caller. Rebuilding reproduces the shipped 168 rows byte for byte. A lefthook catalog-index command regenerates and re-stages both artifact files whenever a staged registry-item.json changes, mirroring the skills-manifest pattern, so adding or removing an item keeps the index in sync without anyone remembering to. Verified end to end: staging a new item took the artifact 168 to 169 rows and staged it in 0.80s. * fix(cli): refuse a half-downloaded vector cache The two artifact files have to agree on how many rows there are, and until now nothing checked that before writing them. A truncated or wrong-model response landed in the cache and only failed at load, on every later search, until someone cleared it by hand. The pair is now checked first and refused as a unit, and the cache is created 0o700 with 0o600 files rather than inheriting the umask of a directory the caller may have pointed anywhere. Also lifts the capture setup the two preview generators had drifted into sharing into scripts/preview-capture.ts, and splits the vector builders batching and packing out of main. Both were findings the audit attributed to this branch. * fix(cli): keep the catalog vitest run with the tests it runs Restacking took the base package.json wholesale, which dropped the vitest dependency and the scripts/catalog run this PR adds. Both belong here rather than under it. * fix(cli): stop the declined model download from happening anyway Answering no to the on-device download offer recorded no and warned, then carried on. The guard below it is localModelConsent() !== false, which the decline had just made false, so it was skipped rather than taken: control reached recordLocalModelConsent(true), overwrote the answer with yes, and fetched the 32 MB model the user had refused. Next run it never asked again. No test could catch it. The stub pinned localModelStatus to ready, so the prompt never fired, and recordLocalModelConsent was a no-op that recorded nothing. Two tests now cover the offer, and they need three things the old stubs did not model: the run has to look like a terminal, because off one the command treats --on-device as the consent and never asks; the ONNX probe has to answer true, or an accepted offer returns at the runtime guard before it can download; and the status has to follow the recorded answer, or the second offer later in the run fires as well. Removing the return makes the decline test fail. * fix(catalog): let someone without the model still add a component The pre-commit hook rebuilds the search index, and rebuilding needs the 32 MB embedding model. An outside contributor adding a registry item does not have it, so their commit died inside the ONNX loader on an ENOENT naming a path they never set, and the CI gate then told them to run the command that had just crashed. The model is an opt-in for search, not a build dependency, so nobody is charged for it to contribute. The builder checks first and explains itself, exiting 3 for cannot as distinct from 1 for failed. The hook treats 3 as skip and lets the commit through. The gate now names both paths: regenerate if you have the model, leave it if you do not and a maintainer will. Verified both ways: with no model the builder explains and the hook exits 0; with the model it still regenerates byte-identically. * docs: say that anyone can add a registry item, and stop hand-editing a generated file Two defects, one of them the reason 64 stale entries survived in registry.json. The checklist told contributors to add their item to registry/registry.json. That file is generated from the item directories, so an entry added by hand survives until the next regeneration and then vanishes, and one left behind for a directory that no longer exists is worse: hyperframes add resolves the name and then fails on missing files. Both CONTRIBUTING.md and the agent-facing skill reference now run the generator instead. Nothing said contribution was maintainer-only, but nothing said it was not either, and two steps do need assets an outside contributor has no reason to install. Those are now named in a table with what happens if you do not have them, matching how the preview image was already handled. The search index is the new one: the model behind it is a 32 MB opt-in for search, not a build dependency. * fix(cli): harden on-device catalog search * fix(cli): refresh stale catalog vectors * test: create catalog vector temp dirs securely
This commit is contained in:
Miguel Ángel
GitHub
parent
f28bc80a1d
commit
68205dbbc1
20
.github/workflows/ci.yml
vendored
20
.github/workflows/ci.yml
vendored
@ -34,6 +34,7 @@ jobs:
|
||||
code: ${{ steps.filter.outputs.code }}
|
||||
cli: ${{ steps.filter.outputs.cli }}
|
||||
skills: ${{ steps.filter.outputs.skills }}
|
||||
catalog_index: ${{ steps.filter.outputs.catalog_index }}
|
||||
codex_plugin: ${{ steps.filter.outputs.codex_plugin }}
|
||||
gcp_beginframe: ${{ steps.filter.outputs.gcp_beginframe }}
|
||||
studio: ${{ steps.filter.outputs.studio }}
|
||||
@ -52,6 +53,10 @@ jobs:
|
||||
with:
|
||||
token: ""
|
||||
filters: |
|
||||
catalog_index:
|
||||
- "registry/registry.json"
|
||||
- "registry/catalog-artifact/**"
|
||||
- "scripts/catalog/check-artifact-coverage.ts"
|
||||
code:
|
||||
- "packages/**"
|
||||
- "scripts/**"
|
||||
@ -363,6 +368,21 @@ jobs:
|
||||
# `gen:skills-manifest --check`, which compares per-skill content hashes; the
|
||||
# manifest carries no version/timestamp, so it only fails on real content
|
||||
# drift. bun runs the TS script directly, no install needed.
|
||||
catalog-index-coverage:
|
||||
name: "Catalog: search index covers the registry"
|
||||
needs: changes
|
||||
if: needs.changes.outputs.catalog_index == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 3
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
# Comparing two name lists needs neither the embedding model nor a
|
||||
# network call, so this stays a seconds-long gate. Regenerating the
|
||||
# vectors does need the model, which is why CI checks rather than fixes.
|
||||
- name: Verify every searchable registry item has a vector
|
||||
run: bun scripts/catalog/check-artifact-coverage.ts
|
||||
|
||||
skills-manifest:
|
||||
name: "Skills: manifest in sync"
|
||||
needs: changes
|
||||
|
||||
@ -91,11 +91,36 @@ Blocks don't need `demo.html` — they are already standalone compositions.
|
||||
|
||||
### Checklist for new items
|
||||
|
||||
**Anyone can add an item.** Nothing here needs commit access, and the two steps
|
||||
that do need something a contributor may not have are handled by a maintainer
|
||||
before merge, listed at the end.
|
||||
|
||||
1. Create `registry/<blocks|components>/<name>/registry-item.json` following the [schema](packages/core/schemas/registry-item.json)
|
||||
2. Add the item to `registry/registry.json`
|
||||
3. For components: include a `demo.html`
|
||||
4. Run `npx hyperframes lint` and `npx hyperframes validate` on your HTML
|
||||
5. Test the install flow: `hyperframes add <name> --dir /tmp/test-project`
|
||||
2. For components: include a `demo.html`
|
||||
3. Run `npx hyperframes lint` and `npx hyperframes validate` on your HTML
|
||||
4. Test the install flow: `hyperframes add <name> --dir /tmp/test-project`
|
||||
5. Regenerate the manifest: `npx tsx scripts/generate-registry-items.ts`
|
||||
|
||||
`registry/registry.json` is generated from the item directories, so edit it with
|
||||
that script rather than by hand. An entry added by hand survives until the next
|
||||
regeneration and then disappears; entries left behind for directories that no
|
||||
longer exist are worse, because `hyperframes add <name>` resolves the name and
|
||||
then fails on missing files.
|
||||
|
||||
### What a maintainer finishes for you
|
||||
|
||||
Two things need assets an outside contributor is not expected to install. Open
|
||||
the pull request without them and say so; neither blocks review.
|
||||
|
||||
| Thing | If you have it | If you do not |
|
||||
| ----------------------------------------------- | --------------------------------------------------------- | ----------------------------------------------------------------------- |
|
||||
| The search index (`registry/catalog-artifact/`) | The pre-commit hook rebuilds and stages it | The hook skips, CI names the gap, a maintainer regenerates before merge |
|
||||
| The catalog preview image | Internal contributors run `scripts/upload-docs-images.sh` | Attach the preview MP4 to the PR instead |
|
||||
|
||||
The search index needs a 32 MB embedding model, which is an opt-in for catalog
|
||||
search rather than a build dependency. Until it is regenerated your item is
|
||||
findable by word search and not by meaning, which is the same state as any item
|
||||
published since a user last refreshed their copy.
|
||||
|
||||
### Auto-generated docs
|
||||
|
||||
|
||||
1
bun.lock
1
bun.lock
@ -18,6 +18,7 @@
|
||||
"oxlint": "^1.56.0",
|
||||
"tsx": "^4.21.0",
|
||||
"typescript": "^5.0.0",
|
||||
"vitest": "^3.2.4",
|
||||
"yaml": "^2.9.0",
|
||||
},
|
||||
},
|
||||
|
||||
@ -21,6 +21,19 @@
|
||||
"format": "uri",
|
||||
"description": "Registry homepage URL."
|
||||
},
|
||||
"catalogArtifact": {
|
||||
"type": "object",
|
||||
"description": "Published on-device vector artifact for this registry.",
|
||||
"required": ["revision"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"revision": {
|
||||
"type": "string",
|
||||
"pattern": "^[a-f0-9]{64}$",
|
||||
"description": "SHA-256 identity of the searchable corpus and embedding contract."
|
||||
}
|
||||
}
|
||||
},
|
||||
"items": {
|
||||
"type": "array",
|
||||
"description": "Items in this registry. Each entry is a shorthand reference; the full item manifest lives at <type-dir>/<name>/registry-item.json.",
|
||||
|
||||
13
lefthook.yml
13
lefthook.yml
@ -17,6 +17,19 @@ pre-commit:
|
||||
# enforces the same via the "Skills: manifest in sync" job). Churn-free:
|
||||
# the generator rewrites only when a content hash actually changed.
|
||||
run: bun packages/cli/scripts/gen-skills-manifest.ts && git add skills-manifest.json
|
||||
catalog-index:
|
||||
glob: "registry/*/*/registry-item.json"
|
||||
# Rebuild the on-device search vectors when an item's searchable text
|
||||
# changes, then re-stage them so registry/catalog-artifact/ never drifts
|
||||
# from the registry (CI enforces the same via the "Catalog: search index
|
||||
# covers the registry" job). Churn-free: identical inputs re-embed to
|
||||
# identical bytes, so an unrelated registry edit leaves no diff.
|
||||
# Exit 3 is "no embedding model here", which is the normal case for an
|
||||
# outside contributor. Their commit must not be blocked over a 32 MB
|
||||
# opt-in they were never asked to install; CI names the gap instead.
|
||||
run: |
|
||||
bun scripts/catalog/build-local-vectors.ts || { [ $? -eq 3 ] && exit 0; exit 1; }
|
||||
git add registry/registry.json registry/catalog-artifact/local-vectors.json registry/catalog-artifact/local-vectors.bin
|
||||
typecheck:
|
||||
glob: "*.{ts,tsx}"
|
||||
run: cd packages/core && bunx tsc --noEmit && cd ../studio && bunx tsc --noEmit && cd ../.. && bunx tsc --noEmit -p scripts/tsconfig.json
|
||||
|
||||
@ -48,7 +48,7 @@
|
||||
"player:perf": "bun run --filter @hyperframes/player perf",
|
||||
"format:check": "oxfmt --check .",
|
||||
"knip": "knip",
|
||||
"test:scripts": "node --import tsx --test scripts/check-tracked-artifacts.test.mjs scripts/check-no-main-deletions.test.mjs scripts/check-docs-snippet-motion.test.mjs scripts/registry-target-paths.test.mjs scripts/check-workspace-contracts.test.mjs scripts/check-package-cycles.test.mjs scripts/check-cli-process-ownership.test.mjs scripts/package-subpaths.test.mjs scripts/validate-release-channel.test.mjs scripts/publish-workflow.test.mjs scripts/install-workspace-dependencies.test.mjs scripts/draft-changelog.test.ts scripts/set-version.test.ts scripts/release-prepare.test.ts scripts/cli-options.test.ts scripts/changelog-weekly.test.ts scripts/claude-plugin-compression.test.ts scripts/studio-runtime-smoke.test.mjs scripts/verify-packed-manifests.test.mjs scripts/lint-skills.test.mjs packages/gcp-cloud-run/check-dockerfile-workspaces.test.mjs",
|
||||
"test:scripts": "node --import tsx --test scripts/check-tracked-artifacts.test.mjs scripts/check-no-main-deletions.test.mjs scripts/check-docs-snippet-motion.test.mjs scripts/registry-target-paths.test.mjs scripts/check-workspace-contracts.test.mjs scripts/check-package-cycles.test.mjs scripts/check-cli-process-ownership.test.mjs scripts/package-subpaths.test.mjs scripts/validate-release-channel.test.mjs scripts/publish-workflow.test.mjs scripts/install-workspace-dependencies.test.mjs scripts/draft-changelog.test.ts scripts/set-version.test.ts scripts/release-prepare.test.ts scripts/cli-options.test.ts scripts/changelog-weekly.test.ts scripts/claude-plugin-compression.test.ts scripts/studio-runtime-smoke.test.mjs scripts/verify-packed-manifests.test.mjs scripts/lint-skills.test.mjs packages/gcp-cloud-run/check-dockerfile-workspaces.test.mjs && vitest run scripts/catalog/",
|
||||
"typecheck:scripts": "tsc --noEmit -p scripts/tsconfig.json",
|
||||
"test:skills": "node --test 'skills/**/*.test.mjs'",
|
||||
"generate:previews": "tsx scripts/generate-template-previews.ts",
|
||||
@ -72,6 +72,7 @@
|
||||
"oxlint": "^1.56.0",
|
||||
"tsx": "^4.21.0",
|
||||
"typescript": "^5.0.0",
|
||||
"vitest": "^3.2.4",
|
||||
"yaml": "^2.9.0"
|
||||
},
|
||||
"resolutions": {
|
||||
|
||||
432
packages/cli/src/commands/catalog.test.ts
Normal file
432
packages/cli/src/commands/catalog.test.ts
Normal file
@ -0,0 +1,432 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { countUnindexed, pickByName } from "./catalog.js";
|
||||
|
||||
/** The whole registry, which is what "in this registry" has to be measured against. */
|
||||
const registryNames = new Set(["fade-through", "whip-pan", "count-up"]);
|
||||
const item = (name: string): { name: string } => ({ name });
|
||||
|
||||
describe("pickByName", () => {
|
||||
it("counts only the ranked names this registry has no item for", () => {
|
||||
const { ranked, missing } = pickByName(
|
||||
[item("fade-through"), item("whip-pan"), item("count-up")],
|
||||
["whip-pan", "fade-through", "accordion", "alert-dialog"],
|
||||
registryNames,
|
||||
);
|
||||
|
||||
expect(ranked.map((entry) => entry.name)).toEqual(["whip-pan", "fade-through"]);
|
||||
// accordion and alert-dialog are in the ranking artifact and nowhere in the
|
||||
// registry: a real skew between two separately published generations.
|
||||
expect(missing).toBe(2);
|
||||
});
|
||||
|
||||
it("does not count moves the user's own filter removed", () => {
|
||||
// `items` is what survived --type/--tag; the registry still has the rest.
|
||||
const { ranked, missing } = pickByName(
|
||||
[item("fade-through")],
|
||||
["whip-pan", "fade-through", "count-up", "accordion"],
|
||||
registryNames,
|
||||
);
|
||||
|
||||
expect(ranked.map((entry) => entry.name)).toEqual(["fade-through"]);
|
||||
// whip-pan and count-up are installable, just filtered out. Only accordion
|
||||
// is genuinely absent, and filtering must not inflate that number.
|
||||
expect(missing).toBe(1);
|
||||
});
|
||||
|
||||
it("reports nothing missing when the whole ranking is installable", () => {
|
||||
const { missing } = pickByName(
|
||||
[item("fade-through")],
|
||||
["fade-through", "whip-pan"],
|
||||
registryNames,
|
||||
);
|
||||
|
||||
expect(missing).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("countUnindexed", () => {
|
||||
it("counts the registry moves the on-device index holds no vector for", () => {
|
||||
// The move published after the artifact was fetched. Meaning search cannot
|
||||
// rank it at all, which is the failure this number exists to expose.
|
||||
expect(countUnindexed(registryNames, ["fade-through"])).toBe(2);
|
||||
});
|
||||
|
||||
it("reports nothing when the index covers the registry", () => {
|
||||
expect(countUnindexed(registryNames, ["count-up", "whip-pan", "fade-through"])).toBe(0);
|
||||
});
|
||||
|
||||
it("does not let names the registry dropped paper over a gap", () => {
|
||||
// The artifact holds two names this registry cannot install and is missing
|
||||
// two it can. Comparing sizes rather than membership would call that even.
|
||||
expect(countUnindexed(registryNames, ["fade-through", "accordion", "alert-dialog"])).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
// ── The command envelope ────────────────────────────────────────────────────
|
||||
// The JSON envelope is the surface an agent reads, so under-coverage and the
|
||||
// score are pinned where they are actually published rather than only at the
|
||||
// helper that computes them.
|
||||
|
||||
const state = vi.hoisted(() => ({
|
||||
registry: [] as Array<{ name: string; type: string; tags?: string[] }>,
|
||||
artifactRevision: "revision-current",
|
||||
cachedVectorRevision: "revision-current",
|
||||
vectorFetches: 0,
|
||||
vectorFetchSucceeds: true,
|
||||
ranking: null as Array<{ name: string; score: number }> | null,
|
||||
rankingError: null as Error | null,
|
||||
indexed: [] as string[],
|
||||
// The consent path. Static stubs could not reach it: with the status pinned
|
||||
// to "ready" the prompt never fires, so the answer was never a variable and
|
||||
// the decline branch was never executed by any test.
|
||||
modelStatus: "ready" as "ready" | "not-asked" | "declined" | "unavailable",
|
||||
confirmAnswer: true as boolean,
|
||||
consentRecorded: [] as boolean[],
|
||||
downloads: 0,
|
||||
runtimeAvailable: true,
|
||||
}));
|
||||
|
||||
vi.mock("../registry/resolver.js", () => ({
|
||||
loadAllItems: async (entries: Array<{ name: string; type: string; tags?: string[] }>) =>
|
||||
entries.map((entry) => ({
|
||||
name: entry.name,
|
||||
type: entry.type,
|
||||
title: entry.name,
|
||||
description: `${entry.name} description`,
|
||||
tags: entry.tags ?? [],
|
||||
})),
|
||||
}));
|
||||
|
||||
vi.mock("../registry/remote.js", () => ({
|
||||
fetchRegistryManifest: async () => ({
|
||||
items: state.registry,
|
||||
catalogArtifact: { revision: state.artifactRevision },
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock("@clack/prompts", () => ({
|
||||
confirm: async () => state.confirmAnswer,
|
||||
isCancel: (value: unknown) => value === null,
|
||||
}));
|
||||
|
||||
vi.mock("../registry/localModel.js", () => ({
|
||||
// "ready" is a user who opted into the on-device tier at some point. Every
|
||||
// later search takes that tier with no flag, which is how a frozen artifact
|
||||
// goes on answering forever. "not-asked" is the first run, the one that asks.
|
||||
// Recording an answer is what stops the CLI asking again, so the stub has to
|
||||
// move with it. Pinned to "not-asked" the second offer later in the run also
|
||||
// fires, and the double prompt looks like a product bug rather than a stub
|
||||
// that does not model the contract.
|
||||
localModelStatus: () => {
|
||||
const answer = state.consentRecorded.at(-1);
|
||||
return {
|
||||
status: answer === false ? "declined" : answer === true ? "ready" : state.modelStatus,
|
||||
};
|
||||
},
|
||||
ensureLocalModel: async () => {
|
||||
state.downloads += 1;
|
||||
if (state.modelStatus === "unavailable") state.modelStatus = "ready";
|
||||
return true;
|
||||
},
|
||||
recordLocalModelConsent: (enabled: boolean) => {
|
||||
state.consentRecorded.push(enabled);
|
||||
},
|
||||
downloadOfferMessage: () => "offer",
|
||||
nonInteractiveConsentMessage: () => "consent",
|
||||
}));
|
||||
|
||||
vi.mock("../registry/localEmbedder.js", () => ({
|
||||
// The native runtime is present in these tests. Left unmocked it answers
|
||||
// false under vitest, and every accepted offer returns at the runtime guard
|
||||
// before it can download, which looks like the download being skipped.
|
||||
localRuntimeAvailable: async () => state.runtimeAvailable,
|
||||
}));
|
||||
|
||||
vi.mock("../registry/localSemantic.js", () => ({
|
||||
localSemanticRanking: async () => {
|
||||
if (state.rankingError) throw state.rankingError;
|
||||
return state.ranking;
|
||||
},
|
||||
localVectorNames: () => state.indexed,
|
||||
cachedLocalVectorRevision: () => state.cachedVectorRevision,
|
||||
hasLocalVectors: () => true,
|
||||
fetchLocalVectors: async (_registry: string, options: { expectedRevision?: string } = {}) => {
|
||||
state.vectorFetches += 1;
|
||||
if (state.vectorFetchSucceeds && options.expectedRevision !== undefined) {
|
||||
state.cachedVectorRevision = options.expectedRevision;
|
||||
}
|
||||
return state.vectorFetchSucceeds;
|
||||
},
|
||||
}));
|
||||
|
||||
const block = (name: string, tags?: string[]): { name: string; type: string; tags?: string[] } => ({
|
||||
name,
|
||||
type: "hyperframes:block",
|
||||
tags,
|
||||
});
|
||||
const component = (name: string): { name: string; type: string } => ({
|
||||
name,
|
||||
type: "hyperframes:component",
|
||||
});
|
||||
|
||||
interface Envelope {
|
||||
tier: string;
|
||||
dropped: number;
|
||||
unindexed: number;
|
||||
top_score?: number;
|
||||
shown: number;
|
||||
warnings?: string[];
|
||||
}
|
||||
|
||||
async function runCatalog(args: Record<string, unknown>): Promise<string> {
|
||||
const command = (await import("./catalog.js")).default as unknown as {
|
||||
run: (context: { args: Record<string, unknown> }) => Promise<void>;
|
||||
};
|
||||
const lines: string[] = [];
|
||||
const log = vi.spyOn(console, "log").mockImplementation((...parts: unknown[]) => {
|
||||
lines.push(parts.map(String).join(" "));
|
||||
});
|
||||
try {
|
||||
await command.run({ args });
|
||||
} finally {
|
||||
log.mockRestore();
|
||||
}
|
||||
// Colour is decoration; assertions are about the words. The escape byte is
|
||||
// built rather than written: as a literal or as \u001B it is a control
|
||||
// character in the source, which the lint rules reject either way.
|
||||
const ansi = new RegExp(`${String.fromCharCode(27)}\\[[0-9;]*m`, "g");
|
||||
return lines.join("\n").replace(ansi, "");
|
||||
}
|
||||
|
||||
async function runEnvelope(args: Record<string, unknown>): Promise<Envelope> {
|
||||
return JSON.parse(await runCatalog({ json: true, ...args })) as Envelope;
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
state.modelStatus = "ready";
|
||||
state.artifactRevision = "revision-current";
|
||||
state.cachedVectorRevision = "revision-current";
|
||||
state.vectorFetches = 0;
|
||||
state.vectorFetchSucceeds = true;
|
||||
state.rankingError = null;
|
||||
state.confirmAnswer = true;
|
||||
state.consentRecorded = [];
|
||||
state.downloads = 0;
|
||||
state.runtimeAvailable = true;
|
||||
state.registry = [block("count-up"), block("fade-through"), component("whip-pan")];
|
||||
state.indexed = ["count-up", "fade-through", "whip-pan"];
|
||||
state.ranking = [
|
||||
{ name: "count-up", score: 0.71 },
|
||||
{ name: "whip-pan", score: 0.42 },
|
||||
{ name: "fade-through", score: 0.31 },
|
||||
];
|
||||
});
|
||||
|
||||
describe("catalog --json meaning search", () => {
|
||||
it("reports the registry moves meaning search cannot see", async () => {
|
||||
// Published after the user's artifact was fetched: in the registry, absent
|
||||
// from the index, and therefore unreturnable by any query.
|
||||
state.registry.push(block("split-screen"));
|
||||
|
||||
const envelope = await runEnvelope({ query: "make a number count up" });
|
||||
|
||||
expect(envelope.tier).toBe("on-device");
|
||||
expect(envelope.unindexed).toBe(1);
|
||||
});
|
||||
|
||||
it("reports nothing unindexed when the artifact matches the registry", async () => {
|
||||
const envelope = await runEnvelope({ query: "make a number count up" });
|
||||
|
||||
expect(envelope.unindexed).toBe(0);
|
||||
});
|
||||
|
||||
it("measures unindexed against the unfiltered registry under --type", async () => {
|
||||
// The missing move is a component; the user asked for blocks. Their filter
|
||||
// is not the index being stale, and it must not hide a stale index either.
|
||||
state.registry.push(component("push-in"));
|
||||
|
||||
const envelope = await runEnvelope({ query: "make a number count up", type: "block" });
|
||||
|
||||
expect(envelope.shown).toBe(2);
|
||||
expect(envelope.unindexed).toBe(1);
|
||||
});
|
||||
|
||||
it("keeps dropped counted against the unfiltered registry under --type", async () => {
|
||||
// accordion is the only name the registry genuinely lacks. whip-pan is
|
||||
// installable and merely filtered out, so it is not a drop.
|
||||
state.ranking = [
|
||||
{ name: "count-up", score: 0.71 },
|
||||
{ name: "whip-pan", score: 0.62 },
|
||||
{ name: "accordion", score: 0.55 },
|
||||
{ name: "fade-through", score: 0.31 },
|
||||
];
|
||||
|
||||
const envelope = await runEnvelope({ query: "make a number count up", type: "block" });
|
||||
|
||||
expect(envelope.dropped).toBe(1);
|
||||
expect(envelope.shown).toBe(2);
|
||||
});
|
||||
|
||||
it("carries the score of the best result it actually showed", async () => {
|
||||
// The top-ranked name is not installable here, so reporting the ranking's
|
||||
// own head would describe a row the caller never received.
|
||||
state.ranking = [
|
||||
{ name: "accordion", score: 0.93 },
|
||||
{ name: "count-up", score: 0.71 },
|
||||
{ name: "whip-pan", score: 0.42 },
|
||||
{ name: "fade-through", score: 0.31 },
|
||||
];
|
||||
|
||||
const envelope = await runEnvelope({ query: "make a number count up" });
|
||||
|
||||
expect(envelope.top_score).toBeCloseTo(0.71);
|
||||
});
|
||||
|
||||
it("omits the score on the word tier, whose scale is not the same one", async () => {
|
||||
state.ranking = null;
|
||||
|
||||
const envelope = await runEnvelope({ query: "count up" });
|
||||
|
||||
expect(envelope.tier).toBe("words");
|
||||
expect(envelope.top_score).toBeUndefined();
|
||||
// Word matching ranks the live registry listing, so it is never stale.
|
||||
expect(envelope.unindexed).toBe(0);
|
||||
});
|
||||
|
||||
it("finds registry tags on the word tier", async () => {
|
||||
state.modelStatus = "declined";
|
||||
state.ranking = null;
|
||||
state.registry = [block("fade-through", ["transition"]), block("count-up", ["number"])];
|
||||
|
||||
const envelope = await runEnvelope({ query: "transition" });
|
||||
|
||||
expect(envelope.tier).toBe("words");
|
||||
expect(envelope.shown).toBe(1);
|
||||
});
|
||||
|
||||
it("carries an on-device runtime failure into the JSON envelope", async () => {
|
||||
state.rankingError = new Error("model could not load");
|
||||
|
||||
const envelope = await runEnvelope({ query: "count up" });
|
||||
|
||||
expect(envelope.tier).toBe("words");
|
||||
expect(envelope.warnings).toEqual(["on-device search did not run: model could not load"]);
|
||||
});
|
||||
|
||||
it("refreshes a changed vector revision under existing consent", async () => {
|
||||
state.cachedVectorRevision = "revision-previous";
|
||||
|
||||
const envelope = await runEnvelope({ query: "count up" });
|
||||
|
||||
expect(envelope.tier).toBe("on-device");
|
||||
expect(state.vectorFetches).toBe(1);
|
||||
expect(state.cachedVectorRevision).toBe("revision-current");
|
||||
expect(state.consentRecorded).toEqual([]);
|
||||
});
|
||||
|
||||
it("replaces a changed model revision under existing consent", async () => {
|
||||
state.modelStatus = "unavailable";
|
||||
|
||||
const envelope = await runEnvelope({ query: "count up" });
|
||||
|
||||
expect(envelope.tier).toBe("on-device");
|
||||
expect(state.downloads).toBe(1);
|
||||
expect(state.consentRecorded).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps the previous vectors and reports a failed routine refresh", async () => {
|
||||
state.cachedVectorRevision = "revision-previous";
|
||||
state.vectorFetchSucceeds = false;
|
||||
|
||||
const envelope = await runEnvelope({ query: "count up" });
|
||||
|
||||
expect(envelope.tier).toBe("on-device");
|
||||
expect(state.cachedVectorRevision).toBe("revision-previous");
|
||||
expect(envelope.warnings).toEqual([
|
||||
"on-device search is using the previous catalog vectors because the update failed",
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("catalog meaning search, on a terminal", () => {
|
||||
it("says how much is missing and what to run about it", async () => {
|
||||
state.registry.push(block("split-screen"));
|
||||
|
||||
const output = await runCatalog({ query: "make a number count up" });
|
||||
|
||||
expect(output).toContain("1 of 4 moves are missing from the on-device index");
|
||||
expect(output).toContain("Re-run with --on-device to refresh it.");
|
||||
});
|
||||
|
||||
it("says nothing when the index covers the registry", async () => {
|
||||
const output = await runCatalog({ query: "make a number count up" });
|
||||
|
||||
expect(output).not.toContain("missing from the on-device index");
|
||||
});
|
||||
});
|
||||
|
||||
describe("the on-device download offer", () => {
|
||||
// The offer only exists for someone who can answer it. Off a terminal the
|
||||
// caller must add --yes explicitly, so a test that forgets the terminal
|
||||
// never reaches the prompt and passes for the wrong reason.
|
||||
const asATerminal = async (run: () => Promise<string>): Promise<string> => {
|
||||
const descriptor = Object.getOwnPropertyDescriptor(process.stdout, "isTTY");
|
||||
Object.defineProperty(process.stdout, "isTTY", { value: true, configurable: true });
|
||||
try {
|
||||
return await run();
|
||||
} finally {
|
||||
if (descriptor) Object.defineProperty(process.stdout, "isTTY", descriptor);
|
||||
else delete (process.stdout as unknown as { isTTY?: boolean }).isTTY;
|
||||
}
|
||||
};
|
||||
|
||||
it("downloads nothing and records no consent when the offer is declined", async () => {
|
||||
// The whole point of asking. Nothing below this line may fetch 32 MB.
|
||||
state.modelStatus = "not-asked";
|
||||
state.confirmAnswer = false;
|
||||
|
||||
const output = await asATerminal(() =>
|
||||
runCatalog({ query: "make a number count up", "on-device": true }),
|
||||
);
|
||||
|
||||
expect(state.downloads).toBe(0);
|
||||
expect(state.consentRecorded).toEqual([false]);
|
||||
expect(output).not.toContain("offer");
|
||||
});
|
||||
|
||||
it("downloads once when the offer is accepted", async () => {
|
||||
state.modelStatus = "not-asked";
|
||||
state.confirmAnswer = true;
|
||||
|
||||
await asATerminal(() => runCatalog({ query: "make a number count up", "on-device": true }));
|
||||
|
||||
expect(state.downloads).toBe(1);
|
||||
expect(state.consentRecorded).toEqual([true]);
|
||||
});
|
||||
|
||||
it("keeps a decline sticky until explicit --yes consent", async () => {
|
||||
state.modelStatus = "not-asked";
|
||||
state.confirmAnswer = false;
|
||||
|
||||
await asATerminal(() => runCatalog({ query: "count up", "on-device": true }));
|
||||
state.confirmAnswer = true;
|
||||
await asATerminal(() => runCatalog({ query: "count up", "on-device": true }));
|
||||
|
||||
expect(state.downloads).toBe(0);
|
||||
expect(state.consentRecorded).toEqual([false]);
|
||||
|
||||
await asATerminal(() => runCatalog({ query: "count up", "on-device": true, yes: true }));
|
||||
expect(state.downloads).toBe(1);
|
||||
expect(state.consentRecorded).toEqual([false, true]);
|
||||
});
|
||||
|
||||
it("does not treat non-interactive output as download consent", async () => {
|
||||
state.modelStatus = "not-asked";
|
||||
|
||||
await runEnvelope({ query: "count up", "on-device": true });
|
||||
|
||||
expect(state.downloads).toBe(0);
|
||||
expect(state.consentRecorded).toEqual([]);
|
||||
});
|
||||
});
|
||||
@ -1,4 +1,3 @@
|
||||
import { failCommand, finishCommand } from "../utils/commandResult.js";
|
||||
import { defineCommand } from "citty";
|
||||
import type { Example } from "./_examples.js";
|
||||
|
||||
@ -13,10 +12,124 @@ export const examples: Example[] = [
|
||||
import * as clack from "@clack/prompts";
|
||||
import { type ItemType } from "@hyperframes/core";
|
||||
import { c } from "../ui/colors.js";
|
||||
import { listRegistryItems, loadAllItems } from "../registry/resolver.js";
|
||||
import { loadAllItems } from "../registry/resolver.js";
|
||||
import { fetchRegistryManifest } from "../registry/remote.js";
|
||||
import { loadProjectConfig, DEFAULT_PROJECT_CONFIG } from "../utils/projectConfig.js";
|
||||
import { resolve } from "node:path";
|
||||
import { finishCommand } from "../utils/commandResult.js";
|
||||
import { runAdd } from "./add.js";
|
||||
import { searchByWords } from "../registry/localSearch.js";
|
||||
import {
|
||||
downloadOfferMessage,
|
||||
ensureLocalModel,
|
||||
type LocalModelStatus,
|
||||
localModelStatus,
|
||||
nonInteractiveConsentMessage,
|
||||
recordLocalModelConsent,
|
||||
} from "../registry/localModel.js";
|
||||
import { localRuntimeAvailable } from "../registry/localEmbedder.js";
|
||||
import {
|
||||
cachedLocalVectorRevision,
|
||||
fetchLocalVectors,
|
||||
hasLocalVectors,
|
||||
localSemanticRanking,
|
||||
localVectorNames,
|
||||
} from "../registry/localSemantic.js";
|
||||
|
||||
/**
|
||||
* Get the offline tier ready, and report every reason it could not be.
|
||||
*
|
||||
* A per-run opt-in, so an agent or CI run can reach the offline tier at all:
|
||||
* the only other route is a prompt that fires exclusively on a terminal.
|
||||
*
|
||||
* Warnings are returned as well as printed so `--json` can carry the same
|
||||
* reasons the terminal shows.
|
||||
*/
|
||||
// a consent gate: each branch is a distinct reason the tier cannot run, and each has to be reported separately
|
||||
// fallow-ignore-next-line complexity
|
||||
async function prepareOnDeviceTier(opts: {
|
||||
assumedYes: boolean;
|
||||
artifactRevision?: string;
|
||||
canPrompt: boolean;
|
||||
registry: string;
|
||||
registryNames: ReadonlySet<string>;
|
||||
status: LocalModelStatus;
|
||||
}): Promise<string[]> {
|
||||
const warnings: string[] = [];
|
||||
const warn = (message: string): void => {
|
||||
warnings.push(message);
|
||||
console.error(message);
|
||||
};
|
||||
|
||||
const status = opts.status;
|
||||
if (!opts.assumedYes && status.status === "declined") {
|
||||
warn(
|
||||
"on-device search skipped: the model download was previously declined. Re-run with --yes to consent.",
|
||||
);
|
||||
return warnings;
|
||||
}
|
||||
|
||||
if (!opts.assumedYes && status.status === "not-asked" && !opts.canPrompt) {
|
||||
warn(nonInteractiveConsentMessage());
|
||||
return warnings;
|
||||
}
|
||||
|
||||
if (!opts.assumedYes && status.status === "not-asked") {
|
||||
const answer = await clack.confirm({
|
||||
message: downloadOfferMessage(),
|
||||
initialValue: true,
|
||||
});
|
||||
if (clack.isCancel(answer) || answer !== true) {
|
||||
recordLocalModelConsent(false);
|
||||
warn("on-device search skipped: the download was declined.");
|
||||
// Return, or the decline is the only thing that does not happen: the
|
||||
// runtime check below is skipped precisely because consent is now false,
|
||||
// control reaches recordLocalModelConsent(true), and the answer is
|
||||
// overwritten with yes before the download it refused.
|
||||
return warnings;
|
||||
}
|
||||
}
|
||||
|
||||
if (!(await localRuntimeAvailable())) {
|
||||
// Checked before downloading. Fetching 32 MB and then discovering the
|
||||
// runtime is missing wastes the bandwidth the consent was granted for.
|
||||
warn(
|
||||
"on-device search needs the native ONNX runtime, which a single-file build cannot load. " +
|
||||
"Install the CLI normally (npm i -g hyperframes) to use this tier.",
|
||||
);
|
||||
return warnings;
|
||||
}
|
||||
|
||||
if (status.status === "declined" || status.status === "not-asked") {
|
||||
recordLocalModelConsent(true);
|
||||
}
|
||||
const model = await ensureLocalModel();
|
||||
const revisionStale =
|
||||
opts.artifactRevision !== undefined && cachedLocalVectorRevision() !== opts.artifactRevision;
|
||||
if (
|
||||
!hasLocalVectors() ||
|
||||
revisionStale ||
|
||||
countUnindexed(opts.registryNames, localVectorNames()) > 0
|
||||
) {
|
||||
await fetchLocalVectors(opts.registry, { expectedRevision: opts.artifactRevision });
|
||||
}
|
||||
// Deliberately not the fetch's own answer. A refresh that fails still leaves
|
||||
// the previous vectors on disk, and those still rank: reporting the tier
|
||||
// unavailable there would be false, and the search that follows says what is
|
||||
// actually wrong with them.
|
||||
const vectors = hasLocalVectors();
|
||||
if (!model || !vectors) {
|
||||
warn(
|
||||
`on-device search unavailable: ${!model ? "model" : "catalog vectors"} could not be fetched`,
|
||||
);
|
||||
} else if (
|
||||
opts.artifactRevision !== undefined &&
|
||||
cachedLocalVectorRevision() !== opts.artifactRevision
|
||||
) {
|
||||
warn("on-device search is using the previous catalog vectors because the update failed");
|
||||
}
|
||||
return warnings;
|
||||
}
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
@ -40,7 +153,27 @@ export default defineCommand({
|
||||
type: "boolean",
|
||||
description: "Interactive picker — select an item to install",
|
||||
},
|
||||
query: {
|
||||
type: "string",
|
||||
description:
|
||||
"Search by meaning when the on-device model is on, otherwise by name, title, description and tags",
|
||||
},
|
||||
yes: {
|
||||
type: "boolean",
|
||||
alias: "y",
|
||||
description: "Assume yes for prompts this run, including the on-device model download",
|
||||
},
|
||||
"on-device": {
|
||||
type: "boolean",
|
||||
// Consent for the model download is otherwise only reachable through a
|
||||
// prompt that fires on a TTY, which leaves every agent and CI run unable
|
||||
// to opt in at all.
|
||||
description:
|
||||
"Use on-device meaning search; pass --yes to approve a first non-interactive download",
|
||||
},
|
||||
},
|
||||
// one flag-parsing entry point feeding three output paths (json, interactive, table); splitting those is its own change
|
||||
// fallow-ignore-next-line complexity
|
||||
async run({ args }) {
|
||||
const json = args.json === true;
|
||||
const interactive = args["human-friendly"] === true;
|
||||
@ -52,13 +185,17 @@ export default defineCommand({
|
||||
else if (args.type === "component") typeFilter = "hyperframes:component";
|
||||
else if (args.type) {
|
||||
console.error(`Invalid --type: "${args.type}". Use "block" or "component".`);
|
||||
failCommand();
|
||||
finishCommand(1);
|
||||
}
|
||||
|
||||
const entries = await listRegistryItems(typeFilter ? { type: typeFilter } : undefined, {
|
||||
baseUrl: config.registry,
|
||||
});
|
||||
const filtered = entries.filter((e) => e.type !== "hyperframes:example");
|
||||
// Asked for the whole manifest on purpose: its item list defines coverage,
|
||||
// and its artifact revision is the one owner of vector freshness.
|
||||
const manifest = await fetchRegistryManifest(config.registry);
|
||||
const entries = manifest?.items ?? [];
|
||||
const artifactRevision = manifest?.catalogArtifact?.revision;
|
||||
const catalog = entries.filter((e) => e.type !== "hyperframes:example");
|
||||
const registryNames = new Set(catalog.map((e) => e.name));
|
||||
const filtered = typeFilter ? catalog.filter((e) => e.type === typeFilter) : catalog;
|
||||
|
||||
if (filtered.length === 0) {
|
||||
if (json) console.log("[]");
|
||||
@ -69,13 +206,76 @@ export default defineCommand({
|
||||
const items = await loadAllItems(filtered, { baseUrl: config.registry });
|
||||
|
||||
const tagFilter = args.tag?.toLowerCase();
|
||||
const matching = tagFilter
|
||||
const tagged = tagFilter
|
||||
? items.filter((item) => item.tags?.some((t) => t.toLowerCase() === tagFilter))
|
||||
: items;
|
||||
|
||||
const query = typeof args.query === "string" ? args.query.trim() : "";
|
||||
// Collected rather than only printed, so --json can carry the same reasons
|
||||
// the terminal shows. A machine that asked for a tier deserves to be told
|
||||
// it did not run.
|
||||
const searchContext = query ? { status: localModelStatus() } : null;
|
||||
const routineUpdate =
|
||||
searchContext?.status.status === "unavailable" ||
|
||||
(searchContext?.status.status === "ready" &&
|
||||
artifactRevision !== undefined &&
|
||||
cachedLocalVectorRevision() !== artifactRevision);
|
||||
const shouldPrepare = args["on-device"] === true || routineUpdate;
|
||||
let warnings: string[] = [];
|
||||
let effectiveStatus = searchContext?.status;
|
||||
if (searchContext && shouldPrepare) {
|
||||
warnings = await prepareOnDeviceTier({
|
||||
assumedYes: args.yes === true,
|
||||
artifactRevision,
|
||||
canPrompt: process.stdout.isTTY === true && !json,
|
||||
registry: config.registry,
|
||||
registryNames,
|
||||
status: searchContext.status,
|
||||
});
|
||||
// A successful preparation can move not-asked/unavailable to ready. Read
|
||||
// the state owner again rather than carrying the pre-download snapshot.
|
||||
effectiveStatus = localModelStatus();
|
||||
}
|
||||
const searched = effectiveStatus
|
||||
? await applySearch(tagged, query, registryNames, effectiveStatus)
|
||||
: null;
|
||||
if (searched) warnings.push(...searched.warnings);
|
||||
const matching = searched ? searched.items : tagged;
|
||||
|
||||
if (matching.length === 0) {
|
||||
if (json) console.log("[]");
|
||||
else console.log(`No items match tag "${args.tag}".`);
|
||||
// An empty result is exactly when the tier matters most: nothing found on
|
||||
// the weakest tier means something different from nothing found on the
|
||||
// best one.
|
||||
if (json && query) {
|
||||
console.log(
|
||||
JSON.stringify(
|
||||
{
|
||||
query,
|
||||
tier: tierToken(searched),
|
||||
tier_detail: tierDetail(searched),
|
||||
dropped: searched?.missing ?? 0,
|
||||
unindexed: searched?.unindexed ?? 0,
|
||||
...(searched?.topScore != null ? { top_score: searched.topScore } : {}),
|
||||
shown: 0,
|
||||
total: tagged.length,
|
||||
...(warnings.length ? { warnings } : {}),
|
||||
results: [],
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
} else if (json) console.log("[]");
|
||||
else {
|
||||
// Name whichever filter actually emptied the list. Reporting a tag
|
||||
// miss for a query miss sends people to fix the wrong thing.
|
||||
const criteria = [
|
||||
query ? `query "${query}"` : null,
|
||||
args.tag ? `tag "${args.tag}"` : null,
|
||||
].filter(Boolean);
|
||||
console.log(`No items match ${criteria.join(" and ")}.`);
|
||||
}
|
||||
if (query) await offerLocalModel(0, json, config.registry, artifactRevision);
|
||||
return;
|
||||
}
|
||||
|
||||
@ -89,10 +289,69 @@ export default defineCommand({
|
||||
...("dimensions" in item && item.dimensions ? { dimensions: item.dimensions } : {}),
|
||||
...("duration" in item && item.duration ? { duration: item.duration } : {}),
|
||||
}));
|
||||
console.log(JSON.stringify(output, null, 2));
|
||||
if (!query) {
|
||||
// A plain listing has no tier and no drop count, and this array shape
|
||||
// is already released. Leave it alone.
|
||||
console.log(JSON.stringify(output, null, 2));
|
||||
return;
|
||||
}
|
||||
console.log(
|
||||
JSON.stringify(
|
||||
{
|
||||
query,
|
||||
tier: tierToken(searched),
|
||||
tier_detail: tierDetail(searched),
|
||||
dropped: searched?.missing ?? 0,
|
||||
unindexed: searched?.unindexed ?? 0,
|
||||
...(searched?.topScore != null ? { top_score: searched.topScore } : {}),
|
||||
shown: output.length,
|
||||
total: tagged.length,
|
||||
...(warnings.length ? { warnings } : {}),
|
||||
results: output,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
if (searched) {
|
||||
// Name the search that actually ran. A user seeing worse results has to
|
||||
// be able to tell which tier produced them.
|
||||
const how = tierDetail(searched);
|
||||
const unshowable =
|
||||
searched.missing > 0
|
||||
? ` · ${searched.missing} ranked ${searched.missing === 1 ? "move" : "moves"} not in this registry`
|
||||
: "";
|
||||
console.log(c.dim(` ${matching.length} of ${tagged.length} moves · ${how}${unshowable}`));
|
||||
if (searched.unindexed > 0) {
|
||||
// The costly direction, so it gets a line of its own rather than a
|
||||
// suffix: these moves cannot come back from meaning search at any rank,
|
||||
// for any query, and a reader has to learn what to run about it.
|
||||
// Silent when the index covers the registry.
|
||||
const remedy =
|
||||
args["on-device"] === true
|
||||
? "The published index is behind this registry."
|
||||
: "Re-run with --on-device to refresh it.";
|
||||
console.log(
|
||||
c.warn(
|
||||
` ${searched.unindexed} of ${registryNames.size} moves are missing from the ` +
|
||||
`on-device index, so meaning search cannot return them. ${remedy}`,
|
||||
),
|
||||
);
|
||||
}
|
||||
if (searched.localMode === "words") {
|
||||
// Suppressed when on-device was asked for and refused: telling someone
|
||||
// to pass a flag one line after explaining that flag cannot work here
|
||||
// reads as the tool arguing with itself.
|
||||
if (warnings.length === 0) {
|
||||
reportLocalModelOption(json);
|
||||
await offerLocalModel(matching.length, json, config.registry, artifactRevision);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (interactive) {
|
||||
const options = matching.map((item) => ({
|
||||
value: item.name,
|
||||
@ -152,3 +411,207 @@ export default defineCommand({
|
||||
console.log(c.dim(`${matching.length} items. Run "hyperframes add <name>" to install.`));
|
||||
},
|
||||
});
|
||||
|
||||
/**
|
||||
* Resolve ranked names against the items this registry actually has.
|
||||
*
|
||||
* `missing` counts the ranked names this registry has no item for at all,
|
||||
* measured against `registryNames` — the whole catalog, not the post-filter
|
||||
* `items`. It is reported rather than swallowed: the catalog artifact and the
|
||||
* registry are published separately, so they can be different generations, and
|
||||
* the moves lost that way are the top-ranked ones.
|
||||
*
|
||||
* A move the user's own --type or --tag removed is not one of them. It is in
|
||||
* the registry and installable; they excluded it. Counting it here made every
|
||||
* filtered search report a skew that was not there, and made a real skew
|
||||
* indistinguishable from a filter doing its job.
|
||||
*/
|
||||
export function pickByName<T extends { name: string }>(
|
||||
items: T[],
|
||||
names: string[],
|
||||
registryNames: ReadonlySet<string>,
|
||||
): { ranked: T[]; missing: number } {
|
||||
const byName = new Map(items.map((item) => [item.name, item]));
|
||||
const ranked = names
|
||||
.map((name) => byName.get(name))
|
||||
.filter((item): item is T => item !== undefined);
|
||||
return { ranked, missing: names.filter((name) => !registryNames.has(name)).length };
|
||||
}
|
||||
|
||||
/**
|
||||
* Registry moves the on-device index holds no vector for.
|
||||
*
|
||||
* The counterpart to `pickByName`'s `missing`, and the direction that costs
|
||||
* something. `missing` is over-coverage: names the artifact ranks that this
|
||||
* registry cannot install, which only wastes a rank. This is under-coverage:
|
||||
* moves the registry has that were never embedded, so meaning search cannot
|
||||
* return them at any rank, for any query, and until now nothing in the output
|
||||
* said so. The vectors are fetched once and never invalidated, so every move
|
||||
* published since that fetch lands here.
|
||||
*
|
||||
* Measured against the unfiltered registry, matching `missing`: a --type or
|
||||
* --tag filter removing a move is the user narrowing their own search, not an
|
||||
* index that cannot see it.
|
||||
*/
|
||||
export function countUnindexed(
|
||||
registryNames: ReadonlySet<string>,
|
||||
indexedNames: Iterable<string>,
|
||||
): number {
|
||||
const indexed = new Set(indexedNames);
|
||||
let count = 0;
|
||||
for (const name of registryNames) {
|
||||
if (!indexed.has(name)) count += 1;
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
interface SearchOutcome<T> {
|
||||
items: T[];
|
||||
localMode: LocalMode;
|
||||
warnings: string[];
|
||||
/** Over-coverage: ranked names this registry has no item for. */
|
||||
missing: number;
|
||||
/**
|
||||
* Under-coverage: registry moves the tier that answered could not see.
|
||||
*
|
||||
* Always zero for word matching, which ranks the live registry listing and
|
||||
* so cannot be stale. Non-zero only on the on-device tier, whose vectors are
|
||||
* a separately published artifact that drifts from the registry.
|
||||
*/
|
||||
unindexed: number;
|
||||
/**
|
||||
* Similarity of the best result actually shown, when the answering tier
|
||||
* produces one. Null for word matching, whose score is a different and
|
||||
* non-comparable scale. Deliberately not a threshold: it is the signal a
|
||||
* caller needs to judge a ranker that ships without one.
|
||||
*/
|
||||
topScore: number | null;
|
||||
}
|
||||
|
||||
async function applySearch<
|
||||
T extends { name: string; title: string; description: string; tags?: string[] },
|
||||
>(
|
||||
items: T[],
|
||||
query: string,
|
||||
registryNames: ReadonlySet<string>,
|
||||
status: LocalModelStatus,
|
||||
): Promise<SearchOutcome<T>> {
|
||||
const warnings: string[] = [];
|
||||
// On-device meaning search, when the user opted into the model. Free and
|
||||
// offline, and it answers phrasings word matching cannot reach.
|
||||
if (status.status === "ready") {
|
||||
try {
|
||||
const ranking = await localSemanticRanking(query);
|
||||
if (ranking) {
|
||||
const { ranked, missing } = pickByName(
|
||||
items,
|
||||
ranking.map((entry) => entry.name),
|
||||
registryNames,
|
||||
);
|
||||
const best = ranked[0];
|
||||
if (best) {
|
||||
const scoreByName = new Map(ranking.map((entry) => [entry.name, entry.score]));
|
||||
return {
|
||||
items: ranked.slice(0, 25),
|
||||
localMode: "local-model",
|
||||
warnings,
|
||||
missing,
|
||||
unindexed: countUnindexed(registryNames, localVectorNames()),
|
||||
topScore: scoreByName.get(best.name) ?? null,
|
||||
};
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
// A broken model costs ranking quality, never the command. It does not
|
||||
// get to cost it silently: a tier the user switched on that quietly does
|
||||
// not run is indistinguishable from one that ran badly.
|
||||
const warning = `on-device search did not run: ${error instanceof Error ? error.message : "unknown error"}`;
|
||||
warnings.push(warning);
|
||||
console.error(warning);
|
||||
}
|
||||
}
|
||||
|
||||
// Shared vocabulary, not substring presence. A user asking to "make the
|
||||
// pace feel faster" shares no literal substring with any description, so
|
||||
// the old test returned nothing at all for exactly the phrasing people use.
|
||||
const words = searchByWords(
|
||||
query,
|
||||
items,
|
||||
(item) => `${item.name} ${item.title} ${item.description} ${(item.tags ?? []).join(" ")}`,
|
||||
);
|
||||
// Word matching ranks the items in hand, so nothing can go missing.
|
||||
return { items: words, localMode: "words", warnings, missing: 0, unindexed: 0, topScore: null };
|
||||
}
|
||||
|
||||
/**
|
||||
* Which tier answered, as a stable token.
|
||||
*
|
||||
* Deliberately not the printed sentence: "on-device meaning search" is written
|
||||
* for a person and will be reworded. A machine consumer needs something that
|
||||
* will not move underneath it.
|
||||
*/
|
||||
function tierToken(searched: { localMode: LocalMode } | null): "on-device" | "words" {
|
||||
return searched?.localMode === "local-model" ? "on-device" : "words";
|
||||
}
|
||||
|
||||
/** The same tier, as the sentence a person reads. */
|
||||
function tierDetail(searched: { localMode: LocalMode } | null): string {
|
||||
return searched?.localMode === "local-model" ? "on-device meaning search" : "local word match";
|
||||
}
|
||||
|
||||
type LocalMode = "local-model" | "words";
|
||||
|
||||
/**
|
||||
* Offer the on-device model when word matching came up thin, and only then.
|
||||
*
|
||||
* Asking on first run would interrupt people the free tier already serves.
|
||||
* Asking here puts the evidence in front of them: they can see what word
|
||||
* matching returned before deciding whether 33 MB is worth it.
|
||||
*/
|
||||
/**
|
||||
* Nobody to ask, so say what to ask for.
|
||||
*
|
||||
* Without this a scripted run sits on word matching with no indication that a
|
||||
* better offline tier exists and is one question away.
|
||||
*/
|
||||
function reportLocalModelOption(json: boolean): void {
|
||||
if (!json && process.stdout.isTTY) return;
|
||||
if (localModelStatus().status !== "not-asked") return;
|
||||
console.error(nonInteractiveConsentMessage());
|
||||
}
|
||||
|
||||
async function offerLocalModel(
|
||||
matchCount: number,
|
||||
json: boolean,
|
||||
registryBaseUrl: string,
|
||||
artifactRevision?: string,
|
||||
): Promise<void> {
|
||||
if (json || !process.stdout.isTTY) return;
|
||||
if (localModelStatus().status !== "not-asked") return;
|
||||
// Deliberately not gated on the number of results. That gate was set when the
|
||||
// catalog was small; against 411 moves a word match nearly always returns
|
||||
// more than a handful, so it had quietly become unreachable and nobody was
|
||||
// ever told the offline tier exists. Reaching the weakest tier is the signal.
|
||||
|
||||
const answer = await clack.confirm({
|
||||
message: downloadOfferMessage(matchCount),
|
||||
initialValue: true,
|
||||
});
|
||||
if (clack.isCancel(answer)) return;
|
||||
recordLocalModelConsent(answer === true);
|
||||
if (answer !== true) return;
|
||||
|
||||
// The vectors come from the registry rather than the package, so consent is
|
||||
// also the moment to fetch them. A failure here is reported: the alternative
|
||||
// is an offline tier the user turned on that silently never ranks anything.
|
||||
const vectors =
|
||||
hasLocalVectors() ||
|
||||
(await fetchLocalVectors(registryBaseUrl, { expectedRevision: artifactRevision }));
|
||||
console.log(
|
||||
c.dim(
|
||||
vectors
|
||||
? " Run the search again to download the model and rank by meaning."
|
||||
: " Could not fetch the catalog vectors; offline ranking stays off until they are available.",
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
4419
packages/cli/src/registry/__fixtures__/wordpiece-reference.json
Normal file
4419
packages/cli/src/registry/__fixtures__/wordpiece-reference.json
Normal file
File diff suppressed because it is too large
Load Diff
122
packages/cli/src/registry/localEmbedder.ts
Normal file
122
packages/cli/src/registry/localEmbedder.ts
Normal file
@ -0,0 +1,122 @@
|
||||
/**
|
||||
* On-device embeddings for local semantic search.
|
||||
*
|
||||
* Runs bge-small-en-v1.5 through the ONNX runtime the CLI already depends on,
|
||||
* with the WordPiece tokenizer in `wordpiece.ts`. Nothing is sent anywhere and
|
||||
* nothing costs money once the model is cached.
|
||||
*
|
||||
* Two details are specific to bge and easy to miss. Pooling takes the CLS token
|
||||
* rather than a mean over the sequence, and short queries carry an instruction
|
||||
* prefix that passages do not. Getting either wrong degrades retrieval quietly
|
||||
* rather than failing, so both are asserted in tests.
|
||||
*/
|
||||
|
||||
import { readFileSync } from "node:fs";
|
||||
|
||||
import {
|
||||
LOCAL_MODEL_DIMENSIONS,
|
||||
QUERY_INSTRUCTION,
|
||||
localModelPath,
|
||||
localTokenizerPath,
|
||||
} from "./localModel.js";
|
||||
import { configFromTokenizerJson, encode } from "./wordpiece.js";
|
||||
|
||||
export interface LocalEmbedder {
|
||||
embed(texts: string[], options?: { isQuery?: boolean }): Promise<number[][]>;
|
||||
}
|
||||
|
||||
/** Long inputs are truncated rather than rejected; the model has a 512 token limit. */
|
||||
const MAX_TOKENS = 512;
|
||||
|
||||
/**
|
||||
* Is the native ONNX runtime reachable in this install?
|
||||
*
|
||||
* It cannot be bundled: a single-file build leaves `onnxruntime-node` as an
|
||||
* unresolved import, so on-device search is unavailable there no matter what
|
||||
* else is on disk. Checking costs one dynamic import and saves a pointless
|
||||
* 32 MB download.
|
||||
*/
|
||||
export async function localRuntimeAvailable(): Promise<boolean> {
|
||||
try {
|
||||
await import("onnxruntime-node");
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export async function loadLocalEmbedder(): Promise<LocalEmbedder> {
|
||||
// Imported lazily so the CLI does not pay for the ONNX runtime on every run.
|
||||
const ort = await import("onnxruntime-node");
|
||||
const config = configFromTokenizerJson(readFileSync(localTokenizerPath(), "utf-8"));
|
||||
const session = await ort.InferenceSession.create(localModelPath());
|
||||
|
||||
return {
|
||||
async embed(texts, options) {
|
||||
if (texts.length === 0) return [];
|
||||
const prefix = options?.isQuery ? QUERY_INSTRUCTION : "";
|
||||
const encodings = texts.map((text) => truncate(encode(prefix + text, config)));
|
||||
const width = Math.max(...encodings.map((e) => e.ids.length));
|
||||
|
||||
const batch = encodings.length;
|
||||
const ids = new BigInt64Array(batch * width);
|
||||
const mask = new BigInt64Array(batch * width);
|
||||
const types = new BigInt64Array(batch * width);
|
||||
encodings.forEach((encoding, row) => {
|
||||
encoding.ids.forEach((id, column) => {
|
||||
const at = row * width + column;
|
||||
ids[at] = BigInt(id);
|
||||
mask[at] = 1n;
|
||||
types[at] = 0n;
|
||||
});
|
||||
// Remaining positions stay zero: padded ids with a zero attention mask,
|
||||
// which the model must not attend to.
|
||||
});
|
||||
|
||||
const dims = [batch, width];
|
||||
const output = await session.run({
|
||||
input_ids: new ort.Tensor("int64", ids, dims),
|
||||
attention_mask: new ort.Tensor("int64", mask, dims),
|
||||
token_type_ids: new ort.Tensor("int64", types, dims),
|
||||
});
|
||||
|
||||
const hidden = output["last_hidden_state"];
|
||||
if (!hidden) throw new Error("model returned no last_hidden_state");
|
||||
const data = hidden.data as Float32Array;
|
||||
const hiddenSize = hidden.dims[2] as number;
|
||||
if (hiddenSize !== LOCAL_MODEL_DIMENSIONS) {
|
||||
throw new Error(
|
||||
`model produced ${hiddenSize} dimensions, expected ${LOCAL_MODEL_DIMENSIONS}`,
|
||||
);
|
||||
}
|
||||
|
||||
return encodings.map((_, row) => {
|
||||
// CLS pooling: bge trains the first token as the sequence representation.
|
||||
// Mean pooling here would produce vectors that look fine and rank worse.
|
||||
const start = row * width * hiddenSize;
|
||||
return normalize(Array.from(data.subarray(start, start + hiddenSize)));
|
||||
});
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function truncate(encoding: ReturnType<typeof encode>): ReturnType<typeof encode> {
|
||||
if (encoding.ids.length <= MAX_TOKENS) return encoding;
|
||||
// Keep the closing separator so the sequence still ends the way the model expects.
|
||||
const ids = [
|
||||
...encoding.ids.slice(0, MAX_TOKENS - 1),
|
||||
encoding.ids[encoding.ids.length - 1] as number,
|
||||
];
|
||||
return { ids, attentionMask: ids.map(() => 1), tokenTypeIds: ids.map(() => 0) };
|
||||
}
|
||||
|
||||
function normalize(vector: number[]): number[] {
|
||||
const norm = Math.sqrt(vector.reduce((sum, value) => sum + value * value, 0));
|
||||
return norm === 0 ? vector : vector.map((value) => value / norm);
|
||||
}
|
||||
|
||||
export function cosine(a: number[], b: number[]): number {
|
||||
let sum = 0;
|
||||
for (let i = 0; i < a.length; i += 1) sum += (a[i] as number) * (b[i] as number);
|
||||
return sum;
|
||||
}
|
||||
206
packages/cli/src/registry/localModel.test.ts
Normal file
206
packages/cli/src/registry/localModel.test.ts
Normal file
@ -0,0 +1,206 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const readConfig = vi.fn();
|
||||
const writeConfig = vi.fn();
|
||||
const existsSync = vi.fn();
|
||||
const readFileSync = vi.fn();
|
||||
const unlinkSync = vi.fn();
|
||||
const downloadFile = vi.fn();
|
||||
const digest = vi.fn();
|
||||
|
||||
vi.mock("../telemetry/config.js", () => ({
|
||||
readConfig: () => readConfig(),
|
||||
writeConfig: (c: unknown) => writeConfig(c),
|
||||
}));
|
||||
vi.mock("node:fs", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("node:fs")>()),
|
||||
existsSync: (p: string) => existsSync(p),
|
||||
mkdirSync: vi.fn(),
|
||||
readFileSync: (p: string) => readFileSync(p),
|
||||
unlinkSync: (p: string) => unlinkSync(p),
|
||||
}));
|
||||
vi.mock("node:crypto", () => ({
|
||||
createHash: () => {
|
||||
const hash = {
|
||||
update: () => hash,
|
||||
digest: () => digest(),
|
||||
};
|
||||
return hash;
|
||||
},
|
||||
}));
|
||||
vi.mock("../utils/download.js", () => ({
|
||||
downloadFile: (...args: unknown[]) => downloadFile(...args),
|
||||
}));
|
||||
|
||||
const {
|
||||
LOCAL_MODEL_DIMENSIONS,
|
||||
LOCAL_MODEL_ARTIFACTS,
|
||||
LOCAL_MODEL_SIZE_MB,
|
||||
QUERY_INSTRUCTION,
|
||||
downloadOfferMessage,
|
||||
ensureLocalModel,
|
||||
isLocalModelReady,
|
||||
localModelConsent,
|
||||
localModelStatus,
|
||||
localTokenizerPath,
|
||||
localModelPath,
|
||||
recordLocalModelConsent,
|
||||
} = await import("./localModel.js");
|
||||
|
||||
function returnMatchingDigests(rounds = 1): void {
|
||||
for (let round = 0; round < rounds; round += 1) {
|
||||
for (const artifact of LOCAL_MODEL_ARTIFACTS) digest.mockReturnValueOnce(artifact.sha256);
|
||||
}
|
||||
}
|
||||
|
||||
function makeDownloadsAppearOnDisk(): void {
|
||||
const present = new Set<string>();
|
||||
existsSync.mockImplementation((path: string) => present.has(path));
|
||||
downloadFile.mockImplementation(async (_url: string, path: string) => {
|
||||
present.add(path);
|
||||
});
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
readConfig.mockReturnValue({});
|
||||
existsSync.mockReturnValue(false);
|
||||
writeConfig.mockReset();
|
||||
readFileSync.mockReturnValue(Buffer.from("artifact"));
|
||||
unlinkSync.mockReset();
|
||||
downloadFile.mockReset();
|
||||
digest.mockReset();
|
||||
});
|
||||
|
||||
afterEach(() => vi.clearAllMocks());
|
||||
|
||||
describe("consent state", () => {
|
||||
it("reports never-asked as undefined", () => {
|
||||
expect(localModelConsent()).toBeUndefined();
|
||||
});
|
||||
|
||||
it("persists a yes", () => {
|
||||
readConfig.mockReturnValue({ telemetryEnabled: true });
|
||||
recordLocalModelConsent(true);
|
||||
expect(writeConfig).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ localEmbeddingEnabled: true }),
|
||||
);
|
||||
});
|
||||
|
||||
it("persists a no without discarding other settings", () => {
|
||||
readConfig.mockReturnValue({ telemetryEnabled: false, anonymousId: "abc" });
|
||||
recordLocalModelConsent(false);
|
||||
expect(writeConfig).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ localEmbeddingEnabled: false, anonymousId: "abc" }),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("readiness", () => {
|
||||
it("needs both the model and its tokenizer", () => {
|
||||
// A model without its tokenizer cannot turn text into tensors, so half a
|
||||
// download is not a usable state.
|
||||
existsSync.mockImplementation((p: string) => p === localModelPath());
|
||||
expect(isLocalModelReady()).toBe(false);
|
||||
|
||||
existsSync.mockImplementation((p: string) => p === localTokenizerPath());
|
||||
expect(isLocalModelReady()).toBe(false);
|
||||
|
||||
existsSync.mockReturnValue(true);
|
||||
returnMatchingDigests();
|
||||
expect(isLocalModelReady()).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects present files whose digest does not match the pinned revision", () => {
|
||||
existsSync.mockReturnValue(true);
|
||||
digest.mockReturnValue("wrong");
|
||||
|
||||
expect(isLocalModelReady()).toBe(false);
|
||||
});
|
||||
|
||||
it("bounds each download and rejects a digest mismatch before reporting ready", async () => {
|
||||
makeDownloadsAppearOnDisk();
|
||||
digest.mockReturnValue("wrong");
|
||||
|
||||
expect(await ensureLocalModel()).toBe(false);
|
||||
expect(downloadFile).toHaveBeenCalledWith(
|
||||
expect.stringContaining("model_quantized.onnx"),
|
||||
localModelPath(),
|
||||
{ maxBytes: 34_014_426 },
|
||||
);
|
||||
expect(unlinkSync).toHaveBeenCalledWith(localModelPath());
|
||||
});
|
||||
|
||||
it("accepts only two bounded downloads that match both pinned digests", async () => {
|
||||
makeDownloadsAppearOnDisk();
|
||||
returnMatchingDigests(2);
|
||||
|
||||
expect(await ensureLocalModel()).toBe(true);
|
||||
expect(downloadFile).toHaveBeenNthCalledWith(
|
||||
2,
|
||||
expect.stringContaining("tokenizer.json"),
|
||||
localTokenizerPath(),
|
||||
{ maxBytes: 711_396 },
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("status", () => {
|
||||
it("is declined when the user said no, even if files somehow exist", () => {
|
||||
readConfig.mockReturnValue({ localEmbeddingEnabled: false });
|
||||
existsSync.mockReturnValue(true);
|
||||
expect(localModelStatus()).toEqual({ status: "declined" });
|
||||
});
|
||||
|
||||
it("is ready when consented and downloaded", () => {
|
||||
readConfig.mockReturnValue({ localEmbeddingEnabled: true });
|
||||
existsSync.mockReturnValue(true);
|
||||
returnMatchingDigests();
|
||||
expect(localModelStatus()).toEqual({ status: "ready" });
|
||||
});
|
||||
|
||||
it("is not-asked when never answered and nothing downloaded", () => {
|
||||
expect(localModelStatus()).toEqual({ status: "not-asked" });
|
||||
});
|
||||
|
||||
it("is unavailable when consented but not yet downloaded", () => {
|
||||
readConfig.mockReturnValue({ localEmbeddingEnabled: true });
|
||||
expect(localModelStatus().status).toBe("unavailable");
|
||||
});
|
||||
|
||||
it("never downloads or prompts by itself", () => {
|
||||
// The caller owns the prompt, because only it knows whether word matching
|
||||
// already answered well enough to make the offer pointless.
|
||||
expect(() => localModelStatus()).not.toThrow();
|
||||
expect(writeConfig).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe("the offer text", () => {
|
||||
it("names the cost rather than implying a privacy tradeoff", () => {
|
||||
const message = downloadOfferMessage(0);
|
||||
expect(message).toContain(`${LOCAL_MODEL_SIZE_MB} MB`);
|
||||
expect(message).toMatch(/stays on your machine/);
|
||||
expect(message).toMatch(/nothing is sent/);
|
||||
});
|
||||
|
||||
it("leads with what the user just saw", () => {
|
||||
expect(downloadOfferMessage(0)).toMatch(/^No matches/);
|
||||
expect(downloadOfferMessage(1)).toMatch(/^Only 1 match\b/);
|
||||
expect(downloadOfferMessage(3)).toMatch(/^Only 3 matches/);
|
||||
});
|
||||
|
||||
it("does not claim word search already ran for an explicit on-device request", () => {
|
||||
expect(downloadOfferMessage()).toMatch(/^Use on-device meaning search/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("model contract", () => {
|
||||
it("uses the dimension the bundled vectors are built at", () => {
|
||||
expect(LOCAL_MODEL_DIMENSIONS).toBe(384);
|
||||
});
|
||||
|
||||
it("carries the query instruction bge retrieval expects", () => {
|
||||
// Omitting this measurably degrades short-query retrieval for bge models.
|
||||
expect(QUERY_INSTRUCTION).toMatch(/searching relevant passages/);
|
||||
});
|
||||
});
|
||||
184
packages/cli/src/registry/localModel.ts
Normal file
184
packages/cli/src/registry/localModel.ts
Normal file
@ -0,0 +1,184 @@
|
||||
/**
|
||||
* The on-device embedding model behind local semantic search.
|
||||
*
|
||||
* Downloading roughly 33 MB to someone's disk is worth asking about. This is
|
||||
* not a privacy question, because nothing leaves the machine, so the prompt
|
||||
* says what it actually costs: disk and bandwidth. Borrowing privacy language
|
||||
* here would misdescribe it.
|
||||
*
|
||||
* The CLI already downloads a much larger model for background removal without
|
||||
* asking, and that is defensible there: running that command is itself a
|
||||
* request for a model. Searching a catalog is not, so a download arriving
|
||||
* mid-search would be a surprise. That difference is why this asks and that
|
||||
* does not.
|
||||
*/
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
import { existsSync, mkdirSync, readFileSync, unlinkSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { downloadFile } from "../utils/download.js";
|
||||
import { readConfig, writeConfig } from "../telemetry/config.js";
|
||||
|
||||
/**
|
||||
* bge-small-en-v1.5: 62.17 MTEB average against 62.3 for the hosted model this
|
||||
* substitutes for, at 384 dimensions instead of 1536. Chosen for being close to
|
||||
* parity while small enough to ship, not for being a weaker tier.
|
||||
*/
|
||||
export const LOCAL_MODEL_ID = "bge-small-en-v1.5";
|
||||
export const LOCAL_MODEL_DIMENSIONS = 384;
|
||||
/** Measured: 32 MB quantized model plus a 0.7 MB tokenizer. */
|
||||
export const LOCAL_MODEL_SIZE_MB = 33;
|
||||
|
||||
/** bge retrieval expects short queries to carry this prefix; passages do not. */
|
||||
export const QUERY_INSTRUCTION = "Represent this sentence for searching relevant passages: ";
|
||||
|
||||
const MODELS_DIR = join(homedir(), ".hyperframes", "models");
|
||||
|
||||
/**
|
||||
* Where the two halves come from.
|
||||
*
|
||||
* Pinned to a revision rather than to `main`: a model that silently changes
|
||||
* under a cached vector set would return confident nonsense, because the
|
||||
* catalog vectors were produced by a specific set of weights.
|
||||
*/
|
||||
const MODEL_REPO = "Xenova/bge-small-en-v1.5";
|
||||
export const LOCAL_MODEL_REVISION = "ea104dacec62c0de699686887e3f920caeb4f3e3";
|
||||
|
||||
/**
|
||||
* The quantized export, 32 MB, not the 126 MB full-precision one.
|
||||
*
|
||||
* The size this feature quotes has always described the quantized build. The
|
||||
* catalog vectors must be produced by this same file: embedding the catalog
|
||||
* with one precision and the query with another degrades ranking silently,
|
||||
* which is the failure mode this whole feature keeps having to defend against.
|
||||
*/
|
||||
interface ModelFile {
|
||||
readonly url: string;
|
||||
readonly dest: () => string;
|
||||
readonly bytes: number;
|
||||
readonly sha256: string;
|
||||
}
|
||||
|
||||
export const LOCAL_MODEL_ARTIFACTS: ReadonlyArray<ModelFile> = [
|
||||
{
|
||||
url: `https://huggingface.co/${MODEL_REPO}/resolve/${LOCAL_MODEL_REVISION}/onnx/model_quantized.onnx`,
|
||||
dest: () => localModelPath(),
|
||||
bytes: 34_014_426,
|
||||
sha256: "6c9c6101a956d62dfb5e7190c538226c0c5bb9cb27b651234b6df063ee7dbfe4",
|
||||
},
|
||||
{
|
||||
url: `https://huggingface.co/${MODEL_REPO}/resolve/${LOCAL_MODEL_REVISION}/tokenizer.json`,
|
||||
dest: () => localTokenizerPath(),
|
||||
bytes: 711_396,
|
||||
sha256: "d241a60d5e8f04cc1b2b3e9ef7a4921b27bf526d9f6050ab90f9267a1f9e5c66",
|
||||
},
|
||||
];
|
||||
|
||||
export type LocalModelDecision = boolean | undefined;
|
||||
|
||||
export function localModelConsent(): LocalModelDecision {
|
||||
return readConfig().localEmbeddingEnabled;
|
||||
}
|
||||
|
||||
export function recordLocalModelConsent(enabled: boolean): void {
|
||||
writeConfig({ ...readConfig(), localEmbeddingEnabled: enabled });
|
||||
}
|
||||
|
||||
export function localModelPath(): string {
|
||||
return join(MODELS_DIR, `${LOCAL_MODEL_ID}.onnx`);
|
||||
}
|
||||
|
||||
export function localTokenizerPath(): string {
|
||||
return join(MODELS_DIR, `${LOCAL_MODEL_ID}.tokenizer.json`);
|
||||
}
|
||||
|
||||
function modelFileIsValid(file: ModelFile): boolean {
|
||||
const path = file.dest();
|
||||
if (!existsSync(path)) return false;
|
||||
try {
|
||||
return createHash("sha256").update(readFileSync(path)).digest("hex") === file.sha256;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/** Both halves must match the pinned revision. Presence alone cannot establish readiness. */
|
||||
export function isLocalModelReady(): boolean {
|
||||
return LOCAL_MODEL_ARTIFACTS.every(modelFileIsValid);
|
||||
}
|
||||
|
||||
export type LocalModelStatus =
|
||||
| { status: "ready" }
|
||||
| { status: "declined" }
|
||||
| { status: "not-asked" }
|
||||
| { status: "unavailable"; reason: string };
|
||||
|
||||
/**
|
||||
* Resolve whether local semantic search can run right now.
|
||||
*
|
||||
* Deliberately does not prompt. The caller owns that, because only the caller
|
||||
* knows whether word matching already answered well enough to make the offer
|
||||
* pointless. Consent has one owner, which is the lesson the smart-search flag
|
||||
* taught when it was checked in two places and the override became a no-op.
|
||||
*/
|
||||
export function localModelStatus(): LocalModelStatus {
|
||||
const consent = localModelConsent();
|
||||
if (consent === false) return { status: "declined" };
|
||||
if (isLocalModelReady()) return { status: "ready" };
|
||||
if (consent === undefined) return { status: "not-asked" };
|
||||
return { status: "unavailable", reason: "model not downloaded yet" };
|
||||
}
|
||||
|
||||
/**
|
||||
* Put the model on disk, once.
|
||||
*
|
||||
* Returns false rather than throwing: a failed download costs the offline tier,
|
||||
* never the command, and the caller says so. Both halves are fetched before
|
||||
* either is reported ready, because a model without its tokenizer embeds
|
||||
* nothing and would fail later at a less obvious place.
|
||||
*/
|
||||
export async function ensureLocalModel(): Promise<boolean> {
|
||||
if (isLocalModelReady()) return true;
|
||||
try {
|
||||
mkdirSync(MODELS_DIR, { recursive: true });
|
||||
for (const file of LOCAL_MODEL_ARTIFACTS) {
|
||||
if (modelFileIsValid(file)) continue;
|
||||
const dest = file.dest();
|
||||
await downloadFile(file.url, dest, { maxBytes: file.bytes });
|
||||
if (!modelFileIsValid(file)) {
|
||||
unlinkSync(dest);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return isLocalModelReady();
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* What to print when nobody can be asked.
|
||||
*
|
||||
* An agent or CI run has no one to prompt, and it must not decide a download
|
||||
* onto someone's disk by itself. So it is told what to ask and which flag
|
||||
* records the answer. Passing the flag is the human's yes, not the agent's.
|
||||
*/
|
||||
export function nonInteractiveConsentMessage(): string {
|
||||
return (
|
||||
`On-device search needs a one-time ${LOCAL_MODEL_SIZE_MB} MB download that stays on this machine. ` +
|
||||
"Nothing is sent anywhere. Ask the person you are working for, and pass --on-device --yes once they agree."
|
||||
);
|
||||
}
|
||||
|
||||
/** What the prompt should say. Costs, not privacy: nothing is sent anywhere. */
|
||||
export function downloadOfferMessage(matchCount?: number): string {
|
||||
const found =
|
||||
matchCount === undefined
|
||||
? "Use on-device meaning search."
|
||||
: matchCount === 0
|
||||
? "No matches from word search."
|
||||
: `Only ${matchCount} match${matchCount === 1 ? "" : "es"} from word search.`;
|
||||
return `${found} Download a ${LOCAL_MODEL_SIZE_MB} MB search model for better offline results? It stays on your machine and nothing is sent.`;
|
||||
}
|
||||
97
packages/cli/src/registry/localSearch.test.ts
Normal file
97
packages/cli/src/registry/localSearch.test.ts
Normal file
@ -0,0 +1,97 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { rankByWords, searchByWords, tokenize } from "./localSearch.js";
|
||||
|
||||
interface Item {
|
||||
name: string;
|
||||
text: string;
|
||||
}
|
||||
|
||||
const ITEMS: Item[] = [
|
||||
{ name: "whip-pan", text: "A fast camera whip blurs between two shots at speed." },
|
||||
{ name: "number-wheel", text: "A rolling digit counter settles on its final value." },
|
||||
{ name: "text-reveal", text: "Type reveals line by line beneath a mask." },
|
||||
];
|
||||
|
||||
const textOf = (item: Item) => item.text;
|
||||
|
||||
describe("tokenize", () => {
|
||||
it("drops stop words", () => {
|
||||
expect(tokenize("the camera and the shot")).toEqual(["camera", "shot"]);
|
||||
});
|
||||
|
||||
it("drops tokens of three characters or fewer", () => {
|
||||
expect(tokenize("ab abc abcd")).toEqual(["abc", "abcd"]);
|
||||
});
|
||||
|
||||
it("lowercases and strips punctuation and digits", () => {
|
||||
expect(tokenize("Camera, pushes 42 times!")).toEqual(["camera", "pushes", "times"]);
|
||||
});
|
||||
|
||||
it("returns nothing for a query made only of stop words", () => {
|
||||
expect(tokenize("the and or of")).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("rankByWords", () => {
|
||||
it("ranks the entry sharing the most vocabulary first", () => {
|
||||
expect(rankByWords("rolling counter digit", ITEMS, textOf)[0]?.item.name).toBe("number-wheel");
|
||||
});
|
||||
|
||||
it("returns every item, not only matches", () => {
|
||||
expect(rankByWords("camera", ITEMS, textOf)).toHaveLength(ITEMS.length);
|
||||
});
|
||||
|
||||
it("sorts non-matching items last with a zero score", () => {
|
||||
const ranked = rankByWords("rolling counter", ITEMS, textOf);
|
||||
expect(ranked.at(-1)?.score).toBe(0);
|
||||
});
|
||||
|
||||
it("does not let the wordiest entry win on length alone", () => {
|
||||
// Without the sqrt divisor this padded entry outranks the real match.
|
||||
const padded: Item[] = [
|
||||
...ITEMS,
|
||||
{
|
||||
name: "padded",
|
||||
text: `camera ${Array.from({ length: 300 }, (_, i) => `filler${i}`).join(" ")}`,
|
||||
},
|
||||
];
|
||||
expect(rankByWords("fast camera whip speed", padded, textOf)[0]?.item.name).toBe("whip-pan");
|
||||
});
|
||||
|
||||
it("breaks ties on descending name, matching the evaluation", () => {
|
||||
const tied: Item[] = [
|
||||
{ name: "alpha", text: "nothing shared here" },
|
||||
{ name: "zulu", text: "nothing shared here" },
|
||||
];
|
||||
expect(rankByWords("unrelated", tied, textOf).map((s) => s.item.name)).toEqual([
|
||||
"zulu",
|
||||
"alpha",
|
||||
]);
|
||||
});
|
||||
|
||||
it("treats a stop-word-only query as matching nothing rather than everything", () => {
|
||||
expect(rankByWords("the and of", ITEMS, textOf).every((s) => s.score === 0)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("searchByWords", () => {
|
||||
it("keeps only entries sharing at least one token", () => {
|
||||
expect(searchByWords("rolling counter", ITEMS, textOf).map((i) => i.name)).toEqual([
|
||||
"number-wheel",
|
||||
]);
|
||||
});
|
||||
|
||||
it("beats substring matching on a natural phrasing", () => {
|
||||
// The motivating case. No description contains this phrase, so a substring
|
||||
// test returns nothing; shared vocabulary still finds the speed entry.
|
||||
const phrase = "make the shot feel fast";
|
||||
const substring = ITEMS.filter((i) => i.text.toLowerCase().includes(phrase.toLowerCase()));
|
||||
expect(substring).toHaveLength(0);
|
||||
expect(searchByWords(phrase, ITEMS, textOf)[0]?.name).toBe("whip-pan");
|
||||
});
|
||||
|
||||
it("returns nothing when no vocabulary is shared", () => {
|
||||
expect(searchByWords("quantum entanglement", ITEMS, textOf)).toEqual([]);
|
||||
});
|
||||
});
|
||||
75
packages/cli/src/registry/localSearch.ts
Normal file
75
packages/cli/src/registry/localSearch.ts
Normal file
@ -0,0 +1,75 @@
|
||||
/**
|
||||
* Local catalog search that costs nothing and needs no account.
|
||||
*
|
||||
* Replaces a substring test. `description.includes(query)` cannot answer "make
|
||||
* the pace suddenly feel faster" because no description contains that phrase,
|
||||
* so the honest result was zero matches. Scoring shared vocabulary answers it
|
||||
* partially, offline, with no model and no network.
|
||||
*
|
||||
* The scorer is the one the retrieval evaluation used, reproduced so the local
|
||||
* arm and the remote fallback rank identically rather than merely similarly:
|
||||
* lowercase alphabetic tokens, stop words dropped, anything three characters or
|
||||
* shorter dropped, and the shared-token count divided by the square root of the
|
||||
* entry's token count. That divisor is load-bearing. Without it the wordiest
|
||||
* entry wins every query on sheer surface area.
|
||||
*
|
||||
* Ties break on descending name, matching the evaluation's sort.
|
||||
*/
|
||||
|
||||
/** Dropped from both sides before scoring. Without this every entry matches on "the". */
|
||||
const STOP = new Set(
|
||||
(
|
||||
"the a an and or of to in on at is are be it its for with that this as by from into " +
|
||||
"one two must not no all over under across while when where which who whom whose they " +
|
||||
"them their we our you your he she his her but if then than so such can may might will " +
|
||||
"would should each other another same both few more most some any every"
|
||||
).split(" "),
|
||||
);
|
||||
|
||||
export function tokenize(text: string): string[] {
|
||||
const words = text.toLowerCase().match(/[a-z]+/g) ?? [];
|
||||
return words.filter((word) => word.length > 2 && !STOP.has(word));
|
||||
}
|
||||
|
||||
export interface Scored<T> {
|
||||
item: T;
|
||||
score: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Rank every item by shared vocabulary, best first.
|
||||
*
|
||||
* Returns all items rather than only matches, so a caller can decide where to
|
||||
* cut. Items scoring zero sort last.
|
||||
*/
|
||||
export function rankByWords<T>(
|
||||
query: string,
|
||||
items: T[],
|
||||
textOf: (item: T) => string,
|
||||
): Scored<T>[] {
|
||||
const want = new Set(tokenize(query));
|
||||
if (want.size === 0) return items.map((item) => ({ item, score: 0 }));
|
||||
|
||||
return items
|
||||
.map((item) => {
|
||||
const have = new Set(tokenize(textOf(item)));
|
||||
let shared = 0;
|
||||
for (const token of want) if (have.has(token)) shared += 1;
|
||||
// sqrt normalization: a longer entry has more chances to overlap, and
|
||||
// without this the wordiest blurb ranks first for every query.
|
||||
return { item, score: shared / (Math.sqrt(have.size) || 1) };
|
||||
})
|
||||
.sort((a, b) => b.score - a.score || nameOf(b.item).localeCompare(nameOf(a.item)));
|
||||
}
|
||||
|
||||
/** Only items sharing at least one token, best first. */
|
||||
export function searchByWords<T>(query: string, items: T[], textOf: (item: T) => string): T[] {
|
||||
return rankByWords(query, items, textOf)
|
||||
.filter((scored) => scored.score > 0)
|
||||
.map((scored) => scored.item);
|
||||
}
|
||||
|
||||
function nameOf(item: unknown): string {
|
||||
const named = item as { name?: unknown };
|
||||
return typeof named?.name === "string" ? named.name : "";
|
||||
}
|
||||
110
packages/cli/src/registry/localSemantic.test.ts
Normal file
110
packages/cli/src/registry/localSemantic.test.ts
Normal file
@ -0,0 +1,110 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { cachedLocalVectorRevision, fetchLocalVectors } from "./localSemantic.js";
|
||||
import { LOCAL_MODEL_DIMENSIONS } from "./localModel.js";
|
||||
|
||||
describe("fetchLocalVectors", () => {
|
||||
let dir: string;
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), "hf-vec-"));
|
||||
});
|
||||
afterEach(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
/** A metadata/matrix pair that agrees: one name, one row of the real width. */
|
||||
const servePair = (names: string[], dimensions: number, floats: number, revision?: string) =>
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async (url: string) => ({
|
||||
ok: true,
|
||||
arrayBuffer: async () =>
|
||||
url.endsWith(".json")
|
||||
? new TextEncoder().encode(JSON.stringify({ names, dimensions, revision })).buffer
|
||||
: new Float32Array(floats).buffer,
|
||||
})),
|
||||
);
|
||||
|
||||
it("writes both files into the cache directory", async () => {
|
||||
servePair(["whip-pan"], LOCAL_MODEL_DIMENSIONS, LOCAL_MODEL_DIMENSIONS);
|
||||
expect(await fetchLocalVectors("http://registry.test/", { directory: dir })).toBe(true);
|
||||
expect(fetch).toHaveBeenCalledWith(expect.any(String), {
|
||||
signal: expect.any(AbortSignal),
|
||||
});
|
||||
expect(existsSync(join(dir, "local-vectors.bin"))).toBe(true);
|
||||
expect(existsSync(join(dir, "local-vectors.json"))).toBe(true);
|
||||
});
|
||||
|
||||
it("caches nothing when the matrix is short of the names it claims", async () => {
|
||||
// Half a download is the case worth refusing: written, it loads as an
|
||||
// error on every later search until someone clears the cache by hand.
|
||||
servePair(["whip-pan", "rack-focus"], LOCAL_MODEL_DIMENSIONS, LOCAL_MODEL_DIMENSIONS);
|
||||
expect(await fetchLocalVectors("http://registry.test/", { directory: dir })).toBe(false);
|
||||
expect(existsSync(join(dir, "local-vectors.bin"))).toBe(false);
|
||||
expect(existsSync(join(dir, "local-vectors.json"))).toBe(false);
|
||||
});
|
||||
|
||||
it("caches nothing when the vectors came from a different model", async () => {
|
||||
servePair(["whip-pan"], 1536, 1536);
|
||||
expect(await fetchLocalVectors("http://registry.test/", { directory: dir })).toBe(false);
|
||||
expect(existsSync(join(dir, "local-vectors.json"))).toBe(false);
|
||||
});
|
||||
|
||||
it("reports failure instead of throwing, so the command survives", async () => {
|
||||
// A tier the user switched on that silently never runs is the failure
|
||||
// being guarded: the caller needs a false to be able to say so.
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () => ({ ok: false, arrayBuffer: async () => new ArrayBuffer(0) })),
|
||||
);
|
||||
expect(await fetchLocalVectors("http://registry.test", { directory: dir })).toBe(false);
|
||||
});
|
||||
|
||||
it("reports failure when the network throws", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () => {
|
||||
throw new Error("offline");
|
||||
}),
|
||||
);
|
||||
expect(await fetchLocalVectors("http://registry.test", { directory: dir })).toBe(false);
|
||||
});
|
||||
|
||||
it("refuses an unexpected revision without replacing the previous pair", async () => {
|
||||
mkdirSync(dir, { recursive: true });
|
||||
writeFileSync(
|
||||
join(dir, "local-vectors.json"),
|
||||
JSON.stringify({ names: ["old"], dimensions: LOCAL_MODEL_DIMENSIONS, revision: "old" }),
|
||||
);
|
||||
writeFileSync(join(dir, "local-vectors.bin"), new Float32Array(LOCAL_MODEL_DIMENSIONS));
|
||||
servePair(["new"], LOCAL_MODEL_DIMENSIONS, LOCAL_MODEL_DIMENSIONS, "old");
|
||||
|
||||
expect(
|
||||
await fetchLocalVectors("http://registry.test", {
|
||||
directory: dir,
|
||||
expectedRevision: "new",
|
||||
}),
|
||||
).toBe(false);
|
||||
expect(JSON.parse(readFileSync(join(dir, "local-vectors.json"), "utf-8"))).toEqual({
|
||||
names: ["old"],
|
||||
dimensions: LOCAL_MODEL_DIMENSIONS,
|
||||
revision: "old",
|
||||
});
|
||||
});
|
||||
|
||||
it("reads the revision only from a complete cached pair", () => {
|
||||
mkdirSync(dir, { recursive: true });
|
||||
writeFileSync(
|
||||
join(dir, "local-vectors.json"),
|
||||
JSON.stringify({ names: ["whip-pan"], dimensions: LOCAL_MODEL_DIMENSIONS, revision: "r1" }),
|
||||
);
|
||||
expect(cachedLocalVectorRevision(dir)).toBeUndefined();
|
||||
|
||||
writeFileSync(join(dir, "local-vectors.bin"), new Float32Array(LOCAL_MODEL_DIMENSIONS));
|
||||
expect(cachedLocalVectorRevision(dir)).toBe("r1");
|
||||
});
|
||||
});
|
||||
229
packages/cli/src/registry/localSemantic.ts
Normal file
229
packages/cli/src/registry/localSemantic.ts
Normal file
@ -0,0 +1,229 @@
|
||||
/**
|
||||
* Rank the catalog by meaning, on the user's machine, for free.
|
||||
*
|
||||
* Sits between word matching and the hosted endpoint. Word matching cannot
|
||||
* connect "make the pace feel faster" to a whip pan because they share no
|
||||
* words; the hosted endpoint can, but needs an account and a network. This
|
||||
* closes that gap with a 33 MB model the user opted into.
|
||||
*
|
||||
* The vectors here are 384-dimension and were produced by a different model
|
||||
* than the hosted 1536-dimension set. The two are not comparable and are never
|
||||
* mixed: a query is embedded by whichever model produced the vectors it is
|
||||
* being compared against.
|
||||
*/
|
||||
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { cosine, loadLocalEmbedder } from "./localEmbedder.js";
|
||||
import { LOCAL_MODEL_DIMENSIONS, isLocalModelReady } from "./localModel.js";
|
||||
|
||||
interface LocalVectorSet {
|
||||
names: string[];
|
||||
dimensions: number;
|
||||
vectors: Float32Array;
|
||||
}
|
||||
|
||||
interface LocalVectorMetadata {
|
||||
names: string[];
|
||||
dimensions: number;
|
||||
revision?: string;
|
||||
}
|
||||
|
||||
export interface FetchLocalVectorOptions {
|
||||
directory?: string;
|
||||
expectedRevision?: string;
|
||||
}
|
||||
|
||||
const CATALOG_ARTIFACT_TIMEOUT_MS = 30_000;
|
||||
|
||||
/** Where the bundled vector set lives, overridable for development. */
|
||||
function localVectorDirectory(): string {
|
||||
return (
|
||||
process.env["HYPERFRAMES_CATALOG_ARTIFACT_DIR"] ?? join(homedir(), ".hyperframes", "catalog")
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Put the catalog vectors in the user's cache, once.
|
||||
*
|
||||
* They are fetched rather than bundled: every install would otherwise carry a
|
||||
* copy of a file only people who opt into offline ranking will ever read, and
|
||||
* the vectors have to track the registry anyway, so shipping them inside the
|
||||
* package would freeze them to the release instead.
|
||||
*
|
||||
* Returns false rather than throwing. A download that fails costs the offline
|
||||
* tier, never the command, and the caller says so.
|
||||
*/
|
||||
/**
|
||||
* Do a freshly fetched metadata/matrix pair describe the same index?
|
||||
*
|
||||
* `names.length * dimensions` floats, at four bytes each, is the whole
|
||||
* contract. A pair that fails it is a truncated download or a different
|
||||
* model, never something worth caching.
|
||||
*/
|
||||
function vectorPairAgrees(fetched: Array<[string, Buffer]>): boolean {
|
||||
const meta = fetched.find(([file]) => file === "local-vectors.json")?.[1];
|
||||
const bin = fetched.find(([file]) => file === "local-vectors.bin")?.[1];
|
||||
if (!meta || !bin) return false;
|
||||
try {
|
||||
const parsed = JSON.parse(meta.toString("utf-8")) as {
|
||||
names?: string[];
|
||||
dimensions?: number;
|
||||
};
|
||||
if (parsed.dimensions !== LOCAL_MODEL_DIMENSIONS) return false;
|
||||
return bin.byteLength === (parsed.names?.length ?? -1) * LOCAL_MODEL_DIMENSIONS * 4;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/** The registry manifest and fetched pair must describe the same generation. */
|
||||
function vectorRevisionAgrees(
|
||||
fetched: Array<[string, Buffer]>,
|
||||
expectedRevision?: string,
|
||||
): boolean {
|
||||
if (expectedRevision === undefined) return true;
|
||||
const meta = fetched.find(([file]) => file === "local-vectors.json")?.[1];
|
||||
if (!meta) return false;
|
||||
try {
|
||||
const parsed = JSON.parse(meta.toString("utf-8")) as { revision?: unknown };
|
||||
return parsed.revision === expectedRevision;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export async function fetchLocalVectors(
|
||||
registryBaseUrl: string,
|
||||
options: FetchLocalVectorOptions = {},
|
||||
): Promise<boolean> {
|
||||
const directory = options.directory ?? localVectorDirectory();
|
||||
const base = registryBaseUrl.replace(/\/+$/, "");
|
||||
try {
|
||||
mkdirSync(directory, { recursive: true, mode: 0o700 });
|
||||
// Downloaded in full before anything is written. The two files have to
|
||||
// agree on how many rows there are, so a fetch that fails halfway through
|
||||
// must leave the previous pair intact rather than pairing a new name list
|
||||
// with an old matrix, which loads as an error instead of as stale data.
|
||||
const fetched: Array<[string, Buffer]> = [];
|
||||
for (const file of ["local-vectors.json", "local-vectors.bin"] as const) {
|
||||
const response = await fetch(`${base}/catalog-artifact/${file}`, {
|
||||
signal: AbortSignal.timeout(CATALOG_ARTIFACT_TIMEOUT_MS),
|
||||
});
|
||||
if (!response.ok) return false;
|
||||
fetched.push([file, Buffer.from(await response.arrayBuffer())]);
|
||||
}
|
||||
// Check the pair agrees BEFORE either file lands. Writing first and
|
||||
// discovering the mismatch at load time leaves a cache that fails every
|
||||
// subsequent search until someone deletes it by hand, and it is the only
|
||||
// point where a truncated or wrong-model response can still be refused.
|
||||
if (!vectorPairAgrees(fetched) || !vectorRevisionAgrees(fetched, options.expectedRevision)) {
|
||||
return false;
|
||||
}
|
||||
// 0o600: the cache is this user's, and the directory may be world-writable
|
||||
// when the caller overrides it.
|
||||
for (const [file, bytes] of fetched) {
|
||||
writeFileSync(join(directory, file), bytes, { mode: 0o600 });
|
||||
}
|
||||
return hasLocalVectors(directory);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export function hasLocalVectors(directory = localVectorDirectory()): boolean {
|
||||
return (
|
||||
existsSync(join(directory, "local-vectors.bin")) &&
|
||||
existsSync(join(directory, "local-vectors.json"))
|
||||
);
|
||||
}
|
||||
|
||||
function loadLocalVectors(directory = localVectorDirectory()): LocalVectorSet {
|
||||
const meta = JSON.parse(
|
||||
readFileSync(join(directory, "local-vectors.json"), "utf-8"),
|
||||
) as LocalVectorMetadata;
|
||||
const buffer = readFileSync(join(directory, "local-vectors.bin"));
|
||||
const vectors = new Float32Array(buffer.buffer, buffer.byteOffset, buffer.byteLength / 4);
|
||||
|
||||
const expected = meta.names.length * meta.dimensions;
|
||||
if (vectors.length !== expected) {
|
||||
throw new Error(`local vectors hold ${vectors.length} floats, expected ${expected}`);
|
||||
}
|
||||
if (meta.dimensions !== LOCAL_MODEL_DIMENSIONS) {
|
||||
// A dimension mismatch means the vectors and the model disagree, which
|
||||
// produces confident nonsense rather than an error.
|
||||
throw new Error(
|
||||
`local vectors are ${meta.dimensions}-dimension, model produces ${LOCAL_MODEL_DIMENSIONS}`,
|
||||
);
|
||||
}
|
||||
return { names: meta.names, dimensions: meta.dimensions, vectors };
|
||||
}
|
||||
|
||||
/**
|
||||
* The names the on-device index holds, without loading the vector matrix.
|
||||
*
|
||||
* Callers compare this against the live registry to see what meaning search
|
||||
* cannot reach. Reads the metadata file only, so it costs a small JSON parse
|
||||
* rather than the whole matrix, and answers empty rather than throwing: an
|
||||
* absent or unreadable artifact covers nothing, which is a coverage answer
|
||||
* rather than a reason to fail the search that asked.
|
||||
*/
|
||||
export function localVectorNames(directory = localVectorDirectory()): string[] {
|
||||
if (!hasLocalVectors(directory)) return [];
|
||||
try {
|
||||
const meta = JSON.parse(readFileSync(join(directory, "local-vectors.json"), "utf-8")) as {
|
||||
names?: string[];
|
||||
};
|
||||
return meta.names ?? [];
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/** Revision of the last complete pair, absent for pre-revision and unreadable caches. */
|
||||
export function cachedLocalVectorRevision(directory = localVectorDirectory()): string | undefined {
|
||||
if (!hasLocalVectors(directory)) return undefined;
|
||||
try {
|
||||
const meta = JSON.parse(readFileSync(join(directory, "local-vectors.json"), "utf-8")) as {
|
||||
revision?: unknown;
|
||||
};
|
||||
return typeof meta.revision === "string" ? meta.revision : undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Names ranked best first, each with the similarity that placed it there.
|
||||
* Returns null when local semantic search is not available.
|
||||
*
|
||||
* The score rides along because this ranker has no relevance threshold: every
|
||||
* query returns the whole set in some order, so a caller holding names alone
|
||||
* cannot tell a real match from the least-bad row of a query that matched
|
||||
* nothing at all.
|
||||
*/
|
||||
export async function localSemanticRanking(
|
||||
query: string,
|
||||
directory = localVectorDirectory(),
|
||||
): Promise<Array<{ name: string; score: number }> | null> {
|
||||
if (!isLocalModelReady() || !hasLocalVectors(directory)) return null;
|
||||
|
||||
const set = loadLocalVectors(directory);
|
||||
const embedder = await loadLocalEmbedder();
|
||||
const [queryVector] = await embedder.embed([query], { isQuery: true });
|
||||
if (!queryVector) return null;
|
||||
|
||||
const scored = set.names.map((name, row) => {
|
||||
const start = row * set.dimensions;
|
||||
return {
|
||||
name,
|
||||
score: cosine(queryVector, Array.from(set.vectors.subarray(start, start + set.dimensions))),
|
||||
};
|
||||
});
|
||||
|
||||
// Ties break on descending name, matching every other ranking in this system.
|
||||
scored.sort((a, b) => b.score - a.score || b.name.localeCompare(a.name));
|
||||
return scored;
|
||||
}
|
||||
152
packages/cli/src/registry/wordpiece.test.ts
Normal file
152
packages/cli/src/registry/wordpiece.test.ts
Normal file
@ -0,0 +1,152 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { configFromTokenizerJson, encode, normalize, preTokenize, wordPiece } from "./wordpiece.js";
|
||||
|
||||
const REFERENCE_PATH = join(import.meta.dirname, "__fixtures__", "wordpiece-reference.json");
|
||||
const TOKENIZER_PATH = join(
|
||||
homedir(),
|
||||
".hyperframes",
|
||||
"models",
|
||||
"bge-small-en-v1.5.tokenizer.json",
|
||||
);
|
||||
|
||||
/** A tiny vocab keeps the unit tests readable and independent of the 30k file. */
|
||||
const CONFIG = configFromTokenizerJson(
|
||||
JSON.stringify({
|
||||
model: {
|
||||
vocab: {
|
||||
"[UNK]": 0,
|
||||
"[CLS]": 1,
|
||||
"[SEP]": 2,
|
||||
fast: 3,
|
||||
camera: 4,
|
||||
"##era": 5,
|
||||
cam: 6,
|
||||
".": 7,
|
||||
whip: 8,
|
||||
pan: 9,
|
||||
},
|
||||
unk_token: "[UNK]",
|
||||
continuing_subword_prefix: "##",
|
||||
max_input_chars_per_word: 100,
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
describe("normalize", () => {
|
||||
it("lowercases", () => {
|
||||
expect(normalize("Whip PAN")).toBe("whip pan");
|
||||
});
|
||||
|
||||
it("strips accents", () => {
|
||||
expect(normalize("café naïve")).toBe("cafe naive");
|
||||
});
|
||||
|
||||
it("collapses control characters and whitespace variants to spaces", () => {
|
||||
expect(normalize("a\tb\nc")).toBe("a b c");
|
||||
});
|
||||
|
||||
it("isolates CJK characters so each becomes its own token", () => {
|
||||
expect(normalize("a中b")).toBe("a 中 b");
|
||||
});
|
||||
});
|
||||
|
||||
describe("preTokenize", () => {
|
||||
it("splits on whitespace", () => {
|
||||
expect(preTokenize("fast camera pan")).toEqual(["fast", "camera", "pan"]);
|
||||
});
|
||||
|
||||
it("isolates punctuation into its own token", () => {
|
||||
expect(preTokenize("group: transitions.")).toEqual(["group", ":", "transitions", "."]);
|
||||
});
|
||||
|
||||
it("drops empty chunks from repeated spaces", () => {
|
||||
expect(preTokenize("a b")).toEqual(["a", "b"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("wordPiece", () => {
|
||||
it("returns a whole word when the vocab has it", () => {
|
||||
expect(wordPiece("camera", CONFIG)).toEqual(["camera"]);
|
||||
});
|
||||
|
||||
it("splits into continuation pieces, longest match first", () => {
|
||||
// "camera" is present whole, so force the split path with a word that is not.
|
||||
expect(wordPiece("camera", { ...CONFIG, vocab: { cam: 6, "##era": 5 } })).toEqual([
|
||||
"cam",
|
||||
"##era",
|
||||
]);
|
||||
});
|
||||
|
||||
it("marks the whole word unknown when any piece is unmatchable", () => {
|
||||
// Emitting partial pieces would silently change the embedding rather than
|
||||
// failing, so a single bad piece invalidates the word.
|
||||
expect(wordPiece("zzzz", CONFIG)).toEqual(["[UNK]"]);
|
||||
});
|
||||
|
||||
it("marks a word longer than the limit unknown without scanning it", () => {
|
||||
expect(wordPiece("x".repeat(101), CONFIG)).toEqual(["[UNK]"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("encode", () => {
|
||||
it("wraps the sequence in CLS and SEP", () => {
|
||||
const { ids } = encode("fast", CONFIG);
|
||||
expect(ids[0]).toBe(CONFIG.vocab["[CLS]"]);
|
||||
expect(ids.at(-1)).toBe(CONFIG.vocab["[SEP]"]);
|
||||
});
|
||||
|
||||
it("returns masks matching the id length", () => {
|
||||
const encoding = encode("fast camera", CONFIG);
|
||||
expect(encoding.attentionMask).toHaveLength(encoding.ids.length);
|
||||
expect(encoding.tokenTypeIds).toHaveLength(encoding.ids.length);
|
||||
expect(new Set(encoding.attentionMask)).toEqual(new Set([1]));
|
||||
expect(new Set(encoding.tokenTypeIds)).toEqual(new Set([0]));
|
||||
});
|
||||
|
||||
it("encodes empty input as just the special tokens", () => {
|
||||
expect(encode("", CONFIG).ids).toEqual([1, 2]);
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* The test that actually matters.
|
||||
*
|
||||
* Wrong token ids produce plausible embeddings and plausible rankings, so
|
||||
* nothing downstream fails visibly. Comparing against a real tokenizer over the
|
||||
* whole catalog is the only way to know this implementation is correct rather
|
||||
* than merely reasonable.
|
||||
*/
|
||||
describe("parity with the reference tokenizer", () => {
|
||||
const hasFixture = existsSync(REFERENCE_PATH);
|
||||
const hasVocab = existsSync(TOKENIZER_PATH);
|
||||
|
||||
it.runIf(hasFixture && hasVocab)("matches reference ids across the whole corpus", () => {
|
||||
const config = configFromTokenizerJson(readFileSync(TOKENIZER_PATH, "utf-8"));
|
||||
const reference = JSON.parse(readFileSync(REFERENCE_PATH, "utf-8")) as {
|
||||
text: string;
|
||||
ids: number[];
|
||||
}[];
|
||||
expect(reference.length).toBeGreaterThan(500);
|
||||
|
||||
const mismatches = reference
|
||||
.map(({ text, ids }) => ({ text, expected: ids, actual: encode(text, config).ids }))
|
||||
.filter(({ expected, actual }) => expected.join(",") !== actual.join(","));
|
||||
|
||||
expect(
|
||||
mismatches
|
||||
.slice(0, 3)
|
||||
.map(
|
||||
(m) => `${JSON.stringify(m.text.slice(0, 60))}\n want ${m.expected}\n got ${m.actual}`,
|
||||
),
|
||||
).toEqual([]);
|
||||
expect(mismatches).toHaveLength(0);
|
||||
});
|
||||
|
||||
it.skipIf(hasFixture && hasVocab)("is skipped without the fixture and vocab", () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
194
packages/cli/src/registry/wordpiece.ts
Normal file
194
packages/cli/src/registry/wordpiece.ts
Normal file
@ -0,0 +1,194 @@
|
||||
/**
|
||||
* BERT WordPiece tokenization, implemented directly rather than pulled in.
|
||||
*
|
||||
* The obvious dependencies both failed: the standalone tokenizer package ships
|
||||
* builder components with no way to load a `tokenizer.json`, and the full
|
||||
* wrapper costs 359 MB installed because it pins an exact ONNX runtime and ends
|
||||
* up with a second native copy plus an unused web runtime. This is ~120 lines
|
||||
* with no dependencies at all.
|
||||
*
|
||||
* Hand-writing a tokenizer is normally a bad idea, because wrong token ids
|
||||
* still produce plausible-looking embeddings and plausible-looking rankings.
|
||||
* Nothing fails loudly. That is why `wordpiece.parity.test.ts` compares this
|
||||
* implementation's ids against a reference implementation over a corpus rather
|
||||
* than asserting a handful of cases by eye.
|
||||
*
|
||||
* The pipeline reproduces the config in the model's own tokenizer.json:
|
||||
* BertNormalizer (clean text, isolate CJK, strip accents, lowercase), then
|
||||
* BertPreTokenizer (whitespace split, punctuation isolated), then greedy
|
||||
* longest-match-first WordPiece, then [CLS] ... [SEP].
|
||||
*/
|
||||
|
||||
export interface WordPieceConfig {
|
||||
vocab: Record<string, number>;
|
||||
unkToken: string;
|
||||
continuingSubwordPrefix: string;
|
||||
maxInputCharsPerWord: number;
|
||||
clsToken: string;
|
||||
sepToken: string;
|
||||
}
|
||||
|
||||
export interface Encoding {
|
||||
ids: number[];
|
||||
attentionMask: number[];
|
||||
tokenTypeIds: number[];
|
||||
}
|
||||
|
||||
/** Read the pieces this tokenizer needs out of a HuggingFace tokenizer.json. */
|
||||
export function configFromTokenizerJson(raw: string): WordPieceConfig {
|
||||
const parsed = JSON.parse(raw) as {
|
||||
model: {
|
||||
vocab: Record<string, number>;
|
||||
unk_token?: string;
|
||||
continuing_subword_prefix?: string;
|
||||
max_input_chars_per_word?: number;
|
||||
};
|
||||
};
|
||||
const model = parsed.model;
|
||||
if (!model?.vocab) throw new Error("tokenizer.json has no WordPiece vocab");
|
||||
return {
|
||||
vocab: model.vocab,
|
||||
unkToken: model.unk_token ?? "[UNK]",
|
||||
continuingSubwordPrefix: model.continuing_subword_prefix ?? "##",
|
||||
maxInputCharsPerWord: model.max_input_chars_per_word ?? 100,
|
||||
clsToken: "[CLS]",
|
||||
sepToken: "[SEP]",
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* BertNormalizer.
|
||||
*
|
||||
* Control characters are dropped and every whitespace variant collapses to a
|
||||
* plain space. CJK characters are isolated so each becomes its own token, which
|
||||
* is what the reference does even for an English-only model.
|
||||
*/
|
||||
export function normalize(text: string): string {
|
||||
let out = "";
|
||||
for (const char of text) {
|
||||
const code = char.codePointAt(0) as number;
|
||||
if (code === 0 || code === 0xfffd) continue;
|
||||
if (isControl(char, code)) continue;
|
||||
if (isWhitespace(char, code)) {
|
||||
out += " ";
|
||||
continue;
|
||||
}
|
||||
out += isChinese(code) ? ` ${char} ` : char;
|
||||
}
|
||||
// Strip accents after lowercasing: NFD splits a letter from its mark, and the
|
||||
// marks are then dropped. Mirrors strip_accents defaulting to lowercase.
|
||||
return out
|
||||
.toLowerCase()
|
||||
.normalize("NFD")
|
||||
.replace(/\p{Mn}/gu, "");
|
||||
}
|
||||
|
||||
/** BertPreTokenizer: split on whitespace, then break punctuation into its own token. */
|
||||
export function preTokenize(normalized: string): string[] {
|
||||
const words: string[] = [];
|
||||
for (const chunk of normalized.split(/\s+/)) {
|
||||
if (!chunk) continue;
|
||||
let current = "";
|
||||
for (const char of chunk) {
|
||||
if (isPunctuation(char)) {
|
||||
if (current) words.push(current);
|
||||
words.push(char);
|
||||
current = "";
|
||||
} else {
|
||||
current += char;
|
||||
}
|
||||
}
|
||||
if (current) words.push(current);
|
||||
}
|
||||
return words;
|
||||
}
|
||||
|
||||
/** Greedy longest-match-first, the reference WordPiece algorithm. */
|
||||
export function wordPiece(word: string, config: WordPieceConfig): string[] {
|
||||
if ([...word].length > config.maxInputCharsPerWord) return [config.unkToken];
|
||||
|
||||
const pieces: string[] = [];
|
||||
let start = 0;
|
||||
while (start < word.length) {
|
||||
let end = word.length;
|
||||
let match: string | null = null;
|
||||
while (start < end) {
|
||||
const candidate =
|
||||
start === 0
|
||||
? word.slice(start, end)
|
||||
: config.continuingSubwordPrefix + word.slice(start, end);
|
||||
if (candidate in config.vocab) {
|
||||
match = candidate;
|
||||
break;
|
||||
}
|
||||
end -= 1;
|
||||
}
|
||||
// A single unmatchable piece makes the whole word unknown, not just that
|
||||
// piece. Emitting partial pieces here would silently change the embedding.
|
||||
if (match === null) return [config.unkToken];
|
||||
pieces.push(match);
|
||||
start = end;
|
||||
}
|
||||
return pieces;
|
||||
}
|
||||
|
||||
export function encode(text: string, config: WordPieceConfig): Encoding {
|
||||
const tokens = [config.clsToken];
|
||||
for (const word of preTokenize(normalize(text))) tokens.push(...wordPiece(word, config));
|
||||
tokens.push(config.sepToken);
|
||||
|
||||
const unk = config.vocab[config.unkToken] as number;
|
||||
const ids = tokens.map((token) => config.vocab[token] ?? unk);
|
||||
return {
|
||||
ids,
|
||||
attentionMask: ids.map(() => 1),
|
||||
tokenTypeIds: ids.map(() => 0),
|
||||
};
|
||||
}
|
||||
|
||||
function isControl(char: string, code: number): boolean {
|
||||
if (char === "\t" || char === "\n" || char === "\r") return false;
|
||||
return code < 32 || code === 127 || /\p{Cc}|\p{Cf}/u.test(char);
|
||||
}
|
||||
|
||||
function isWhitespace(char: string, code: number): boolean {
|
||||
return (
|
||||
char === " " ||
|
||||
char === "\t" ||
|
||||
char === "\n" ||
|
||||
char === "\r" ||
|
||||
code === 0x0b ||
|
||||
code === 0x0c ||
|
||||
/\p{Zs}/u.test(char)
|
||||
);
|
||||
}
|
||||
|
||||
function isPunctuation(char: string): boolean {
|
||||
const code = char.codePointAt(0) as number;
|
||||
// ASCII non-alphanumerics, plus Unicode punctuation categories only.
|
||||
// Symbol categories are deliberately excluded: the reference leaves a symbol
|
||||
// attached to the preceding token, so "360°" becomes 360 + ##° rather than
|
||||
// two separate words. Including \p{S} here silently changed the embedding of
|
||||
// every description containing one, which the parity corpus caught.
|
||||
const asciiSymbol =
|
||||
(code >= 33 && code <= 47) ||
|
||||
(code >= 58 && code <= 64) ||
|
||||
(code >= 91 && code <= 96) ||
|
||||
(code >= 123 && code <= 126);
|
||||
return asciiSymbol || /\p{P}/u.test(char);
|
||||
}
|
||||
|
||||
// a Unicode block test; the ranges are data, and 16 cyclomatic is what checking them costs
|
||||
// fallow-ignore-next-line complexity
|
||||
function isChinese(code: number): boolean {
|
||||
return (
|
||||
(code >= 0x4e00 && code <= 0x9fff) ||
|
||||
(code >= 0x3400 && code <= 0x4dbf) ||
|
||||
(code >= 0x20000 && code <= 0x2a6df) ||
|
||||
(code >= 0x2a700 && code <= 0x2b73f) ||
|
||||
(code >= 0x2b740 && code <= 0x2b81f) ||
|
||||
(code >= 0x2b820 && code <= 0x2ceaf) ||
|
||||
(code >= 0xf900 && code <= 0xfaff) ||
|
||||
(code >= 0x2f800 && code <= 0x2fa1f)
|
||||
);
|
||||
}
|
||||
@ -386,6 +386,8 @@ function mintConfig(): HyperframesConfig {
|
||||
}
|
||||
|
||||
export interface HyperframesConfig {
|
||||
/** Has the user agreed to download the on-device search model? Undefined means never asked. */
|
||||
localEmbeddingEnabled?: boolean;
|
||||
/** Whether anonymous telemetry is enabled (default: true in production) */
|
||||
telemetryEnabled: boolean;
|
||||
/** Stable anonymous identifier — no PII, just a random UUID */
|
||||
@ -652,6 +654,9 @@ function passthroughFields(parsed: Partial<HyperframesConfig>): Partial<Hyperfra
|
||||
skillsOutdatedCount: parsed.skillsOutdatedCount,
|
||||
skillsMissingCount: parsed.skillsMissingCount,
|
||||
skillsRemovedCount: parsed.skillsRemovedCount,
|
||||
// Consent, so it survives the run that recorded it. Undefined stays
|
||||
// undefined on purpose: it means never asked, which is not the same as no.
|
||||
localEmbeddingEnabled: parsed.localEmbeddingEnabled,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@ -21,6 +21,19 @@
|
||||
"format": "uri",
|
||||
"description": "Registry homepage URL."
|
||||
},
|
||||
"catalogArtifact": {
|
||||
"type": "object",
|
||||
"description": "Published on-device vector artifact for this registry.",
|
||||
"required": ["revision"],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"revision": {
|
||||
"type": "string",
|
||||
"pattern": "^[a-f0-9]{64}$",
|
||||
"description": "SHA-256 identity of the searchable corpus and embedding contract."
|
||||
}
|
||||
}
|
||||
},
|
||||
"items": {
|
||||
"type": "array",
|
||||
"description": "Items in this registry. Each entry is a shorthand reference; the full item manifest lives at <type-dir>/<name>/registry-item.json.",
|
||||
|
||||
@ -128,6 +128,11 @@ export interface RegistryManifest {
|
||||
name: string;
|
||||
/** Registry homepage URL. */
|
||||
homepage: string;
|
||||
/** Published on-device vector artifact, when this registry provides one. */
|
||||
catalogArtifact?: {
|
||||
/** SHA-256 identity of the searchable corpus and embedding contract. */
|
||||
revision: string;
|
||||
};
|
||||
/** Items in this registry. */
|
||||
items: RegistryManifestEntry[];
|
||||
}
|
||||
|
||||
12
registry/catalog-artifact/.gitignore
vendored
Normal file
12
registry/catalog-artifact/.gitignore
vendored
Normal file
@ -0,0 +1,12 @@
|
||||
# Build output, not source. The catalog is derived from the registry by
|
||||
# scripts/catalog/build-catalog-artifact.ts, and committing the hosted copy
|
||||
# would create a stale artifact that a later run silently ranks against.
|
||||
# vectors.json is also ~9 MB, well past the repository's non-LFS file limit;
|
||||
# the hosted tier reads it from the server, never from a checkout.
|
||||
#
|
||||
# The on-device pair is the exception: the CLI fetches local-vectors.bin and
|
||||
# local-vectors.json from this registry when a user opts into offline search,
|
||||
# so they have to be served from here. Ignoring either one leaves the fetch
|
||||
# half-complete and the offline tier silently unable to rank.
|
||||
*.json
|
||||
!local-vectors.json
|
||||
51
registry/catalog-artifact/README.md
Normal file
51
registry/catalog-artifact/README.md
Normal file
@ -0,0 +1,51 @@
|
||||
# Catalog artifact
|
||||
|
||||
Two files ship from here, and only these two. The CLI fetches them over HTTP
|
||||
when a user opts into offline catalog search (`catalog --query ... --on-device`),
|
||||
so they are served from the registry rather than bundled in the package.
|
||||
|
||||
| File | What it is |
|
||||
| -------------------- | ----------------------------------------------------------------------- |
|
||||
| `local-vectors.json` | `{ model, dimensions, names }`. `names` is the row order of the binary. |
|
||||
| `local-vectors.bin` | Float32, row-major, `names.length * dimensions` values, no header. |
|
||||
|
||||
Currently 168 rows at 384 dimensions: 258,048 bytes, one row per installable
|
||||
registry item.
|
||||
|
||||
## Provenance
|
||||
|
||||
No digest, revision or build timestamp is recorded in either file, and the
|
||||
runtime only checks that `dimensions` matches the model it loaded. `model` is a
|
||||
label the build wrote, not a proof. So the only real provenance check is to
|
||||
rebuild the rows and compare them, which works because both inputs are in this
|
||||
repository:
|
||||
|
||||
- the text each row was embedded from is `itemRetrievalText(registry-item.json)`
|
||||
(title, description, tags, joined by newlines, name deliberately excluded),
|
||||
over `registry/blocks/*` and `registry/components/*` sorted by name;
|
||||
- the model is the pinned quantized `bge-small-en-v1.5` ONNX build that
|
||||
`packages/cli/src/registry/localModel.ts` downloads.
|
||||
|
||||
Re-embedding in batches of 16, the batch size the build uses, reproduces the
|
||||
shipped rows exactly (cosine 1.000000). Batch size matters: the same text
|
||||
embedded alone differs at cosine 0.9969, because padding within a batch changes
|
||||
the quantized result. A rebuild that does not match this way was not built from
|
||||
this registry, or not with this model.
|
||||
|
||||
## Rebuilding
|
||||
|
||||
```bash
|
||||
bun scripts/catalog/build-local-vectors.ts
|
||||
```
|
||||
|
||||
It reads `registry/blocks/*` and `registry/components/*` directly, so the
|
||||
rebuild has no input outside this repository. You rarely need to run it by
|
||||
hand: a lefthook `catalog-index` pre-commit command regenerates and re-stages
|
||||
both files whenever a staged `registry-item.json` changes, and CI fails the
|
||||
"Catalog: search index covers the registry" job if the index is ever missing an
|
||||
item.
|
||||
|
||||
`build-catalog-artifact.ts` does **not** produce these files. It builds the
|
||||
3072-dimension hosted artifact from an external shelf file, for the hosted search
|
||||
tier that this repository does not ship. Its `--shelf`, `manifest.json` and
|
||||
`text-embedding-3-large` have nothing to do with `local-vectors.*`.
|
||||
BIN
registry/catalog-artifact/local-vectors.bin
Normal file
BIN
registry/catalog-artifact/local-vectors.bin
Normal file
Binary file not shown.
176
registry/catalog-artifact/local-vectors.json
Normal file
176
registry/catalog-artifact/local-vectors.json
Normal file
@ -0,0 +1,176 @@
|
||||
{
|
||||
"model": "bge-small-en-v1.5",
|
||||
"modelRevision": "ea104dacec62c0de699686887e3f920caeb4f3e3",
|
||||
"dimensions": 384,
|
||||
"revision": "3e0b7c140466db717b69481be5a87e77b1fdec5000b33ca61846ba27e5558238",
|
||||
"names": [
|
||||
"app-showcase",
|
||||
"apple-money-count",
|
||||
"beat-freeze-cut",
|
||||
"blue-sweater-intro-video",
|
||||
"camcorder-hud",
|
||||
"caption-blend-difference",
|
||||
"caption-clip-wipe",
|
||||
"caption-editorial-emphasis",
|
||||
"caption-emoji-pop",
|
||||
"caption-glitch-rgb",
|
||||
"caption-gradient-fill",
|
||||
"caption-highlight",
|
||||
"caption-kinetic-slam",
|
||||
"caption-matrix-decode",
|
||||
"caption-neon-accent",
|
||||
"caption-neon-glow",
|
||||
"caption-parallax-layers",
|
||||
"caption-particle-burst",
|
||||
"caption-pill-karaoke",
|
||||
"caption-texture",
|
||||
"caption-weight-shift",
|
||||
"chromatic-radial-split",
|
||||
"cinematic-zoom",
|
||||
"code-3d-extrude",
|
||||
"code-diff",
|
||||
"code-highlight",
|
||||
"code-morph",
|
||||
"code-particle-assemble",
|
||||
"code-scroll",
|
||||
"code-shader-dissolve",
|
||||
"code-snippet-apple-terminal-basic",
|
||||
"code-snippet-apple-terminal-clear-dark",
|
||||
"code-snippet-apple-terminal-clear-light",
|
||||
"code-snippet-apple-terminal-grass",
|
||||
"code-snippet-apple-terminal-homebrew",
|
||||
"code-snippet-apple-terminal-man-page",
|
||||
"code-snippet-apple-terminal-novel",
|
||||
"code-snippet-apple-terminal-ocean",
|
||||
"code-snippet-apple-terminal-pro",
|
||||
"code-snippet-apple-terminal-red-sands",
|
||||
"code-snippet-apple-terminal-silver-aerogel",
|
||||
"code-snippet-apple-terminal-solid-colors",
|
||||
"code-snippet-dark-2026",
|
||||
"code-snippet-dark-modern",
|
||||
"code-snippet-dark-plus",
|
||||
"code-snippet-flight",
|
||||
"code-snippet-high-contrast",
|
||||
"code-snippet-high-contrast-light",
|
||||
"code-snippet-light-2026",
|
||||
"code-snippet-light-modern",
|
||||
"code-snippet-light-plus",
|
||||
"code-snippet-monokai",
|
||||
"code-snippet-solarized-light",
|
||||
"code-snippet-visual-studio-dark",
|
||||
"code-snippet-visual-studio-light",
|
||||
"code-typing",
|
||||
"cross-warp-morph",
|
||||
"data-chart",
|
||||
"domain-warp-dissolve",
|
||||
"editorial-flash-overlay",
|
||||
"flash-through-white",
|
||||
"flowchart",
|
||||
"flowchart-vertical",
|
||||
"freeze-frame-dressing",
|
||||
"glitch",
|
||||
"grain-overlay",
|
||||
"gravitational-lens",
|
||||
"grid-pixelate-wipe",
|
||||
"hw-arrow",
|
||||
"hw-boil",
|
||||
"hw-box-label",
|
||||
"hw-callout-circle",
|
||||
"hw-frame",
|
||||
"hw-path-text",
|
||||
"hw-pipeline",
|
||||
"hw-scribble-transition",
|
||||
"hw-text-cloud",
|
||||
"hw-title",
|
||||
"hw-underline",
|
||||
"instagram-follow",
|
||||
"ios26-liquid-glass",
|
||||
"light-leak",
|
||||
"liquid-glass-context-menu",
|
||||
"liquid-glass-media-controls",
|
||||
"liquid-glass-notification",
|
||||
"liquid-glass-widgets",
|
||||
"logo-outro",
|
||||
"lower-third-bild",
|
||||
"lt-accent-underline",
|
||||
"lt-bold-block",
|
||||
"lt-clean-bar",
|
||||
"lt-color-block",
|
||||
"lt-dark-card",
|
||||
"lt-kicker-name",
|
||||
"lt-mask-reveal",
|
||||
"lt-side-rule",
|
||||
"lt-soft-pill",
|
||||
"lt-stack-bars",
|
||||
"macos-notification",
|
||||
"macos-tahoe-liquid-glass",
|
||||
"mk-background",
|
||||
"mk-callout-highlight",
|
||||
"mk-clone-wall-transition",
|
||||
"mk-emphasis-type",
|
||||
"mk-line-graph",
|
||||
"mk-placeholder-grid",
|
||||
"mk-progress-stat",
|
||||
"mk-specs-list",
|
||||
"mk-usage-arc",
|
||||
"morph-text",
|
||||
"motion-blur",
|
||||
"news-ticker",
|
||||
"north-korea-locked-down",
|
||||
"nyc-paris-flight",
|
||||
"organic-light-leak-overlay",
|
||||
"parallax-unzoom",
|
||||
"parallax-zoom",
|
||||
"reddit-post",
|
||||
"ridged-burn",
|
||||
"ripple-waves",
|
||||
"sdf-iris",
|
||||
"shimmer-sweep",
|
||||
"spain-map",
|
||||
"spotify-card",
|
||||
"swirl-vortex",
|
||||
"texture-mask-text",
|
||||
"thermal-distortion",
|
||||
"tiktok-follow",
|
||||
"transitions-3d",
|
||||
"transitions-blur",
|
||||
"transitions-cover",
|
||||
"transitions-destruction",
|
||||
"transitions-dissolve",
|
||||
"transitions-distortion",
|
||||
"transitions-grid",
|
||||
"transitions-light",
|
||||
"transitions-mechanical",
|
||||
"transitions-other",
|
||||
"transitions-push",
|
||||
"transitions-radial",
|
||||
"transitions-scale",
|
||||
"ui-3d-reveal",
|
||||
"us-map",
|
||||
"us-map-bubble",
|
||||
"us-map-flow",
|
||||
"us-map-hex",
|
||||
"vfx-iphone-device",
|
||||
"vfx-liquid-background",
|
||||
"vfx-liquid-glass",
|
||||
"vfx-magnetic",
|
||||
"vfx-portal",
|
||||
"vfx-shatter",
|
||||
"vfx-text-cursor",
|
||||
"vignette",
|
||||
"vpn-youtube-spot",
|
||||
"whip-pan",
|
||||
"world-map",
|
||||
"x-post",
|
||||
"yt-camera-move",
|
||||
"yt-circle-pointer",
|
||||
"yt-comment-card",
|
||||
"yt-feather-highlight",
|
||||
"yt-lcd-background",
|
||||
"yt-logo-intro",
|
||||
"yt-lower-third",
|
||||
"yt-prism-title",
|
||||
"yt-screen-warp",
|
||||
"yt-vertical-fill"
|
||||
]
|
||||
}
|
||||
@ -2,6 +2,9 @@
|
||||
"$schema": "https://hyperframes.heygen.com/schema/registry.json",
|
||||
"name": "hyperframes",
|
||||
"homepage": "https://hyperframes.heygen.com",
|
||||
"catalogArtifact": {
|
||||
"revision": "3e0b7c140466db717b69481be5a87e77b1fdec5000b33ca61846ba27e5558238"
|
||||
},
|
||||
"items": [
|
||||
{
|
||||
"name": "warm-grain",
|
||||
|
||||
143
scripts/catalog/build-catalog-artifact.ts
Normal file
143
scripts/catalog/build-catalog-artifact.ts
Normal file
@ -0,0 +1,143 @@
|
||||
/**
|
||||
* Publish the video-primitive catalog artifact.
|
||||
*
|
||||
* Usage:
|
||||
* tsx scripts/catalog/build-catalog-artifact.ts --shelf <path> --revision <sha> [--out <dir>]
|
||||
*
|
||||
* Requires OPENAI_API_KEY. Embedding 424 descriptions is a real, paid call, so
|
||||
* this is invoked deliberately rather than on every commit. The artifact is
|
||||
* verified after writing: a build that cannot be read back is a failed build.
|
||||
*/
|
||||
|
||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
import {
|
||||
buildArtifact,
|
||||
manifestBytes,
|
||||
movesMissingFromRegistry,
|
||||
parseShelf,
|
||||
verifyArtifact,
|
||||
type Embedder,
|
||||
} from "./catalog-artifact.js";
|
||||
|
||||
// Measured on the 405-brief gold set, not chosen from a leaderboard:
|
||||
// 3-large scores 60.5% recall@20 against 54.1% for 3-small, a 6.4 point gain.
|
||||
// Embedding the whole catalog costs a couple of cents, and per-query cost is
|
||||
// negligible, so the larger model is the better default by a wide margin.
|
||||
const EMBEDDING_MODEL = "text-embedding-3-large";
|
||||
const EMBEDDING_DIMENSION = 3072;
|
||||
const BATCH_SIZE = 100;
|
||||
const DEFAULT_REGISTRY_INDEX = "registry/registry.json";
|
||||
const DEFAULT_OUT = "registry/catalog-artifact";
|
||||
|
||||
/**
|
||||
* Read the names the registry can actually serve.
|
||||
*
|
||||
* Absent index means the caller opted out of the check rather than passed it,
|
||||
* so that case is reported by the caller instead of being treated as a pass.
|
||||
*/
|
||||
function registryNames(indexPath: string): string[] | undefined {
|
||||
try {
|
||||
const parsed = JSON.parse(readFileSync(indexPath, "utf8")) as {
|
||||
items?: { name?: string }[];
|
||||
};
|
||||
if (!Array.isArray(parsed.items)) return undefined;
|
||||
return parsed.items
|
||||
.map((item) => item.name)
|
||||
.filter((name): name is string => typeof name === "string");
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function arg(name: string): string | undefined {
|
||||
const index = process.argv.indexOf(`--${name}`);
|
||||
return index === -1 ? undefined : process.argv[index + 1];
|
||||
}
|
||||
|
||||
/** Batched so a 424-move shelf does not depend on one oversized request. */
|
||||
// request assembly plus the error cases a remote embedder can return
|
||||
// fallow-ignore-next-line complexity
|
||||
const openAiEmbedder: Embedder = async (texts) => {
|
||||
const key = process.env.OPENAI_API_KEY;
|
||||
if (!key) throw new Error("OPENAI_API_KEY is not set");
|
||||
|
||||
const vectors: number[][] = [];
|
||||
for (let start = 0; start < texts.length; start += BATCH_SIZE) {
|
||||
const batch = texts.slice(start, start + BATCH_SIZE);
|
||||
const response = await fetch("https://api.openai.com/v1/embeddings", {
|
||||
method: "POST",
|
||||
headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ model: EMBEDDING_MODEL, input: batch }),
|
||||
});
|
||||
if (!response.ok) {
|
||||
throw new Error(`Embedding request failed: ${response.status} ${await response.text()}`);
|
||||
}
|
||||
const body = (await response.json()) as { data: { index: number; embedding: number[] }[] };
|
||||
// Order is documented as preserved, but index is returned explicitly, so
|
||||
// sort by it rather than trusting position.
|
||||
const ordered = [...body.data].sort((a, b) => a.index - b.index);
|
||||
if (ordered.length !== batch.length) {
|
||||
throw new Error(`Expected ${batch.length} embeddings, received ${ordered.length}`);
|
||||
}
|
||||
vectors.push(...ordered.map((item) => item.embedding));
|
||||
}
|
||||
return vectors;
|
||||
};
|
||||
|
||||
// a script entry point
|
||||
// fallow-ignore-next-line complexity
|
||||
async function main(): Promise<void> {
|
||||
const shelfPath = arg("shelf");
|
||||
const revision = arg("revision");
|
||||
const out = arg("out") ?? DEFAULT_OUT;
|
||||
if (!shelfPath || !revision) {
|
||||
throw new Error("Both --shelf and --revision are required");
|
||||
}
|
||||
|
||||
const shelfText = readFileSync(shelfPath, "utf-8");
|
||||
// Only publish moves the registry can serve. The shelf doubles as a design
|
||||
// document, so it lists moves that were specified and never built.
|
||||
const installableNames = registryNames(arg("registry-index") ?? DEFAULT_REGISTRY_INDEX);
|
||||
const built = await buildArtifact({
|
||||
shelfText,
|
||||
...(installableNames ? { installableNames } : {}),
|
||||
sourceRevision: revision,
|
||||
embeddingModel: EMBEDDING_MODEL,
|
||||
embed: openAiEmbedder,
|
||||
expectedDimension: EMBEDDING_DIMENSION,
|
||||
});
|
||||
|
||||
// Verify before writing. Publishing something we cannot read back would put
|
||||
// the burden of discovering it on the consumer at request time.
|
||||
verifyArtifact(built);
|
||||
|
||||
// What is left to report is the gap, not a defect in the artifact: buildArtifact
|
||||
// already dropped these, so the published catalog never contains a move the
|
||||
// registry cannot serve. Reported, not fatal, because the shelf legitimately
|
||||
// leads the registry while moves land.
|
||||
if (installableNames === undefined) {
|
||||
console.warn("warning registry index unreadable; published every shelf move unchecked");
|
||||
} else {
|
||||
const dropped = movesMissingFromRegistry([...parseShelf(shelfText).keys()], installableNames);
|
||||
if (dropped.length > 0) {
|
||||
console.warn(
|
||||
`warning ${dropped.length} shelf moves dropped, no registry item: ${dropped.slice(0, 5).join(", ")}${dropped.length > 5 ? ", ..." : ""}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
mkdirSync(out, { recursive: true });
|
||||
writeFileSync(join(out, "catalog.json"), built.catalogBytes);
|
||||
writeFileSync(join(out, "vectors.json"), built.vectorsBytes);
|
||||
writeFileSync(join(out, "manifest.json"), manifestBytes(built.manifest));
|
||||
|
||||
console.log(`moves ${built.manifest.move_count}`);
|
||||
console.log(`model ${built.manifest.embedding_model}`);
|
||||
console.log(`revision ${built.manifest.source_revision}`);
|
||||
console.log(`version ${built.manifest.payload_sha256}`);
|
||||
console.log(`written ${out}`);
|
||||
}
|
||||
|
||||
await main();
|
||||
146
scripts/catalog/build-local-vectors.ts
Normal file
146
scripts/catalog/build-local-vectors.ts
Normal file
@ -0,0 +1,146 @@
|
||||
/**
|
||||
* Embed the catalog with the on-device model so local search can rank by meaning.
|
||||
*
|
||||
* A second vector set, not a replacement. The hosted vectors are 1536-dimension
|
||||
* and measured; these are 384-dimension and free, and the two are not
|
||||
* interchangeable because vectors from different models cannot be compared.
|
||||
*
|
||||
* Usage:
|
||||
* bun scripts/catalog/build-local-vectors.ts
|
||||
*
|
||||
* Defaults to reading `registry/` and writing `registry/catalog-artifact/`.
|
||||
*
|
||||
* Writes `local-vectors.bin` (float32, row-major, names in manifest order) and
|
||||
* `local-vectors.json` (names and dimensions). Splitting them keeps the payload small enough to ship: 424 moves at
|
||||
* 384 dimensions is about 650 KB as binary against several megabytes as JSON.
|
||||
*/
|
||||
|
||||
import { readFileSync, readdirSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import type { RegistryManifest } from "../../packages/core/src/index.js";
|
||||
|
||||
import {
|
||||
LOCAL_MODEL_DIMENSIONS,
|
||||
LOCAL_MODEL_ID,
|
||||
LOCAL_MODEL_REVISION,
|
||||
} from "../../packages/cli/src/registry/localModel.js";
|
||||
import { loadLocalEmbedder } from "../../packages/cli/src/registry/localEmbedder.js";
|
||||
import { isLocalModelReady } from "../../packages/cli/src/registry/localModel.js";
|
||||
import {
|
||||
catalogFromRegistry,
|
||||
LOCAL_VECTOR_BATCH_SIZE,
|
||||
localVectorRevision,
|
||||
} from "./catalog-artifact.js";
|
||||
|
||||
/** Distinct from 1 so the pre-commit hook can tell "cannot" from "failed". */
|
||||
const EXIT_NO_MODEL = 3;
|
||||
|
||||
function arg(name: string): string | undefined {
|
||||
const index = process.argv.indexOf(`--${name}`);
|
||||
return index === -1 ? undefined : process.argv[index + 1];
|
||||
}
|
||||
|
||||
/**
|
||||
* Embed in batches, in `names` order.
|
||||
*
|
||||
* Batch size is part of the artifact's identity, not a tuning knob: padding
|
||||
* within a batch changes the quantized result, so re-embedding the same text
|
||||
* at a different batch size does not reproduce the shipped rows.
|
||||
*/
|
||||
async function embedInBatches(
|
||||
names: string[],
|
||||
catalog: Record<string, string>,
|
||||
embedder: Awaited<ReturnType<typeof loadLocalEmbedder>>,
|
||||
): Promise<number[][]> {
|
||||
const vectors: number[][] = [];
|
||||
for (let start = 0; start < names.length; start += LOCAL_VECTOR_BATCH_SIZE) {
|
||||
const slice = names.slice(start, start + LOCAL_VECTOR_BATCH_SIZE);
|
||||
// Passages carry no query instruction; only queries do.
|
||||
vectors.push(...(await embedder.embed(slice.map((name) => catalog[name] as string))));
|
||||
process.stdout.write(
|
||||
`\r embedded ${Math.min(start + LOCAL_VECTOR_BATCH_SIZE, names.length)}/${names.length}`,
|
||||
);
|
||||
}
|
||||
process.stdout.write("\n");
|
||||
return vectors;
|
||||
}
|
||||
|
||||
/** Row-major Float32 payload. Row N belongs to `names[N]`, with no header. */
|
||||
function packVectors(names: string[], vectors: number[][]): Float32Array {
|
||||
const flat = new Float32Array(names.length * LOCAL_MODEL_DIMENSIONS);
|
||||
vectors.forEach((vector, row) => {
|
||||
if (vector.length !== LOCAL_MODEL_DIMENSIONS) {
|
||||
throw new Error(`vector for ${names[row]} has ${vector.length} dimensions`);
|
||||
}
|
||||
flat.set(vector, row * LOCAL_MODEL_DIMENSIONS);
|
||||
});
|
||||
return flat;
|
||||
}
|
||||
|
||||
// fallow-ignore-next-line complexity
|
||||
async function main(): Promise<void> {
|
||||
const dir = arg("artifact") ?? "registry/catalog-artifact";
|
||||
const registryDir = arg("registry") ?? "registry";
|
||||
|
||||
// Read the corpus straight from the registry rather than from a catalog.json
|
||||
// built by the hosted-tier script. That file is not in the repo, so the
|
||||
// documented regeneration command used to fail on a missing path, which is
|
||||
// the whole reason the index was allowed to drift.
|
||||
const catalogMap = catalogFromRegistry(
|
||||
registryDir,
|
||||
(path) => readFileSync(path, "utf-8"),
|
||||
(path) =>
|
||||
readdirSync(path, { withFileTypes: true })
|
||||
.filter((e) => e.isDirectory())
|
||||
.map((e) => e.name),
|
||||
);
|
||||
const catalog = Object.fromEntries(catalogMap);
|
||||
const names = Object.keys(catalog).sort();
|
||||
if (names.length === 0) throw new Error(`no registry items found under ${registryDir}`);
|
||||
const revision = localVectorRevision(
|
||||
LOCAL_MODEL_ID,
|
||||
LOCAL_MODEL_REVISION,
|
||||
LOCAL_MODEL_DIMENSIONS,
|
||||
catalogMap,
|
||||
);
|
||||
// Embedding needs the model, and the model is a 32 MB opt-in that most
|
||||
// contributors will not have. Say so and stop, rather than failing inside the
|
||||
// ONNX loader with an ENOENT that names a path nobody set.
|
||||
if (!isLocalModelReady()) {
|
||||
console.error(
|
||||
"The embedding model is not on this machine, so the index cannot be rebuilt here.\n" +
|
||||
"That is fine: adding a registry item does not require it. Open the pull request\n" +
|
||||
"and a maintainer regenerates the index before merge.\n\n" +
|
||||
"To do it yourself, fetch the model once with:\n" +
|
||||
" hyperframes catalog --query anything --on-device\n",
|
||||
);
|
||||
process.exit(EXIT_NO_MODEL);
|
||||
}
|
||||
|
||||
const embedder = await loadLocalEmbedder();
|
||||
|
||||
const vectors = await embedInBatches(names, catalog, embedder);
|
||||
const flat = packVectors(names, vectors);
|
||||
|
||||
writeFileSync(join(dir, "local-vectors.bin"), Buffer.from(flat.buffer));
|
||||
writeFileSync(
|
||||
join(dir, "local-vectors.json"),
|
||||
`${JSON.stringify({ model: LOCAL_MODEL_ID, modelRevision: LOCAL_MODEL_REVISION, dimensions: LOCAL_MODEL_DIMENSIONS, revision, names }, null, 2)}\n`,
|
||||
);
|
||||
const registryPath = join(registryDir, "registry.json");
|
||||
const registry = JSON.parse(readFileSync(registryPath, "utf-8")) as RegistryManifest;
|
||||
writeFileSync(
|
||||
registryPath,
|
||||
`${JSON.stringify({ ...registry, catalogArtifact: { revision } }, null, 2)}\n`,
|
||||
);
|
||||
|
||||
const megabytes = (flat.byteLength / 1024 / 1024).toFixed(2);
|
||||
console.log(`moves ${names.length}`);
|
||||
console.log(`model ${LOCAL_MODEL_ID}`);
|
||||
console.log(`dimensions ${LOCAL_MODEL_DIMENSIONS}`);
|
||||
console.log(`revision ${revision}`);
|
||||
console.log(`payload ${megabytes} MB`);
|
||||
console.log(`written ${dir}`);
|
||||
}
|
||||
|
||||
await main();
|
||||
292
scripts/catalog/catalog-artifact.test.ts
Normal file
292
scripts/catalog/catalog-artifact.test.ts
Normal file
@ -0,0 +1,292 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
import {
|
||||
buildArtifact,
|
||||
LOCAL_VECTOR_BATCH_SIZE,
|
||||
localVectorRevision,
|
||||
MANIFEST_SCHEMA_VERSION,
|
||||
movesMissingFromRegistry,
|
||||
parseShelf,
|
||||
payloadDigest,
|
||||
sha256Hex,
|
||||
sortedNames,
|
||||
verifyArtifact,
|
||||
type Embedder,
|
||||
} from "./catalog-artifact.js";
|
||||
|
||||
const REVISION = "00dda8f35dac0b89defe438d465cfe3d9d941b4f";
|
||||
const MODEL = "text-embedding-3-small";
|
||||
|
||||
const SHELF = `# Video primitive shelf
|
||||
|
||||
Preamble that is not an entry.
|
||||
|
||||
### beta-move
|
||||
|
||||
group: Annotation.
|
||||
what: Second alphabetically, first in the document.
|
||||
use_when: A callout needs to land.
|
||||
avoid_when: The shot is already busy.
|
||||
notes: Not a retrieval field.
|
||||
|
||||
### alpha-move
|
||||
|
||||
group: Motion.
|
||||
what: First alphabetically, second in the document.
|
||||
use_when: Pace needs to lift.
|
||||
avoid_when: Motion would distract.
|
||||
`;
|
||||
|
||||
/** Deterministic stand-in so no test makes a paid call. */
|
||||
const fakeEmbed =
|
||||
(dimension = 4): Embedder =>
|
||||
async (texts) =>
|
||||
texts.map((text, index) => Array.from({ length: dimension }, (_, i) => (index + 1) / (i + 2)));
|
||||
|
||||
async function build(overrides: Partial<Parameters<typeof buildArtifact>[0]> = {}) {
|
||||
return buildArtifact({
|
||||
shelfText: SHELF,
|
||||
sourceRevision: REVISION,
|
||||
embeddingModel: MODEL,
|
||||
embed: fakeEmbed(),
|
||||
...overrides,
|
||||
});
|
||||
}
|
||||
|
||||
describe("shelf contract", () => {
|
||||
it("keys entries by move name and ignores the preamble", () => {
|
||||
const entries = parseShelf(SHELF);
|
||||
expect([...entries.keys()].sort()).toEqual(["alpha-move", "beta-move"]);
|
||||
expect(JSON.stringify([...entries.values()])).not.toContain("Preamble");
|
||||
});
|
||||
|
||||
it("excludes the move name from its own retrieval text", () => {
|
||||
// Including the name would let a move win on its own label, which is what
|
||||
// lexical ranking already does.
|
||||
for (const [name, text] of parseShelf(SHELF)) expect(text).not.toContain(name);
|
||||
});
|
||||
|
||||
it("keeps only the four evaluated fields", () => {
|
||||
expect(parseShelf(SHELF).get("beta-move")?.split("\n")).toEqual([
|
||||
"group: Annotation.",
|
||||
"what: Second alphabetically, first in the document.",
|
||||
"use_when: A callout needs to land.",
|
||||
"avoid_when: The shot is already busy.",
|
||||
]);
|
||||
});
|
||||
|
||||
it("orders embedding by name, not document order", () => {
|
||||
expect(sortedNames(parseShelf(SHELF))).toEqual(["alpha-move", "beta-move"]);
|
||||
});
|
||||
|
||||
it("rejects a duplicate move name", () => {
|
||||
expect(() => parseShelf(`${SHELF}\n### beta-move\n\ngroup: Repeat.\n`)).toThrow(
|
||||
/Duplicate move name/,
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects a shelf with no entries", () => {
|
||||
expect(() => parseShelf("# Heading only\n\nProse.\n")).toThrow(/no entries/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("build contract", () => {
|
||||
it("publishes descriptions and vectors covering the same moves", async () => {
|
||||
const { catalogBytes, vectorsBytes, manifest } = await build();
|
||||
const catalog = JSON.parse(catalogBytes.toString("utf-8"));
|
||||
const vectors = JSON.parse(vectorsBytes.toString("utf-8"));
|
||||
expect(Object.keys(catalog).sort()).toEqual(Object.keys(vectors).sort());
|
||||
expect(manifest.move_count).toBe(2);
|
||||
expect(manifest.embedding_model).toBe(MODEL);
|
||||
expect(manifest.schema_version).toBe(MANIFEST_SCHEMA_VERSION);
|
||||
});
|
||||
|
||||
it("rebuilds byte-identically from an unchanged shelf", async () => {
|
||||
const first = await build();
|
||||
const second = await build();
|
||||
expect(first.catalogBytes.equals(second.catalogBytes)).toBe(true);
|
||||
expect(first.vectorsBytes.equals(second.vectorsBytes)).toBe(true);
|
||||
expect(first.manifest.payload_sha256).toBe(second.manifest.payload_sha256);
|
||||
});
|
||||
|
||||
it("refuses a branch name in place of a resolved commit", async () => {
|
||||
await expect(build({ sourceRevision: "feat-video-primitives" })).rejects.toThrow(
|
||||
/resolved commit SHA/,
|
||||
);
|
||||
});
|
||||
|
||||
it("fails the build when the embedder returns the wrong count", async () => {
|
||||
const short: Embedder = async () => [[0.1, 0.2, 0.3, 0.4]];
|
||||
await expect(build({ embed: short })).rejects.toThrow(/Expected 2 vectors/);
|
||||
});
|
||||
|
||||
it("fails the build when the embedder throws, publishing nothing", async () => {
|
||||
const failing: Embedder = async () => {
|
||||
throw new Error("provider unavailable");
|
||||
};
|
||||
await expect(build({ embed: failing })).rejects.toThrow(/provider unavailable/);
|
||||
});
|
||||
|
||||
it("fails the build on an unexpected dimension", async () => {
|
||||
await expect(build({ expectedDimension: 1536 })).rejects.toThrow(/dimension 4, expected 1536/);
|
||||
});
|
||||
|
||||
it("fails the build on a non-finite value", async () => {
|
||||
const bad: Embedder = async (texts) => texts.map(() => [Number.NaN, 1, 1, 1]);
|
||||
await expect(build({ embed: bad })).rejects.toThrow(/non-finite/);
|
||||
});
|
||||
|
||||
it("fails the build on a zero vector", async () => {
|
||||
const zero: Embedder = async (texts) => texts.map(() => [0, 0, 0, 0]);
|
||||
await expect(build({ embed: zero })).rejects.toThrow(/zero vector/);
|
||||
});
|
||||
|
||||
it("fails the build on inconsistent vector widths", async () => {
|
||||
const ragged: Embedder = async (texts) =>
|
||||
texts.map((_, i) => (i === 0 ? [1, 1, 1, 1] : [1, 1]));
|
||||
await expect(build({ embed: ragged })).rejects.toThrow(/expected 4/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("local vector revision", () => {
|
||||
it("is stable across filesystem traversal order", () => {
|
||||
const first = new Map([
|
||||
["whip-pan", "Fast transition"],
|
||||
["count-up", "Animated number"],
|
||||
]);
|
||||
const reversed = new Map([...first].reverse());
|
||||
|
||||
expect(localVectorRevision("model", "revision", 384, first)).toBe(
|
||||
localVectorRevision("model", "revision", 384, reversed),
|
||||
);
|
||||
});
|
||||
|
||||
it("uses locale-independent code-unit ordering", () => {
|
||||
const corpus = new Map([
|
||||
["ä-item", "Third"],
|
||||
["a-item", "First"],
|
||||
["z-item", "Second"],
|
||||
]);
|
||||
const expectedRows = [
|
||||
["a-item", "First"],
|
||||
["z-item", "Second"],
|
||||
["ä-item", "Third"],
|
||||
];
|
||||
|
||||
expect(localVectorRevision("model", "revision", 384, corpus)).toBe(
|
||||
sha256Hex(
|
||||
JSON.stringify({
|
||||
model: "model",
|
||||
modelRevision: "revision",
|
||||
dimensions: 384,
|
||||
batchSize: LOCAL_VECTOR_BATCH_SIZE,
|
||||
rows: expectedRows,
|
||||
}),
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
it("changes when searchable text changes under the same name", () => {
|
||||
const before = new Map([["whip-pan", "Fast transition"]]);
|
||||
const after = new Map([["whip-pan", "Fast energetic transition"]]);
|
||||
|
||||
expect(localVectorRevision("model", "revision", 384, before)).not.toBe(
|
||||
localVectorRevision("model", "revision", 384, after),
|
||||
);
|
||||
});
|
||||
|
||||
it("changes when the embedding contract changes", () => {
|
||||
const corpus = new Map([["whip-pan", "Fast transition"]]);
|
||||
|
||||
expect(localVectorRevision("model-a", "revision", 384, corpus)).not.toBe(
|
||||
localVectorRevision("model-b", "revision", 384, corpus),
|
||||
);
|
||||
expect(localVectorRevision("model-a", "revision", 384, corpus)).not.toBe(
|
||||
localVectorRevision("model-a", "revision", 768, corpus),
|
||||
);
|
||||
expect(localVectorRevision("model-a", "revision-a", 384, corpus)).not.toBe(
|
||||
localVectorRevision("model-a", "revision-b", 384, corpus),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("verify contract", () => {
|
||||
it("accepts a freshly built artifact", async () => {
|
||||
const built = await build();
|
||||
expect(() => verifyArtifact(built)).not.toThrow();
|
||||
});
|
||||
|
||||
it("rejects a mutated description", async () => {
|
||||
const built = await build();
|
||||
const mutated = Buffer.from(
|
||||
built.catalogBytes.toString("utf-8").replace("Motion.", "Tampered."),
|
||||
);
|
||||
expect(() => verifyArtifact({ ...built, catalogBytes: mutated })).toThrow(
|
||||
/does not match the manifest/,
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects a mutated vector", async () => {
|
||||
const built = await build();
|
||||
const mutated = Buffer.from(built.vectorsBytes.toString("utf-8").replace("0.5", "0.6"));
|
||||
expect(() => verifyArtifact({ ...built, vectorsBytes: mutated })).toThrow(
|
||||
/does not match the manifest/,
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects a manifest whose count disagrees with the payload", async () => {
|
||||
const built = await build();
|
||||
expect(() =>
|
||||
verifyArtifact({ ...built, manifest: { ...built.manifest, move_count: 99 } }),
|
||||
).toThrow(/declares 99/);
|
||||
});
|
||||
|
||||
it("rejects an unknown schema version", async () => {
|
||||
const built = await build();
|
||||
expect(() =>
|
||||
verifyArtifact({ ...built, manifest: { ...built.manifest, schema_version: 99 } }),
|
||||
).toThrow(/schema version/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("consumer parity", () => {
|
||||
it("computes the digest the Python consumer computes", () => {
|
||||
// Pinned from the Python consumer for the same inputs. Both sides hash file
|
||||
// bytes as sha256(sha256(catalog) || sha256(vectors)), so a change to either
|
||||
// implementation fails here rather than producing an artifact the consumer
|
||||
// silently rejects at load time.
|
||||
const digest = payloadDigest(
|
||||
Buffer.from("catalog-bytes", "utf-8"),
|
||||
Buffer.from("vector-bytes", "utf-8"),
|
||||
);
|
||||
expect(digest).toBe("291c30dddc35c705a681081d9c2e2a96314bcccf0d024b77a3692e412adbf103");
|
||||
});
|
||||
|
||||
it("reproduces the evaluated shelf when one is available", () => {
|
||||
// Opt-in: needs the pinned shelf. Guards the ported contract against drift.
|
||||
const shelfPath = process.env.EVAL_SHELF_PATH;
|
||||
if (!shelfPath) return;
|
||||
const entries = parseShelf(readFileSync(join(shelfPath), "utf-8"));
|
||||
expect(entries.size).toBe(424);
|
||||
});
|
||||
});
|
||||
|
||||
describe("movesMissingFromRegistry", () => {
|
||||
it("names the shelf moves no registry item can serve", () => {
|
||||
// The failure this guards produced no error at all: the artifact shipped,
|
||||
// ranked these moves highest, and the client then had nothing to show.
|
||||
expect(movesMissingFromRegistry(["count-up", "number-wheel"], ["count-up"])).toEqual([
|
||||
"number-wheel",
|
||||
]);
|
||||
});
|
||||
|
||||
it("is empty when the registry covers the shelf", () => {
|
||||
expect(movesMissingFromRegistry(["count-up"], ["count-up", "badge-pop"])).toEqual([]);
|
||||
});
|
||||
|
||||
it("reports every move when the registry is empty", () => {
|
||||
expect(movesMissingFromRegistry(["a", "b"], [])).toEqual(["a", "b"]);
|
||||
});
|
||||
});
|
||||
307
scripts/catalog/catalog-artifact.ts
Normal file
307
scripts/catalog/catalog-artifact.ts
Normal file
@ -0,0 +1,307 @@
|
||||
/**
|
||||
* Build the published video-primitive catalog artifact.
|
||||
*
|
||||
* HyperFrames owns the shelf, so HyperFrames owns this job. Descriptions and
|
||||
* their vectors publish together as one version, because a consumer that loads
|
||||
* a new description against an old vector produces a ranking that is wrong in a
|
||||
* way nothing alarms on.
|
||||
*
|
||||
* The retrieval-text contract is inherited from the evaluation and must not
|
||||
* drift: an entry starts at a `### ` heading, only `group`, `what`, `use_when`
|
||||
* and `avoid_when` form the body, the move name is deliberately excluded, and
|
||||
* names sort before embedding. Excluding the name keeps a move from winning on
|
||||
* its own label, which is what lexical ranking already does.
|
||||
*/
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
export const RETRIEVAL_FIELDS = ["group", "what", "use_when", "avoid_when"] as const;
|
||||
export const MANIFEST_SCHEMA_VERSION = 1;
|
||||
/** Padding changes quantized embeddings, so batch size is part of vector identity. */
|
||||
export const LOCAL_VECTOR_BATCH_SIZE = 16;
|
||||
|
||||
export interface CatalogManifest {
|
||||
schema_version: number;
|
||||
source_revision: string;
|
||||
shelf_sha256: string;
|
||||
embedding_model: string;
|
||||
move_count: number;
|
||||
payload_sha256: string;
|
||||
}
|
||||
|
||||
export interface BuiltArtifact {
|
||||
manifest: CatalogManifest;
|
||||
catalogBytes: Buffer;
|
||||
vectorsBytes: Buffer;
|
||||
}
|
||||
|
||||
/** Embed texts in the order given. Injected so tests never make a paid call. */
|
||||
export type Embedder = (texts: string[]) => Promise<number[][]>;
|
||||
|
||||
// tolerant parser for a hand-edited file
|
||||
// fallow-ignore-next-line complexity
|
||||
export function parseShelf(text: string): Map<string, string> {
|
||||
const entries = new Map<string, string>();
|
||||
const blocks = text.split(/^### /m).slice(1);
|
||||
for (const block of blocks) {
|
||||
const lines = block.split("\n");
|
||||
const name = (lines[0] ?? "").trim();
|
||||
if (!name) continue;
|
||||
if (entries.has(name)) throw new Error(`Duplicate move name in shelf: ${name}`);
|
||||
const body = lines
|
||||
.slice(1)
|
||||
.filter((line) => (RETRIEVAL_FIELDS as readonly string[]).includes(line.split(":")[0] ?? ""));
|
||||
entries.set(name, body.join("\n"));
|
||||
}
|
||||
if (entries.size === 0) throw new Error("Shelf contains no entries");
|
||||
return entries;
|
||||
}
|
||||
|
||||
/** Embedding order. Vector N corresponds to element N of this list. */
|
||||
/**
|
||||
* One registry item's retrieval text.
|
||||
*
|
||||
* Title, description and tags only. The name is deliberately excluded: it is
|
||||
* what the query is trying to find, and folding it into the text being matched
|
||||
* rewards items whose name happens to echo the query wording rather than items
|
||||
* that do what was asked.
|
||||
*/
|
||||
export function itemRetrievalText(item: {
|
||||
title?: string;
|
||||
description?: string;
|
||||
tags?: readonly string[];
|
||||
}): string {
|
||||
const parts = [item.title ?? "", item.description ?? "", (item.tags ?? []).join(" ")];
|
||||
return parts
|
||||
.map((part) => part.trim())
|
||||
.filter(Boolean)
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Read every installable item into the same shape parseShelf produces.
|
||||
*
|
||||
* Items with no usable text are skipped rather than embedded empty: an
|
||||
* all-zero-signal entry still occupies a slot in every ranking.
|
||||
*/
|
||||
// walks the registry and skips several kinds of item, each for a different reason
|
||||
// fallow-ignore-next-line complexity
|
||||
export function catalogFromRegistry(
|
||||
registryDir: string,
|
||||
read: (path: string) => string,
|
||||
listDirs: (path: string) => string[],
|
||||
): Map<string, string> {
|
||||
const catalog = new Map<string, string>();
|
||||
for (const type of ["blocks", "components"]) {
|
||||
let names: string[];
|
||||
try {
|
||||
names = listDirs(`${registryDir}/${type}`);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const name of names.sort()) {
|
||||
let item: { title?: string; description?: string; tags?: string[] };
|
||||
try {
|
||||
item = JSON.parse(read(`${registryDir}/${type}/${name}/registry-item.json`)) as typeof item;
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
const text = itemRetrievalText(item);
|
||||
if (text) catalog.set(name, text);
|
||||
}
|
||||
}
|
||||
return catalog;
|
||||
}
|
||||
|
||||
export function sortedNames(entries: Map<string, string>): string[] {
|
||||
return [...entries.keys()].sort();
|
||||
}
|
||||
|
||||
export function sha256Hex(data: Buffer | string): string {
|
||||
return createHash("sha256").update(data).digest("hex");
|
||||
}
|
||||
|
||||
/**
|
||||
* Identity of the searchable corpus and the model contract that embedded it.
|
||||
*
|
||||
* Sorted entries make filesystem traversal order irrelevant. The text stays
|
||||
* in the digest, so changing a title, description, or tag changes the revision
|
||||
* even when every catalog name remains the same.
|
||||
*/
|
||||
export function localVectorRevision(
|
||||
model: string,
|
||||
modelRevision: string,
|
||||
dimensions: number,
|
||||
entries: ReadonlyMap<string, string>,
|
||||
): string {
|
||||
const rows = [...entries.entries()].sort(([left], [right]) =>
|
||||
left < right ? -1 : left > right ? 1 : 0,
|
||||
);
|
||||
return sha256Hex(
|
||||
JSON.stringify({
|
||||
model,
|
||||
modelRevision,
|
||||
dimensions,
|
||||
batchSize: LOCAL_VECTOR_BATCH_SIZE,
|
||||
rows,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Digest the two published files as bytes.
|
||||
*
|
||||
* Deliberately over bytes rather than re-serialized structures. The consumer is
|
||||
* Python and this producer is TypeScript, and the two disagree on JSON number
|
||||
* formatting: a whole-valued float serializes as `1.0` in one and `1` in the
|
||||
* other. Hashing bytes removes the canonicalization question rather than
|
||||
* documenting it. The consumer computes this identically.
|
||||
*/
|
||||
export function payloadDigest(catalogBytes: Buffer, vectorsBytes: Buffer): string {
|
||||
const digest = createHash("sha256");
|
||||
digest.update(createHash("sha256").update(catalogBytes).digest());
|
||||
digest.update(createHash("sha256").update(vectorsBytes).digest());
|
||||
return digest.digest("hex");
|
||||
}
|
||||
|
||||
/** Stable serialization so an unchanged shelf rebuilds byte-identically. */
|
||||
function serialize(record: Record<string, unknown>): Buffer {
|
||||
const sorted: Record<string, unknown> = {};
|
||||
for (const key of Object.keys(record).sort()) sorted[key] = record[key];
|
||||
return Buffer.from(`${JSON.stringify(sorted, null, 2)}\n`, "utf-8");
|
||||
}
|
||||
|
||||
// assembles one artifact from several optional inputs; each branch is an input that may be absent
|
||||
// fallow-ignore-next-line complexity
|
||||
export async function buildArtifact(options: {
|
||||
shelfText: string;
|
||||
sourceRevision: string;
|
||||
embeddingModel: string;
|
||||
embed: Embedder;
|
||||
expectedDimension?: number;
|
||||
/**
|
||||
* Names the registry can serve. When given, shelf moves absent from it are
|
||||
* left out of the artifact: a move that ranks and cannot be installed is
|
||||
* worse than one that never appears, because it occupies a top slot.
|
||||
*/
|
||||
installableNames?: readonly string[];
|
||||
}): Promise<BuiltArtifact> {
|
||||
const { shelfText, sourceRevision, embeddingModel, embed, expectedDimension } = options;
|
||||
if (!/^[0-9a-f]{40}$/i.test(sourceRevision)) {
|
||||
throw new Error(`source_revision must be a resolved commit SHA, got ${sourceRevision}`);
|
||||
}
|
||||
|
||||
const entries = parseShelf(shelfText);
|
||||
if (options.installableNames) {
|
||||
for (const name of movesMissingFromRegistry([...entries.keys()], options.installableNames)) {
|
||||
entries.delete(name);
|
||||
}
|
||||
}
|
||||
const names = sortedNames(entries);
|
||||
const vectors = await embed(names.map((name) => entries.get(name) as string));
|
||||
|
||||
// Every validation runs before anything is written. A build that emits
|
||||
// descriptions and then fails to embed would publish exactly the half-artifact
|
||||
// the publish-together rule exists to prevent.
|
||||
if (vectors.length !== names.length) {
|
||||
throw new Error(`Expected ${names.length} vectors, embedder returned ${vectors.length}`);
|
||||
}
|
||||
const width = vectors[0]?.length ?? 0;
|
||||
if (width === 0) throw new Error("Embedder returned empty vectors");
|
||||
if (expectedDimension !== undefined && width !== expectedDimension) {
|
||||
throw new Error(`Vectors have dimension ${width}, expected ${expectedDimension}`);
|
||||
}
|
||||
vectors.forEach((vector, index) => {
|
||||
if (vector.length !== width) {
|
||||
throw new Error(
|
||||
`Vector for ${names[index]} has dimension ${vector.length}, expected ${width}`,
|
||||
);
|
||||
}
|
||||
if (!vector.every((value) => Number.isFinite(value))) {
|
||||
throw new Error(`Vector for ${names[index]} contains a non-finite value`);
|
||||
}
|
||||
if (!vector.some((value) => value !== 0)) {
|
||||
throw new Error(`Vector for ${names[index]} is a zero vector`);
|
||||
}
|
||||
});
|
||||
|
||||
const catalog: Record<string, string> = {};
|
||||
const vectorMap: Record<string, number[]> = {};
|
||||
names.forEach((name, index) => {
|
||||
catalog[name] = entries.get(name) as string;
|
||||
vectorMap[name] = vectors[index] as number[];
|
||||
});
|
||||
|
||||
const catalogBytes = serialize(catalog);
|
||||
const vectorsBytes = serialize(vectorMap);
|
||||
|
||||
return {
|
||||
catalogBytes,
|
||||
vectorsBytes,
|
||||
manifest: {
|
||||
schema_version: MANIFEST_SCHEMA_VERSION,
|
||||
source_revision: sourceRevision,
|
||||
shelf_sha256: sha256Hex(shelfText),
|
||||
embedding_model: embeddingModel,
|
||||
move_count: names.length,
|
||||
payload_sha256: payloadDigest(catalogBytes, vectorsBytes),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Shelf moves with no matching registry item.
|
||||
*
|
||||
* Ranking is published separately from the items themselves, so the two drift.
|
||||
* A move only present in the shelf ranks well and then cannot be shown or
|
||||
* installed, which reads to the user as a bad search rather than a bad build.
|
||||
*/
|
||||
export function movesMissingFromRegistry(
|
||||
shelfNames: readonly string[],
|
||||
registryNames: readonly string[],
|
||||
): string[] {
|
||||
const known = new Set(registryNames);
|
||||
return shelfNames.filter((name) => !known.has(name));
|
||||
}
|
||||
|
||||
export function manifestBytes(manifest: CatalogManifest): Buffer {
|
||||
return serialize(manifest as unknown as Record<string, unknown>);
|
||||
}
|
||||
|
||||
/** Recompute the published digest and compare it to what the manifest claims. */
|
||||
// one check per way an artifact can be wrong; collapsing them would lose which one failed
|
||||
// fallow-ignore-next-line complexity
|
||||
export function verifyArtifact(input: {
|
||||
manifest: CatalogManifest;
|
||||
catalogBytes: Buffer;
|
||||
vectorsBytes: Buffer;
|
||||
}): void {
|
||||
const { manifest, catalogBytes, vectorsBytes } = input;
|
||||
if (manifest.schema_version !== MANIFEST_SCHEMA_VERSION) {
|
||||
throw new Error(
|
||||
`Manifest schema version ${manifest.schema_version} is not ${MANIFEST_SCHEMA_VERSION}`,
|
||||
);
|
||||
}
|
||||
const catalog = JSON.parse(catalogBytes.toString("utf-8")) as Record<string, string>;
|
||||
const vectors = JSON.parse(vectorsBytes.toString("utf-8")) as Record<string, number[]>;
|
||||
|
||||
const names = Object.keys(catalog);
|
||||
if (names.length !== manifest.move_count) {
|
||||
throw new Error(
|
||||
`Artifact holds ${names.length} moves but the manifest declares ${manifest.move_count}`,
|
||||
);
|
||||
}
|
||||
const missing = names.filter((name) => !(name in vectors));
|
||||
if (missing.length > 0) {
|
||||
throw new Error(
|
||||
`${missing.length} moves have a description but no vector, first is ${missing[0]}`,
|
||||
);
|
||||
}
|
||||
|
||||
const actual = payloadDigest(catalogBytes, vectorsBytes);
|
||||
if (actual !== manifest.payload_sha256) {
|
||||
throw new Error(
|
||||
`Payload digest ${actual} does not match the manifest's ${manifest.payload_sha256}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
110
scripts/catalog/check-artifact-coverage.ts
Normal file
110
scripts/catalog/check-artifact-coverage.ts
Normal file
@ -0,0 +1,110 @@
|
||||
/**
|
||||
* Fails when the committed catalog vector artifact no longer represents the
|
||||
* registry it is meant to search.
|
||||
*
|
||||
* The pre-commit hook rebuilds when the opted-in model is available; CI verifies
|
||||
* freshness when it is not. Removing an item is self-healing at search time,
|
||||
* but every corpus change still invalidates the published revision so stale
|
||||
* vectors cannot pass the gate unnoticed.
|
||||
*
|
||||
* The corpus revision includes title, description, tags, model, and dimensions,
|
||||
* so same-name edits cannot hide behind a name-only coverage check. Computing
|
||||
* it needs no model and no network.
|
||||
*/
|
||||
import { readFileSync, readdirSync } from "node:fs";
|
||||
|
||||
import {
|
||||
LOCAL_MODEL_DIMENSIONS,
|
||||
LOCAL_MODEL_ID,
|
||||
LOCAL_MODEL_REVISION,
|
||||
} from "../../packages/cli/src/registry/localModel.js";
|
||||
import { catalogFromRegistry, localVectorRevision } from "./catalog-artifact.js";
|
||||
|
||||
type RegistryItem = { name: string; type?: string };
|
||||
type Registry = { items: RegistryItem[]; catalogArtifact?: { revision?: string } };
|
||||
type Artifact = { model?: string; dimensions?: number; revision?: string; names?: string[] };
|
||||
|
||||
const REGISTRY = "registry/registry.json";
|
||||
const ARTIFACT = "registry/catalog-artifact/local-vectors.json";
|
||||
|
||||
function read<T>(path: string): T {
|
||||
try {
|
||||
return JSON.parse(readFileSync(path, "utf8")) as T;
|
||||
} catch (error) {
|
||||
console.error(`Could not read ${path}: ${(error as Error).message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
const registry = read<Registry>(REGISTRY);
|
||||
const artifact = read<Artifact>(ARTIFACT);
|
||||
const corpus = catalogFromRegistry(
|
||||
"registry",
|
||||
(path) => readFileSync(path, "utf-8"),
|
||||
(path) =>
|
||||
readdirSync(path, { withFileTypes: true })
|
||||
.filter((entry) => entry.isDirectory())
|
||||
.map((entry) => entry.name),
|
||||
);
|
||||
|
||||
// Only blocks and components are searchable moves. Examples are starter
|
||||
// projects a user scaffolds, never something `catalog` ranks, so the artifact
|
||||
// deliberately carries no vector for them and demanding one would keep this
|
||||
// gate permanently red.
|
||||
const SEARCHABLE = new Set(["hyperframes:block", "hyperframes:component"]);
|
||||
const registryNames = new Set(
|
||||
registry.items.filter((i) => SEARCHABLE.has(i.type ?? "")).map((i) => i.name),
|
||||
);
|
||||
const artifactNames = new Set(artifact.names ?? []);
|
||||
|
||||
const unindexed = [...registryNames].filter((n) => !artifactNames.has(n)).sort();
|
||||
const dropped = [...artifactNames].filter((n) => !registryNames.has(n)).sort();
|
||||
const expectedRevision = localVectorRevision(
|
||||
LOCAL_MODEL_ID,
|
||||
LOCAL_MODEL_REVISION,
|
||||
LOCAL_MODEL_DIMENSIONS,
|
||||
corpus,
|
||||
);
|
||||
const artifactRevisionMatches = artifact.revision === expectedRevision;
|
||||
const registryRevisionMatches = registry.catalogArtifact?.revision === expectedRevision;
|
||||
|
||||
const show = (names: string[]) =>
|
||||
names
|
||||
.slice(0, 10)
|
||||
.map((n) => ` ${n}`)
|
||||
.join("\n") + (names.length > 10 ? `\n ... and ${names.length - 10} more` : "");
|
||||
|
||||
console.log(`registry: ${registryNames.size} searchable items (blocks + components)`);
|
||||
console.log(`artifact: ${artifactNames.size} vectors (${artifact.model ?? "unknown model"})`);
|
||||
|
||||
if (dropped.length > 0) {
|
||||
// Not fatal: the CLI filters these before a user ever sees them.
|
||||
console.log(`\nnote: ${dropped.length} vector(s) name items the registry no longer has.`);
|
||||
console.log(show(dropped));
|
||||
console.log(" These are filtered at search time, so they cost space, not correctness.");
|
||||
}
|
||||
|
||||
if (unindexed.length > 0) {
|
||||
console.error(`\n${unindexed.length} registry item(s) have no vector:`);
|
||||
console.error(show(unindexed));
|
||||
}
|
||||
|
||||
if (!artifactRevisionMatches || !registryRevisionMatches) {
|
||||
console.error("\nThe published vector revision does not match the searchable registry text.");
|
||||
console.error(` expected: ${expectedRevision}`);
|
||||
console.error(` artifact: ${artifact.revision ?? "missing"}`);
|
||||
console.error(` registry: ${registry.catalogArtifact?.revision ?? "missing"}`);
|
||||
}
|
||||
|
||||
if (unindexed.length > 0 || !artifactRevisionMatches || !registryRevisionMatches) {
|
||||
console.error(
|
||||
"\nMeaning search is stale. Word search still uses the live registry.\n\n" +
|
||||
"If you have the embedding model, regenerate and commit the artifact:\n" +
|
||||
" bun scripts/catalog/build-local-vectors.ts\n\n" +
|
||||
"If you do not, leave it: changing a registry item does not require the model,\n" +
|
||||
"and a maintainer regenerates the index before merge.\n",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log("\nEvery searchable field matches the published vector revision.");
|
||||
@ -26,7 +26,7 @@
|
||||
"files": 121
|
||||
},
|
||||
"hyperframes-cli": {
|
||||
"hash": "3eab4c5fc1ae8e26",
|
||||
"hash": "17e2300803951ed7",
|
||||
"files": 11
|
||||
},
|
||||
"hyperframes-core": {
|
||||
@ -42,7 +42,7 @@
|
||||
"files": 3
|
||||
},
|
||||
"hyperframes-registry": {
|
||||
"hash": "3a864992e765b9bf",
|
||||
"hash": "eca45f795c2ff093",
|
||||
"files": 10
|
||||
},
|
||||
"media-use": {
|
||||
|
||||
@ -16,13 +16,14 @@ Run commands as `npx hyperframes ...` unless project instructions provide a wrap
|
||||
## Development loop
|
||||
|
||||
1. **Scaffold:** `npx hyperframes init <project>` or capture a site. In non-TTY mode, pass `--non-interactive --example=<name>`.
|
||||
2. **Author:** write the composition using `/hyperframes-core`.
|
||||
3. **Get fast feedback while editing:** run `npx hyperframes lint` after the first HTML pass and after structural changes.
|
||||
4. **Run the final gate:** run `npx hyperframes check`; it reruns lint before opening the browser. Do not prepend a redundant standalone lint invocation. Add `--snapshots` for annotated overview frames and finding crops.
|
||||
5. **Inspect sub-compositions:** when `index.html` mounts `data-composition-src`, capture midpoint snapshots and inspect each mounted scene.
|
||||
6. **Open the final Studio preview:** run `npx hyperframes preview`, hand the timeline project URL to the user, and ask whether to revise or render.
|
||||
7. **Render only after approval:** use draft quality for iteration and high quality for delivery.
|
||||
8. **Verify the output:** confirm the file exists, is non-empty, and has a plausible duration.
|
||||
2. **Find the move:** before authoring motion by hand, search for a primitive that already does it: `npx hyperframes catalog --query "reveal a headline one line at a time"`. Ask for the effect you want rather than the mechanism you have in mind. Install with `npx hyperframes add <name>` (see `/hyperframes-registry`). Author by hand only once nothing fits.
|
||||
3. **Author:** write the composition using `/hyperframes-core`.
|
||||
4. **Get fast feedback while editing:** run `npx hyperframes lint` after the first HTML pass and after structural changes.
|
||||
5. **Run the final gate:** run `npx hyperframes check`; it reruns lint before opening the browser. Do not prepend a redundant standalone lint invocation. Add `--snapshots` for annotated overview frames and finding crops.
|
||||
6. **Inspect sub-compositions:** when `index.html` mounts `data-composition-src`, capture midpoint snapshots and inspect each mounted scene.
|
||||
7. **Open the final Studio preview:** run `npx hyperframes preview`, hand the timeline project URL to the user, and ask whether to revise or render.
|
||||
8. **Render only after approval:** use draft quality for iteration and high quality for delivery.
|
||||
9. **Verify the output:** confirm the file exists, is non-empty, and has a plausible duration.
|
||||
|
||||
```bash
|
||||
# Fast iteration check; repeat while authoring as needed.
|
||||
@ -61,6 +62,11 @@ Treat tiny unstyled content, canvas-sized icons, missing hero elements, or timel
|
||||
|
||||
## Agent conventions
|
||||
|
||||
- **Search the catalog before writing motion by hand.** `npx hyperframes catalog --query "<the beat, in plain language>"`. Search is entirely local: there is no hosted tier, no account, and the query text is never sent anywhere. By default it ranks on vocabulary shared with the item's name, title and description, which misses any phrasing that does not reuse the catalog's own wording. Add `--on-device` to rank by meaning instead (see the offline tier below).
|
||||
- **Read which tier answered; never infer it from results appearing.** With `--json` the envelope carries `query`, `tier` (`on-device` or `words`), `tier_detail`, `dropped`, `unindexed`, `shown`, `total` and `results`, plus `top_score` when the answering tier produces one and `warnings` when a tier was asked for and could not run. A weak result on `words` is expected; the same result on `on-device` is a bug. `top_score` is on-device only and has no threshold behind it: the ranker returns the whole catalog in some order for every query, so read it as evidence rather than as a pass or fail.
|
||||
- **`dropped` and `unindexed` are opposite skews between the registry and the on-device index, and rewording the query fixes neither.** `dropped` counts ranked names this registry cannot install, so the strongest matches are the ones being lost. `unindexed` counts registry moves the index cannot see at all, which no query can ever return. Refreshing the registry is not the answer to either: its manifest carries a 24h TTL and heals itself, while the vectors are a separately published artifact fetched into `~/.hyperframes/catalog/`. Re-running with `--on-device` refetches that index when `unindexed` is above zero, so that is the remedy to hand the user. A pure over-coverage skew (`dropped` above zero while `unindexed` is zero) does not trigger the refetch; clearing `~/.hyperframes/catalog/` is the only way out of that one. Both counts are of names rather than of results, so either can exceed `total`.
|
||||
- **Offer the offline tier; never enable it silently.** A one-time ~33 MB download (a quantized ONNX build of `bge-small-en-v1.5` plus its tokenizer, pinned to a fixed revision) and the catalog vectors from the registry, both cached under `~/.hyperframes/`, neither added to the project or any package. Once cached it ranks by meaning with nothing sent. Say the size out loud and let the person decide, then pass `--on-device` (with `-y` to skip the prompt) once they agree. The interactive offer only fires on a TTY, and under `--json` nothing about it is printed at all, so in an agent run you have to raise it with the user yourself.
|
||||
|
||||
- Prefer `--json` for agent and CI calls. Server-mode `render`, `preview`, and `play` do not provide ordinary JSON output; `preview --selection --json` and `preview --context --json` are query-mode exceptions.
|
||||
- `doctor --json` always exits zero. Gate on its payload:
|
||||
|
||||
|
||||
@ -85,7 +85,19 @@ See [wiring-components.md](./references/wiring-components.md) for full details.
|
||||
|
||||
## Discovery
|
||||
|
||||
Use the CLI as the primary discovery surface:
|
||||
Use the CLI as the primary discovery surface. **Search by intent before browsing:** the registry holds more items than you can scan by eye, so listing them and matching on names or tags is the slow path, and it fails whenever the author's wording differs from yours.
|
||||
|
||||
```bash
|
||||
# Rank the whole catalog against what the beat should do
|
||||
npx hyperframes catalog --query "reveal a headline one line at a time"
|
||||
npx hyperframes add caption-clip-wipe
|
||||
```
|
||||
|
||||
Search is local and sends nothing. By default it ranks on vocabulary shared with the item's name, title and description, so it only finds items that reuse your words; `--on-device` ranks by meaning instead, after a one-time model download. With `--json` the envelope names which tier answered, so check that rather than assuming a ranking happened.
|
||||
|
||||
Installability is applied after ranking, not before it: a name the vectors carry but this registry cannot serve is dropped from the results and counted in `dropped`, so a non-zero `dropped` means the two are different generations. See `/hyperframes-cli` for the offline tier, the consent gates, and how to refresh a stale index.
|
||||
|
||||
To browse or filter instead of search:
|
||||
|
||||
```bash
|
||||
npx hyperframes catalog
|
||||
|
||||
@ -132,8 +132,12 @@ git checkout -b feat/registry-{name}
|
||||
# 2. Format HTML
|
||||
npx oxfmt registry/{kind}/{name}/*.html
|
||||
|
||||
# 3. Update registry/registry.json — add entry to the "items" array:
|
||||
# { "name": "{name}", "type": "hyperframes:block" } (or "hyperframes:component")
|
||||
# 3. Regenerate registry/registry.json from the item directories.
|
||||
# Do not hand-edit it: an entry added by hand survives until the next
|
||||
# regeneration and then vanishes, and one left behind for a directory that
|
||||
# no longer exists is worse, because `hyperframes add <name>` resolves the
|
||||
# name and then fails on missing files.
|
||||
npx tsx scripts/generate-registry-items.ts
|
||||
|
||||
# 4. Generate catalog docs page
|
||||
npx tsx scripts/generate-catalog-pages.ts
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user