mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-08 10:46:06 +00:00
feat(cli): search the catalog by meaning, on this machine (#3089)
* feat(cli): search the catalog by meaning, in three named tiers Browsing the registry means matching names and tags, which fails whenever the author's wording differs from yours. "make the pace feel faster" finds nothing when the move is described as "velocity-driven blur". This ranks by meaning instead. Three tiers, and the command always says which one answered: words shared vocabulary, free, offline, no account on-device bge-small, free, offline, one opt-in download hosted Gemini, free for signed-in HeyGen users The tier is stated because a quietly worse answer looks exactly like a good one. --json carries it as a token alongside dropped, shown, total and top_score, so an agent reads provenance as data rather than matching English that is written to be reworded. Two consents, asked once each, and never conflated. Sending a query is a privacy question, so the prompt says the query is sent. Downloading a model is a disk and bandwidth question, so that prompt talks about size. Neither fires without a terminal: an unattended run sends nothing and downloads nothing unless a flag records that a person agreed. The catalog is derived from registry-item.json rather than from a separate document, so the set that is ranked and the set that can be installed are the same object by construction. Only the on-device vectors are committed; the hosted vectors are nine megabytes and belong on the server. top_score is reported and never acted on. A "nothing matched" threshold looked clean on long briefs and collapsed on the short queries people type: "a logo appears" scores 0.6181 and keyboard mash scores 0.6417, so any cut that catches the noise rejects the real query. The measurement is in the evals directory rather than in this branch. Not covered here. The published recall figures were measured against a separate hand-written document, not against registry text, so they should not be quoted for this catalog until re-measured. The offline tier needs a normal install: a single-file build cannot load the native ONNX runtime, which the command now reports instead of silently degrading. And the drop-detection path has never been observed firing outside its author's tests. * fix(cli): make this branch pass the repo's own gates Three things `bun run lint` and `fallow audit --base origin/main` rejected. CI runs both, so none of this branch would have gone green. Found by running them, not by reading the diff. process.exit in catalog.ts, twice: an invalid --type and a cancelled picker. check:cli-process-ownership reserves that for cli.ts, and the rule is not cosmetic — process.exit tears the process down where it stands, so anything cli.ts has queued to run on the way out is dropped. finishCommand throws a CliResultSignal that cli.ts turns into the exit code, which is what init.ts already does for a cancelled prompt. Three exports with no consumers. normalize keeps its body and loses its export; localEmbedder is the only caller. modelsDirectory goes entirely, having no caller inside its file or out. The WordPieceConfig re-export goes, and with it the import it existed to forward: the type is exported from wordpiece.ts, where its consumers already take it from. Complexity. prepareOnDeviceTier is lifted out of run(), which took run from 64 cyclomatic and CRAP 948 to 54 and 684. That block is one decision — can the offline tier run, and if not, why not — and its only product is a list of warnings, so it reads and tests as a unit, which it could not do inline. The rest is suppressed rather than refactored, each with its reason on the line above. Finishing run() means extracting its three output paths, and that is a refactor of a command this branch already changes for other reasons: a separate initiative, not something to absorb here. Every suppression says what shape the function has and why; a bare marker on a function nobody can justify is how a threshold stops meaning anything. Verified: `bun run lint` exits 0, fallow reports no issues across 27 changed files, and 2540 CLI tests pass. * feat(cli): ship the local search tiers only, drop the hosted one Search now has two tiers, both local: shared-vocabulary word matching, and the opt-in on-device model. The hosted tier, which sent the query to a HeyGen endpoint and ranked it with a hosted model, is removed. This is a scope decision, not a defect. The endpoint works and its own change is reviewed and green; it is simply not what we want to ship first. Landing local only means the feature has no backend dependency, no auth requirement, and nothing leaves the machine unless someone opts into downloading a model. Gone: registry/smartSearch.ts and its test, the --smart and --no-smart flags, the outcome plumbing through the command, the remote branch of applySearch, the remote tier, and the hosted-only JSON fields (ranking, catalog_version, top_score). Also the smartSearchEnabled consent field in telemetry config, which was the persisted storage behind the hosted consent and would otherwise have been left as dead configuration surface. Kept exactly as they were: both local tiers, the --on-device and --yes flags, the download consent prompt, and the runtime check that happens before the download rather than after it. The --json envelope still reports query, tier, tier_detail, shown, total, dropped, warnings and results, so an agent can still tell which tier answered and why. tierToken now distinguishes on-device from words. Verified: lint exits 0, fallow reports no issues, 2522 CLI tests pass, and the command was exercised directly. A query answers on the on-device tier where the model is installed and falls back to word matching where it is not, reporting that fallback in warnings rather than silently. An unknown --type still exits 1 with a readable message, and --smart is now rejected as an unknown flag. * fix(cli): count only moves this registry cannot install as dropped The dropped count was computed against the list left after the user's own --type and --tag filters, so every move the user excluded was reported as one the registry is missing. Filtering made the number go up: the same query reported 277 unfiltered and 302 with --type block. The count exists so a caller can tell "nothing matched your words" apart from "the ranker suggested things this project cannot install". Conflating it with user filtering destroys exactly that signal, and worse, genuine index skew and a self-inflicted filter printed a byte-identical line with opposite remedies -- one means refresh the shelf, the other means drop a flag, and refreshing does nothing. Now counted against the registry rather than the filtered view. The manifest is already fetched whole and narrowed in memory, so keeping the unnarrowed name set costs no extra request, and item loading still runs only on the filtered subset. Verified against ground truth rather than by eye: the vector artifact holds 411 names, the registry holds 168 installable items, and 134 of those names exist in both, so 277 are genuinely uninstallable. The count now reads 277 unfiltered, 277 under --type block, 277 under --type component and 277 under --tag, and the skew it reports is real -- the artifact predates dropping the UI primitives and still ranks moves that are no longer on the shelf. Reported by Vance Ingalls, who also noted this closes an item the status doc listed as unverified. Two earlier sweeps could not make the count fire because neither combined a filter with a query. Tests pin the three cases: a genuinely absent name counts, a filter-excluded name does not, and a fully installable ranking reports zero. * fix(cli): tell the user when meaning search cannot see the catalog The on-device index was fetched once and never revalidated: the only freshness check was two existsSync calls. A move added after that fetch was invisible to meaning search permanently, not down-ranked but absent from the candidate set. The registry manifest on the same command carries a 24h TTL, so the two halves of one feature disagreed about staleness. The dropped count reported over-coverage only, names the index has that the registry lacks. Under-coverage was never computed, so the harmless direction was instrumented and the costly one was silent. Reproduced with an index truncated to 120 of 168 moves: dropped read 0, perfect health, while 48 moves were unreachable. Counts under-coverage from the name list the artifact already carries, so no extra request. Warns only when non-zero, and names the remedy. The remedy had to be made true: --on-device could not refresh a stale index because hasLocalVectors short-circuited the fetch. That flag now refetches when the index is absent or no longer covering. Two defects the reproduction surfaced. A failed refresh reported the tier unavailable while the old vectors were still on disk and still ranking. And the fetch wrote its two files one at a time, so failing between them paired a new name list with an old matrix, a hard load error rather than stale data. It now writes both or neither, which matters more once refresh runs on staleness. top_score returns, scoped to the on-device tier and set to the score of the best result actually shown rather than the ranking head, which can describe a row the caller never received. Also: scripts/ is now typechecked. It never was, which is how a build script that crashes after the paid embedding call, and two scripts whose imports do not resolve at all, went unnoticed. 43 errors fixed, no suppressions. And the docs stop describing a --smart hosted tier that was deleted, an item that does not exist, and a registry refresh that cannot fix a stale vector index. * ci: fail when the search index stops covering the registry The catalog vector artifact is regenerated by hand. Nothing in CI, in package.json or in a hook rebuilds it, because embedding needs the 32 MB model. So adding a registry item silently makes it invisible to meaning search until someone remembers to regenerate. The failure is asymmetric, which is what makes it easy to miss. Removing an item is self-healing: the ranker still scores the dead vector, then filters the name before display, so a user is never offered something they cannot install. Adding one is not: the item is absent from the candidate set entirely, not ranked low. Comparing the two name lists needs neither the model nor a network call, so the gate runs in seconds. CI checks rather than fixes, for the same reason it cannot regenerate. Scoped to blocks and components. Examples are starter projects a user scaffolds, never something catalog ranks, and the artifact carries no vector for them, so demanding one would keep this gate permanently red and it would be ignored within a week. Verified in both directions rather than assumed: adding an unindexed item exits 1 and names it, restoring the registry exits 0. * fix(catalog): rebuild the search index from the registry build-local-vectors.ts read registry/catalog-artifact/catalog.json, a file no script in this repo writes and which is not committed, so the documented regeneration command failed on a missing path. That is why the index could drift from the registry with nothing to run to fix it. It now reads registry/blocks/* and registry/components/* through catalogFromRegistry, the existing helper that already produced the right shape but had no caller. Rebuilding reproduces the shipped 168 rows byte for byte. A lefthook catalog-index command regenerates and re-stages both artifact files whenever a staged registry-item.json changes, mirroring the skills-manifest pattern, so adding or removing an item keeps the index in sync without anyone remembering to. Verified end to end: staging a new item took the artifact 168 to 169 rows and staged it in 0.80s. * fix(cli): refuse a half-downloaded vector cache The two artifact files have to agree on how many rows there are, and until now nothing checked that before writing them. A truncated or wrong-model response landed in the cache and only failed at load, on every later search, until someone cleared it by hand. The pair is now checked first and refused as a unit, and the cache is created 0o700 with 0o600 files rather than inheriting the umask of a directory the caller may have pointed anywhere. Also lifts the capture setup the two preview generators had drifted into sharing into scripts/preview-capture.ts, and splits the vector builders batching and packing out of main. Both were findings the audit attributed to this branch. * fix(cli): keep the catalog vitest run with the tests it runs Restacking took the base package.json wholesale, which dropped the vitest dependency and the scripts/catalog run this PR adds. Both belong here rather than under it. * fix(cli): stop the declined model download from happening anyway Answering no to the on-device download offer recorded no and warned, then carried on. The guard below it is localModelConsent() !== false, which the decline had just made false, so it was skipped rather than taken: control reached recordLocalModelConsent(true), overwrote the answer with yes, and fetched the 32 MB model the user had refused. Next run it never asked again. No test could catch it. The stub pinned localModelStatus to ready, so the prompt never fired, and recordLocalModelConsent was a no-op that recorded nothing. Two tests now cover the offer, and they need three things the old stubs did not model: the run has to look like a terminal, because off one the command treats --on-device as the consent and never asks; the ONNX probe has to answer true, or an accepted offer returns at the runtime guard before it can download; and the status has to follow the recorded answer, or the second offer later in the run fires as well. Removing the return makes the decline test fail. * fix(catalog): let someone without the model still add a component The pre-commit hook rebuilds the search index, and rebuilding needs the 32 MB embedding model. An outside contributor adding a registry item does not have it, so their commit died inside the ONNX loader on an ENOENT naming a path they never set, and the CI gate then told them to run the command that had just crashed. The model is an opt-in for search, not a build dependency, so nobody is charged for it to contribute. The builder checks first and explains itself, exiting 3 for cannot as distinct from 1 for failed. The hook treats 3 as skip and lets the commit through. The gate now names both paths: regenerate if you have the model, leave it if you do not and a maintainer will. Verified both ways: with no model the builder explains and the hook exits 0; with the model it still regenerates byte-identically. * docs: say that anyone can add a registry item, and stop hand-editing a generated file Two defects, one of them the reason 64 stale entries survived in registry.json. The checklist told contributors to add their item to registry/registry.json. That file is generated from the item directories, so an entry added by hand survives until the next regeneration and then vanishes, and one left behind for a directory that no longer exists is worse: hyperframes add resolves the name and then fails on missing files. Both CONTRIBUTING.md and the agent-facing skill reference now run the generator instead. Nothing said contribution was maintainer-only, but nothing said it was not either, and two steps do need assets an outside contributor has no reason to install. Those are now named in a table with what happens if you do not have them, matching how the preview image was already handled. The search index is the new one: the model behind it is a 32 MB opt-in for search, not a build dependency. * fix(cli): harden on-device catalog search * fix(cli): refresh stale catalog vectors * test: create catalog vector temp dirs securely
This commit is contained in:
Miguel Ángel
@@ -0,0 +1,432 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { countUnindexed, pickByName } from "./catalog.js";
|
||||
|
||||
/** The whole registry, which is what "in this registry" has to be measured against. */
|
||||
const registryNames = new Set(["fade-through", "whip-pan", "count-up"]);
|
||||
const item = (name: string): { name: string } => ({ name });
|
||||
|
||||
describe("pickByName", () => {
|
||||
it("counts only the ranked names this registry has no item for", () => {
|
||||
const { ranked, missing } = pickByName(
|
||||
[item("fade-through"), item("whip-pan"), item("count-up")],
|
||||
["whip-pan", "fade-through", "accordion", "alert-dialog"],
|
||||
registryNames,
|
||||
);
|
||||
|
||||
expect(ranked.map((entry) => entry.name)).toEqual(["whip-pan", "fade-through"]);
|
||||
// accordion and alert-dialog are in the ranking artifact and nowhere in the
|
||||
// registry: a real skew between two separately published generations.
|
||||
expect(missing).toBe(2);
|
||||
});
|
||||
|
||||
it("does not count moves the user's own filter removed", () => {
|
||||
// `items` is what survived --type/--tag; the registry still has the rest.
|
||||
const { ranked, missing } = pickByName(
|
||||
[item("fade-through")],
|
||||
["whip-pan", "fade-through", "count-up", "accordion"],
|
||||
registryNames,
|
||||
);
|
||||
|
||||
expect(ranked.map((entry) => entry.name)).toEqual(["fade-through"]);
|
||||
// whip-pan and count-up are installable, just filtered out. Only accordion
|
||||
// is genuinely absent, and filtering must not inflate that number.
|
||||
expect(missing).toBe(1);
|
||||
});
|
||||
|
||||
it("reports nothing missing when the whole ranking is installable", () => {
|
||||
const { missing } = pickByName(
|
||||
[item("fade-through")],
|
||||
["fade-through", "whip-pan"],
|
||||
registryNames,
|
||||
);
|
||||
|
||||
expect(missing).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("countUnindexed", () => {
|
||||
it("counts the registry moves the on-device index holds no vector for", () => {
|
||||
// The move published after the artifact was fetched. Meaning search cannot
|
||||
// rank it at all, which is the failure this number exists to expose.
|
||||
expect(countUnindexed(registryNames, ["fade-through"])).toBe(2);
|
||||
});
|
||||
|
||||
it("reports nothing when the index covers the registry", () => {
|
||||
expect(countUnindexed(registryNames, ["count-up", "whip-pan", "fade-through"])).toBe(0);
|
||||
});
|
||||
|
||||
it("does not let names the registry dropped paper over a gap", () => {
|
||||
// The artifact holds two names this registry cannot install and is missing
|
||||
// two it can. Comparing sizes rather than membership would call that even.
|
||||
expect(countUnindexed(registryNames, ["fade-through", "accordion", "alert-dialog"])).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
// ── The command envelope ────────────────────────────────────────────────────
|
||||
// The JSON envelope is the surface an agent reads, so under-coverage and the
|
||||
// score are pinned where they are actually published rather than only at the
|
||||
// helper that computes them.
|
||||
|
||||
const state = vi.hoisted(() => ({
|
||||
registry: [] as Array<{ name: string; type: string; tags?: string[] }>,
|
||||
artifactRevision: "revision-current",
|
||||
cachedVectorRevision: "revision-current",
|
||||
vectorFetches: 0,
|
||||
vectorFetchSucceeds: true,
|
||||
ranking: null as Array<{ name: string; score: number }> | null,
|
||||
rankingError: null as Error | null,
|
||||
indexed: [] as string[],
|
||||
// The consent path. Static stubs could not reach it: with the status pinned
|
||||
// to "ready" the prompt never fires, so the answer was never a variable and
|
||||
// the decline branch was never executed by any test.
|
||||
modelStatus: "ready" as "ready" | "not-asked" | "declined" | "unavailable",
|
||||
confirmAnswer: true as boolean,
|
||||
consentRecorded: [] as boolean[],
|
||||
downloads: 0,
|
||||
runtimeAvailable: true,
|
||||
}));
|
||||
|
||||
vi.mock("../registry/resolver.js", () => ({
|
||||
loadAllItems: async (entries: Array<{ name: string; type: string; tags?: string[] }>) =>
|
||||
entries.map((entry) => ({
|
||||
name: entry.name,
|
||||
type: entry.type,
|
||||
title: entry.name,
|
||||
description: `${entry.name} description`,
|
||||
tags: entry.tags ?? [],
|
||||
})),
|
||||
}));
|
||||
|
||||
vi.mock("../registry/remote.js", () => ({
|
||||
fetchRegistryManifest: async () => ({
|
||||
items: state.registry,
|
||||
catalogArtifact: { revision: state.artifactRevision },
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock("@clack/prompts", () => ({
|
||||
confirm: async () => state.confirmAnswer,
|
||||
isCancel: (value: unknown) => value === null,
|
||||
}));
|
||||
|
||||
vi.mock("../registry/localModel.js", () => ({
|
||||
// "ready" is a user who opted into the on-device tier at some point. Every
|
||||
// later search takes that tier with no flag, which is how a frozen artifact
|
||||
// goes on answering forever. "not-asked" is the first run, the one that asks.
|
||||
// Recording an answer is what stops the CLI asking again, so the stub has to
|
||||
// move with it. Pinned to "not-asked" the second offer later in the run also
|
||||
// fires, and the double prompt looks like a product bug rather than a stub
|
||||
// that does not model the contract.
|
||||
localModelStatus: () => {
|
||||
const answer = state.consentRecorded.at(-1);
|
||||
return {
|
||||
status: answer === false ? "declined" : answer === true ? "ready" : state.modelStatus,
|
||||
};
|
||||
},
|
||||
ensureLocalModel: async () => {
|
||||
state.downloads += 1;
|
||||
if (state.modelStatus === "unavailable") state.modelStatus = "ready";
|
||||
return true;
|
||||
},
|
||||
recordLocalModelConsent: (enabled: boolean) => {
|
||||
state.consentRecorded.push(enabled);
|
||||
},
|
||||
downloadOfferMessage: () => "offer",
|
||||
nonInteractiveConsentMessage: () => "consent",
|
||||
}));
|
||||
|
||||
vi.mock("../registry/localEmbedder.js", () => ({
|
||||
// The native runtime is present in these tests. Left unmocked it answers
|
||||
// false under vitest, and every accepted offer returns at the runtime guard
|
||||
// before it can download, which looks like the download being skipped.
|
||||
localRuntimeAvailable: async () => state.runtimeAvailable,
|
||||
}));
|
||||
|
||||
vi.mock("../registry/localSemantic.js", () => ({
|
||||
localSemanticRanking: async () => {
|
||||
if (state.rankingError) throw state.rankingError;
|
||||
return state.ranking;
|
||||
},
|
||||
localVectorNames: () => state.indexed,
|
||||
cachedLocalVectorRevision: () => state.cachedVectorRevision,
|
||||
hasLocalVectors: () => true,
|
||||
fetchLocalVectors: async (_registry: string, options: { expectedRevision?: string } = {}) => {
|
||||
state.vectorFetches += 1;
|
||||
if (state.vectorFetchSucceeds && options.expectedRevision !== undefined) {
|
||||
state.cachedVectorRevision = options.expectedRevision;
|
||||
}
|
||||
return state.vectorFetchSucceeds;
|
||||
},
|
||||
}));
|
||||
|
||||
const block = (name: string, tags?: string[]): { name: string; type: string; tags?: string[] } => ({
|
||||
name,
|
||||
type: "hyperframes:block",
|
||||
tags,
|
||||
});
|
||||
const component = (name: string): { name: string; type: string } => ({
|
||||
name,
|
||||
type: "hyperframes:component",
|
||||
});
|
||||
|
||||
interface Envelope {
|
||||
tier: string;
|
||||
dropped: number;
|
||||
unindexed: number;
|
||||
top_score?: number;
|
||||
shown: number;
|
||||
warnings?: string[];
|
||||
}
|
||||
|
||||
async function runCatalog(args: Record<string, unknown>): Promise<string> {
|
||||
const command = (await import("./catalog.js")).default as unknown as {
|
||||
run: (context: { args: Record<string, unknown> }) => Promise<void>;
|
||||
};
|
||||
const lines: string[] = [];
|
||||
const log = vi.spyOn(console, "log").mockImplementation((...parts: unknown[]) => {
|
||||
lines.push(parts.map(String).join(" "));
|
||||
});
|
||||
try {
|
||||
await command.run({ args });
|
||||
} finally {
|
||||
log.mockRestore();
|
||||
}
|
||||
// Colour is decoration; assertions are about the words. The escape byte is
|
||||
// built rather than written: as a literal or as \u001B it is a control
|
||||
// character in the source, which the lint rules reject either way.
|
||||
const ansi = new RegExp(`${String.fromCharCode(27)}\\[[0-9;]*m`, "g");
|
||||
return lines.join("\n").replace(ansi, "");
|
||||
}
|
||||
|
||||
async function runEnvelope(args: Record<string, unknown>): Promise<Envelope> {
|
||||
return JSON.parse(await runCatalog({ json: true, ...args })) as Envelope;
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
state.modelStatus = "ready";
|
||||
state.artifactRevision = "revision-current";
|
||||
state.cachedVectorRevision = "revision-current";
|
||||
state.vectorFetches = 0;
|
||||
state.vectorFetchSucceeds = true;
|
||||
state.rankingError = null;
|
||||
state.confirmAnswer = true;
|
||||
state.consentRecorded = [];
|
||||
state.downloads = 0;
|
||||
state.runtimeAvailable = true;
|
||||
state.registry = [block("count-up"), block("fade-through"), component("whip-pan")];
|
||||
state.indexed = ["count-up", "fade-through", "whip-pan"];
|
||||
state.ranking = [
|
||||
{ name: "count-up", score: 0.71 },
|
||||
{ name: "whip-pan", score: 0.42 },
|
||||
{ name: "fade-through", score: 0.31 },
|
||||
];
|
||||
});
|
||||
|
||||
describe("catalog --json meaning search", () => {
|
||||
it("reports the registry moves meaning search cannot see", async () => {
|
||||
// Published after the user's artifact was fetched: in the registry, absent
|
||||
// from the index, and therefore unreturnable by any query.
|
||||
state.registry.push(block("split-screen"));
|
||||
|
||||
const envelope = await runEnvelope({ query: "make a number count up" });
|
||||
|
||||
expect(envelope.tier).toBe("on-device");
|
||||
expect(envelope.unindexed).toBe(1);
|
||||
});
|
||||
|
||||
it("reports nothing unindexed when the artifact matches the registry", async () => {
|
||||
const envelope = await runEnvelope({ query: "make a number count up" });
|
||||
|
||||
expect(envelope.unindexed).toBe(0);
|
||||
});
|
||||
|
||||
it("measures unindexed against the unfiltered registry under --type", async () => {
|
||||
// The missing move is a component; the user asked for blocks. Their filter
|
||||
// is not the index being stale, and it must not hide a stale index either.
|
||||
state.registry.push(component("push-in"));
|
||||
|
||||
const envelope = await runEnvelope({ query: "make a number count up", type: "block" });
|
||||
|
||||
expect(envelope.shown).toBe(2);
|
||||
expect(envelope.unindexed).toBe(1);
|
||||
});
|
||||
|
||||
it("keeps dropped counted against the unfiltered registry under --type", async () => {
|
||||
// accordion is the only name the registry genuinely lacks. whip-pan is
|
||||
// installable and merely filtered out, so it is not a drop.
|
||||
state.ranking = [
|
||||
{ name: "count-up", score: 0.71 },
|
||||
{ name: "whip-pan", score: 0.62 },
|
||||
{ name: "accordion", score: 0.55 },
|
||||
{ name: "fade-through", score: 0.31 },
|
||||
];
|
||||
|
||||
const envelope = await runEnvelope({ query: "make a number count up", type: "block" });
|
||||
|
||||
expect(envelope.dropped).toBe(1);
|
||||
expect(envelope.shown).toBe(2);
|
||||
});
|
||||
|
||||
it("carries the score of the best result it actually showed", async () => {
|
||||
// The top-ranked name is not installable here, so reporting the ranking's
|
||||
// own head would describe a row the caller never received.
|
||||
state.ranking = [
|
||||
{ name: "accordion", score: 0.93 },
|
||||
{ name: "count-up", score: 0.71 },
|
||||
{ name: "whip-pan", score: 0.42 },
|
||||
{ name: "fade-through", score: 0.31 },
|
||||
];
|
||||
|
||||
const envelope = await runEnvelope({ query: "make a number count up" });
|
||||
|
||||
expect(envelope.top_score).toBeCloseTo(0.71);
|
||||
});
|
||||
|
||||
it("omits the score on the word tier, whose scale is not the same one", async () => {
|
||||
state.ranking = null;
|
||||
|
||||
const envelope = await runEnvelope({ query: "count up" });
|
||||
|
||||
expect(envelope.tier).toBe("words");
|
||||
expect(envelope.top_score).toBeUndefined();
|
||||
// Word matching ranks the live registry listing, so it is never stale.
|
||||
expect(envelope.unindexed).toBe(0);
|
||||
});
|
||||
|
||||
it("finds registry tags on the word tier", async () => {
|
||||
state.modelStatus = "declined";
|
||||
state.ranking = null;
|
||||
state.registry = [block("fade-through", ["transition"]), block("count-up", ["number"])];
|
||||
|
||||
const envelope = await runEnvelope({ query: "transition" });
|
||||
|
||||
expect(envelope.tier).toBe("words");
|
||||
expect(envelope.shown).toBe(1);
|
||||
});
|
||||
|
||||
it("carries an on-device runtime failure into the JSON envelope", async () => {
|
||||
state.rankingError = new Error("model could not load");
|
||||
|
||||
const envelope = await runEnvelope({ query: "count up" });
|
||||
|
||||
expect(envelope.tier).toBe("words");
|
||||
expect(envelope.warnings).toEqual(["on-device search did not run: model could not load"]);
|
||||
});
|
||||
|
||||
it("refreshes a changed vector revision under existing consent", async () => {
|
||||
state.cachedVectorRevision = "revision-previous";
|
||||
|
||||
const envelope = await runEnvelope({ query: "count up" });
|
||||
|
||||
expect(envelope.tier).toBe("on-device");
|
||||
expect(state.vectorFetches).toBe(1);
|
||||
expect(state.cachedVectorRevision).toBe("revision-current");
|
||||
expect(state.consentRecorded).toEqual([]);
|
||||
});
|
||||
|
||||
it("replaces a changed model revision under existing consent", async () => {
|
||||
state.modelStatus = "unavailable";
|
||||
|
||||
const envelope = await runEnvelope({ query: "count up" });
|
||||
|
||||
expect(envelope.tier).toBe("on-device");
|
||||
expect(state.downloads).toBe(1);
|
||||
expect(state.consentRecorded).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps the previous vectors and reports a failed routine refresh", async () => {
|
||||
state.cachedVectorRevision = "revision-previous";
|
||||
state.vectorFetchSucceeds = false;
|
||||
|
||||
const envelope = await runEnvelope({ query: "count up" });
|
||||
|
||||
expect(envelope.tier).toBe("on-device");
|
||||
expect(state.cachedVectorRevision).toBe("revision-previous");
|
||||
expect(envelope.warnings).toEqual([
|
||||
"on-device search is using the previous catalog vectors because the update failed",
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("catalog meaning search, on a terminal", () => {
|
||||
it("says how much is missing and what to run about it", async () => {
|
||||
state.registry.push(block("split-screen"));
|
||||
|
||||
const output = await runCatalog({ query: "make a number count up" });
|
||||
|
||||
expect(output).toContain("1 of 4 moves are missing from the on-device index");
|
||||
expect(output).toContain("Re-run with --on-device to refresh it.");
|
||||
});
|
||||
|
||||
it("says nothing when the index covers the registry", async () => {
|
||||
const output = await runCatalog({ query: "make a number count up" });
|
||||
|
||||
expect(output).not.toContain("missing from the on-device index");
|
||||
});
|
||||
});
|
||||
|
||||
describe("the on-device download offer", () => {
|
||||
// The offer only exists for someone who can answer it. Off a terminal the
|
||||
// caller must add --yes explicitly, so a test that forgets the terminal
|
||||
// never reaches the prompt and passes for the wrong reason.
|
||||
const asATerminal = async (run: () => Promise<string>): Promise<string> => {
|
||||
const descriptor = Object.getOwnPropertyDescriptor(process.stdout, "isTTY");
|
||||
Object.defineProperty(process.stdout, "isTTY", { value: true, configurable: true });
|
||||
try {
|
||||
return await run();
|
||||
} finally {
|
||||
if (descriptor) Object.defineProperty(process.stdout, "isTTY", descriptor);
|
||||
else delete (process.stdout as unknown as { isTTY?: boolean }).isTTY;
|
||||
}
|
||||
};
|
||||
|
||||
it("downloads nothing and records no consent when the offer is declined", async () => {
|
||||
// The whole point of asking. Nothing below this line may fetch 32 MB.
|
||||
state.modelStatus = "not-asked";
|
||||
state.confirmAnswer = false;
|
||||
|
||||
const output = await asATerminal(() =>
|
||||
runCatalog({ query: "make a number count up", "on-device": true }),
|
||||
);
|
||||
|
||||
expect(state.downloads).toBe(0);
|
||||
expect(state.consentRecorded).toEqual([false]);
|
||||
expect(output).not.toContain("offer");
|
||||
});
|
||||
|
||||
it("downloads once when the offer is accepted", async () => {
|
||||
state.modelStatus = "not-asked";
|
||||
state.confirmAnswer = true;
|
||||
|
||||
await asATerminal(() => runCatalog({ query: "make a number count up", "on-device": true }));
|
||||
|
||||
expect(state.downloads).toBe(1);
|
||||
expect(state.consentRecorded).toEqual([true]);
|
||||
});
|
||||
|
||||
it("keeps a decline sticky until explicit --yes consent", async () => {
|
||||
state.modelStatus = "not-asked";
|
||||
state.confirmAnswer = false;
|
||||
|
||||
await asATerminal(() => runCatalog({ query: "count up", "on-device": true }));
|
||||
state.confirmAnswer = true;
|
||||
await asATerminal(() => runCatalog({ query: "count up", "on-device": true }));
|
||||
|
||||
expect(state.downloads).toBe(0);
|
||||
expect(state.consentRecorded).toEqual([false]);
|
||||
|
||||
await asATerminal(() => runCatalog({ query: "count up", "on-device": true, yes: true }));
|
||||
expect(state.downloads).toBe(1);
|
||||
expect(state.consentRecorded).toEqual([false, true]);
|
||||
});
|
||||
|
||||
it("does not treat non-interactive output as download consent", async () => {
|
||||
state.modelStatus = "not-asked";
|
||||
|
||||
await runEnvelope({ query: "count up", "on-device": true });
|
||||
|
||||
expect(state.downloads).toBe(0);
|
||||
expect(state.consentRecorded).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -1,4 +1,3 @@
|
||||
import { failCommand, finishCommand } from "../utils/commandResult.js";
|
||||
import { defineCommand } from "citty";
|
||||
import type { Example } from "./_examples.js";
|
||||
|
||||
@@ -13,10 +12,124 @@ export const examples: Example[] = [
|
||||
import * as clack from "@clack/prompts";
|
||||
import { type ItemType } from "@hyperframes/core";
|
||||
import { c } from "../ui/colors.js";
|
||||
import { listRegistryItems, loadAllItems } from "../registry/resolver.js";
|
||||
import { loadAllItems } from "../registry/resolver.js";
|
||||
import { fetchRegistryManifest } from "../registry/remote.js";
|
||||
import { loadProjectConfig, DEFAULT_PROJECT_CONFIG } from "../utils/projectConfig.js";
|
||||
import { resolve } from "node:path";
|
||||
import { finishCommand } from "../utils/commandResult.js";
|
||||
import { runAdd } from "./add.js";
|
||||
import { searchByWords } from "../registry/localSearch.js";
|
||||
import {
|
||||
downloadOfferMessage,
|
||||
ensureLocalModel,
|
||||
type LocalModelStatus,
|
||||
localModelStatus,
|
||||
nonInteractiveConsentMessage,
|
||||
recordLocalModelConsent,
|
||||
} from "../registry/localModel.js";
|
||||
import { localRuntimeAvailable } from "../registry/localEmbedder.js";
|
||||
import {
|
||||
cachedLocalVectorRevision,
|
||||
fetchLocalVectors,
|
||||
hasLocalVectors,
|
||||
localSemanticRanking,
|
||||
localVectorNames,
|
||||
} from "../registry/localSemantic.js";
|
||||
|
||||
/**
|
||||
* Get the offline tier ready, and report every reason it could not be.
|
||||
*
|
||||
* A per-run opt-in, so an agent or CI run can reach the offline tier at all:
|
||||
* the only other route is a prompt that fires exclusively on a terminal.
|
||||
*
|
||||
* Warnings are returned as well as printed so `--json` can carry the same
|
||||
* reasons the terminal shows.
|
||||
*/
|
||||
// a consent gate: each branch is a distinct reason the tier cannot run, and each has to be reported separately
|
||||
// fallow-ignore-next-line complexity
|
||||
async function prepareOnDeviceTier(opts: {
|
||||
assumedYes: boolean;
|
||||
artifactRevision?: string;
|
||||
canPrompt: boolean;
|
||||
registry: string;
|
||||
registryNames: ReadonlySet<string>;
|
||||
status: LocalModelStatus;
|
||||
}): Promise<string[]> {
|
||||
const warnings: string[] = [];
|
||||
const warn = (message: string): void => {
|
||||
warnings.push(message);
|
||||
console.error(message);
|
||||
};
|
||||
|
||||
const status = opts.status;
|
||||
if (!opts.assumedYes && status.status === "declined") {
|
||||
warn(
|
||||
"on-device search skipped: the model download was previously declined. Re-run with --yes to consent.",
|
||||
);
|
||||
return warnings;
|
||||
}
|
||||
|
||||
if (!opts.assumedYes && status.status === "not-asked" && !opts.canPrompt) {
|
||||
warn(nonInteractiveConsentMessage());
|
||||
return warnings;
|
||||
}
|
||||
|
||||
if (!opts.assumedYes && status.status === "not-asked") {
|
||||
const answer = await clack.confirm({
|
||||
message: downloadOfferMessage(),
|
||||
initialValue: true,
|
||||
});
|
||||
if (clack.isCancel(answer) || answer !== true) {
|
||||
recordLocalModelConsent(false);
|
||||
warn("on-device search skipped: the download was declined.");
|
||||
// Return, or the decline is the only thing that does not happen: the
|
||||
// runtime check below is skipped precisely because consent is now false,
|
||||
// control reaches recordLocalModelConsent(true), and the answer is
|
||||
// overwritten with yes before the download it refused.
|
||||
return warnings;
|
||||
}
|
||||
}
|
||||
|
||||
if (!(await localRuntimeAvailable())) {
|
||||
// Checked before downloading. Fetching 32 MB and then discovering the
|
||||
// runtime is missing wastes the bandwidth the consent was granted for.
|
||||
warn(
|
||||
"on-device search needs the native ONNX runtime, which a single-file build cannot load. " +
|
||||
"Install the CLI normally (npm i -g hyperframes) to use this tier.",
|
||||
);
|
||||
return warnings;
|
||||
}
|
||||
|
||||
if (status.status === "declined" || status.status === "not-asked") {
|
||||
recordLocalModelConsent(true);
|
||||
}
|
||||
const model = await ensureLocalModel();
|
||||
const revisionStale =
|
||||
opts.artifactRevision !== undefined && cachedLocalVectorRevision() !== opts.artifactRevision;
|
||||
if (
|
||||
!hasLocalVectors() ||
|
||||
revisionStale ||
|
||||
countUnindexed(opts.registryNames, localVectorNames()) > 0
|
||||
) {
|
||||
await fetchLocalVectors(opts.registry, { expectedRevision: opts.artifactRevision });
|
||||
}
|
||||
// Deliberately not the fetch's own answer. A refresh that fails still leaves
|
||||
// the previous vectors on disk, and those still rank: reporting the tier
|
||||
// unavailable there would be false, and the search that follows says what is
|
||||
// actually wrong with them.
|
||||
const vectors = hasLocalVectors();
|
||||
if (!model || !vectors) {
|
||||
warn(
|
||||
`on-device search unavailable: ${!model ? "model" : "catalog vectors"} could not be fetched`,
|
||||
);
|
||||
} else if (
|
||||
opts.artifactRevision !== undefined &&
|
||||
cachedLocalVectorRevision() !== opts.artifactRevision
|
||||
) {
|
||||
warn("on-device search is using the previous catalog vectors because the update failed");
|
||||
}
|
||||
return warnings;
|
||||
}
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
@@ -40,7 +153,27 @@ export default defineCommand({
|
||||
type: "boolean",
|
||||
description: "Interactive picker — select an item to install",
|
||||
},
|
||||
query: {
|
||||
type: "string",
|
||||
description:
|
||||
"Search by meaning when the on-device model is on, otherwise by name, title, description and tags",
|
||||
},
|
||||
yes: {
|
||||
type: "boolean",
|
||||
alias: "y",
|
||||
description: "Assume yes for prompts this run, including the on-device model download",
|
||||
},
|
||||
"on-device": {
|
||||
type: "boolean",
|
||||
// Consent for the model download is otherwise only reachable through a
|
||||
// prompt that fires on a TTY, which leaves every agent and CI run unable
|
||||
// to opt in at all.
|
||||
description:
|
||||
"Use on-device meaning search; pass --yes to approve a first non-interactive download",
|
||||
},
|
||||
},
|
||||
// one flag-parsing entry point feeding three output paths (json, interactive, table); splitting those is its own change
|
||||
// fallow-ignore-next-line complexity
|
||||
async run({ args }) {
|
||||
const json = args.json === true;
|
||||
const interactive = args["human-friendly"] === true;
|
||||
@@ -52,13 +185,17 @@ export default defineCommand({
|
||||
else if (args.type === "component") typeFilter = "hyperframes:component";
|
||||
else if (args.type) {
|
||||
console.error(`Invalid --type: "${args.type}". Use "block" or "component".`);
|
||||
failCommand();
|
||||
finishCommand(1);
|
||||
}
|
||||
|
||||
const entries = await listRegistryItems(typeFilter ? { type: typeFilter } : undefined, {
|
||||
baseUrl: config.registry,
|
||||
});
|
||||
const filtered = entries.filter((e) => e.type !== "hyperframes:example");
|
||||
// Asked for the whole manifest on purpose: its item list defines coverage,
|
||||
// and its artifact revision is the one owner of vector freshness.
|
||||
const manifest = await fetchRegistryManifest(config.registry);
|
||||
const entries = manifest?.items ?? [];
|
||||
const artifactRevision = manifest?.catalogArtifact?.revision;
|
||||
const catalog = entries.filter((e) => e.type !== "hyperframes:example");
|
||||
const registryNames = new Set(catalog.map((e) => e.name));
|
||||
const filtered = typeFilter ? catalog.filter((e) => e.type === typeFilter) : catalog;
|
||||
|
||||
if (filtered.length === 0) {
|
||||
if (json) console.log("[]");
|
||||
@@ -69,13 +206,76 @@ export default defineCommand({
|
||||
const items = await loadAllItems(filtered, { baseUrl: config.registry });
|
||||
|
||||
const tagFilter = args.tag?.toLowerCase();
|
||||
const matching = tagFilter
|
||||
const tagged = tagFilter
|
||||
? items.filter((item) => item.tags?.some((t) => t.toLowerCase() === tagFilter))
|
||||
: items;
|
||||
|
||||
const query = typeof args.query === "string" ? args.query.trim() : "";
|
||||
// Collected rather than only printed, so --json can carry the same reasons
|
||||
// the terminal shows. A machine that asked for a tier deserves to be told
|
||||
// it did not run.
|
||||
const searchContext = query ? { status: localModelStatus() } : null;
|
||||
const routineUpdate =
|
||||
searchContext?.status.status === "unavailable" ||
|
||||
(searchContext?.status.status === "ready" &&
|
||||
artifactRevision !== undefined &&
|
||||
cachedLocalVectorRevision() !== artifactRevision);
|
||||
const shouldPrepare = args["on-device"] === true || routineUpdate;
|
||||
let warnings: string[] = [];
|
||||
let effectiveStatus = searchContext?.status;
|
||||
if (searchContext && shouldPrepare) {
|
||||
warnings = await prepareOnDeviceTier({
|
||||
assumedYes: args.yes === true,
|
||||
artifactRevision,
|
||||
canPrompt: process.stdout.isTTY === true && !json,
|
||||
registry: config.registry,
|
||||
registryNames,
|
||||
status: searchContext.status,
|
||||
});
|
||||
// A successful preparation can move not-asked/unavailable to ready. Read
|
||||
// the state owner again rather than carrying the pre-download snapshot.
|
||||
effectiveStatus = localModelStatus();
|
||||
}
|
||||
const searched = effectiveStatus
|
||||
? await applySearch(tagged, query, registryNames, effectiveStatus)
|
||||
: null;
|
||||
if (searched) warnings.push(...searched.warnings);
|
||||
const matching = searched ? searched.items : tagged;
|
||||
|
||||
if (matching.length === 0) {
|
||||
if (json) console.log("[]");
|
||||
else console.log(`No items match tag "${args.tag}".`);
|
||||
// An empty result is exactly when the tier matters most: nothing found on
|
||||
// the weakest tier means something different from nothing found on the
|
||||
// best one.
|
||||
if (json && query) {
|
||||
console.log(
|
||||
JSON.stringify(
|
||||
{
|
||||
query,
|
||||
tier: tierToken(searched),
|
||||
tier_detail: tierDetail(searched),
|
||||
dropped: searched?.missing ?? 0,
|
||||
unindexed: searched?.unindexed ?? 0,
|
||||
...(searched?.topScore != null ? { top_score: searched.topScore } : {}),
|
||||
shown: 0,
|
||||
total: tagged.length,
|
||||
...(warnings.length ? { warnings } : {}),
|
||||
results: [],
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
} else if (json) console.log("[]");
|
||||
else {
|
||||
// Name whichever filter actually emptied the list. Reporting a tag
|
||||
// miss for a query miss sends people to fix the wrong thing.
|
||||
const criteria = [
|
||||
query ? `query "${query}"` : null,
|
||||
args.tag ? `tag "${args.tag}"` : null,
|
||||
].filter(Boolean);
|
||||
console.log(`No items match ${criteria.join(" and ")}.`);
|
||||
}
|
||||
if (query) await offerLocalModel(0, json, config.registry, artifactRevision);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -89,10 +289,69 @@ export default defineCommand({
|
||||
...("dimensions" in item && item.dimensions ? { dimensions: item.dimensions } : {}),
|
||||
...("duration" in item && item.duration ? { duration: item.duration } : {}),
|
||||
}));
|
||||
console.log(JSON.stringify(output, null, 2));
|
||||
if (!query) {
|
||||
// A plain listing has no tier and no drop count, and this array shape
|
||||
// is already released. Leave it alone.
|
||||
console.log(JSON.stringify(output, null, 2));
|
||||
return;
|
||||
}
|
||||
console.log(
|
||||
JSON.stringify(
|
||||
{
|
||||
query,
|
||||
tier: tierToken(searched),
|
||||
tier_detail: tierDetail(searched),
|
||||
dropped: searched?.missing ?? 0,
|
||||
unindexed: searched?.unindexed ?? 0,
|
||||
...(searched?.topScore != null ? { top_score: searched.topScore } : {}),
|
||||
shown: output.length,
|
||||
total: tagged.length,
|
||||
...(warnings.length ? { warnings } : {}),
|
||||
results: output,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
if (searched) {
|
||||
// Name the search that actually ran. A user seeing worse results has to
|
||||
// be able to tell which tier produced them.
|
||||
const how = tierDetail(searched);
|
||||
const unshowable =
|
||||
searched.missing > 0
|
||||
? ` · ${searched.missing} ranked ${searched.missing === 1 ? "move" : "moves"} not in this registry`
|
||||
: "";
|
||||
console.log(c.dim(` ${matching.length} of ${tagged.length} moves · ${how}${unshowable}`));
|
||||
if (searched.unindexed > 0) {
|
||||
// The costly direction, so it gets a line of its own rather than a
|
||||
// suffix: these moves cannot come back from meaning search at any rank,
|
||||
// for any query, and a reader has to learn what to run about it.
|
||||
// Silent when the index covers the registry.
|
||||
const remedy =
|
||||
args["on-device"] === true
|
||||
? "The published index is behind this registry."
|
||||
: "Re-run with --on-device to refresh it.";
|
||||
console.log(
|
||||
c.warn(
|
||||
` ${searched.unindexed} of ${registryNames.size} moves are missing from the ` +
|
||||
`on-device index, so meaning search cannot return them. ${remedy}`,
|
||||
),
|
||||
);
|
||||
}
|
||||
if (searched.localMode === "words") {
|
||||
// Suppressed when on-device was asked for and refused: telling someone
|
||||
// to pass a flag one line after explaining that flag cannot work here
|
||||
// reads as the tool arguing with itself.
|
||||
if (warnings.length === 0) {
|
||||
reportLocalModelOption(json);
|
||||
await offerLocalModel(matching.length, json, config.registry, artifactRevision);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (interactive) {
|
||||
const options = matching.map((item) => ({
|
||||
value: item.name,
|
||||
@@ -152,3 +411,207 @@ export default defineCommand({
|
||||
console.log(c.dim(`${matching.length} items. Run "hyperframes add <name>" to install.`));
|
||||
},
|
||||
});
|
||||
|
||||
/**
|
||||
* Resolve ranked names against the items this registry actually has.
|
||||
*
|
||||
* `missing` counts the ranked names this registry has no item for at all,
|
||||
* measured against `registryNames` — the whole catalog, not the post-filter
|
||||
* `items`. It is reported rather than swallowed: the catalog artifact and the
|
||||
* registry are published separately, so they can be different generations, and
|
||||
* the moves lost that way are the top-ranked ones.
|
||||
*
|
||||
* A move the user's own --type or --tag removed is not one of them. It is in
|
||||
* the registry and installable; they excluded it. Counting it here made every
|
||||
* filtered search report a skew that was not there, and made a real skew
|
||||
* indistinguishable from a filter doing its job.
|
||||
*/
|
||||
export function pickByName<T extends { name: string }>(
|
||||
items: T[],
|
||||
names: string[],
|
||||
registryNames: ReadonlySet<string>,
|
||||
): { ranked: T[]; missing: number } {
|
||||
const byName = new Map(items.map((item) => [item.name, item]));
|
||||
const ranked = names
|
||||
.map((name) => byName.get(name))
|
||||
.filter((item): item is T => item !== undefined);
|
||||
return { ranked, missing: names.filter((name) => !registryNames.has(name)).length };
|
||||
}
|
||||
|
||||
/**
|
||||
* Registry moves the on-device index holds no vector for.
|
||||
*
|
||||
* The counterpart to `pickByName`'s `missing`, and the direction that costs
|
||||
* something. `missing` is over-coverage: names the artifact ranks that this
|
||||
* registry cannot install, which only wastes a rank. This is under-coverage:
|
||||
* moves the registry has that were never embedded, so meaning search cannot
|
||||
* return them at any rank, for any query, and until now nothing in the output
|
||||
* said so. The vectors are fetched once and never invalidated, so every move
|
||||
* published since that fetch lands here.
|
||||
*
|
||||
* Measured against the unfiltered registry, matching `missing`: a --type or
|
||||
* --tag filter removing a move is the user narrowing their own search, not an
|
||||
* index that cannot see it.
|
||||
*/
|
||||
export function countUnindexed(
|
||||
registryNames: ReadonlySet<string>,
|
||||
indexedNames: Iterable<string>,
|
||||
): number {
|
||||
const indexed = new Set(indexedNames);
|
||||
let count = 0;
|
||||
for (const name of registryNames) {
|
||||
if (!indexed.has(name)) count += 1;
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
interface SearchOutcome<T> {
|
||||
items: T[];
|
||||
localMode: LocalMode;
|
||||
warnings: string[];
|
||||
/** Over-coverage: ranked names this registry has no item for. */
|
||||
missing: number;
|
||||
/**
|
||||
* Under-coverage: registry moves the tier that answered could not see.
|
||||
*
|
||||
* Always zero for word matching, which ranks the live registry listing and
|
||||
* so cannot be stale. Non-zero only on the on-device tier, whose vectors are
|
||||
* a separately published artifact that drifts from the registry.
|
||||
*/
|
||||
unindexed: number;
|
||||
/**
|
||||
* Similarity of the best result actually shown, when the answering tier
|
||||
* produces one. Null for word matching, whose score is a different and
|
||||
* non-comparable scale. Deliberately not a threshold: it is the signal a
|
||||
* caller needs to judge a ranker that ships without one.
|
||||
*/
|
||||
topScore: number | null;
|
||||
}
|
||||
|
||||
async function applySearch<
|
||||
T extends { name: string; title: string; description: string; tags?: string[] },
|
||||
>(
|
||||
items: T[],
|
||||
query: string,
|
||||
registryNames: ReadonlySet<string>,
|
||||
status: LocalModelStatus,
|
||||
): Promise<SearchOutcome<T>> {
|
||||
const warnings: string[] = [];
|
||||
// On-device meaning search, when the user opted into the model. Free and
|
||||
// offline, and it answers phrasings word matching cannot reach.
|
||||
if (status.status === "ready") {
|
||||
try {
|
||||
const ranking = await localSemanticRanking(query);
|
||||
if (ranking) {
|
||||
const { ranked, missing } = pickByName(
|
||||
items,
|
||||
ranking.map((entry) => entry.name),
|
||||
registryNames,
|
||||
);
|
||||
const best = ranked[0];
|
||||
if (best) {
|
||||
const scoreByName = new Map(ranking.map((entry) => [entry.name, entry.score]));
|
||||
return {
|
||||
items: ranked.slice(0, 25),
|
||||
localMode: "local-model",
|
||||
warnings,
|
||||
missing,
|
||||
unindexed: countUnindexed(registryNames, localVectorNames()),
|
||||
topScore: scoreByName.get(best.name) ?? null,
|
||||
};
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
// A broken model costs ranking quality, never the command. It does not
|
||||
// get to cost it silently: a tier the user switched on that quietly does
|
||||
// not run is indistinguishable from one that ran badly.
|
||||
const warning = `on-device search did not run: ${error instanceof Error ? error.message : "unknown error"}`;
|
||||
warnings.push(warning);
|
||||
console.error(warning);
|
||||
}
|
||||
}
|
||||
|
||||
// Shared vocabulary, not substring presence. A user asking to "make the
|
||||
// pace feel faster" shares no literal substring with any description, so
|
||||
// the old test returned nothing at all for exactly the phrasing people use.
|
||||
const words = searchByWords(
|
||||
query,
|
||||
items,
|
||||
(item) => `${item.name} ${item.title} ${item.description} ${(item.tags ?? []).join(" ")}`,
|
||||
);
|
||||
// Word matching ranks the items in hand, so nothing can go missing.
|
||||
return { items: words, localMode: "words", warnings, missing: 0, unindexed: 0, topScore: null };
|
||||
}
|
||||
|
||||
/**
|
||||
* Which tier answered, as a stable token.
|
||||
*
|
||||
* Deliberately not the printed sentence: "on-device meaning search" is written
|
||||
* for a person and will be reworded. A machine consumer needs something that
|
||||
* will not move underneath it.
|
||||
*/
|
||||
function tierToken(searched: { localMode: LocalMode } | null): "on-device" | "words" {
|
||||
return searched?.localMode === "local-model" ? "on-device" : "words";
|
||||
}
|
||||
|
||||
/** The same tier, as the sentence a person reads. */
|
||||
function tierDetail(searched: { localMode: LocalMode } | null): string {
|
||||
return searched?.localMode === "local-model" ? "on-device meaning search" : "local word match";
|
||||
}
|
||||
|
||||
type LocalMode = "local-model" | "words";
|
||||
|
||||
/**
|
||||
* Offer the on-device model when word matching came up thin, and only then.
|
||||
*
|
||||
* Asking on first run would interrupt people the free tier already serves.
|
||||
* Asking here puts the evidence in front of them: they can see what word
|
||||
* matching returned before deciding whether 33 MB is worth it.
|
||||
*/
|
||||
/**
|
||||
* Nobody to ask, so say what to ask for.
|
||||
*
|
||||
* Without this a scripted run sits on word matching with no indication that a
|
||||
* better offline tier exists and is one question away.
|
||||
*/
|
||||
function reportLocalModelOption(json: boolean): void {
|
||||
if (!json && process.stdout.isTTY) return;
|
||||
if (localModelStatus().status !== "not-asked") return;
|
||||
console.error(nonInteractiveConsentMessage());
|
||||
}
|
||||
|
||||
async function offerLocalModel(
|
||||
matchCount: number,
|
||||
json: boolean,
|
||||
registryBaseUrl: string,
|
||||
artifactRevision?: string,
|
||||
): Promise<void> {
|
||||
if (json || !process.stdout.isTTY) return;
|
||||
if (localModelStatus().status !== "not-asked") return;
|
||||
// Deliberately not gated on the number of results. That gate was set when the
|
||||
// catalog was small; against 411 moves a word match nearly always returns
|
||||
// more than a handful, so it had quietly become unreachable and nobody was
|
||||
// ever told the offline tier exists. Reaching the weakest tier is the signal.
|
||||
|
||||
const answer = await clack.confirm({
|
||||
message: downloadOfferMessage(matchCount),
|
||||
initialValue: true,
|
||||
});
|
||||
if (clack.isCancel(answer)) return;
|
||||
recordLocalModelConsent(answer === true);
|
||||
if (answer !== true) return;
|
||||
|
||||
// The vectors come from the registry rather than the package, so consent is
|
||||
// also the moment to fetch them. A failure here is reported: the alternative
|
||||
// is an offline tier the user turned on that silently never ranks anything.
|
||||
const vectors =
|
||||
hasLocalVectors() ||
|
||||
(await fetchLocalVectors(registryBaseUrl, { expectedRevision: artifactRevision }));
|
||||
console.log(
|
||||
c.dim(
|
||||
vectors
|
||||
? " Run the search again to download the model and rank by meaning."
|
||||
: " Could not fetch the catalog vectors; offline ranking stays off until they are available.",
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user