ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-10-30 09:16:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(cli): explain how catalog search works and which model it downloads (#4462)

Browse Source 
* docs(cli): explain how catalog search works and which model it downloads

* docs(cli): name the model catalog search was chosen against

* docs(cli): keep only live rules in the word-search header comment

* docs(cli): correct the hosted vector size in the local vector build comment
This commit is contained in:
Miguel Ángel
2026-09-25 04:43:54 +00:00
committed by GitHub
parent ad87313ee2
commit 7c634426fd
10 changed files with 178 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/developers/catalog-search.mdx
+128
View File
@@ -0,0 +1,128 @@
---
title: "How catalog search works"
description: "What `hyperframes catalog --query` does, the on-device model behind meaning search, what it downloads, and how its index is built."
---
`hyperframes catalog --query "<what you want>"` finds catalog blocks and components by what they do,
not only by name. It runs on your machine in one of two ways, and neither sends your query anywhere.
| Search | When it runs | Needs |
| --- | --- | --- |
| **Word match** | By default, and whenever the model below is not on this machine | Nothing |
| **On-device meaning search** | Once you have agreed to a one-time 33 MB download | The model, a small index, and the ONNX runtime |
With `--query --json`, the output names the one that ran: `"tier": "words"` or `"tier": "on-device"`.
## Word match
Scores each item by the words it shares with your query across its name, title, description and
tags, leaving out common words and words of one or two letters. A word that matches an item's name or
title counts for more than one in its description, a word few items use counts for more than one most
items use, plurals match their singular ("pulses" finds "pulse"), and compound words match both ways
("countdown" and "count down"). It needs no model and no network, and it can only find items that use
your words.
## On-device meaning search
Turns your query and every catalog item into vectors of numbers and returns the items whose vectors
point the same way as the query's, so "make the pace feel suddenly faster" can find a move whose
description never uses those words. It shows the 25 closest items, best first, and `--json` adds
`top_score`, the similarity of the first result.
### Turning it on
The download is never started without a yes:
- In a terminal, the first word-match search asks once whether to download the model, after showing
what word match found. Your answer is remembered. A yes fetches the catalog vectors right away and
the model on your next search.
- `--on-device` gets meaning search ready on this run, prompting first in a terminal if you have not
been asked yet.
- Agents, CI and `--json` runs are never prompted. They are told to ask the person they work for;
`--on-device --yes` is that person's yes, and records it.
- After a no, `--on-device` alone is skipped with a message; `--on-device --yes` changes the answer.
Once the model is on your machine, every `catalog --query` uses meaning search. If it cannot run, the
CLI says why and falls back to word match.
### The model
| | |
| --- | --- |
| Model | `bge-small-en-v1.5`, the quantized ONNX build from `Xenova/bge-small-en-v1.5` |
| Pinned to | revision `ea104dacec62c0de699686887e3f920caeb4f3e3`, never `main` |
| Vector size | 384 numbers per item |
| Download | 32 MB model and a 0.7 MB tokenizer, 33 MB in total |
It was chosen because its average score on the MTEB benchmark is close to OpenAI's
`text-embedding-3-small` (62.17 against 62.3) with vectors a quarter of the size (384 numbers instead
of 1,536), and it is small enough to download on request. It is pinned to one revision because the
catalog's vectors were made with those exact weights: a model that changed underneath them would still
return results, just wrong ones.
The quantized build is also what makes the catalog vectors, so the query and the catalog always come
from the same file.
### What is downloaded, and where
| What | From | Stored in |
| --- | --- | --- |
| `model_quantized.onnx` (34,014,426 bytes) | huggingface.co, at the pinned revision | `~/.hyperframes/models/bge-small-en-v1.5.onnx` |
| `tokenizer.json` (711,396 bytes) | huggingface.co, at the pinned revision | `~/.hyperframes/models/bge-small-en-v1.5.tokenizer.json` |
| `onnxruntime-node` 1.21.1 | npm, on first use | `~/.cache/hyperframes/optional` |
| `local-vectors.json` and `local-vectors.bin` | the catalog registry, `catalog-artifact/` | `~/.hyperframes/catalog` |
Both model files are checked against pinned SHA-256 hashes. A file that does not match is deleted and
the search falls back to word match, so a partial or altered download is never used. The runtime is
installed before the model, so a machine that cannot run it does not spend the 33 MB first.
The catalog vectors carry a `revision`. When the registry publishes a new one, the CLI fetches the new
pair before the next meaning search, so newly published items become findable without a CLI update.
For development, `HYPERFRAMES_CATALOG_ARTIFACT_DIR` replaces `~/.hyperframes/catalog` as the folder
the vectors are kept in.
### What is compared
- **Each catalog item** is represented by its title, its description and its tags, one per line. Its
name is deliberately left out.
- **Your query** gets the prefix bge expects on short searches:
`Represent this sentence for searching relevant passages: `.
- Each vector is the model's first-token output scaled to length 1, and items are ranked by cosine
similarity to the query.
## Privacy
Meaning search runs entirely on your machine: your query is turned into a vector locally and compared
with vectors already on disk. The only network traffic is the one-time downloads above and the refresh
of the catalog vectors, none of which carries your query.
When a search finds nothing useful, the CLI prints a `hyperframes feedback --search-miss` command you
can run to report the gap. That command sends the query, and only if you run it.
## How the index is built
The vectors ship with the registry, one row per installable block and component, built by:
```bash
bun scripts/catalog/build-local-vectors.ts
```
It reads `registry/blocks/*` and `registry/components/*` from the repository, sorts them by name, and
embeds them 16 at a time with the pinned model. The batch size matters: within a batch the texts are
padded to the same length, which changes the quantized result, so the same text embedded alone comes
out slightly different. Rebuilding in batches of 16 reproduces the shipped rows
exactly, which is how to check that an index came from this registry and this model.
The `revision` is a SHA-256 over the model, its revision, the vector size, the batch size and every
item's name and text. It is written into `local-vectors.json` and into the registry's `registry.json`, which is
how the CLI knows its copy is out of date.
### Rebuilding it
Contributors adding a catalog item rarely need to. `bun run generate:catalog` rebuilds the index along
with the rest of the generated catalog, downloading the pinned model first if needed; the catalog
publication pull request does the same after source changes merge. Without the model, the build script
stops and says so rather than failing midway, and the "Catalog: search index covers the registry" CI
check fails if any searchable item is missing from the index. See
[`registry/catalog-artifact/README.md`](https://github.com/heygen-com/hyperframes/blob/main/registry/catalog-artifact/README.md)
for the file format.
docs/docs.json
+1
View File
@@ -851,6 +851,7 @@
"pages": [
"developers/cli",
"packages/cli",
"developers/catalog-search",
"packages/lint"
]
},
docs/packages/cli.mdx
+5
View File
@@ -177,6 +177,11 @@ and tags, shaped for agents to parse; `--json` is the structured form. Each
dimensions, duration and `preview` (`video` and `poster` URLs) when it has them.
`--human-friendly` opens a picker that runs `add` on whatever you select.
`--query "<what you want>"` searches by description instead of listing
everything, and `--on-device` switches that search from word match to meaning
search after a one-time 33 MB download. What each does, what is downloaded and
what stays on your machine: [How catalog search works](/developers/catalog-search).
## Bring in source material
### `capture`
packages/cli/src/registry/localModel.ts
+3 -3
View File
@@ -22,9 +22,9 @@ import { downloadFile } from "../utils/download.js";
import { readConfig, writeConfig } from "../telemetry/config.js";
/**
* bge-small-en-v1.5: 62.17 MTEB average against 62.3 for the hosted model this
* substitutes for, at 384 dimensions instead of 1536. Chosen for being close to
* parity while small enough to ship, not for being a weaker tier.
* bge-small-en-v1.5: 62.17 MTEB average against 62.3 for text-embedding-3-small,
* the model it was chosen against, at 384 dimensions instead of 1536. Chosen for
* being close to parity while small enough to ship, not for being a weaker tier.
*/
export const LOCAL_MODEL_ID = "bge-small-en-v1.5";
export const LOCAL_MODEL_DIMENSIONS = 384;
packages/cli/src/registry/localSearch.ts
+13 -17
View File
@@ -1,31 +1,27 @@
/**
* Local catalog search that costs nothing and needs no account.
*
* Replaces a substring test. `description.includes(query)` cannot answer "make
* the pace suddenly feel faster" because no description contains that phrase,
* so the honest result was zero matches. Scoring shared vocabulary answers it
* partially, offline, with no model and no network.
* Scoring shared vocabulary answers "make the pace suddenly feel faster"
* partially, offline, with no model and no network, where a substring test
* finds nothing because no description contains that phrase.
*
* The scorer is the one the retrieval evaluation used: lowercase tokens, stop
* words dropped, anything three characters or shorter dropped, and the shared
* token count divided by the square root of the entry's token count. That
* divisor is load-bearing. Without it the wordiest entry wins every query on
* sheer surface area.
* words dropped, anything two characters or shorter dropped, and the shared
* token count divided by the square root of the entry's token count. Without
* that divisor the wordiest entry wins every query on sheer surface area.
*
* Two things sit on top of that, both measured against a fixed eval set of
* catalog queries rather than adjusted by feel:
*
* 1. A token matching an item's NAME or TITLE counts for more than one
* matching its description. Without this, searching "typewriter effect on a
* title" ranked the item literally called `typewriter` seventh, behind
* entries that merely mention typing. An author who types a move's name is
* giving the strongest signal available and it was being averaged away.
* matching its description, so searching "typewriter effect on a title"
* ranks the item called `typewriter` above entries that merely mention
* typing. A name the author types is the strongest signal available.
*
* 2. Plurals fold to their singular on both sides. "a stat that counts up and
* then pulses once" shares no token with a description reading "lands with
* a restrained scale pulse", because `counts` is not `count` and `pulses`
* is not `pulse`. Adding detail to a query made results strictly worse,
* which is the opposite of what a search should do.
* 2. Plurals fold to their singular on both sides, so "a stat that counts up
* and then pulses once" matches a description reading "lands with a
* restrained scale pulse". Adding detail to a query must not make results
* worse.
*
* Ties break on descending name, matching the evaluation's sort.
*/
packages/cli/src/registry/localSemantic.ts
+1 -1
View File
@@ -7,7 +7,7 @@
* closes that gap with a 33 MB model the user opted into.
*
* The vectors here are 384-dimension and were produced by a different model
* than the hosted 1536-dimension set. The two are not comparable and are never
* than the hosted 3072-dimension set. The two are not comparable and are never
* mixed: a query is embedded by whichever model produced the vectors it is
* being compared against.
*/
registry/catalog-artifact/README.md
+23 -18
View File
@@ -1,24 +1,29 @@
# Catalog artifact
Two files ship from here, and only these two. The CLI fetches them over HTTP
when a user opts into offline catalog search (`catalog --query ... --on-device`),
so they are served from the registry rather than bundled in the package.
The CLI fetches these files over HTTP when a user opts into on-device catalog
search (`catalog --query ... --on-device`), so they are served from the registry
rather than bundled in the package. How the search uses them is in
[How catalog search works](https://hyperframes.heygen.com/developers/catalog-search).
| File | What it is |
| -------------------- | ----------------------------------------------------------------------- |
| `local-vectors.json` | `{ model, dimensions, names }`. `names` is the row order of the binary. |
| `local-vectors.bin` | Float32, row-major, `names.length * dimensions` values, no header. |
| File | What it is |
| -------------------- | ------------------------------------------------------------------------------------------------ |
| `local-vectors.json` | `{ model, modelRevision, dimensions, revision, names }`. `names` is the row order of the binary. |
| `local-vectors.bin` | Float32, row-major, `names.length * dimensions` values, no header. |
Currently 168 rows at 384 dimensions: 258,048 bytes, one row per installable
registry item.
One row per installable registry item, at 384 dimensions.
`media-vectors.*` are the same kind of index over the bundled sound effects in
`skills/media-use/audio/assets/sfx/manifest.json`, built with `--manifest`.
Everything below is about `local-vectors.*`.
## Provenance
No digest, revision or build timestamp is recorded in either file, and the
runtime only checks that `dimensions` matches the model it loaded. `model` is a
label the build wrote, not a proof. So the only real provenance check is to
rebuild the rows and compare them, which works because both inputs are in this
repository:
`revision` is a sha256 over the model, its revision, the dimensions, the batch
size and every item's name and embedded text, and the build writes the same value
to `registry.json` as `catalogArtifact.revision`. The CLI refetches its cached copy
when the two differ. That names the inputs the build claims, not that the floats
came from them, so the only real provenance check is to rebuild the rows and
compare them, which works because both inputs are in this repository:
- the text each row was embedded from is `itemRetrievalText(registry-item.json)`
(title, description, tags, joined by newlines, name deliberately excluded),
@@ -40,10 +45,10 @@ bun scripts/catalog/build-local-vectors.ts
It reads `registry/blocks/*` and `registry/components/*` directly, so the
rebuild has no input outside this repository. You rarely need to run it by
hand: a lefthook `catalog-index` pre-commit command regenerates and re-stages
both files whenever a staged `registry-item.json` changes, and CI fails the
"Catalog: search index covers the registry" job if the index is ever missing an
item.
hand: `bun run generate:catalog` runs it after downloading the pinned model, the
catalog publication PR regenerates these files after source changes merge, and CI
fails the "Catalog: search index covers the registry" job if the index is ever
missing an item.
`build-catalog-artifact.ts` does **not** produce these files. It builds the
3072-dimension hosted artifact from an external shelf file, for the hosted search
scripts/catalog/build-local-vectors.ts
+1 -1
View File
@@ -3,7 +3,7 @@
/**
* Embed the catalog with the on-device model so local search can rank by meaning.
*
* A second vector set, not a replacement. The hosted vectors are 1536-dimension
* A second vector set, not a replacement. The hosted vectors are 3072-dimension
* and measured; these are 384-dimension and free, and the two are not
* interchangeable because vectors from different models cannot be compared.
*
skills-manifest.json
+1 -1
View File
@@ -30,7 +30,7 @@
"files": 7
},
"hyperframes-cli": {
"hash": "d994bb25706982ee",
"hash": "ce021b1e5053d507",
"files": 11
},
"hyperframes-core": {
skills/hyperframes-cli/SKILL.md
+2 -2
View File
@@ -69,12 +69,12 @@ Treat tiny unstyled content, canvas-sized icons, missing hero elements, or timel
## Agent conventions
- **Search the catalog before writing motion by hand.** `npx hyperframes catalog --query "<the beat, in plain English>"`. Search is entirely local: there is no hosted tier, no account, and the query text is never sent anywhere. By default it ranks on vocabulary shared with the item's name, title and description, which misses any phrasing that does not reuse the catalog's own wording. Add `--on-device` to rank by meaning instead (see the offline tier below).
- **Search the catalog before writing motion by hand.** `npx hyperframes catalog --query "<the beat, in plain English>"`. Search is entirely local: there is no hosted tier, no account, and the query text is never sent anywhere. By default it ranks on vocabulary shared with the item's name, title, description and tags, which misses any phrasing that does not reuse the catalog's own wording. Add `--on-device` to rank by meaning instead (see the offline tier below).
- **Query in English even when the video is not.** Both tiers index an English catalog, so a query in another script produces no searchable terms and returns nothing. Describe the move in English; the on-screen copy stays in whatever language the video needs. `No searchable words in query` means exactly this and is not a missing component, so do not report it as a catalog gap.
- **Read which tier answered; never infer it from results appearing.** With `--json` the envelope carries `query`, `tier` (`on-device` or `words`), `tier_detail`, `dropped`, `unindexed`, `shown`, `total` and `results`, plus `top_score` when the answering tier produces one and `warnings` when a tier was asked for and could not run, or when a search returned nothing and a better tier is still waiting on someone's consent. A weak result on `words` is expected; the same result on `on-device` is a bug. `top_score` is on-device only and has no threshold behind it: the ranker returns the whole catalog in some order for every query, so read it as evidence rather than as a pass or fail.
- **`dropped` and `unindexed` are opposite skews between the registry and the on-device index, and rewording the query fixes neither.** `dropped` counts ranked names this registry cannot install, so the strongest matches are the ones being lost. `unindexed` counts registry moves the index cannot see at all, which no query can ever return. Refreshing the registry is not the answer to either: its manifest carries a 24h TTL and heals itself, while the vectors are a separately published artifact fetched into `~/.hyperframes/catalog/`. Re-running with `--on-device` refetches that index when `unindexed` is above zero, so that is the remedy to hand the user. A pure over-coverage skew (`dropped` above zero while `unindexed` is zero) does not trigger the refetch; clearing `~/.hyperframes/catalog/` is the only way out of that one. Both counts are of names rather than of results, so either can exceed `total`.
- **When a search comes back with nothing worth installing, say so.** `npx hyperframes feedback --search-miss "<the query you ran>" --wanted "<the move you needed>" --tier <the tier that answered>`. You do not have to assemble that line: `catalog --query` prints it pre-filled, and every `--json` search envelope carries it as `report_gap` with the query and tier already correct — fill in `--wanted` and send. This is the only path that sends a query anywhere, and it is a separate deliberate command precisely so plain `catalog --query` keeps its promise of sending nothing. **Report on either tier**, whenever the results do not do the thing; do not hold out for the on-device tier, which needs a consented 33 MB download and is therefore off in most agent runs — waiting for it means never reporting at all. The tier rides along in the report, so a vocabulary miss stays distinguishable from a meaning miss without you having to judge which one you hit. What comes back is a list of moves the catalog does not have yet, read directly rather than guessed from install counts, so the phrasing that matters is the effect you wanted, not the item name you imagined. It carries no rating and never lands in the rating metric.
- **Offer the offline tier; never enable it silently.** A one-time ~33 MB download (a quantized ONNX build of `bge-small-en-v1.5` plus its tokenizer, pinned to a fixed revision) and the catalog vectors from the registry, both cached under `~/.hyperframes/`, neither added to the project or any package. Once cached it ranks by meaning with nothing sent. Say the size out loud and let the person decide, then pass `--on-device` (with `-y` to skip the prompt) once they agree. The interactive offer only fires on a TTY. Under `--json` there is no prompt, but a search that found nothing puts the same ask in `warnings`, so read that array and put the decision to the user yourself.
- **Offer the offline tier; never enable it silently.** A one-time ~33 MB download (a quantized ONNX build of `bge-small-en-v1.5` plus its tokenizer, pinned to a fixed revision) and the catalog vectors from the registry, both cached under `~/.hyperframes/`, neither added to the project or any package. Once cached it ranks by meaning with nothing sent. Say the size out loud and let the person decide, then pass `--on-device` (with `-y` to skip the prompt) once they agree. The interactive offer only fires on a TTY. Under `--json` there is no prompt, but a search that found nothing puts the same ask in `warnings`, so read that array and put the decision to the user yourself. When the person asks what the download is, why this model, or what leaves the machine, point them to https://hyperframes.heygen.com/developers/catalog-search.
- Prefer `--json` for agent and CI calls. Server-mode `render`, `preview`, and `play` do not provide ordinary JSON output; `preview --selection --json` and `preview --context --json` are query-mode exceptions.
- `doctor --json` always exits zero. Gate on its payload: