feat(skills): wake-word-training — teach Hermes to train custom wake words

Adds an optional skill so Hermes can build a custom openWakeWord model for a
phrase other than the bundled "hey hermes" (defaults to "hey <profile>"). The
skill drives openWakeWord's own trainer via the terminal tool — no heavy
training stack ships in core, and the existing resolver already loads the
resulting .onnx by path, so there are zero changes to the wake runtime.

- SKILL.md: end-to-end procedure (synthetic positives, augmentation/negatives,
  train, export into the profile's ~/.hermes/wakewords/, wire config, restart).
- scripts/generate_positives.py: OpenAI-TTS positives (default when
  OPENAI_API_KEY is set) with import-lazy synthesis; pure planning helpers.
- scripts/make_training_config.py: emits the openWakeWord training YAML.
- references/platforms.md: Colab fast path + macOS caveats + dataset sources.
- tests: frontmatter contract + pure helper logic, stdlib-only, no network.
- docs: point the wake-word "different phrase" section at the skill.
This commit is contained in:
Brooklyn Nicholson 2026-07-24 10:11:38 -05:00
parent 80e78eaf86
commit 576c090d2f
6 changed files with 560 additions and 0 deletions

View File

@ -0,0 +1,142 @@
---
name: wake-word-training
description: "Train a custom on-device wake word from synthetic speech."
version: 1.0.0
author: Brooklyn <brooklyn> & Hermes Agent
license: MIT
platforms: [linux, macos]
metadata:
hermes:
tags: [Voice, Wake Word, openWakeWord, On-Device, Training]
category: productivity
related_skills: []
---
# Wake Word Training Skill
Trains a custom openWakeWord model so Hermes answers to a phrase other than the
bundled "hey hermes" — e.g. "hey morgane" or "hey <profile>". Detection stays
100% on-device; only the optional voice synthesis (OpenAI TTS) touches the
network, and only during training. This skill drives openWakeWord's own
training pipeline through the `terminal` tool — it does not reimplement it.
Custom models are trained on **synthetic speech**: you never record yourself.
A TTS engine speaks the phrase a few thousand times across many voices, the
clips are augmented with noise/room impulses, and a small classifier is trained
on top of openWakeWord's frozen feature extractor. The output is a sub-100 KB
`.onnx` the existing wake runtime loads with zero code changes.
## When to Use
- The user wants Hermes to wake on a different phrase ("hey morgane", a
nickname, "hey <profile>").
- The user asks to "make/train my own wake word".
Do **not** use this for the default "hey hermes" — that model already ships.
For a phrase you already have an `.onnx` for, skip training and jump to
**Wire It Up**.
## Prerequisites
- **Time & hardware.** Fast on Linux/GPU (~1 hr) or a free Colab GPU. On macOS
it is single-threaded and slow (~35 hr) — see `references/platforms.md`.
Prefer Colab if the user just wants the file quickly.
- **Disk.** Several GB for augmentation/negative datasets (room impulses,
AudioSet/FMA noise, LibriSpeech negatives). Use a scratch dir, not the repo.
- **Voices.** `OPENAI_API_KEY` for high-quality OpenAI TTS positives (~$0.04 a
run); otherwise the offline Piper generator openWakeWord ships with.
- **A scratch training venv**, created ad hoc via `terminal` — never install the
heavy training stack into Hermes's own environment.
## How to Run
Confirm the phrase and where the model belongs, then train, then wire it up.
Resolve the profile-aware model directory once and reuse it:
```bash
python -c "from hermes_constants import get_hermes_home; print(get_hermes_home() / 'wakewords')"
```
Default the phrase to `hey <profile>` (the active profile's name; `hey hermes`
is the default profile) unless the user gives one. The model file is
`<phrase-slug>.onnx` (lowercase, spaces → underscores), e.g. `hey_morgane.onnx`.
## Quick Reference
| Step | Command / action |
| --- | --- |
| Model dir | `get_hermes_home() / 'wakewords'` (profile-aware) |
| Positives (OpenAI) | `scripts/generate_positives.py --phrase "hey morgane" --out-dir <dir>` |
| Training config | `scripts/make_training_config.py --phrase "hey morgane" --out <cfg.yml>` |
| Train | openWakeWord's `train.py --training_config <cfg.yml>` (in the scratch venv) |
| Wire up | set `wake_word.openwakeword.model` + `phrase`, then restart the listener |
## Procedure
1. **Agree on the phrase and paths.** Slugify the phrase for the filename.
Resolve the model dir with the one-liner above and `mkdir -p` it.
2. **Make a scratch training dir + venv** somewhere with room (not the repo):
```bash
mkdir -p /tmp/oww-train && cd /tmp/oww-train
python -m venv .venv && . .venv/bin/activate
pip install openwakeword
```
3. **Generate positives.**
- *OpenAI TTS (default when `OPENAI_API_KEY` is set):* run
`generate_positives.py` — it speaks the phrase across many voices with
small text/pacing variations and writes 16 kHz mono WAVs.
- *Offline:* let openWakeWord's own Piper generator produce them (the
training config's `target_phrase` drives it). See `references/platforms.md`.
4. **Fetch augmentation + negative data.** Follow openWakeWord's automatic
training notebook (room impulses, AudioSet/FMA noise, LibriSpeech/precomputed
negatives). `references/platforms.md` has the exact dataset sources and the
Colab shortcut that downloads them for you.
5. **Write the training config** with `make_training_config.py` (point it at the
positives/negatives/output dirs), then run openWakeWord's `train.py` against
it. Cross-check the emitted YAML against openWakeWord's `custom_model.yml` for
your installed version.
6. **Place the model.** Copy the exported `<slug>.onnx` into the resolved
`wakewords` dir.
7. **Wire It Up.** In the active profile's `~/.hermes/config.yaml`, set:
- `wake_word.openwakeword.model` → the absolute `.onnx` path
- `wake_word.phrase` → the human label ("hey morgane")
- `wake_word.enabled``true`
Then restart the listener so it reloads: `/wake` off then on, or restart the
app/gateway. (Wake config is not part of the prompt, so no cache concern.)
## Pitfalls
- **Don't install the training stack into Hermes's venv.** It's large and
macOS-flaky; keep it in the throwaway scratch venv.
- **openWakeWord needs its base feature models** (melspectrogram + embedding)
for any model — the trainer downloads them; don't delete them.
- **macOS bus errors** from openWakeWord's threadpool + mmap: train
single-threaded, or use Colab. See `references/platforms.md`.
- **Config `model` must be an absolute path** (or a bundled/built-in name). A
bare filename won't resolve.
- **Sensitivity.** If the new phrase over/under-triggers, tune
`wake_word.sensitivity` (01, higher = stricter) before retraining.
## Verification
- The `.onnx` exists in the profile's `wakewords` dir and is > 50 KB.
- Smoke-test detection before enabling it live:
```bash
python - <<'PY'
import wave, numpy as np
from openwakeword.model import Model
m = Model(wakeword_models=["/abs/path/hey_morgane.onnx"], inference_framework="onnx")
a = np.frombuffer(wave.open("a_clip_of_the_phrase.wav").readframes(1<<20), np.int16)
print("best:", max(max(m.predict(a[i:i+1280]).values()) for i in range(0, len(a)-1280, 1280)))
PY
```
A clear utterance should score well above `wake_word.sensitivity`.
- After wiring up, `/wake` on and say the phrase — Hermes opens a fresh session.

View File

@ -0,0 +1,59 @@
# Platforms, Datasets & the Fast Path
openWakeWord's automatic trainer is heavy (torch + a synthetic-speech +
augmentation stack) and its speed depends entirely on where it runs. Pick the
environment first, then follow the data steps.
## Where to train
| Environment | Time | Notes |
| --- | --- | --- |
| **Colab (free GPU)** | ~1 hr | Easiest. Recommended when the user just wants the `.onnx` quickly. |
| Linux + CUDA GPU | ~1 hr | Full local control. |
| Linux CPU | ~2 hr | Fine, just slower. |
| **macOS** | ~35 hr | Single-threaded only — see the bus-error note below. |
### Colab shortcut (recommended)
openWakeWord ships an automatic-training notebook that installs the stack,
downloads every dataset below, generates positives, trains, and exports the
model — in one run on a free GPU:
<https://colab.research.google.com/drive/1q1oe2zOyZp7UsB3jJiQ1IFn8z5YfjwEb>
Set the target phrase (and, if using the OpenAI-TTS positives from this skill,
upload the generated WAVs and point the config's positives dir at them), run it,
download the `.onnx`, then follow **Wire It Up** in `SKILL.md`.
### macOS caveat
openWakeWord's data pipeline uses a threadpool over memory-mapped files that
triggers bus errors on macOS. Train **single-threaded** (set the trainer's
worker/`n_jobs` to 1) or, better, use Colab. Expect 35 hours locally.
## Datasets the trainer needs
Four kinds of data (openWakeWord's notebook pulls them from HuggingFace):
1. **Room impulse responses**`davidscripka/MIT_environmental_impulse_responses`
(reverb augmentation). → `./mit_rirs`
2. **Background noise** — a slice of `agkphysics/AudioSet` (16 kHz). → `./audioset_16k`
3. **False-positive / music negatives**`rudraml/fma` (start with ~1 hr).
`./fma`
4. **Precomputed openWakeWord negative features** — from the openWakeWord
releases; used as generic negatives + validation for early stopping.
Base feature models (`melspectrogram`, `embedding`) are fetched automatically by
openWakeWord — don't delete them. Budget several GB of scratch space; keep a
local copy to reuse across runs.
## Positives: OpenAI TTS vs Piper
- **OpenAI TTS** (`generate_positives.py`, default when `OPENAI_API_KEY` is set):
10 voices, small text/pacing variations, ~$0.04 per run, higher quality. Point
the training config's positives dir at the generated WAVs.
- **Piper** (offline): openWakeWord's built-in generator synthesizes positives
from the config's `target_phrase`. Fully local, no key. Note: Piper's sample
generator is best-supported on Linux.
More positives → better accuracy, with smooth diminishing returns. A few
thousand is the practical floor; the notebook defaults to 5,000.

View File

@ -0,0 +1,136 @@
#!/usr/bin/env python3
"""Generate synthetic positive samples of a wake phrase for openWakeWord.
OpenAI TTS (``gpt-4o-mini-tts``) speaks the phrase across many voices with small
text/pacing variations, resampled to the 16 kHz mono WAV openWakeWord trains on.
This is the higher-quality alternative to openWakeWord's built-in Piper
generator; use it when ``OPENAI_API_KEY`` is set.
Heavy deps (openai, soundfile, numpy, scipy) are imported lazily inside the
synthesis path so the pure planning helpers stay import-light and unit-testable.
"""
from __future__ import annotations
import argparse
import re
# OpenAI's standard TTS voice set — cycling these gives speaker diversity, which
# is what drives custom wake-word robustness (openWakeWord's own models lean on
# exactly this kind of multi-voice synthetic data).
OPENAI_VOICES = (
"alloy", "ash", "ballad", "coral", "echo",
"fable", "onyx", "nova", "sage", "shimmer",
)
TARGET_SAMPLE_RATE = 16_000
def slugify(phrase: str) -> str:
"""`"Hey Morgane!"` → `"hey_morgane"` — the model filename stem."""
s = re.sub(r"[^a-z0-9]+", "_", phrase.strip().lower())
return s.strip("_") or "wake_word"
def build_variations(phrase: str) -> list[str]:
"""Small textual variations so the TTS doesn't render one frozen prosody."""
core = phrase.strip().rstrip(".!?,")
# Punctuation nudges cadence/intonation; duplicates are dropped by dict order.
seen = {core: None, f"{core}.": None, f"{core}!": None, f"{core}...": None}
return list(seen)
def voice_list(engine: str = "openai") -> list[str]:
if engine == "openai":
return list(OPENAI_VOICES)
raise ValueError(f"unknown engine: {engine!r}")
def plan_clips(phrase: str, count: int, voices: list[str]) -> list[tuple[str, str, str]]:
"""Deterministic (voice, text, filename) plan for *count* clips.
Voices and text variations are cycled independently so the set stays evenly
balanced across speakers regardless of *count*. Filenames are zero-padded and
unique, ready to drop into openWakeWord's positive-samples directory.
"""
if count < 1:
return []
if not voices:
raise ValueError("need at least one voice")
variations = build_variations(phrase)
stem = slugify(phrase)
width = max(4, len(str(count - 1)))
plan = []
for i in range(count):
voice = voices[i % len(voices)]
text = variations[i % len(variations)]
plan.append((voice, text, f"{stem}_{i:0{width}d}.wav"))
return plan
def _resample_to_16k_mono(pcm, src_rate: int):
"""int16 mono PCM at *src_rate* → int16 mono PCM at 16 kHz (lazy scipy)."""
import numpy as np
from scipy.signal import resample
if src_rate == TARGET_SAMPLE_RATE:
return pcm
n = round(len(pcm) * TARGET_SAMPLE_RATE / src_rate)
out = resample(pcm.astype("float32"), n)
return np.clip(out, -32768, 32767).astype("int16")
def _synthesize(plan, out_dir, api_key): # pragma: no cover - network/heavy deps
"""Call OpenAI TTS for each planned clip and write a 16 kHz mono WAV."""
import io
import wave
from pathlib import Path
import numpy as np
import soundfile as sf
from openai import OpenAI
client = OpenAI(api_key=api_key)
out = Path(out_dir)
out.mkdir(parents=True, exist_ok=True)
for voice, text, name in plan:
resp = client.audio.speech.create(
model="gpt-4o-mini-tts", voice=voice, input=text, response_format="wav"
)
data, rate = sf.read(io.BytesIO(resp.read()), dtype="int16")
if getattr(data, "ndim", 1) == 2: # stereo → mono
data = data.mean(axis=1).astype("int16")
data = _resample_to_16k_mono(np.asarray(data), rate)
with wave.open(str(out / name), "wb") as w:
w.setnchannels(1)
w.setsampwidth(2)
w.setframerate(TARGET_SAMPLE_RATE)
w.writeframes(np.asarray(data).tobytes())
return len(plan)
def main(argv=None) -> int:
import os
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--phrase", required=True, help='e.g. "hey morgane"')
ap.add_argument("--out-dir", required=True, help="positive-samples directory")
ap.add_argument("--count", type=int, default=500, help="clips to generate")
ap.add_argument("--engine", default="openai", choices=["openai"])
args = ap.parse_args(argv)
api_key = (os.getenv("OPENAI_API_KEY") or "").strip()
if not api_key:
ap.error(
"OPENAI_API_KEY not set. Set it, or use openWakeWord's offline Piper "
"generator instead (see the skill's references/platforms.md)."
)
plan = plan_clips(args.phrase, args.count, voice_list(args.engine))
written = _synthesize(plan, args.out_dir, api_key)
print(f"wrote {written} positive samples to {args.out_dir}")
return 0
if __name__ == "__main__": # pragma: no cover
raise SystemExit(main())

View File

@ -0,0 +1,85 @@
#!/usr/bin/env python3
"""Emit an openWakeWord automatic-training YAML for a custom phrase.
Writes the config openWakeWord's ``train.py`` consumes, pre-filled with the
target phrase, model name, and the positives/negatives/output paths. It covers
the common fields; always cross-check against openWakeWord's ``custom_model.yml``
for your installed version, since the schema evolves.
"""
from __future__ import annotations
import argparse
import re
def slugify(phrase: str) -> str:
s = re.sub(r"[^a-z0-9]+", "_", phrase.strip().lower())
return s.strip("_") or "wake_word"
def build_config(
phrase: str,
output_dir: str,
*,
positives_dir: str | None = None,
background_dir: str = "./audioset_16k",
rir_dir: str = "./mit_rirs",
false_positive_dir: str = "./fma",
n_samples: int = 5_000,
n_samples_val: int = 1_000,
steps: int = 10_000,
) -> dict:
"""Build the openWakeWord training-config dict.
When *positives_dir* is given (OpenAI-TTS clips already generated), the
trainer uses those instead of synthesizing its own with Piper.
"""
model_name = slugify(phrase)
cfg: dict = {
"target_phrase": [phrase.strip()],
"model_name": model_name,
"output_dir": output_dir,
"n_samples": n_samples,
"n_samples_val": n_samples_val,
"steps": steps,
"target_accuracy": 0.7,
"target_recall": 0.5,
# Augmentation + negative sources (see references/platforms.md).
"rir_paths": [rir_dir],
"background_paths": [background_dir],
"false_positive_validation_data_path": false_positive_dir,
"augmentation_rounds": 1,
"layer_size": 32,
}
if positives_dir:
# Reuse pre-generated positive clips rather than Piper-synthesizing them.
cfg["custom_positive_samples_dir"] = positives_dir
return cfg
def main(argv=None) -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--phrase", required=True, help='e.g. "hey morgane"')
ap.add_argument("--out", required=True, help="path to write the YAML config")
ap.add_argument("--output-dir", default="./oww_out", help="trainer output dir")
ap.add_argument("--positives-dir", default=None, help="pre-generated positives")
ap.add_argument("--steps", type=int, default=10_000)
args = ap.parse_args(argv)
import yaml
cfg = build_config(
args.phrase,
args.output_dir,
positives_dir=args.positives_dir,
steps=args.steps,
)
with open(args.out, "w", encoding="utf-8") as f:
yaml.safe_dump(cfg, f, sort_keys=False)
print(f"wrote training config for {cfg['model_name']!r} to {args.out}")
return 0
if __name__ == "__main__": # pragma: no cover
raise SystemExit(main())

View File

@ -0,0 +1,129 @@
"""Tests for the wake-word-training optional skill.
Pure-logic + frontmatter contract only no network, no heavy deps. The scripts'
synthesis paths (openai/soundfile/scipy) are import-lazy, so importing the
helpers here needs nothing beyond stdlib.
"""
import importlib.util
import re
from pathlib import Path
import pytest
import yaml
REPO_ROOT = Path(__file__).resolve().parents[2]
SKILL_DIR = REPO_ROOT / "optional-skills" / "productivity" / "wake-word-training"
def _load(module_name: str):
path = SKILL_DIR / "scripts" / f"{module_name}.py"
spec = importlib.util.spec_from_file_location(f"_wwt_{module_name}", path)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod
@pytest.fixture(scope="module")
def positives():
return _load("generate_positives")
@pytest.fixture(scope="module")
def cfg_mod():
return _load("make_training_config")
# ── Frontmatter contract ─────────────────────────────────────────────────
def _frontmatter() -> dict:
text = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8")
m = re.match(r"^---\n(.*?)\n---\n", text, re.DOTALL)
assert m, "SKILL.md must open with a YAML frontmatter block"
return yaml.safe_load(m.group(1))
def test_description_is_short_one_sentence():
desc = _frontmatter()["description"]
assert len(desc) <= 60, f"description is {len(desc)} chars (max 60)"
assert desc.endswith("."), "description must end with a period"
def test_platforms_declared():
# POSIX-ish training tooling — must declare supported platforms.
assert _frontmatter().get("platforms"), "platforms gating is required"
def test_author_credits_a_human_first():
author = _frontmatter()["author"]
assert not author.lower().startswith("hermes"), "credit the human first"
# ── generate_positives pure helpers ──────────────────────────────────────
@pytest.mark.parametrize(
"phrase,expected",
[("Hey Morgane!", "hey_morgane"), (" Hey Hermes ", "hey_hermes"), ("!!!", "wake_word")],
)
def test_slugify(positives, phrase, expected):
assert positives.slugify(phrase) == expected
def test_variations_are_unique_and_nonempty(positives):
v = positives.build_variations("hey morgane")
assert v and len(v) == len(set(v))
assert all(x.lower().startswith("hey morgane") for x in v)
def test_plan_balances_voices_and_is_unique(positives):
voices = positives.voice_list("openai")
plan = positives.plan_clips("hey morgane", 25, voices)
assert len(plan) == 25
filenames = [name for _, _, name in plan]
assert len(set(filenames)) == 25, "filenames must be unique"
# Voices cycle evenly: with 25 clips over 10 voices, counts differ by <= 1.
from collections import Counter
counts = Counter(v for v, _, _ in plan)
assert max(counts.values()) - min(counts.values()) <= 1
def test_plan_empty_for_nonpositive_count(positives):
assert positives.plan_clips("hey morgane", 0, ["alloy"]) == []
def test_plan_requires_a_voice(positives):
with pytest.raises(ValueError):
positives.plan_clips("hey morgane", 5, [])
def test_unknown_engine_rejected(positives):
with pytest.raises(ValueError):
positives.voice_list("piper")
# ── make_training_config pure helper ─────────────────────────────────────
def test_config_sets_phrase_and_derived_name(cfg_mod):
cfg = cfg_mod.build_config("Hey Morgane", "/tmp/out")
assert cfg["target_phrase"] == ["Hey Morgane"]
assert cfg["model_name"] == "hey_morgane"
assert cfg["output_dir"] == "/tmp/out"
assert isinstance(cfg["n_samples"], int) and cfg["n_samples"] > 0
def test_config_uses_custom_positives_only_when_given(cfg_mod):
without = cfg_mod.build_config("hey morgane", "/tmp/out")
assert "custom_positive_samples_dir" not in without
with_pos = cfg_mod.build_config("hey morgane", "/tmp/out", positives_dir="/tmp/pos")
assert with_pos["custom_positive_samples_dir"] == "/tmp/pos"
def test_config_is_yaml_serializable(cfg_mod):
cfg = cfg_mod.build_config("hey morgane", "/tmp/out", positives_dir="/tmp/pos")
round_tripped = yaml.safe_load(yaml.safe_dump(cfg))
assert round_tripped == cfg

View File

@ -124,6 +124,15 @@ wake_word:
model: ~/.hermes/wakewords/computer.onnx # or a built-in name like hey_jarvis
```
Or let Hermes train one for you. Install the **wake-word-training** skill and
ask it to make a phrase (defaults to `hey <profile>`); it drives openWakeWord's
trainer end-to-end — synthetic voices (OpenAI TTS or Piper), model export into
your profile's `~/.hermes/wakewords/`, and the config wiring:
```bash
hermes skills install official/productivity/wake-word-training
```
Training references:
- [openWakeWord](https://github.com/dscripka/openWakeWord)