fix/relay-per-platform-caps
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ben Barclay
|
baae55bb32 |
fix(relay): per-platform capability descriptors for multi-platform gateways
One relay adapter fronts N platforms on one WS, but the capability surface (MAX_MESSAGE_LENGTH / message_len_fn) was a scalar from whichever descriptor resolved the handshake — and the transport's read loop OVERWROTE it on every descriptor frame (last-writer-wins). A Discord chat on a gateway whose applied descriptor was Telegram's inherited the 4,096-char cap and over-sent into Discord's 2,000-char API 400 (observed live: 2,543/2,641-char replies silently lost while inbound kept working). - ws_transport: accumulate one descriptor per platform in _descriptors_by_platform (exposed via descriptor_for_platform); the FIRST descriptor of a connection generation stays the session default instead of last-writer-wins; the map resets on re-dial. - BasePlatformAdapter: new max_message_length_for_chat / message_len_fn_for_chat hooks defaulting to the scalar surface (native single-platform adapters unchanged). - RelayAdapter: overrides resolve the chat's platform from _platform_by_chat (the same map per-frame egress uses) and look up that platform's negotiated descriptor; falls back to the scalar for unknown chats/transports. - stream_consumer (streaming budget, _raw_message_limit, fallback-continuation chunking) + run.py tool-progress limit now resolve per-chat. Tests: tests/gateway/relay/test_relay_per_platform_caps.py (7) — verified fail-without/pass-with (all 7 fail with the fix stashed). Relay + stream consumer suites green (213 + 223). |
||
|
Ben Barclay
|
ef6ce56cad |
fix(cron): reconcile external provider after a claimed direct run (#70479)
A direct run (cronjob action='run' / webhook-triggered manual fire) takes the job's fire_claim and advances next_run_at. When it races the external provider's scheduled fire for the same occurrence, Chronos loses the claim and — by design — does not re-arm (the winner owns the re-arm). But the direct-run winner never notified the provider, so the NAS one-shot for the consumed occurrence was left stale forever and the recurring job silently stopped firing. Observed in production: a managed 1-minute review job stalled for 20 hours because a GitHub-webhook direct run claimed the job 2s before the Chronos fire arrived; every subsequent occurrence was orphaned while /api/status stayed green. Fix: after a *claimed* direct execution completes (success or failure — next_run_at advances at claim time either way), call _notify_provider_jobs_changed_safe() so the active provider re-arms the post-run next_run_at. No-op for the built-in ticker; claim-lost direct runs still never notify (the winning scheduler owns the re-arm). |
||
|
Ben Barclay
|
d8cd7ac2e7 |
fix(relay): declare explicit relevance policy when require_mention is configured false (#69816)
Companion to gateway-gateway#160 (connector absent-row default flips to mention-gated / requireAddress=true). relay_relevance_policy() previously returned None whenever the projected policy was all-falsy, on the premise that 'the connector's quiet default already matches'. Once the connector defaults requireAddress to true, that premise inverts for one case: an agent EXPLICITLY configured with require_mention: false would stay silent and get mention-gated anyway, with no way out. The nothing-to-declare check now keys on 'require_mention is unset' rather than 'require_mention is falsy': an explicit false IS a configured knob and is declared to the connector; a genuinely unconfigured platform still declares nothing and inherits the connector's default. No wire/contract change. |
||
|
Ben BarclayandVictor Kyriazakos
|
305a3c7424 |
fix(relay): restore streaming delivery, Slack command parity, and status clearing (salvage of #69716) (#69747)
* fix(gateway): restore relay streaming delivery * fix(relay): route Slack parent commands before session gates * fix(relay): clear Slack typing status after turns --------- Co-authored-by: Victor Kyriazakos <victor@rocketfueldev.com> |
||
|
Ben Barclay
|
e5b83633ec |
feat(relay): egress typing indicators through the connector (op="typing") (#69721)
The base class spawns _keep_typing for every adapter — a 2s refresh loop
that runs for the whole turn, including while the stream consumer sends
the response — but RelayAdapter inherited BasePlatformAdapter's no-op
send_typing. The loop ran every turn and emitted nothing, so relay-fronted
chats (hosted Discord/Telegram/Signal/Slack) never showed 'is typing…'.
The rest of the pipe already existed on both sides: OutboundOp "typing"
is in the wire contract (contract_version 1), and every connector-side
sender implements it (Discord POST /channels/{id}/typing, Telegram
sendChatAction, Signal sendTyping, Slack assistant status).
This bridges the tick onto the existing outbound frame, mirroring send():
- _with_scope re-attaches the tenant discriminator (metadata.scope_id, or
user_id for DMs) — the connector's routedEgressGuard wraps ALL ops, so
an undiscriminated typing frame is declined like a bare send would be.
- the Phase 1.5 per-frame platform tag routes typing through the platform
the chat lives on for multi-platform gateways.
Best-effort by design: transport errors are swallowed (typing is cosmetic;
the next tick retries), no transport is a silent no-op, and stop_typing
stays the base no-op (platform indicators self-expire). Additive within
contract_version 1 — an older connector returns an unsupported-op result
we ignore. Contract doc §4 updated (typing now carries metadata?).
|
||
|
Ben
|
5d747a91c4 |
fix(slack): humanize inbound user mentions + ground bot identity
Slack delivers user mentions as opaque IDs (<@U123>). The agent had no way to tell one participant from another — or from itself — so it could misread a mention of a human as a self-mention and answer messages addressed to that person (the "bot thinks it's @someone-else" bug). Two cooperating fixes: - _humanize_user_mentions rewrites remaining <@UID> tokens (the bot's own mention is stripped earlier) to @DisplayName in the trigger text and reply_to_text — the Slack equivalent of Discord's clean_content. Handles the labelled <@UID|handle> form; unresolvable IDs fall back to the raw ID. - _build_identity_prompt injects an ephemeral per-turn system-prompt line via the channel_prompt seam (applied at API-call time, never persisted — prompt caching preserved) naming the bot's own workspace handle (per-team in multi-workspace installs) so the agent has a positive "that's me" anchor. Salvaged from #55340 by @benbarclay, rebased over the workspace-scoped user-name cache (team_id-aware resolution) on main. |
||
|
Ben Barclay
|
49423f8084 |
fix(desktop-auth): make RFC 8252 native login work end-to-end at runtime
The native-app (RFC 8252) login passed its unit tests but failed in real Electron runtime — the tests mocked the exact seams that were broken. Four runtime defects, each proven against a live gated gateway: 1. Lockfile drift: apps/desktop declared @assistant-ui/react + @assistant-ui/react-streamdown but package-lock.json didn't place them, so `npm ci` (CI + every fresh checkout) failed to install them → Vite "Failed to resolve @assistant-ui/react". Reconcile the lockfile. 2. Double JSON encoding: postJsonNoAuth pre-JSON.stringify'd the body before fetchJson (which stringifies again), so /auth/native/token received a JSON string, not an object → gateway 422 "Input should be a valid dictionary" → native login silently fell back to the embedded webview. 3. Cookie-only liveness gate: buildRemoteConnection (and the Settings connected indicator) treated "signed in" as "has OAuth cookie". The native flow stores a bearer and sets no cookie, so a completed native login looped the UI into needsOauthLogin. Accept native token OR cookie. 4. Cookie-only REST path: the hermes:api handler routed oauth-mode REST through the cookie partition only. A cookieless native session → 401 no_cookie on every API call. Prefer the native bearer (with transparent refresh), else cookie — mirroring mintGatewayWsTicket, which was already bearer-aware. The three decision points (2–4) are extracted into a pure native-auth-decisions.ts with regression tests, since the mocked flow tests could not catch them. Verified live: system-browser login → cookieless bearer → connected chat, no embedded webview. |
||
|
Ben Barclay
|
7857d8737c |
feat(dashboard-auth): RFC 8252 native desktop sign-in (system browser + PKCE, no webview/cookies)
The Desktop app can now sign in to a gated gateway using the user's SYSTEM browser and OAuth 2.0 for Native Apps (RFC 8252) instead of an embedded Electron BrowserWindow, and authenticates with bearer tokens it holds itself instead of relying on HttpOnly browser session cookies. Why brokered: the upstream IDP (Nous Portal) binds client_id to the gateway instance and only permits redirect_uris on the gateway's own origin, so a desktop loopback redirect can't be a direct Portal client. The gateway therefore acts as the authorization server TO the desktop and an OAuth client TO the Portal, reusing the existing PKCE start_login/complete_login provider path unchanged. Server (Ben's dashboard-auth lane): - native_flow.py: in-memory broker — binds the desktop's PKCE challenge to a completed Session, mints a single-use, short-TTL, PKCE-verified gateway authorization code. Constant-time compare, single-use (consumed before the PKCE check so a wrong verifier can't be retried), capacity-bounded. - routes.py: GET /auth/native/authorize (starts the brokered PKCE login, loopback-only redirect_uri, S256-only), POST /auth/native/token (loopback code + verifier -> tokens in the JSON body, never Set-Cookie), POST /auth/native/refresh (desktop-held RT rotation). /auth/callback branches to mint a loopback code + 302 to 127.0.0.1 when a broker_state rides the PKCE cookie; the cookie/SPA path is untouched. - middleware.py: the gate accepts Authorization: Bearer <access_token>, verified via the same verify_session provider stack (no cookie set/read), with the same "provider unreachable -> 503, not logout" semantics. - web_server.py /api/status: advertise auth_flows (["cookie","native_pkce"]) so clients can detect the capability; native_pkce only when a brokerable OAuth provider is registered. Desktop (Ben's lane): - native-oauth.ts: pure PKCE/capability/URL/callback/token helpers. - native-oauth-login.ts: loopback-listener orchestration (system browser via openExternal, ephemeral 127.0.0.1 listener, state/PKCE verification), all I/O injected for testability. - main.ts: capability-gated oauth-login IPC — native flow when advertised, automatic fallback to the existing embedded-webview cookie flow otherwise; tokens stored encrypted (safeStorage/OS keychain), REST + ws-ticket authenticated by bearer, transparent refresh, logout clears both shapes. Tests: 18 server pytest (broker unit + full authorize->callback->token E2E + cookieless bearer auth of a gated route + ws-ticket mint + capability advertisement + refresh); desktop node --test/vitest for both pure modules (PKCE, capability detection, callback CSRF, loopback round trip, timeout, browser-open failure). Electron project typechecks clean. Docs: website/docs/guides/desktop-native-signin.md. |
||
|
Ben Barclay
|
f4df260f26 |
fix(relay): attach metadata.user_id on guild replies for egress fallback (#68320)
The relay adapter re-attaches an egress discriminator on outbound replies so the connector can resolve the owning tenant. It captured scope_id for scoped (guild) messages and user_id for DMs, but as MUTUALLY EXCLUSIVE: a scoped inbound hit an early return, so the author's user_id was never recorded, and _with_scope only attached user_id when there was no scope_id. Guild replies therefore went out with scope_id only. That's fine while the guild has a provision-time route row. But a MANAGED Discord agent joins guilds dynamically (the shared bot is added to / removed from servers at runtime), and GATEWAY_RELAY_ROUTE_KEYS — the only thing that writes guild route rows — is a self-hosted, static field never stamped for managed agents. So their guild has no route row, the connector's guild-route lookup misses, and with no user_id on the frame there's nothing to fall back to → every guild reply is declined "discord egress declined: target not routed to an onboarded tenant" even though INBOUND resolved the same guild fine (via the author-first SharedSocketRouter.targets() fallback). Fix: capture the authentic author user_id for EVERY inbound (DM and scoped alike) and re-attach it on the outbound reply alongside scope_id. The connector consults it only on a route/scope miss, so carrying both never overrides routing-table resolution. This is the gateway half of the paired gateway-gateway change (makeDiscordTenantOf guild-route-miss author-binding fallback); together they make guild replies resolve the same observed-author way inbound already does. Tests (tests/gateway/relay/test_relay_adapter.py): a guild reply now carries both scope_id AND user_id; a scoped inbound with no author still yields scope_id only (never invents one). Verified fail-without / pass-with. |
||
|
Ben Barclay
|
f0aae14c68 |
fix(desktop): retry OAuth cookie read on cold-start jar race (#67769)
A `persist:` partition's cookie store hydrates lazily, so the first cookies.get() on a fresh launch can return empty for a signed-in user. That false-negative made hasLiveOauthSession() throw "not signed in", which on the no-retry initial boot path surfaced as the transient "Hermes couldn't start" OAuth overlay that always cleared on Retry. hasLiveOauthSession now reads once (no added latency on the happy path); only on an empty read does it warm the store (flushStorageData + a throwaway get, memoized) and re-read with a bounded ~180ms backoff before trusting the negative. Genuinely signed-out users still resolve false quickly and get the overlay. Fixes the whole class: the same function backs the reconnect path and the Settings connected indicator. |
||
|
Ben Barclay
|
11eaa77daf | fix(mcp): serialize hosted oauth reauthorization | ||
|
Ben Barclay
|
b09f1ba770 | fix(mcp): reject invalid dashboard oauth callbacks | ||
|
Ben Barclay
|
05dea7be04 | fix(mcp): complete OAuth through hosted dashboards | ||
|
Ben Barclay
|
03885e0aa1 |
feat(skills): add mcp-oauth-remote-gateway skill
Add an optional skill for connecting OAuth-gated remote MCP servers (Better Stack, Linear, Cloudflare, Datadog, Stripe, etc.) when Hermes runs as a remote gateway, where the built-in browser OAuth flow cannot capture the 127.0.0.1 callback. Covers the manual RFC 7591 DCR + RFC 7636 PKCE + authorization_code flow, writing tokens in Hermes' HermesTokenStorage schema, the dashboard-first escalation path, and a diagnostic script + pitfalls for refresh/session-revocation recovery. |
||
|
Ben
|
d542894adf |
fix(webhook): default to dual-stack bind so 6PN (IPv6) can reach the adapter
The webhook adapter defaulted to host='0.0.0.0' — IPv4 only. On Fly.io hosted agents the edge router (hermes-agent-router) reverse-proxies public webhook traffic to <app>.internal:8644 over 6PN, Fly's private network, which is IPv6-only (.internal resolves to an fdaa:… address). An IPv4-only listener is unreachable there, so public webhook POSTs to https://<agent>.agents.nousresearch.com/webhooks/<route> never landed on the adapter — the router's dial was refused. Fix: DEFAULT_HOST = None, which makes aiohttp/asyncio create_server bind BOTH address families. '::' is NOT a valid substitute: on hosts where the kernel sets bindv6only=1 (verified on Fly machines) it yields an IPv6-only socket, breaking the IPv4 loopback /health check and the AF_INET port-conflict probe in connect(). None binds per-family regardless of the sysctl. An explicit empty-string/null host in config now also normalises to None (dual-stack) rather than an invalid host=''. Users can still pin a specific host via platforms.webhook.extra.host. Validated live on a Fly staging agent: with this default and no config override, the adapter binds both v4 and v6 (127.0.0.1:8644 and [::1]:8644 both answer), and a public signed webhook POST through the router returns 202 (valid sig) / 401 (bad sig) instead of the router's 502. Tests: new TestDualStackBind asserts the None default, config resolution (missing/empty→None, pinned preserved), and a real dual-stack bind opens both AF_INET and AF_INET6 listeners. Red-proof: these fail on the old '0.0.0.0' default. |
||
|
Ben Barclay
|
2ea39daeb1 | fix(gateway): share relay adapter in multiplex mode (#65366) | ||
|
Ben Barclay
|
6a35f9e667 | fix(container): keep named multiplex gateway slots down (#65368) | ||
|
Ben Barclay
|
3f2a389c7e |
fix(auth): apply newer hosted bootstrap session (#64612)
* fix(auth): apply newer hosted bootstrap session * fix(auth): validate rebootstrap replacement seeds |
||
|
Ben Barclay
|
5fc2d9e64d |
docs(auth): scrub Fly.io host detail from quarantine-log comment (#60145)
hermes-agent is public/OSS; the forensic-logging comment in
_quarantine_nous_oauth_state named 'Fly' (the specific managed-hosting compute
provider) twice. Reword generically ('a hosted agent', 'a managed log drain may
be WARNING-only') — the behaviour is unchanged, only the comment. Follows the
same scrub applied to the boot re-seed helper (#59983) before merge; this one
slipped through in #59976.
|
||
|
Ben Barclay
|
bb5fc723b6 |
feat(relay): consume channel context from the connector (#64649)
Phase 3 of relay-channel-context (gateway/agent side, single PR). The connector (gateway-gateway #122/#123/#124) now attaches read-only surrounding channel/group context to an addressed relay turn; this wires the gateway to consume it. - descriptor.py: additive optional supports_context (default False) on CapabilityDescriptor. from_json already filters unknown keys, so this is back-compat both directions within contract_version 1. - ws_transport.py: _event_from_wire maps the connector's read-only context[] array into the EXISTING MessageEvent.channel_context field via a new _render_relay_context() helper — reusing the same read-only injection path history-backfill uses (run.py prepends channel_context ahead of the trigger message). Never raises; absent/empty/malformed -> channel_context unset (byte-identical to today). - docs/relay-connector-contract.md: document supports_context in the §2 descriptor table (fixes the contract-doc conformance test) + the context/context_error inbound fields in §3. - tests: descriptor default/round-trip/forward-compat; _render_relay_context rendering + malformed-safe; _event_from_wire context->channel_context mapping + the read-only invariant (trigger text untouched). RELAY-ONLY: only gateway/relay/* + the shared MessageEvent consumption via its existing channel_context field. No native adapter touched. |
||
|
Ben Barclay
|
9884b4faad |
fix(cron/chronos): cache PyJWKClient across fires to stop JWKS fetch storm (#64641)
The inbound cron-fire verifier constructed a fresh PyJWKClient on every fire, discarding the client's key cache and forcing a synchronous JWKS HTTP GET to the portal on each fire. Under a burst of concurrent fires (a hosted instance with several cron jobs firing in the same window) this fanned out into N simultaneous JWKS fetches that the portal rate-limited (HTTP 403 -> verification fails -> agent 401), or that blocked the event loop long enough that the fire webhook could not return its 202 before the relay's 30s timeout (observed in prod as relay 504s concentrated on high-job-count instances). Cache one PyJWKClient per JWKS URL at module scope (double-checked lock) so the signing keys are reused across fires; NAS keys rotate rarely, so the steady state is zero JWKS fetches per fire. Regression test proves 5 fires -> 1 client construction (was 5). |
||
|
Ben
|
c101207b99 |
feat(desktop): Hermes Cloud connection mode — one sign-in, agent discovery, silent connect
Adds a third "Hermes Cloud" gateway mode to the desktop app: one portal
sign-in auto-discovers the agents on your account and connects to any of
them with no second interactive prompt.
- Electron: widen connection mode to 'local' | 'remote' | 'cloud', routed
through a centralized modeIsRemoteLike() so every resolution site treats
cloud exactly like remote; portal discovery (GET /api/agents over the
OAuth partition), Privy-cookie liveness, multi-org picker (NAS 409), and a
silent per-agent /oauth cascade (load protected root, not /login).
- Persist a cloudOrg on the cloud block; unselect cloud on mode switch.
- Renderer: Hermes Cloud ModeCard + agent picker (signed-out/loading/empty/
list), org picker, Change-org, connected-highlight + Connected pill.
- i18n (en + zh full; ja/zh-hant inherit via defineLocale), Cloud icon.
- IPC: hermes☁️{status,login,logout,discover,agent-sign-in}.
Salvage of #55402 onto current main: the original branch predates the
desktop electron .cjs -> .ts migration (
|
||
|
Ben Barclay
|
f64e4f4f57 |
feat(gateway): generic OIDC client-credentials relay provisioning (NAS-free) (#60730)
For air-gapped / self-hosted-IdP deploys with NO Nous Portal, let the gateway
obtain its caller-identity bearer from a generic OAuth2 client_credentials grant
against the operator's own IdP (e.g. Microsoft Entra ID) instead of only
resolve_nous_access_token(). The connector's OIDC tenant resolver reads a claim
(default tid) off that token as the tenant.
- gateway/relay: new canonical _resolve_relay_identity_token() — client_credentials
when gateway.idp.token_url (or GATEWAY_RELAY_IDP_* env) is set, else Nous Portal
(unchanged default). Wired into self_provision_relay().
- hermes_cli/gateway_enroll: _resolve_identity_token() delegates to the canonical
resolver so the enroll CLI and the runtime self-provision path share ONE impl.
Config via gateway.idp.{token_url,client_id,client_secret,scope} in config.yaml
(env override GATEWAY_RELAY_IDP_*). No behaviour change when unset.
Tests: tests/gateway/relay/test_identity_token_resolver.py (6 — mode selection,
request shape, config/env precedence, fail-closed). Relay suite 162 pass.
Validated via the cross-repo gateway<->connector live E2E (provision, managed
self-provision, inbound round-trip, /link) against a connector running the OIDC
tenant resolver with zero NAS config.
|
||
|
Ben Barclay
|
75de0057bc |
feat(gateway): GATEWAY_MULTIPLEX_PROFILES env override for multiplex flag (#60589)
The connector now depends on the single multiplexed gateway for per-profile
relay routing, so hosted deployments need to FORCE multiplexing on regardless
of the image's config.yaml. gateway.multiplex_profiles was config.yaml-only,
which a user could leave unset or flip off.
Add GATEWAY_MULTIPLEX_PROFILES as a standard operator override on top of the
existing config key — the same 'config.yaml is canonical, env is the operator
override' pattern the Telegram/Signal require_mention bridges use:
env (recognized token) > config.yaml (top-level or nested gateway.*) > False
- gateway/config.py: _env_multiplex_profiles_override() resolves the env var
tri-state — recognized truthy/falsy token → bool; unset/blank/unrecognized
→ None (fall through to config). Blank is deliberately None, not False, so a
provisioned-but-unpopulated Fly secret ('') can't shadow a config.yaml opt-in
(the empty-secret trap). Wired into GatewayConfig.from_dict so every consumer
(run.py, session.py via self.config) sees the resolved value.
- hermes_cli/gateway.py: the named-profile-start guard
(_guard_named_profile_under_multiplexer) reads config.yaml directly, so it
gets the SAME env precedence — otherwise env-forced multiplex would leave the
guard blind and someone could start a conflicting per-profile gateway that
double-binds a bot token. Env-forced-on trips the guard even with no
config.yaml key; env-forced-off disables it over a config opt-in.
Tests: full 3-tier precedence in test_config.py (incl. the discriminating
env-overrides-config cases + the empty/whitespace/unrecognized fall-through
trap + resolver tri-state), mutation-verified (flipping precedence fails
exactly the two env-wins tests); guard env cases in test_multiplex_lifecycle.py.
Force-on is safe on a single-profile instance: session keys stay byte-identical
(agent:main) and the _run_agent wrapper installs the per-turn secret scope, so
the fail-closed get_secret() path is satisfied.
|
||
|
Ben Barclay
|
4e4a69cbf7 |
feat(relay): carry routed profile from the connector wire source (#60586)
The multiplex machinery already routes an inbound message to a profile via SessionSource.profile (build_session_key namespacing + the per-turn config/credential scope in SessionStore._resolve_profile_for_key). But the relay path never populated it: _event_from_wire rebuilt the SessionSource field-by-field and dropped any 'profile' the connector sent, so a Team-Gateway (connector + relay) message could not be routed to a specific profile the way the /p/<profile>/ HTTP prefix and per-credential polling adapters already can. Stamp source.profile from the wire payload in _event_from_wire. This is the last missing link for NAS-driven per-profile routing over the relay in multiplex mode; the connector populating the field ships separately (gateway-gateway contract adds the optional wire field). Back-compat: absent 'profile' → None → legacy agent:main namespace, byte-identical to today for every single-profile gateway. |
||
|
Ben Barclay
|
8d66e78844 |
feat(dashboard): expose profile names + gateway_mode on gated /api/status (#60585)
The profile+gateway topology added in #60537 sits entirely behind the loopback/--insecure auth gate. But a hosted agent (Hermes Cloud) binds non-loopback with OAuth, so should_require_auth is True, and NAS reads /api/status over the network (fly-provider.ts getInstanceRuntimeStatus) with no session token. On that gated path the whole topology block was omitted, so the Portal could never render the profile list. Split the topology readout by sensitivity: - profile NAMES (profiles) + gateway_mode are low-sensitivity product surface and now ride the always-public status body, surviving the auth gate so NAS/the Portal can enumerate profiles. - the per-gateway detail (gateways[], carrying host ports) is deployment recon and stays gated alongside hermes_home / config_path / env_path / gateway_pid / gateway_health_url. The collector now runs unconditionally (still in the executor, off the event loop). No new fields; only the gate placement changes. |
||
|
Ben BarclayandSquabbyZ
|
4b9d9b205b |
fix(dashboard): use loopback host for in-container WebSocket client (#58993) [salvage #59682] (#60092)
* fix(dashboard): use loopback host for in-container WebSocket client (#58993) Fixes #58993 - the in-container Dashboard's WebSocket client was dialing the bind host (0.0.0.0) instead of 127.0.0.1, hijacking the host browser when the container port was exposed. * `hermes_cli/web_server.py::resolve_dashboard_ws_url()` now substitutes 127.0.0.1 for any 0.0.0.0 bind host discovered via the existing `find_unused_port` / `get_listen_address` path. LAN IPs and explicit `DASHBOARD_WS_HOST` overrides pass through unchanged. * Existing tests preserved (no regression on the explicit-bind case). Tests in `tests/dashboard/test_ws_client_host.py` cover: - Bind host 0.0.0.0 → ws URL uses 127.0.0.1 - Bind host 127.0.0.1 → ws URL uses 127.0.0.1 (no regression) - Bind host 192.168.1.5 → ws URL preserves the LAN IP - DASHBOARD_WS_HOST env override wins over auto-detection AI-assisted fix by https://github.com/SquabbyZ/peaks-loop (cherry picked from commit 5501dd38d660619729ebcad85b75a8f46f8deb38) * chore(release): map SquabbyZ email for AUTHOR_MAP attribution (#59682) --------- Co-authored-by: SquabbyZ <601709253@qq.com> |
||
|
Ben
|
5eac665252 |
feat(status): expose nous_session_valid on /api/status for hosted-agent self-heal
A hosted agent whose Nous bootstrap session dies terminally (invalid_grant / quarantine) looks HEALTHY to every liveness/connectivity probe — the machine, relay ws, and dashboard all stay up — yet every inference turn hard-fails with a provider-auth error until a human re-logs-in. Nothing currently surfaces that condition to NAS. Add get_nous_session_validity() (valid|terminal|unknown), classified from local auth-store state (no working token required), and report it on the public /api/status payload. NAS's 2-min health sweep reads it and re-mints the bootstrap session in place on 'terminal'. Anti-flap: only a terminal failure (relogin_required / persisted quarantine marker with tokens cleared) maps to 'terminal'; transient/mid-rotation blips and merely-expiring tokens report 'unknown' so a healthy box never triggers a spurious re-mint. Part of the hosted-agent bootstrap-session self-heal (NAS side reads this field). |
||
|
Ben
|
182256206a |
test: drop worktree-path sanity guard that fails in CI
The test_module_resolves_to_this_worktree guard asserted auth.__file__ contained 'worktrees/bootstrap-h2-logging' — a local dev crutch to defeat the editable- install trap (venv points at the main checkout). In CI the code lives at /home/runner/work/... so the assertion always fails. It never belonged in the committed suite; the 5 behavioural tests are what matter. |
||
|
Ben
|
444dc0da89 |
feat(auth): log forensic detail at Nous quarantine so terminal auth death is visible
A NAS-hosted Fly agent's Nous bootstrap session can take a terminal invalid_grant and get quarantined in _quarantine_nous_oauth_state, which clears the dead tokens from auth.json. Until now this quarantine was completely silent: the only signal was a downstream "No access token found" WARNING once the credential pool was already empty, which is too late to root-cause. Because the Fly log drain is WARNING-only, nothing about the terminal death reached centralized logging, and a real incident could not be diagnosed because the evidence was never recorded. Emit a WARNING+ forensic record AT the quarantine point, before the token material is cleared. Fields: refresh_token hash prefix (12-char SHA-256 hex, correlates to NAS's refreshTokenHash), client_id, agent_key_id, error code, reason, auth.json path/size/mtime/exists, and whether the token was already past its own expiry. WARNING level is deliberate — INFO never reaches the Fly drain. Redaction safety (load-bearing): the log dict is built only from computed values (hash prefix, sizes, booleans). No raw refresh_token, access_token, or agent_key bytes are ever passed into the log call, avoiding Hermes's known credential-literal corruption bug class. A test asserts the raw refresh token substring is absent from all emitted log output. Note: no session_id field exists on Nous auth state; provenance is captured via client_id + agent_key_id, which are non-secret routing identifiers. |
||
|
Ben Barclay
|
536ffedbf4 |
feat(docker): re-seed a terminally-dead Nous bootstrap session on boot (#59983)
The stage2-hook auth.json seed is first-boot-only ([ ! -f auth.json ]) to avoid clobbering rotated refresh tokens on restart. That guard means a container whose Nous bootstrap session took a terminal invalid_grant (tokens cleared, providers.nous.last_auth_error.relogin_required stamped) cannot recover from a restart — it stays unauthenticated until the credential is replaced. Add a self-heal path: an orchestrator that manages the container supplies a freshly-issued session via HERMES_AUTH_JSON_REBOOTSTRAP (distinct from the create-only *_BOOTSTRAP var). On boot, scripts/docker_rebootstrap_nous_session.py swaps ONLY the providers.nous entry, and ONLY when the on-disk entry is provably terminal (quarantine marker + no usable tokens). Healthy/rotating/absent/ unparseable auth.json is always a no-op, so the env is safe to leave set across restarts and never clobbers a good token. Pure stdlib, runs as its own subprocess, always exits 0 so a re-seed error never fails the boot. Reuses the same terminal predicate as get_nous_session_validity() so we re-seed only a session that is genuinely dead. |
||
|
Ben
|
590a19332e |
fix(skills): don't request Brotli for the centralized skills index
The Skills Hub 'Browse Hub' landing page and index-backed search render
empty on fresh deployments (e.g. Fly.io VPS agents) with no stale cache.
Root cause: the centralized index at /docs/api/skills-index.json is a
large body (~34MB, tens of MB compressed) served with Content-Encoding:
br. httpx's streaming Brotli decoder — backed by brotlicffi 1.2.0.1,
which is pinned so aiohttp can decode Discord attachments — trips over
its own output_buffer_limit on a payload this size and raises:
DecodingError("brotli: decoder process called with data when
'can_accept_more_data()' is False")
_load_hermes_index() catches that (DecodingError is an httpx.HTTPError
subclass) and silently falls back to the on-disk cache. On a fresh box
that cache never existed, so HermesIndexSource.is_available is False,
the index contributes 0 skills, and the hub landing page — which is
built solely from an empty-query index search — is blank. Existing
installs only appear to work because they serve a (possibly weeks-)stale
cached index instead.
Fix: request 'gzip, deflate' on the index fetch so httpx never
negotiates the broken Brotli path, and retry once with 'identity' if a
DecodingError still occurs (defends against a proxy that ignores the
header). Falls through to the stale cache only when both attempts fail.
Verified on a live staging VPS agent: index_available flips False->True
and the featured landing list repopulates from 0 to 12.
Also un-freezes already-deployed images: skills added after an image was
built (e.g. the 'unbroker' optional skill) become reachable again via
the index, which is the whole point of the centralized catalog.
|
||
|
Ben Barclay
|
9be39de0f2 |
fix(auth): make HERMES_PORTAL_BASE_URL/NOUS_PORTAL_BASE_URL bypass the Portal host allowlist (#56864)
Ben caught that the initial approach (widening _NOUS_PORTAL_ALLOWED_HOSTS to
include the staging host) was the wrong fix -- env vars are supposed to
override the allowlist, mirroring how NOUS_INFERENCE_BASE_URL already
bypasses _ALLOWED_NOUS_INFERENCE_HOSTS via _nous_inference_env_override().
The actual bug: both resolve_nous_access_token and
resolve_nous_runtime_credentials read
`_optional_base_url(state.get("portal_base_url")) or os.getenv(...) or ...`
-- a plain `or` chain where the STORED state value wins first (short-circuits
before the env vars are even read), and then whichever value won gets run
through the same _NOUS_PORTAL_ALLOWED_HOSTS gate regardless of its source.
So a hosted agent stamped with HERMES_PORTAL_BASE_URL=<staging> in its env
AND a staging portal_base_url already persisted to auth.json would still
get silently rewritten to prod on every refresh, because the env var never
even got a chance to be consulted.
Revert the previous _NOUS_PORTAL_ALLOWED_HOSTS widening entirely --
staying prod-only preserves the allowlist's actual job (rejecting an
untrusted network-provided portal_base_url persisted to auth.json by a
compromised Portal response).
Add _nous_portal_env_override() (mirrors _nous_inference_env_override())
and restructure both call sites so the env override is checked FIRST and,
when set, wins outright and skips the allowlist gate entirely -- the
allowlist only ever runs against the fallback (stored-state-or-default)
path now.
Rewrote tests/hermes_cli/test_nous_portal_staging_allowlist.py to test the
actual fix: the helper function, and an end-to-end
resolve_nous_access_token proof that the env override wins even when state
ALSO has the staging host stored (the exact incident shape), that it wins
over a stored PROD host too, and that the allowlist's heal-to-prod
behaviour for an untrusted stored value is preserved when no override is
set.
|
||
|
Ben
|
751a300fca |
docs(cron): scope in_channel to channels; document DM continuation knob
Live DM testing showed a reply to a DM cron brief did NOT continue the job. Root cause: for a 1:1 DM the governing knob is dm_top_level_threads_as_sessions (default True), NOT reply_in_thread / cron_continuable_surface. Under the default, each top-level DM keys to a per-message session (…:dm:<chat>:<ts>), so a reply mints a new ts and can never converge with the flat …:dm:<chat> session the cron seed creates. A 1:1 DM has no thread-vs-timeline split, so "in_channel" has no coherent meaning for a DM — cron_continuable_surface is a channel concept and is a no-op for DMs. DM continuation is governed entirely by dm_top_level_threads_as_sessions: - false → all top-level DMs share …:dm:<chat> → seed + reply converge → works - true (default) → per-message sessions → no continuation (cron or interactive) Option A (chosen): document the requirement; no code change (the flat-DM seed from the prior commit already lands correctly when the knob is false). Adds a ":::note 1:1 DMs" admonition to cron.md + the zh-Hans mirror. Verification (real inbound handler, not a hard-coded assumption — the mistake that made the earlier DM E2E falsely pass): tests/manual/cron_inchannel_dm_e2e.py drives the REAL _handle_slack_message for a top-level DM under both knob values and asserts false→converges (…:dm:D_TESTDM == seed), true→diverges (…:dm:D_TESTDM:<ts>). See decisions.md D9. |
||
|
Ben
|
2c84fb42b0 |
fix(cron/slack): CREATE the flat session for in_channel (mirror only appends)
Live testing exposed a real bug: an in_channel continuable cron delivered
flat to the channel (✅) but the reply did NOT continue the job — the bot
had no brief in context and confabulated the answer.
Root cause: mirror_to_session only APPENDS to a session that already
exists (_find_session_id → no-op when none matches); it never CREATEs one.
A flat (slack, chat_id, None) row is only created when a human posts a
top-level message the bot processes — a cron chat_postMessage delivery
never goes through the inbound handler, so the row is absent and the brief
is silently dropped. The prior impl relied on the bare mirror (F5/OQ-1
concluded "deletion only" — wrong).
Fix: _seed_cron_channel_session mirrors _seed_cron_thread_session —
get_or_create_session FIRST (chat_type = "dm" if is_dm else "group",
thread_id=None), keyed to the ORIGIN USER'S id, then mirror. The channel
session key embeds user_id (…:group:<chat>:<user>), so a system:cron id
would key the seed away from the reply; the origin user's id makes seed
key == inbound reply key. DM key ignores user_id but needs chat_type=dm
to match the prefix. Wired into the in_channel branch after delivery;
suppresses the generic mirror to avoid double-write.
DM validated (per request): the seeded key equals the inbound DM reply key
for a 1:1 DM; continuation works there too.
Tests:
- Rewrote the in_channel tests to use a real _session_store and the origin
user_id; assert get_or_create_session is called with the flat, correctly-
keyed source. Prove-fail: (a) reverting the create step and (b) seeding
with system:cron each turn a targeted test RED; restore → GREEN.
- +2 direct _seed_cron_channel_session unit tests asserting the KEY-MATCH
invariant (seed key == inbound reply key) via build_session_key, for both
channel and DM.
- Rewrote tests/manual/cron_inchannel_e2e.py to drive a REAL SessionStore +
real mirror_to_session + real _find_session_id + real build_session_key
(no session-layer mocks — the old mocked E2E is exactly why the bug
shipped). Asserts the brief lands in the transcript and the reply resolves
to the same session, for BOTH channel and 1:1 DM.
Full relevant sweep: 283 passed.
|
||
|
Ben
|
4b4349eb9a |
feat(cron/slack): flat in-channel continuable cron delivery surface
Add a per-platform `cron_continuable_surface` extra key (`thread` default | `in_channel`) so a continuable cron job can deliver FLAT into a Slack channel — no dedicated thread — and still be replied-to. In `in_channel` mode the scheduler skips the thread-open branch (leaves `thread_id=None`); the shipped origin-mirror then seeds the `(slack, chat_id, None)` shared-channel session — the same bucket `reply_in_thread: false` routes inbound channel replies to — so a plain channel reply continues the job in context. Design: specs/cron-inchannel-continuable (D1–D7, F5). Model B (shared-channel session), NOT anchoring to the delivery `ts` — on Slack replying to a specific message IS threading, so a `ts` anchor would only relocate the thread, never deliver true threadless continuable. - gateway/platforms/base.py: `supports_inchannel_continuable` capability flag (default False → unsupported platforms fail SAFE to `thread`). - plugins/platforms/slack/adapter.py: flag=True; `_cron_continuable_surface()` resolver (coerces to the two-value enum); `_warn_if_inchannel_without_flat_reply` connect-time warning (D5: warn, not hard-require — the misconfig fails safe). - gateway/config.py: shared-key bridge line (top-level OR nested config). - cron/scheduler.py: read the key generically from platform config, gate the `in_channel` branch on the adapter capability flag, skip thread-open. No new seed function (reuses the existing mirror — G6). Pairing (docs): `in_channel` + `reply_in_thread: false` + `require_mention: false` (or a free-response channel). Missing `reply_in_thread: false` fails safe to a threaded continuation. Gateway-side config flag — `/restart` to apply; NO Slack app reinstall. Tests (from inside the worktree, PYTHONPATH=$PWD): - +6 cron scheduler tests (in_channel skips thread-open; seeds flat channel session with thread_id=None; thread-mode regression; fail-safe on unsupported platform; value coercion). Prove-fail: removing the `and not in_channel_surface` guard turns the two load-bearing tests RED; restore → GREEN. - +10 slack resolver/capability/warning tests; +2 config-bridge tests. - tests/manual/cron_inchannel_e2e.py: offline E2E driving BOTH real legs (delivery seed + inbound reply keying) → both converge on (slack, C, None). - No regressions: test_slack.py 216 passed alone; broader sweep green (4 pre-existing cross-file-ordering failures reproduce identically on pristine origin/main). Docs: cron.md + slack.md + zh-Hans mirrors of both. |
||
|
Ben Barclay
|
c71f816956 |
fix(compression): clear all per-session state in on_session_end, not just _previous_summary
The original cross-session contamination fix (#38788) only cleared _previous_summary in on_session_end(), but on_session_reset() clears 14+ per-session variables. When a session ends (cron exit, gateway expiry, session-id rotation) and the compressor instance is reused, the surviving stale state causes: - _ineffective_compression_count surviving → next session skips compression prematurely (anti-thrashing guard misfires) - _summary_failure_cooldown_until surviving → next session blocks summary generation for an unrelated transient error - _last_compress_aborted surviving → callers think compression is still aborted - _last_aux_model_failure_* surviving → stale error warnings shown - _last_summary_dropped_count / _last_summary_fallback_used surviving → misleading user warnings - _context_probed / _context_probe_persistable surviving → stale context-probe state Also fix on_session_reset() which was missing _last_compress_aborted clearing — a /new or /reset would inherit the aborted flag from the prior conversation. Add 6 targeted tests covering the leak vectors and a parity test ensuring on_session_end and on_session_reset always clear the same surface. |
||
|
Ben
|
e71f9ad0bb |
fix(tui): close busy-flag race that stuck queue-mode back-to-back sends
Under display.busy_input_mode: queue, sending two messages back-to-back
hung the session on 'Analyzing…' until a manual Ctrl+C.
The submit path only marked the session busy inside the .then of an
async input.detect_drop RPC. dispatchSubmission routes queue-vs-send on
getUiState().busy, so a second Enter inside that RPC window read
busy===false and raced a second prompt.submit down the send path
instead of enqueuing locally. The gateway accepts the mid-turn submit
as a success ({status:'queued'}, not an error), and the client's only
re-queue recovery is gated on catching a 'session busy' error — which
never fires — so the message became invisible to the client-side drain
effect and the UI stayed busy forever.
Extract the ready-prompt submit into a pure submissionCore module and
mark the session busy synchronously at the choke point, before the
detect_drop round-trip, closing the gap for every caller (mainline
submit, queue-edit picks, drain, interpolation). Verified the real
gateway already queues+drains both turns correctly, so the fix is
purely client-side. Adds submissionCore.test.ts whose regression
assertions fail without the synchronous busy and pass with it.
|
||
|
Ben
|
7c7b489813 |
feat(slack): render markdown tables as native Block Kit table blocks
Replace the interim monospace table fallback with Slack's native `table` block (rows of rich_text cells). Addresses the core ask in #18918. - _table_block(): builds type:"table" with rich_text cells, so inline formatting (bold, links, code) renders inside cells. - Column alignment parsed from the markdown separator row (:---, :-:, --:) into column_settings (left = default/null-skip, center/right emitted). - Escaped pipes (\\|) are not treated as column separators. - Respects Slack's table limits (100 rows / 20 cols / 10k aggregate chars); oversized or unparseable tables gracefully fall back to aligned monospace (rich_text_preformatted), so a big table never breaks the message. Docs (EN + zh-Hans) updated to describe native tables + the fallback. Tests: native table shape, alignment->column_settings, inline-formatted cells, oversized/too-wide monospace fallback, escaped-pipe cell. Prove- failed against a stubbed _table_block (native-table tests fail, fallback tests stay green). All existing Slack tests still pass. |
||
|
Ben
|
b080b93ad8 |
feat(slack): opt-in Block Kit rendering for agent messages
Add platforms.slack.extra.rich_blocks (default off). When enabled, the final agent message is sent as Slack Block Kit blocks — section headers, dividers, and true nested lists via rich_text — instead of flat mrkdwn. - New plugins/platforms/slack/block_kit.py: pure markdown->blocks renderer (headers, dividers, nested ordered/bullet lists, blockquotes, fenced code; pipe-tables as aligned monospace since Block Kit has no robust table block). Enforces Slack's 50-block / 3000-char section limits and returns None to fall back to plain text on empty/oversized/unexpected input. Never raises. - adapter.send(): render blocks on the single-chunk primary message; a text= fallback is ALWAYS sent alongside (notifications/accessibility). - adapter.edit_message(): blocks only on finalize=True, so intermediate streaming edits stay plain mrkdwn (no per-flush block re-derivation). - Docs (EN + zh-Hans) + config example. Send-side only: no app reinstall. Tests: pure-renderer unit suite + adapter integration suite (blocks present when on, plain text when off, text fallback always set, finalize gating, multi-chunk fallback). Prove-failed against a stubbed renderer. |
||
|
Ben
|
5f7deeba84 |
fix(gateway): suppress NO_REPLY/[SILENT] markers on the streaming path
The agent emits a bare control marker (NO_REPLY / [SILENT] / …) when it
intentionally chooses not to reply. The gateway's whole-response filter
(is_intentional_silence_agent_result) suppresses this on the non-streaming
delivery path, but the streaming path (GatewayStreamConsumer) had no silence
awareness: it edited the raw marker onto the screen delta-by-delta and
finalized it BEFORE the whole-response filter could run. On any
streaming-capable adapter (Slack, Telegram, Discord, …) users saw a literal
'NO_REPLY' message leak into chat.
Fix (contained in the stream consumer + a shared predicate; no new config,
no platform-specific code):
- gateway/response_filters.py: add is_partial_silence_marker() — the
streaming counterpart to is_intentional_silence_response(), sharing the
same marker set and canonicalization so the two never drift.
- gateway/stream_consumer.py:
- Mid-stream hold-back: defer edits while the accumulated buffer is still a
prefix of a silence marker, so a partial marker never flashes on an
interval tick.
- On stream end (got_done): if the final buffer is exactly a marker, retract
any preview already shown (best-effort delete_message, reusing the
_try_fresh_final cleanup path) and leave the delivery flags False so the
gateway's own filter turns the marker into '' and no fallback send fires.
Substantive prose that merely mentions a marker is still delivered normally.
Tests: tests/gateway/test_stream_consumer_silence.py — predicate truth table
+ end-to-end run() suppression (single-shot + token-by-token), preview
retraction, no-delete-support best-effort, [SILENT] parity, and
prose-passthrough. Prove-fail verified by reverting only the consumer change
(the 4 behavioral tests fail: 'NO_REPLY'/'[SILENT]' leaks).
|
||
|
Ben Barclay
|
729bbb7a30 |
refactor(relay): purge platform-specific scope terminology from the relay adapter (D-Q2.5c) (#56016)
The gateway HALF of the D-Q2.5c cleanup (connector half: gateway-gateway #92). Scope is STRICTLY the relay adapter (gateway/relay/) — session.py and every native platform adapter are untouched (SessionSource.guild_id remains for their use; it is NOT relay-only). Within gateway/relay/, drop the D-Q2.5 wire dual-write/dual-read alias AND genericize all platform-specific (Discord "guild") scope terminology: - ws_transport._event_from_wire: read scope_id only (drop the ?? guild_id fallback). - adapter._with_scope: emit scope_id only on outbound metadata (drop the guild_id dual-write); genericize the "GUILD reply" docstring to "SCOPED reply". - adapter._capture_scope: read source.scope_id only; rename the local `guild` var to `scope`; genericize the docstring + the _scope_by_chat/_dm_user_by_chat field comments ("guild_id (Discord)" -> "scope_id (server/workspace scope)"). - __init__.relay_route_keys docstring: "guild_ids" -> "scope_ids". - The ONE real Discord `guild_id` kept: the raw inbound interaction payload field (payload.get("guild_id")), which is Discord's own wire field, mapped straight into the generic scope_id slot — unchanged. Contract doc (docs/relay-connector-contract.md): reframe the `guild_id` row as a legacy alias the connector no longer reads (session.py's agent-wide to_dict() still emits it for non-relay persistence, so it stays documented + wire-present but ignored) — accurate, and keeps the to_dict()-vs-doc conformance test green. Tests (relay only): migrate the wire-key writes + assertions guild_id -> scope_id across test_relay_adapter / _ws_transport / _passthrough / _roundtrip / _roundtrip_telegram / _multiplatform; keep raw Discord `type:2` interaction payloads' guild_id (real Discord field) and the conformance test's guild_id parametrize (validates the kept legacy field stays wire-reachable). Gate: 156 relay tests pass, ruff clean. Cross-repo E2E — all 14 drivers pass BOTH ways: connector#92 (scope_id-only) x agent-main (still dual-reads) AND connector#92 x this worktree (scope_id-only). Deploy-order-safe either way. |
||
|
Ben
|
98d550e035 |
feat(debug): support /debug [nous|local] in the CLI/TUI slash command
The --nous flag was only wired into the argparse `hermes debug share` subcommand. The /debug slash command (classic CLI + TUI, both via process_command -> _handle_debug_command) built a hardcoded args namespace with no `nous` attribute, so it always took the default paste.rs path. Pass cmd_original through to _handle_debug_command and parse an optional destination word: /debug -> public paste (default, unchanged) /debug nous -> Nous-internal S3 /debug local -> stdout, no upload local wins over nous (never touches the network); unknown words fall back to the default. Add args_hint="[nous|local]" so help/autocomplete surface it. New TestDebugSlashCommand covers the parsing + dispatch. |
||
|
Ben
|
89653db403 |
feat(debug): drop dead confirm step from --nous upload (stateless NAS)
NAS PR #349 (merged) ships a stateless presigned-PUT endpoint: the only route is POST /api/diagnostics/upload-url, and the object's existence in S3 is the only state. There is no /api/diagnostics/confirm route — confirming live against the merged preview returns 404. The client's confirm_upload() therefore fired a guaranteed-404 request on every --nous upload (harmless, since errors were swallowed, but dead). Remove it and simplify share_to_nous() to the 2-step mint + PUT flow that matches the shipped contract. Drop the corresponding TestConfirmUpload class and confirm assertions; add a test that the share succeeds even when the response carries no id (we no longer depend on it). The separately-flagged cross-repo requirement from #349's review -- sizeBytes is now REQUIRED and signed into the presigned URL's ContentLength -- was already satisfied: share_to_nous() sends len(bundle) as sizeBytes and urllib sets a matching Content-Length on the PUT. Verified against the live merged preview (missing sizeBytes -> 400 invalid_body; present -> 503 dark). Tested: pytest tests/hermes_cli/test_diagnostics_upload.py tests/hermes_cli/test_debug.py -> 95 passed. |
||
|
Ben Barclay
|
51eeb70cb8 |
feat(debug): add --nous flag to upload diagnostics to Nous S3
`hermes debug share --nous` uploads the (force-redacted) debug bundle to Nous-internal S3 storage via a presigned URL minted by the Nous account service, instead of a public paste. The bundle is private — viewable only by Nous staff / allowlisted mods through a Google-OAuth-gated viewer — and auto-deletes after 14 days. The paste.rs path is unchanged and remains the default. - hermes_cli/diagnostics_upload.py (new): stdlib-urllib NAS client — request_upload_url(), put_bundle(), confirm_upload() (best-effort), share_to_nous() orchestrator. Base URL via HERMES_DIAGNOSTICS_BASE_URL (default https://portal.nousresearch.com). - hermes_cli/debug.py: extract collect_share_bundle() from build_debug_share() so the Nous path reuses the exact same redaction/collection (paste.rs behaviour unchanged); add build_nous_bundle() producing the gzipped {"format":"hermes-debug-share/1","redacted":...,"files":...} envelope the discord-support viewer parses; add the --nous run path with a privacy notice and a clean fallback (suggest --local) on failure. - hermes_cli/main.py: add the --nous flag + help/epilog entry on `debug share`. - tests: test_diagnostics_upload.py (new) mocks urllib; test_debug.py adds bundle/Nous coverage. 97 passing. |
||
|
Ben Barclay
|
05ac16778b |
feat(gateway): per-platform typing_indicator toggle
Add a generic per-platform PlatformConfig.typing_indicator flag (default True) that gates the _keep_typing refresh loop in _process_message_background. When false, the loop is never spawned, so no typing/"is thinking…" status is shown on that platform — message delivery is otherwise unchanged. Mirrors the gateway_restart_notification contract exactly: dataclass field + to_dict/from_dict (with extra-fallback resolution) + shared-key bridge in load_gateway_config, so 'slack: typing_indicator: false' under platforms works without a separate block. Generic by design — the same key works for every platform (Slack 'is thinking…', Telegram/Discord/Signal typing). Motivated by users who find Slack's assistant 'is thinking…' status noisy (it also briefly disables the compose box, via the Assistant API). |
||
|
Ben
|
d3f2931b8c |
fix(container-boot): autostart a gateway stranded in 'draining' state
A gateway hard-killed while draining (a container/VM recreate SIGTERMs it before _stop_impl reaches its terminal-state persist) leaves gateway_state.json frozen at 'draining'. With no explicit desired_state to fall back to, container_boot read that transient value literally, found it not in _AUTOSTART_STATES, and left the gateway DOWN on every subsequent boot — dashboard up, messaging silently dark. Observed on a relay-opted-in staging instance (2026-06): the s6 gateway-default slot kept its 'down' marker across recreates and the gateway never came back. 'draining' is a transient sub-state of RUNNING (written by the drain watcher / scale-to-zero go-dormant path), never an operator stop and never a failed boot. Normalise it to 'running' in the gateway_state fallback so a stranded drain marker reads as the run-intent it represents. This extends gateway/run.py's #42675 handling (persist 'running' on an unexpected signal) to the case where the gateway died before persisting anything at all. 'starting'/'startup_failed' are deliberately NOT normalised — those mean a mid-boot death and must stay down to avoid the crash-loop the down-marker guard prevents. An explicit desired_state still wins verbatim, so an operator stop survives a transient 'draining' runtime value. Tests: draining named-profile + default-root autostart (both fail without the fix), plus a guard that an explicit desired_state=stopped still blocks a draining runtime. |
||
|
Ben
|
184c10cf97 |
fix(slack): warn when configured token is a user token, not a bot token
A Slack user/legacy token (xoxp-...) makes auth.test resolve to the installing human's member ID with no bot_id, so the adapter binds its identity (_bot_user_id / _team_bot_user_ids) to that human. Every "is this the bot?" check then misfires: that person's <@...> mentions wake the bot and are stripped as the bot's own mention, so the agent is genuinely told it was @mentioned and replies to messages merely addressed to that human (symptom: bot responds to "@trevor ..." and insists it was explicitly mentioned). There is no runtime API error to catch — a user token still sends/receives — so the only detectable moment is connect time. Add a warning-only nudge (_warn_if_not_bot_token) alongside the existing group-DM scope nudge: when auth.test resolves a user_id but no bot_id, log that the token is a user token and to use the xoxb-... Bot User OAuth Token. Warning-only: does not block a working-but-misconfigured install. Fires once per workspace per process. |
||
|
Ben Barclay
|
53a75f147f |
feat(dashboard_auth): support confidential clients (client_secret) in self-hosted OIDC (#55344)
The self-hosted OIDC dashboard provider was public-client + PKCE only, with two `# TODO(confidential-client)` seams. Authentik and Keycloak commonly default a new OIDC client to *confidential*, whose token endpoint rejects an unauthenticated exchange (`invalid_client`) — so a self-hoster who accepts their IDP's default could not complete dashboard login without manually flipping the client to public. Add optional confidential-client support: - New optional `client_secret` (env `HERMES_DASHBOARD_OIDC_CLIENT_SECRET`, or `dashboard.oauth.self_hosted.client_secret`; env-wins-config, empty treated as unset). It is a credential, so docs steer operators to the `.env` file; config.yaml is supported only for precedence symmetry. - `_token_endpoint_auth()` selects `client_secret_basic` (HTTP Basic header) vs `client_secret_post` (form body) from the IDP's advertised `token_endpoint_auth_methods_supported`, defaulting to basic (the OIDC default) when absent. Applied to complete_login, refresh_session, and revoke_session (RFC 7009 §2.1). - PKCE is sent in BOTH modes — the secret is client authentication layered on top, never a replacement (OAuth 2.1 / RFC 9700 keep PKCE mandatory). - Basic header url-encodes client_id/secret before base64 per RFC 6749 §2.3.1, so reserved chars (`:`, `@`, space) round-trip correctly. Non-breaking: with no secret configured the provider is a pure public PKCE client, byte-identical to prior behaviour (no Authorization header, no client_secret in the body). The secret is never logged — register() reports only a `confidential=<bool>` flag. Tests: 16 new cases covering basic/post selection, default-when-absent, public-unchanged contract, PKCE-preserved, reserved-char url-encoding, blank-secret-is-public, refresh + revoke auth, no-secret-in-logs, and env/config register wiring. Full dashboard-auth suite (nous provider, middleware, gate, cookies, WS, 401-reauth, status endpoint) — 396 tests — green, proving no existing auth path regressed. |
||
|
Ben Barclay
|
3a55f66602 |
refactor(relay): adopt scope_id wire key (guild_id → scope_id dual-read/write) (#55289)
Gateway half of relay-platform-parity Phase 2.5 (D-Q2.5). The relay wire's platform-neutral scope discriminator is renamed guild_id → scope_id; this is the hermes-agent side of the cross-repo wire-compatible migration. - SessionSource: scope_id is canonical; guild_id kept as @deprecated alias. __post_init__ mirrors the two so all existing SessionSource(guild_id=...) constructors across native adapters keep working unchanged. to_dict dual-WRITES scope_id+guild_id; from_dict dual-READS scope_id ?? guild_id. - relay/adapter.py: capture + outbound metadata dual-read/write scope_id. - relay/ws_transport.py: _frame_to_event dual-reads scope_id ?? guild_id. - docs/relay-connector-contract.md: document scope_id (canonical) + guild_id (deprecated alias) in the §3 SessionSource field table (conformance test). 250 relay+session+contract tests green. Solo lane (relay). |
||
|
Ben Barclay
|
b963d3238b |
feat(gateway): suppress home-channel shutdown broadcast on flagged drains (#54824)
Add a generic suppress_notification flag to the drain-request marker. When a drain that ends in process exit (e.g. a NAS auto-update image migration on the always-on Hermes Cloud fleet) is flagged, the gateway skips ONLY the home-channel 'gateway shutting down' broadcast — the operator-flavoured ping that would otherwise fire on every routine auto-update, dozens of times a day. The per-active-session interrupt ping is ALWAYS kept: on a drained shutdown it's empty by construction, and in the force-interrupt (deadline-exceeded) case it carries the user-valuable 'your task was cut off, message me to resume' hint. The gateway stays agnostic about WHY a drain is quiet (generic boolean, not a kind enum); the policy of which drain causes set the flag lives in the caller (NAS). Default-false so legacy/operator drains behave exactly as before. The reader reuses the NS-570 epoch-staleness check so an orphaned marker on the durable volume can never silence a fresh gateway's legitimate broadcast. - drain_control.py: write_drain_request gains suppress_notification; new drain_notification_suppressed() reader (current-epoch + truthy flag). - web_server.py: /api/gateway/drain reads + echoes the flag. - run.py: _notify_active_sessions_of_shutdown skips the home-channel loop only. Tests prove: flag round-trips; home-channel suppressed when set, kept when unset; active-session ping always fires; stale/legacy/corrupt markers never suppress. |
||
|
Ben Barclay
|
f53ba9bb54 |
fix(s6): dot-prefix gateway staging dir so svscan ignores it mid-build (#54834)
The register path builds each profile-gateway slot in a sibling staging dir under /run/service (the scandir s6-svscan watches), then atomically renames it to the live gateway-<profile> name. The staging dir was named gateway-<profile>.tmp — a NON-dotfile — so a concurrent `s6-svscanctl -a` rescan (fired by the cont-init reconciler registering gateway-default, or by a sibling register) would supervise the half-built slot the moment it had a valid type/run: s6-supervise spawns AS ROOT and mkdirs supervise/ root-owned 0700, then the in-flight _seed_supervise_skeleton early-returns on the now-existing supervise/ and the next `mkdir supervise/event` hits PermissionError. That is the arm64-only CI flake on test_s6_unregister_removes_service_dir_in_live_container (PermissionError: /run/service/gateway-phase3test.tmp/supervise/event) — arm64-only because the native-arm runner's wider scheduling jitter lets the rescan land inside the ~ms seed window; amd64 ran 30/30 clean. Fix: dot-prefix the staging dir (.gateway-<profile>.tmp) in both register paths (S6ServiceManager.register_profile_gateway and container_boot._register_service). s6-svscan skips any scandir entry whose name begins with '.', so the half-built slot can never be supervised mid-build. The atomic rename to the dotless live name is unchanged. Verified on a real s6 image (amd64): a non-dotted staging dir is picked up by an svscanctl -a rescan (SUPERVISED owner=root) while a dot-prefixed one is ignored (NOT-SUPERVISED). Added a docker-harness regression test that asserts both, plus a unit test that the staging dir is dot-prefixed. |
||
|
Ben
|
f5ecbe1ec6 |
feat(dashboard): auto-initiate portal SSO redirect on unauthenticated load
When the dashboard gateway has no local session cookie, it rendered a click-through /login interstitial — even though the Nous portal's /oauth/authorize auto-approves any current member of the dashboard's org and is a silent 302 when the user already holds a portal session. For the common case (clicking a hosted-agent dashboard link while signed in to the portal) that interstitial click is pure friction. This makes the gate auto-initiate the OAuth redirect on an unauthenticated HTML document load instead of rendering the interstitial, when exactly one interactive provider is registered. A one-shot loop-guard cookie (hermes_sso_attempt, 60s TTL) ensures that a genuinely absent portal session (the portal bounces back still-unauthenticated) falls back to the /login page after exactly one bounce rather than ping-ponging forever. The marker is cleared on a successful callback and whenever the gate falls back to /login. Security: this removes a human CLICK, not a security check. The redirect lands on the existing /auth/login route and runs the unchanged PKCE auth-code flow; token verification, audience checks, redirect-URI match, and org-membership checks are all untouched. /api/* fetches still get the 401 JSON envelope (never a 302 a fetch() would follow opaquely), and with two or more providers the /login chooser still renders. Phase 1 of the cloud-auto-discovery work. |
||
|
Ben Barclay
|
eddfecd2ce |
fix(vision): cap vision_analyze fan-out concurrency process-wide
A single agent turn can fan out N vision_analyze calls at once — the classic trigger is "analyze every frame of this video", where ffmpeg explodes a clip into dozens of frames and the model calls vision_analyze on each. Every call does a CPU-heavy base64-encode/resize burst AND holds a long-lived LLM stream open. The tool executor runs concurrent tool calls on a per-session ThreadPoolExecutor (_MAX_TOOL_WORKERS=8), and multiple agent sessions share one process (the dashboard runs the agent in-process), so there was no global ceiling. In prod (June 2026) a video-frame fan-out pinned a worker thread at ~100% CPU and starved the shared asyncio event loop that also serves the dashboard's /api/status liveness probe, flapping the instance to UNHEALTHY even though nothing had crashed. Add a process-global threading.BoundedSemaphore that bounds how many vision analyses run concurrently across the whole process, held across the entire analysis (image load + encode + LLM call) in the single _handle_vision_analyze chokepoint (covers both the native fast path and the legacy aux-LLM path). It is a threading semaphore, NOT asyncio: each vision call is dispatched through model_tools._run_async on a per-thread event loop, so an asyncio primitive bound to one loop cannot coordinate across them. The acquire is offloaded via run_in_executor so waiting for a slot never blocks the calling loop. Default: min(host CPUs, 4), floored at 1 — respect the host's concurrency, or lower. Override via auxiliary.vision.max_concurrency (config.yaml) or HERMES_VISION_MAX_CONCURRENCY (env). Values < 1 are ignored so the cap can never be disabled into an unbounded fan-out. Tests: bounded-fan-out regression guard + a control proving it would fail without the cap; resolver tests for host-cpu default, ceiling clamp, low-cpu host, env override, and sub-1 rejection. Pre-existing handler tests updated for the now-async _handle_vision_analyze. Verified via the real registry.dispatch -> _run_async per-thread-loop path (16 concurrent calls, peak bounded to cap). |
||
|
Ben
|
4125cc3b7c |
fix(slack): subscribe to message.mpim + mpim scopes so group DMs work
Group DMs (multi-person DMs, channel_type=mpim) were never delivered to
the Slack bot. The adapter already classifies mpim as a DM and replies
ambiently (adapter.py:2526, is_dm = channel_type in {im, mpim}), but the
generated app manifest only subscribed to message.im / im:history — the
1:1 DM pair. Without the message.mpim event subscription Slack drops
group-DM messages before the adapter ever sees them, so 1:1 DMs worked
while group-DM ambient mode was dead.
Add message.mpim to bot_events and mpim:history (the scope that event
requires per Slack docs) + mpim:read (mirrors im:read for the
conversations.info classification call) to bot_scopes. Update the
SLACK_BOT_TOKEN / SLACK_APP_TOKEN setup-help strings and the Slack docs
(EN + zh-Hans: scope table, event table, troubleshooting) so existing
installs are told to add the new scopes and reinstall.
Reported by an enterprise customer. Note: this is a manifest/scope
change, so it only takes effect after the app is reinstalled and the
new scopes are accepted.
Tests: assert message.mpim + mpim:history + mpim:read are in the
manifest (with and without assistant mode); both fail on current main
and pass with this change.
|
||
|
Ben Barclay
|
1289f12812 |
fix(memory): lazy-install supermemory + mem0 SDKs like honcho/hindsight
The supermemory and mem0 memory providers shipped third-party SDKs
(supermemory / mem0ai) that are not core dependencies, but — unlike the
honcho and hindsight providers — they imported those SDKs directly with
no tools.lazy_deps.ensure() preflight and had no LAZY_DEPS allowlist
entry. On the published Docker image the agent venv is sealed
(HERMES_DISABLE_LAZY_INSTALLS=1) and lazy installs are redirected to a
writable durable target (HERMES_LAZY_INSTALL_TARGET). honcho/hindsight
route through ensure() and install fine there; supermemory/mem0 never
called it, so their SDK was never installed on a hosted instance and the
provider silently reported itself unavailable even with the API key set.
Fixes:
- Add memory.supermemory + memory.mem0 to the LAZY_DEPS allowlist
(tools/lazy_deps.py), pinned to current PyPI releases.
- Call ensure('memory.<x>', prompt=False) at each SDK-import chokepoint
(_SupermemoryClient.__init__; Mem0MemoryProvider._create_backend),
mirroring honcho's wrapped try/except shape.
- Drop the SDK-import gate from supermemory's is_available() — it was a
chicken-and-egg trap (provider never loaded on a sealed venv, so
ensure() never ran). Now key-presence only, like honcho/mem0.
- Add matching pyproject extras [supermemory]/[mem0]; update the
lazy-covered-extras contract test (excluded from [all] by policy).
Tests prove each path fails without the fix and the real sealed-venv
durable-target gate accepts both features.
|
||
|
Ben
|
20b03d9aee |
i18n: add Custom Keys strings to all locale files
The env translation block is type-checked across every locale (tsc -b), so the 8 new customKeys strings must exist in all of them, not just en/zh. Add translated entries to the remaining 14 locales (de, es, fr, it, ja, ko, pt, ru, tr, uk, hu, ga, af, zh-hant). |
||
|
Ben
|
1c75e7c9d8 |
feat(dashboard): list & add arbitrary custom .env keys on the Keys page
The Keys page only rendered env vars present in a catalog (OPTIONAL_ENV_VARS or the provider catalog); any other key a user set in .env was invisible, and there was no way to add an arbitrary env var from the GUI (e.g. to inject a var a skill or MCP server needs). Backend: GET /api/env now also emits a row for every on-disk .env key that isn't in any catalog, flagged category="custom" + custom=true and password-masked (an unrecognised key could hold anything, so it's redacted and reveal-gated like any secret). Channel-managed credentials stay excluded. The write (PUT /api/env) and reveal (POST /api/env/reveal) paths already handle arbitrary keys, with the existing env-name guard + denylist (PATH, LD_PRELOAD, PYTHONPATH, …) enforced server-side — no new write surface. Frontend: a new "Custom Keys" section lists those custom rows and carries an add-a-key form (client-side name validation mirroring the backend regex; the new row reuses the normal edit/save flow, so on save it round-trips back from the backend as a durable custom row). i18n added for en + zh + types. Tests: behavior-contract coverage that an unknown .env key surfaces as a masked custom row and a catalogued key does not — verified to fail on the pre-fix backend. |
||
|
Ben BarclayandTranquil-Flow
|
8fe800ee1a |
fix(file-tools): sanitize host/relative cwd override before it reaches container sandbox (#54447) (#54616)
(cherry picked from commit 82132f7911ecf71f27ee5657870bf4105cecf8e2) Co-authored-by: Tranquil-Flow <66773372+Tranquil-Flow@users.noreply.github.com> |
||
|
Ben Barclay
|
e1f4098b9f |
docs(cron): document explicit per-channel delivery targets for all platforms (#54630)
The cron delivery table only showed Discord/Telegram with explicit target syntax and described Slack and every other platform as home-channel-only. In fact the generic platform:<target> routing in _resolve_single_delivery_target resolves explicit targets for every platform: Slack (#channel / channel ID / channel:thread_ts), Matrix (room/user IDs), Feishu (chat:thread), WhatsApp (JID / E.164), Signal (group / E.164), SMS, Email, and Weixin all have dedicated explicit- target branches in _parse_target_ref; the remaining platforms accept a generic platform:<chat_id> passthrough. Update the Delivery Model table (en + zh-Hans) to show the real per-platform syntax, document #channel name resolution via the channel directory, and note the Slack thread_ts nuance. Docs-only. |
||
|
Ben Barclay
|
0943e2a272 |
fix(cron): don't report a false 'gateway not running' on external-provider instances (#54600)
`hermes cron status` (and the create/list 'gateway not running' nag) judge whether cron will fire purely from the in-process ticker's heartbeat file + a live gateway PID. That heuristic is correct for the built-in ticker but WRONG for an external provider like Chronos: Chronos arms exactly one external one-shot per job and is fired by a NAS-mediated webhook (POST /api/cron/fire). Its `start()` returns immediately and it deliberately runs no 60s loop and writes no ticker heartbeat — that's the whole point of scale-to-zero (the machine is at zero between fires). So on a perfectly healthy Chronos instance, `cron status` always printed '✗ Gateway is not running — cron jobs will NOT fire' (or a STALLED-ticker warning), and `cron create` always appended the 'jobs won't fire automatically' nag — both false. Verified live on a staging Chronos instance: jobs fired and completed on schedule via the relay while `cron status` insisted the gateway wasn't running and the heartbeat was 370s+ stale. Fix: resolve the active provider (offline — `resolve_cron_scheduler`, whose `is_available()` contract forbids network) and, for any non-builtin provider, report the managed-scheduler state instead of the ticker heuristics, and suppress the ticker-only 'gateway not running' warning. The built-in path is byte-unchanged. Active-job summary is factored into a shared helper so both paths print it identically. New tests prove both directions (chronos: no false negative even with no gateway PID / no heartbeat; builtin: historical warning preserved) and fail without the fix. |
||
|
Ben
|
dee41d0716 |
feat(dashboard): catalogue all memory-provider API keys in OPTIONAL_ENV_VARS
The dashboard Keys page and `hermes setup` render API-key rows from OPTIONAL_ENV_VARS, but only Honcho had an entry — so Hindsight, Supermemory, Mem0, RetainDB, ByteRover, and OpenViking read their keys straight from os.environ yet had no place to set them in the GUI. Add catalog entries (category=tool, password-masked, with get-key URLs and the tool each powers) for all six, plus the relevant base-URL/endpoint companions. Pure declaration: the generic GET /api/env endpoint, the save/reveal write path, and the sandbox env blocklist (which auto-derives from tool-category OPTIONAL_ENV_VARS) all pick these up with no further wiring. Adds a behavior-contract test asserting every memory provider's primary credential key is catalogued, tool-categorised, and password-masked. |
||
|
Ben Barclayandangelos
|
7cfa2fa13f |
fix(docker): gate resource limit flags on cgroup controller availability (#54516)
On hosts where the cgroup v2 cpu/memory/pids controllers are not delegated to the docker/podman process (unprivileged Proxmox LXCs, some rootless and nested setups), --pids-limit/--cpus/--memory cause every container start to fail with OCI runtime error / exit 126, breaking terminal + execute_code. - Add _cgroup_limits_available(image): one-shot, host-wide cached probe that spawns a throwaway container from the sandbox image itself (sleep 0) with all three flags together, mirroring the existing _storage_opt_supported probe-and-degrade pattern. - Remove --pids-limit from static _BASE_SECURITY_ARGS; apply it (default 256 via _DEFAULT_PIDS_LIMIT) in resource_args gated on the probe. - Gate --cpus and --memory on the same probe. Behavior unchanged on cgroup-capable hosts; graceful degradation with a one-time warning where controllers aren't delegated. Fixes #6568. (cherry picked from commit c933880b7ee2ce4d1167e0f89caa2d233db5639f) Co-authored-by: angelos <angelos@oikos.lan.home.malaiwah.com> |
||
|
Ben Barclay
|
1466eab4ee |
test(docker): wait for cont-init to finish before privilege-drop shim tests (#54026)
The docker-exec privilege-drop shim tests started a sleep container and released the fixture as soon as `docker exec <c> true` returned 0. On s6-overlay that succeeds almost immediately — ~0.05s in measurement — long before the `01-hermes-setup` cont-init hook (docker/stage2-hook.sh) has finished seeding + `chown hermes:hermes` config.yaml and running the Python config migration (cont-init only fully settles at ~9.8s under arm64 QEMU emulation). `test_shim_opt_out_keeps_root` wipes config.yaml, writes it as root with HERMES_DOCKER_EXEC_AS_ROOT=1, and asserts root:root ownership. When the fixture released the test inside that ~10s window, stage2-hook's boot-time `chown hermes:hermes config.yaml` raced the root-written file and reset it to hermes:hermes — failing the assertion. The window is invisible on native amd64 (stage2-hook completes in a blink) but wide open under the arm64 build's QEMU emulation, which is why only build-arm64 flaked while build-amd64 stayed green. Replace the responsiveness poll with a wait on the canonical 'cont-init finished' signal: $HERMES_HOME/logs/container-boot.log gaining a `profile=default` line, written by 02-reconcile-profiles which s6 runs strictly after 01-hermes-setup. Mirrors the readiness pattern already used in test_container_restart.py. Also bumps the readiness timeout 20s->60s to cover slow emulation. No production code change — test-only hardening of a timing race. |
||
|
Ben Barclay
|
fbf748b282 |
fix(dashboard-auth): follow redirects on self-hosted OIDC discovery (#53399)
The self-hosted OIDC provider fetched the discovery document with a bare httpx.get(). httpx defaults to follow_redirects=False (unlike curl -L or the requests library), so when an IDP answers GET /.well-known/openid-configuration with a 3xx — Authentik canonicalises the .well-known path, and any IDP behind a reverse proxy doing an http→https upgrade redirects too — the bare redirect (empty body) tripped the status != 200 guard and raised 'OIDC discovery returned 302', which routes.py maps to the provider_unreachable audit event and a 503. The browser surfaced 'Auth provider self-hosted unreachable'. The user's smoking gun (curl -o writing zero bytes from inside the container) is exactly a redirect with no body — the same wall the code hit. Add follow_redirects=True to the discovery GET only. It's safe: the issuer-pin check and _require_https_or_loopback still validate the resolved document and every endpoint, so a redirect can't smuggle in a bad issuer or a cleartext endpoint. The token/revocation POSTs deliberately keep the no-follow default (they carry an auth code / refresh token and the endpoint is already the canonical absolute URL). Existing discovery tests mocked httpx.get with a canned 200 and never exercised a real 3xx. Add a regression test that runs a real loopback server returning a 302 on the .well-known path — fails without the fix (ProviderError: discovery returned 302), passes with it. |
||
|
Ben
|
8ab7246c45 |
fix(gateway): stamp drain marker with instantiation epoch so a durable-volume restart clears it (NS-570)
The external-drain marker .drain_request.json is written under HERMES_HOME, which on Hermes Cloud is a persistent Fly volume (/opt/data). A begin-drain marker therefore SURVIVES the post-update machine restart. But the disruptive lifecycle actions a drain protects (auto-update / image migrate / env edit / profile change) all restart the machine — which is exactly the signal the drain is over. The freshly-restarted gateway re-read the orphaned marker on its startup reconcile and parked itself back in 'draining', refusing every new turn indefinitely (NS-570: ~52 min until manually cleared). Fix: stamp the marker with an identity of THIS container/VM instantiation (kernel boot_id + PID 1 start time, read from /proc) and treat a marker whose epoch differs from the current instantiation as absent. A deliberate restart → new PID 1 → new epoch → stale marker ignored → gateway boots 'running'. A marker written during the current instantiation (the live drain) still matches; an s6 respawn of just the gateway (PID 1/init unchanged) keeps the same epoch, so an in-flight drain is still honoured (D4a reversibility preserved). The staleness check is lenient and never fail-closed: a legacy marker with no epoch, a corrupt/contentless marker, or an environment with no /proc (epoch unavailable) all degrade to the original presence-only behaviour. NAS is untouched — it only ever POSTs begin/cancel-drain over HTTP; the marker file is purely gateway-internal IPC. The fix is entirely within gateway/drain_control.py; the watcher and the dashboard endpoint go through the same drain_requested()/write_drain_request() chokepoints and need no functional change. |
||
|
Ben
|
19b2624404 |
feat(gateway): external drain trigger + accept-gating (begin/cancel + control channel)
Tasks 2.1 + 2.2 + 2.3 of the safe-shutdown plan — the reversible
quiesce-without-restart machinery NAS drives during a lifecycle action (D4a).
These ship together because the endpoint, the control channel, and the gateway
state machine are one coherent slice.
2.2 — control channel (gateway/drain_control.py, new):
The dashboard has no HTTP path into a running gateway (guardrails: "there is NO
external control channel into a running gateway"); restart/drain is driven only
by markers the gateway reacts to. So begin/cancel-drain writes/removes a
presence-based marker .drain_request.json (HERMES_HOME-scoped, atomic write,
never-raises read; a corrupt marker reads as present-contentless → fail-safe
toward quiescing). This is Q-B option A.
2.2 — gateway state machine (gateway/run.py):
- _external_drain_active flag, DISTINCT from the shutdown _draining flag: this
one does NOT exit the process and is fully reversible.
- _enter_external_drain / _exit_external_drain: idempotent transitions that
flip gateway_state→draining / →running via _update_runtime_status (preserving
the live active_agents count). exit refuses to revert to running during a
real shutdown or after the loop stops (shutdown wins).
- _drain_control_watcher: 1s background task (modelled on _handoff_watcher)
reconciling accept-state with the marker; honours a marker that survived a
restart on its first tick. Registered alongside the other watchers in start.
- New-turn accept gate in _handle_message, placed BEFORE the session-slot
claim: when draining, refuse to START a new turn (so active_agents can only
fall → no TOCTOU race), while in-flight turns finish untouched. Internal/
system events (restart-recovery replays, bg-process completions) bypass it.
2.1 — endpoint (hermes_cli/web_server.py):
POST /api/gateway/drain {action: drain|cancel}. Authenticated by the Task-2.0a
token seam (the drain plugin registered this exact path as a token route);
attributes the request to the verified token principal. Begin writes the
marker, cancel removes it — the gateway process owns the actual transition.
Force-override (D6) is NOT here; it maps onto the existing immediate
/api/gateway/restart force path.
Tests (mocked — necessary-not-sufficient; the HARD live gate Q-B is next):
- tests/gateway/test_external_drain_control.py — marker contract (write/clear/
read/corrupt/atomic), state machine (enter/exit/idempotency/shutdown-wins/
loop-stopped), watcher reconcile-enter-then-exit, new-turn refusal, and
in-flight-not-interrupted. 15 tests.
- tests/hermes_cli/test_web_server.py — /api/gateway/drain begin/default-begin/
cancel/cancel-idempotent/bad-action-400. 6 tests.
- dashboard.drain_auth config section already added in 2.0b commit.
All touched suites green: 301 (gateway+auth) + 9 (web_server endpoints) passed.
Intentionally deferred:
- HARD live-validation gate (Q-B): real isolated `hermes gateway run`, drive a
real begin-drain marker, prove the 5-point checklist a–e.
- Spec-doc status flip + Phase-2 PR.
Build status: external-drain, restart-drain, status, dashboard-auth, drain-plugin,
token-auth, and web_server-endpoint suites green.
|
||
|
Ben
|
2e322466b1 |
feat(dashboard-auth): drain shared-bearer-secret provider plugin
Task 2.0b: the concrete shared-bearer-secret auth provider, the FIRST consumer
of the generic token-auth capability (Task 2.0a). Implements decisions.md Q-A.
plugins/dashboard_auth/drain/ (bundled, discovered like dashboard_auth/basic):
- DrainSecretProvider: non-interactive provider, supports_token=True. Verifies
an inbound Authorization bearer token against a per-agent shared secret with
hmac.compare_digest (constant-time, no timing oracle) and, on a match,
vouches for the caller as the "drain-control" principal scoped to "drain".
The five interactive ABC methods raise NotImplementedError; verify_session
returns None (stacks harmlessly in the cookie-verify loop).
- assess_secret_strength(): fail-closed entropy gate. Rejects secrets shorter
than 43 url-safe-b64 chars (~256 bits), with < 16 distinct characters, or
below 128 bits Shannon entropy — so a weak/structured/repeated secret can
never be silently accepted. Enforced both at register() (friendly skip
reason) and in __init__ (raises — defence in depth).
- register(ctx): no-op + skip reason when HERMES_DASHBOARD_DRAIN_SECRET is
unset; rejects a weak secret fail-closed (drain endpoint stays gated). On a
strong secret, registers the provider AND opts /api/gateway/drain into the
generic token-auth seam via register_token_route().
Config: the secret is a CREDENTIAL → carried via HERMES_DASHBOARD_DRAIN_SECRET
(per-agent, provisioned by NAS at deploy). Behavioural knobs only
(dashboard.drain_auth.{scope,min_secret_chars}) live in config.yaml — added to
DEFAULT_CONFIG with the .env-is-for-secrets rationale documented inline.
Tests: tests/plugins/dashboard_auth/test_drain_provider.py — entropy gate
(strong pass; empty/short/repeated/few-distinct/custom-min reject), verify_token
(match → scoped principal, wrong/empty → None, custom scope), protocol
compliance, interactive-methods-raise, and register() (skip-no-secret,
fail-closed-weak-secret, strong-env-secret registers + route opt-in, config
scope + min_secret_chars). 21 new tests; drain + token-auth suites 44 passed.
Verified the plugin is discovered as dashboard_auth/drain alongside basic/nous.
Intentionally deferred:
- The begin/cancel-drain endpoint handler itself — Task 2.1.
- The dashboard→gateway control channel — Task 2.2.
Build status: dashboard-auth + drain-plugin suites green.
|
||
|
Ben
|
cb9cb6ba1c |
feat(dashboard-auth): generic non-interactive API-token capability
Task 2.0a of the safe-shutdown drain-coordination plan. Widens the dashboard auth framework GENERICALLY to support non-interactive (service-to-service) bearer-token auth, mirroring the existing supports_password precedent. This is a reusable capability — any future machine-credential provider plugs in without core changes (decisions.md Q-C). The drain bearer-secret plugin (Task 2.0b) is the first consumer, not the definition. - base.py: add TokenPrincipal dataclass (the token analog of Session) + supports_token capability flag + verify_token() on the ABC (default raises NotImplementedError so a misconfigured provider fails loud). Contract mirrors verify_session stacking: return None for unrecognised tokens (never raise), raise ProviderError only on a genuine backing-store outage. - registry.py: list_token_providers() — the supports_token subset, in registration order. Empty when none registered (token routes fail closed). - token_auth.py (new): route-agnostic seam. Routes opt in via register_token_route(exact path); token_auth_middleware owns the auth decision for those routes only — authenticate via stacked providers, attach request.state.token_principal + token_authenticated, pass through. 401 on missing/unrecognised token, 503 when a provider was unreachable, untouched passthrough for non-token routes. Fails closed (never open). - web_server.py: install the seam OUTERMOST (registered last → runs first). Both downstream gates (legacy auth_middleware + gated_auth_middleware) honour request.state.token_authenticated and skip enforcement, so a token-authed service request is never bounced to /login. - audit.py: TOKEN_AUTH_SUCCESS / TOKEN_AUTH_FAILURE events. Tests: tests/hermes_cli/test_dashboard_token_auth.py — ABC flag default, verify_token NotImplementedError, registry filter, bearer extraction (case-insensitive scheme, malformed/non-bearer → ""), provider stacking (first-match-wins, unreachable-remembered, unreachable-then-valid, buggy provider doesn't crash the gate), and the seam's passthrough/401/503/ fail-closed behaviour. 29 new tests; full dashboard-auth suite 169 passed. Intentionally deferred: - The concrete shared-bearer-secret provider plugin — Task 2.0b. - The begin/cancel-drain endpoint that registers itself as a token route — Task 2.1. Build status: dashboard-auth + plugin-hook suites green. |
||
|
Ben Barclay
|
6e4e5967f7 |
feat(relay): multi-platform-per-agent — list identity, provision-loop, N-hello, per-frame egress (Phase 1.5) (#52830)
Cut over the agent half of Shape A (D-Q1.5a/b.1/c) to front a SET of platforms on
one relay WS:
- relay_platform_identities() parses GATEWAY_RELAY_PLATFORMS (list) +
GATEWAY_RELAY_BOT_IDS (JSON keyed map {platform:{botId,username?}}). Cut over
from the scalar GATEWAY_RELAY_PLATFORM/_BOT_ID (no fallback, D-Q1.5c).
- self_provision_relay() loops one /relay/provision per platform under one
gatewayId+secret, partial-failure-tolerant.
- WebSocketRelayTransport takes the identity SET, sends one hello per identity
(connector accumulates the advertised set), and stamps the per-frame
OutboundFrame.platform + its matching advertised botId on outbound.
- RelayAdapter remembers each chat's underlying source.platform (mirroring the
existing guild/dm scope capture) and tags the reply's egress platform.
- send_relay_policy() declares one relevance policy per fronted platform (the
connector keys policy by (tenant,platform,instanceId)).
Single-platform deploys are byte-identical on the wire (1-element list, no per-frame
tag -> connector session-default fallback). typecheck/ruff clean; relay unit 221 pass
(+10 new); all 15 cross-repo E2E drivers green vs connector origin/main.
|
||
|
Ben Barclay
|
cb7d1f68f8 |
fix(relay): accept is_reconnect kwarg in RelayAdapter.connect (#52911)
The gateway reconnect watcher (gateway/run.py) recovers a platform after a
fatal adapter error by building a fresh adapter and calling
connect(is_reconnect=True). Every BasePlatformAdapter implements
connect(*, is_reconnect: bool = False) for this — except RelayAdapter, whose
connect() was bare. So the watcher's recovery path raised:
TypeError: connect() got an unexpected keyword argument 'is_reconnect'
Observed live on a hosted staging agent: after a fatal relay adapter error the
watcher could never re-establish relay, so the shared-bot inbound never reached
the gateway and Discord DMs stopped (dashboard surfaced the TypeError).
Relay deliberately ignores the flag: the #46621 server-side-queue-preservation
concern doesn't apply, because relay's outage buffer is the connector's durable
buffer (replayed on the transport's re-handshake), not a gateway-side queue the
adapter owns. Routine WS drops are already handled by the transport's own
reconnect supervisor (WebSocketRelayTransport, reconnect=True); the watcher path
is fatal-error recovery, and the fatal handler disconnect()s the old adapter
(cancelling its supervisor) before a fresh adapter+transport is built, so there
is no double-dial.
Adds two regression tests (both proven red without the fix): connect(is_reconnect=True)
reaches the same transport-less RuntimeError instead of TypeError, and the
signature matches BasePlatformAdapter.connect.
|
||
|
Ben Barclay
|
dedf5643d8 |
fix(gateway): scale-to-zero never armed — arm-gate counted disabled placeholder platforms (#52831)
The scale-to-zero idle watcher never started on a correctly-opted-in, relay-only instance, so the gateway never ran its idle decision, never called go_dormant(), and never sent going_idle to the connector. Fly's autostop still suspended the machine on traffic-idle, but the connector never flipped the instance to buffered-only — so an inbound DM took the live delivery path, found no live session for the suspended machine, and was dropped fail-closed with no wake poke. The machine slept and never woke. Root cause: _scale_to_zero_should_arm() passed list(config.platforms.keys()) to messaging_is_relay_only_or_absent(). config.platforms is pre-seeded with a DISABLED placeholder PlatformConfig for every known platform (telegram, discord, slack, matrix, …), so the key set is always the full ~20-entry catalog regardless of what the instance actually runs. The relay-only check discarded "relay", saw the disabled placeholders as live direct-socket platforms, and returned False — so should_arm() was False and the watcher was never created. Verified live on a staging instance: config.platforms keys = [telegram, discord, slack, mattermost, matrix, relay] with only relay enabled=True; should_arm() = False. Fix: filter config.platforms to ENABLED entries before the relay-only check, mirroring the adapter-connect loop which already gates on `if not platform_config.enabled: continue`. This arms off the same notion of "active platform" the rest of start() already uses — no parallel concept. Also add a one-line not-armed diagnostic: when an instance IS opted in (the HERMES_SCALE_TO_ZERO stamp is set) but the watcher still doesn't arm, log why (relay_only_or_absent, the enabled platforms, wake_url present/missing). A non-opted instance stays silent. The arm path previously logged only on success, so a failed arm was invisible. Tests: the existing pure-helper tests passed bare names so they never exercised the call site that feeds the placeholder-laden config. Add behaviour-contract tests against the REAL _scale_to_zero_should_arm with a realistic config.platforms (relay enabled + others disabled). The F25 regression test (relay-only + disabled placeholders must arm) and the no-platform case are RED without this fix, GREEN with it; the genuinely-enabled-direct-platform / not-opted-in / no-wake-url cases stay correctly non-arming so the filter can't over-broaden. Wake mechanism itself verified healthy independently (direct wakeUrl GET resumed a suspended staging instance in 1.15s, clean resume signature). |
||
|
Ben Barclay
|
d6269da7fd |
fix(gateway): harden scale-to-zero dormancy guards (#52359)
Block scale-to-zero suspend while background async delegations are active, and restore runtime status to running on real inbound after a dormant wake.\n\nAdd regression coverage for both review findings. |
||
|
Ben Barclay
|
736e981abf |
fix(auth): honor NOUS_INFERENCE_BASE_URL env override for Nous OAuth sessions (#52270)
The host-allowlist hardening (#30611) plus the refresh heal (#49735) left the documented NOUS_INFERENCE_BASE_URL dev/staging escape hatch unreachable for OAuth sessions, despite three code comments asserting it still works. Root cause — resolution precedence in resolve_nous_runtime_credentials: inference_base_url = ( _optional_base_url(state.get("inference_base_url")) # stored — wins or os.getenv("NOUS_INFERENCE_BASE_URL") # env — unreachable or DEFAULT_NOUS_INFERENCE_URL ) A staging OAuth login persists its inference_base_url, but the allowlist rejects the staging host and the refresh heal rewrites the stored value to the production default. The stored (now prod) value is then read BEFORE the env var, so the override never takes effect — every request 401s against prod or is pinned to prod, and setting the env var does nothing. Fix: the user-set env override is the most-trusted source, so consult it FIRST for the URL used to build the client / returned to callers — while keeping the PERSISTED value the validated, network-provenance one (the override is a runtime overlay, never written to auth.json, so unsetting it cleanly reverts to prod). Applied at both chokepoints: - resolve_nous_runtime_credentials (no-refresh read path AND refresh path) - the nous_portal proxy adapter, which re-validates the resolver's returned base_url against the prod allowlist as defense-in-depth and would otherwise reject a legitimate staging override at the forward boundary. New _nous_inference_env_override() / split of stored-vs-effective URL keep the threat model intact: Portal-returned URLs are still allowlist-validated at every network site, and the env path stays ungated (trusted OS user). Also folds in the no-refresh read-path heal (supersedes the approach in the open #50265): a poisoned stored staging host now heals to the prod default on read even when no refresh fires. Tests: TestEnvOverrideWins (env wins on read + refresh paths; override never persisted; poisoned stored heals) and TestProxyAdapterEnvOverride. Verified the 4 behavioral tests fail against pre-fix code and pass with the fix; full inference-validation + nous-provider suites green (85 passed). E2E-validated against a real temp HERMES_HOME exercising the real resolver + proxy adapter: resolver→staging, persisted→prod, proxy→staging, unset→reverts to prod. |
||
|
Ben Barclay
|
72ae163250 |
fix(relay): authorize relay-delivered events by delivery, not source.platform (#52306)
* fix(relay): authorize relay-delivered events by delivery, not source.platform The #52190 upstream-authz fix keyed _is_user_authorized off source.platform via _adapter_authorization_is_upstream(source.platform). But a relay *message* inbound carries the UNDERLYING platform (source.platform == discord/telegram/...), NOT Platform.RELAY, because ws_transport._event_from_wire maps the connector's wire payload (platform="discord") straight onto SessionSource for session-keying and egress. The relay adapter is registered only under Platform.RELAY, so adapters.get(Platform.DISCORD) misses, the trusted-upstream branch is skipped, and the user hits the env-allowlist default-deny: WARNING gateway.run: Unauthorized user: <id> (<name>) on discord (Live staging bug: alpha tester linked successfully, then every follow-up DM was silently dropped.) Fix: the authentic trust signal is that the event was delivered over the per-instance-authenticated relay WS, not which platform it underlies. Add a wire-INVISIBLE SessionSource.delivered_via_upstream_relay flag, stamped by the relay transport in _event_from_wire, and authorize on it. The flag is excluded from to_dict/from_dict so a peer can neither forge it across the wire nor have it restored from persistence. The existing adapter-flag check is retained for events whose source.platform IS Platform.RELAY (interaction-passthrough). A direct Discord event on a multiplexing gateway (direct + relay adapters) is unmarked and still default-denies. * fix(relay): use identity check on delivery marker to avoid MagicMock fail-open A MagicMock() source (used by test_signal.py and other gateway tests) auto- vivifies source.delivered_via_upstream_relay as a truthy Mock, which a bare truthiness check would treat as authorized — flipping test_signal_in_allowlist_maps from False to True. The marker is a real bool on SessionSource, so check 'is True' explicitly: refuses to authorize any non-bool stand-in, defensive against accidental fail-open. |
||
|
Ben
|
0c3f197cff |
fix(relay): re-attach DM author user_id on outbound for connector egress
A DM reply carries no guild_id, so the connector's egress guard cannot resolve the owning tenant from metadata.guild_id and declines the send with "discord egress declined: target not routed to an onboarded tenant" — the bug behind "the bot never replies in DMs". Guild replies are unaffected (they carry guild_id), which is why the guild path worked end-to-end while DMs looked broken. The connector now resolves a DM reply's tenant from the recipient's author binding (gateway-gateway #67, resolveByUser keyed on metadata.user_id) — the outbound counterpart to inbound Phase 7a author-first resolution. But it needs the recipient user_id ON the outbound action, and the adapter only re-attached guild_id (_capture_scope/_with_scope), no-op for DMs (the docstring even said so). This extends the adapter's inbound-scope capture: for a DM (no guild_id) remember chat_id -> the authentic author user_id we observed, and re-attach it as metadata.user_id on outbound. Guild capture is unchanged and wins when present; user_id is the DM-only fallback. The id is the one the connector observed inbound (never gateway-asserted), so the trust invariant holds. +4 unit tests (DM reply re-attaches user_id + no guild_id; unknown chat invents nothing; explicit user_id preserved; guild reply never carries user_id). Proved load-bearing (reverting the re-attach fails the DM test). 144 relay tests pass, ruff clean. Pairs with gateway-gateway #67 (the connector-side resolver). Together they close the DM-reply egress gap end-to-end. |
||
|
Ben Barclay
|
c15945655f |
fix(terminal): sanitize host/relative cwd OVERRIDE before it reaches docker run -w (#50636)
terminal_tool() resolves a per-task cwd override that WINS over config["cwd"]:
cwd = overrides.get("cwd") or config["cwd"]
config["cwd"] is sanitized for container backends in _get_env_config() (host
prefixes /Users//home//C:\\/C:/ and relative paths are replaced with the
backend default /root). But the override was applied RAW — it was never run
through that guard. The gateway/TUI registers the host launch dir as a cwd
override for workspace tracking (tui_gateway/server.py _register_session_cwd
-> _terminal_task_cwd -> _session_cwd -> os.getcwd()), so on a container
backend a host path leaked straight to `docker run -w <host-path>`:
- Windows desktop: -w C:\Users\<user> -> container fails to start (exit 125)
- POSIX: -w /home/<user> -> same
The ACP adapter translates its override cwd (acp_adapter/session.py
_translate_acp_cwd), but the gateway path did neither translation nor
sanitization, so the override bypassed the one guard that would have caught it.
Fix: extract the host/relative-path predicate into a shared
_is_unusable_container_cwd() helper (so the existing _get_env_config()
sanitizer and the new guard can't drift), and re-apply it to the *resolved*
cwd at the override-resolution site. Valid in-container override paths
(RL/benchmark sandboxes that set cwd to /workspace, /root, ...) are absolute
non-host paths and pass through untouched.
Tests: unit-pin the predicate (Windows backslash/forwardslash, POSIX home,
macOS /Users, relative, valid container paths) AND an E2E call-site pin that
drives terminal_tool() with a host-path override registered and asserts the
cwd reaching _create_environment is sanitized. Mutation-verified: reverting
the call-site guard makes the two host-path E2E tests fail (showing the raw
host path leaking) while the valid-/workspace-override test stays green.
|
||
|
Ben
|
d1cac0e5ef |
feat(gateway): scale-to-zero idle detection + dormant-quiesce (Phase 0)
The gateway-side BEHAVIOUR layer that consumes the relay scale-to-zero primitives (gateway-gateway Phase 5): the gateway decides it is idle and drives the relay transport dormant so the platform (Fly autostop:"suspend") can suspend the now-traffic-idle machine, which wakes on the connector's wakeUrl poke (decisions.md Q3=C', D1-D13). - gateway/scale_to_zero.py: pure helpers — scale_to_zero_enabled (the NAS Labs HERMES_SCALE_TO_ZERO stamp, D11/Q8=A), parse_idle_timeout_seconds (config.yaml gateway.scale_to_zero.idle_timeout_minutes, D2), messaging_is_relay_only_or_absent (F6/D1), should_arm (D1/D11/§3.4(1)), is_idle (D2/D3/F7). - gateway/run.py: _last_inbound_at clock stamped on user inbound in _handle_message (F13); the arm-gate + idle predicate + the _scale_to_zero_watcher dormant sequence (mark draining -> adapter go_dormant() -> cooldown), started only when armed. Deliberately NOT the stop path and NOT mark_resume_pending (F12/D13). - tools/process_registry.py: has_any_active() for the bg-work guard (D3/F7). - hermes_cli/config.py: gateway.scale_to_zero.idle_timeout_minutes default 5. Tests: 38 pure-logic + 6 watcher (incl. bg-work regression guard proven RED). Full relay + scale-to-zero suites: 184 passed. The 20 unrelated failures in the broader run are PRE-EXISTING on origin/main (custom-provider/tools tests), confirmed via a pristine baseline worktree. |
||
|
Ben
|
96af4bec30 |
feat(relay): add go_dormant() transport mode for scale-to-zero (0.E0)
Net-new WebSocketRelayTransport.go_dormant() + RelayAdapter.go_dormant() — the third transport mode the scale-to-zero behaviour layer needs, distinct from both disconnect() and an unexpected close (decisions.md D12/F14): - disconnect() sets _closing=True and CANCELS the reconnect supervisor (terminal "shutting down for good") -> a suspended machine never re-dials on wake, stranding its buffered backlog. - an unexpected close re-dials IMMEDIATELY -> the socket never stays down, so the platform proxy never suspends the machine. go_dormant(): going_idle->ack (reuse go_idle), then close the socket WITHOUT setting _closing, so the reader's fall-through still arms the reconnect supervisor (wake path stays live) but on the longer _dormant_redial_s cadence so it doesn't fight the platform suspend window. A successful re-dial clears _dormant. Honors the §3.4 wake->reconnect->drain contract. Tests: 6 new in test_relay_going_idle.py incl. the F14 regression guard (routing dormancy through disconnect() fails exactly the 4 wake-path tests). Full relay suite 140 passed. |
||
|
Benandhelix4u
|
538c419d2e |
fix(gateway): scope dashboard liveness fallback to the profile
PR #52151 hardened the runtime-status liveness check to trust a readable live process command line over stale gateway_state.json argv, so a recycled PID now owned by an s6 supervisor no longer counts as a running gateway. That fix is correct but incomplete for the reported symptom: the web dashboard showed a named profile's gateway green while `hermes -p <name> gateway status` showed it stopped. Two further issues: 1. Cross-profile PID reuse. In per-profile Docker supervision, one profile's stale `gateway_state.json` can record a PID the OS later recycled onto a DIFFERENT profile's live gateway. That PID's command line still `looks_like_gateway`, so the dead profile was reported running. The recorded argv has its `-p <name>` selector stripped in-process by `_apply_profile_override`, so it cannot disambiguate; the live `/proc` cmdline still carries it. `get_runtime_status_running_pid` now accepts an `expected_home` and validates the live command line belongs to THAT profile (mirroring `hermes_cli.gateway._matches_current_profile`, the logic the CLI scan path already uses — which is why the CLI was correct). `_check_gateway_running` passes the enumerated profile dir. 2. The existing regression test `test_gateway_running_check_falls_back_to_ runtime_state` used the live pytest PID with a gateway-shaped record; once the live cmdline became authoritative it no longer looked like a gateway. Updated to mock the live cmdline to the real separate-process scenario it describes. The active-profile path (`get_running_pid`) is intentionally left unscoped: it is lock-verified and any live gateway cmdline is acceptable there. Multiplex mode is unaffected — `running` state is only ever written to a gateway's own home, never a secondary served profile's. Adds coverage for: cross-profile PID reuse (named + default), matching profile cmdline (`-p`, `--profile`, explicit HERMES_HOME=), the bare default gateway, and the unreadable-cmdline cross-platform fallback. Each new cross-profile assertion fails without the profile scope and passes with it. Co-authored-by: helix4u <4317663+helix4u@users.noreply.github.com> |
||
|
Ben
|
d335164833 |
fix(relay): authorize relay inbound via connector-enforced upstream authz
A hosted instance fronted by the Team Gateway connector dropped EVERY relay
message as "Unauthorized user" and the agent never replied — despite the
message routing correctly through the connector to the instance.
Root cause: gateway authorization (_is_user_authorized) had no notion of
upstream-enforced authz. Platform.RELAY matches no {PLATFORM}_ALLOWED_USERS
allowlist and isn't in the HA/WEBHOOK always-authorized set, so a relay user
with no env allowlist configured hit the default-deny ("No user allowlists
configured. All unauthorized users will be denied."). The message was received,
then silently denied before reaching the agent.
This is incorrect for relay: the connector authenticates the gateway's WS with
a per-instance secret and performs owner-only author-binding resolution BEFORE
delivering. A message only reaches this gateway because the connector resolved
it to THIS instance's bound user (user_instance_binding), keyed on the author id
the connector OBSERVED off the event — never a gateway claim. The authorization
decision is already made by a trusted, authenticated upstream; there is no local
RELAY_ALLOWED_USERS allowlist to consult, and default-denying for its absence is
the bug.
Fix: add a generic BasePlatformAdapter.authorization_is_upstream capability
(default False) that the relay adapter overrides to True, plus a dedicated
trusted branch in _is_user_authorized that honors it. This is delegation to a
trusted upstream, NOT a fail-open: it fires only for an adapter that explicitly
declares the flag; every direct network-exposed adapter leaves it False and the
env-allowlist default-deny (SECURITY.md §2.6) is unchanged. Distinct from
enforces_own_access_policy, which mirrors a LOCAL config-driven allowlist —
this delegates to an authenticated upstream's decision.
Tests: behavior contract that the base defaults False, the relay adapter
declares True, a relay user (group + DM) is authorized with no env allowlist,
and crucially a non-upstream adapter with no allowlist still default-denies
(guards against the fix becoming a blanket fail-open). 6 new tests; relay +
authz + config-policy suites green (134 + 90).
Found via live staging debug of the Discord self-serve onboarding flow.
|
||
|
Ben
|
41b9b7e719 |
test(lazy-deps): make durable-target tests network-free
CI test shard has no PyPI egress: the real 'pip install packaging==20.9' in test_core_package_is_not_shadowed failed (the pypi.org reachability probe passed but the actual install didn't), failing slice 2/6. - Prove the anti-shadow invariant deterministically: synthesize a fake 'packaging' in the durable target with a sentinel and assert the import still resolves to the core copy (TestCoreNeverShadowed). No network. - Cover the install wire offline: stub subprocess and assert --target + --constraint are built in durable mode and absent in venv-scoped mode (TestInstallArgConstruction). - Gate the genuine PyPI install behind HERMES_RUN_NETWORK_TESTS=1 (opt-in, skipped in CI) instead of a flaky reachability probe that doesn't predict install success. |
||
|
Ben
|
cbd6ba1bdd |
fix(docker): redirect lazy installs to a durable target so opt-in backends work in the immutable image (#51136)
The published Docker image seals the agent venv (root-owned, read-only /opt/hermes) and sets HERMES_DISABLE_LAZY_INSTALLS=1 so a runtime install can't mutate and brick the core. But opt-in backends (Firecrawl web search, Exa, Feishu, ...) deliberately keep their SDKs in tools/lazy_deps.py and out of [all] (pyproject policy 2026-05-12: one quarantined release must not break every install). The two policies collided: the SDK isn't baked in AND can't lazy-install, so the default Firecrawl web_search/web_extract fail out of the box in Docker (#51136), as do Exa (#49445) and Feishu (#50205). Fix the whole class instead of baking in one backend: when HERMES_LAZY_INSTALL_TARGET is set, lazy installs are redirected to a writable dir on the durable /opt/data volume via `pip/uv install --target`, and that dir is APPENDED to the end of sys.path. Because the core venv always wins name collisions, a package installed this way can only ADD new modules — it can never shadow, downgrade, or break a module the core ships. The worst a bad/incompatible backend package can do is fail to import and report itself unavailable; the agent core stays healthy. That structural guarantee is what made it safe to seal the venv, and it is preserved here even with installs re-enabled. - tools/lazy_deps.py: durable-target mode — `--target` install + core-pinned `--constraint` file (shared deps resolve to core's versions, conflicts fail loudly at install time), append-only sys.path activation, ABI/Python-version stamp that wipes the store if an image rebuild bumps the interpreter, and a reworked gate so HERMES_DISABLE_LAZY_INSTALLS=1 redirects (rather than hard- blocks) when a target is set. security.allow_lazy_installs=false still disables installs in every mode. - hermes_bootstrap.py: activate the durable target on sys.path at first import (before any backend imports its SDK) so packages installed on a previous run are importable on this run. - Dockerfile: set HERMES_LAZY_INSTALL_TARGET=/opt/data/lazy-packages. - docker/stage2-hook.sh: seed + chown the dir on the data volume. - tests: real-install E2E proving installs land in the target, import cleanly, don't leak into the sealed venv, and that a core package is never shadowed; ABI-stamp wipe/preserve; gate matrix; Dockerfile/stage2 contract test. Fixes #51136 |
||
|
Ben
|
8446c15706 |
docs(chronos): pin hop-1 auth to the hosted-agent bootstrap token
The wire contract said hop 1 uses "the agent's existing Nous Portal
access token" but didn't name WHICH of an agent's two identities that is.
A hosted agent never holds an `agent:{instanceId}` OAuth client (that
shape is minted only by the interactive dashboard auth-code grant); its
own outbound portal calls use the bootstrap-session token (client
`hermes-cli-vps`) planted in auth.json on first boot. NAS must resolve
the instance id from either an `agent:{id}` client OR the bootstrap
session (AgentInstance.bootstrapSessionId), not gate on `agent:*` alone —
which 403'd every real hosted-agent provision in prod.
Documents the NAS-side fix (resolveAgentCronInstanceId) so the contract
and the implementation agree.
|
||
|
Ben
|
c93b9f9057 |
feat(relay): terminal 4401 (opt-out) → clean "Relay disabled" state
Phase 7 Unit 7d-B. When an operator opts an instance OUT of the Team Gateway
relay (Unit 7b deprovision), the connector revokes the per-gateway secret and
closes the gateway's WS with 4401. The reconnect supervisor previously treated
EVERY close as retryable, so the live process spun "retrying 4401" forever and
the dashboard showed a red error — opt-out looked like a failure.
Now a 4401 close that arrives AFTER a successful handshake is recognized as a
terminal credential revocation:
- ws_transport.py: track `_handshake_succeeded` (set when a descriptor is
received); on a 4401 close after a prior success, latch `auth_revoked` and do
NOT spawn the reconnect supervisor. A 4401 BEFORE any successful handshake
stays retryable (cold-start / not-yet-provisioned race, not a revocation).
New `auth_revoked` property + a websockets-version-safe close-code reader
(prefers `.rcvd`/`.sent` Close frames; `.code` is deprecated in websockets 13+).
- adapter.py: a revocation monitor turns `transport.auth_revoked` into a clean,
NON-retryable `relay_disabled` fatal and notifies the gateway's fatal-error
handler (so the adapter is removed and NOT queued for reconnection — the
credential is dead until the instance is recreated). Monitor is cancelled on
disconnect; only started when the transport exposes `auth_revoked` (prod WS).
- run.py: `_handle_adapter_fatal_error` maps the `relay_disabled` code to a
`disabled` platform_state (not `fatal`/`retrying`).
- web: PlatformsCard renders the `disabled` state with a neutral outline badge,
a PowerOff icon, and muted (not destructive-red) text + message. New optional
`status.disabled` i18n string ("Disabled").
Also bundles the Phase 7 contract-doc update (this doc is authoritative in
hermes-agent): docs/relay-connector-contract.md gains an "Author-first
resolution + the account-link (DM) path" section documenting the
multi-tenant-guild rule (D-7.2 — route by authenticated author binding, never by
guild; unlinked → fail-closed), the `/link <code>` DM flow, and the
connector-authoritative opt-out + terminal-4401 behavior this PR implements.
Tests: +2 ws_transport (4401-after-handshake terminal / no-reconnect;
4401-before-handshake stays retryable) and +2 adapter (revocation → non-retryable
relay_disabled fatal + handler fired; no-revocation → no fatal). 138 relay tests
pass (incl. the contract-doc conformance test); ruff clean; web tsc clean.
Phase 7 Unit 7d-B (relay-adapter solo lane). Q17 → Option 2; Option 3 (live
de-register, no recreate) + the restart-re-provision hole deferred post-alpha.
|
||
|
Ben
|
31bced1607 |
fix(profiles): detect a separate-process gateway in profile status
The dashboard Profiles view showed "Gateway stopped" for a gateway that
is in fact running — while the sidebar status strip and `hermes gateway
status` (CLI) both correctly showed it running. Reported on v0.17.0
running the gateway + dashboard in one Docker container.
Root cause: three liveness surfaces with three detection strengths, all
reading the same `gateway.pid`:
- `hermes gateway status` -> find_gateway_pids() (process-table scan)
- sidebar /api/status -> get_running_pid() + gateway_state.json PID
fallback + health-URL probe
- Profiles view -> _check_gateway_running() = get_running_pid()
ONLY, no fallback
`get_running_pid()` short-circuits to None the moment the runtime lock
(`gateway.lock`) doesn't register as held by the *calling* process —
which is always true when the reader is a separate process from the
gateway (the dashboard is its own s6 service in the container), and also
for any launch-service-managed gateway that left a fresh
`gateway_state.json` but no live PID file. So the Profiles view alone
reported the live gateway as stopped.
Fix: give _check_gateway_running the same fallback the sidebar already
has — after the pid-file/lock check misses, validate the PID recorded in
that profile's gateway_state.json against the live process table via the
existing get_runtime_status_running_pid(). read_runtime_status() gains an
optional path arg so a profile's state file can be read without mutating
the process-global HERMES_HOME (preserving the contextvar-based profile
isolation the dashboard relies on). Backward compatible: every existing
caller passes no argument.
Tests: a regression test that fails pre-fix (live gateway, lock check
returns None -> must still report running) and a guard test that a
'stopped' state file is never reported running even with a live PID.
|
||
|
Ben Barclay
|
935f2bc48d |
docs(relay): add §3.4 — obligations on a future scale-to-zero behaviour layer (#51633)
The contract already documents the scale-to-zero PRIMITIVES (§3.2 going-idle/ buffered-flip, §3.3 wake poke) and what's out of scope. This adds the missing half: the contract FROM the primitives TO the behaviour layer — the guarantees a separate scale-to-zero workstream must honour to consume them safely (register a wakeUrl before suspend; drain+ack before teardown; keep the reconnect loop live; treat suspended != down in the health model; don't assume exactly-once/ prompt wake; suspend only when genuinely idle, composing with the existing drain machine). Docs-only; lets the independent scale-to-zero stream build against a written contract instead of re-reading the connector. |
||
|
Ben Barclay
|
6e88f7b6f7 |
feat(relay): Phase 5 Unit C — wake primitive (gateway side) (#51595)
Register a per-instance wakeUrl and forward it to the connector at self-provision so a suspended gateway can be poked awake when buffered work arrives (pairs with the connector-side WakePoker). - relay_wake_url() resolver (env GATEWAY_RELAY_WAKE_URL, then gateway.relay_wake_url in config.yaml), mirroring relay_instance_id() - thread wake_url through _post_provision (adds wakeUrl to the body only when set) + self_provision_relay (resolve, forward, log) - hermes gateway enroll --wake-url <url> persists GATEWAY_RELAY_WAKE_URL - document the §5.2 wake poke in relay-connector-contract.md §3.3 - tests: relay_wake_url resolution (env/config/absent), provision forwarding, body-only-when-set (6 new; 130 relay tests pass) The actual reconnect+drain on wake is Unit B's loop; this unit only wires the wake SIGNAL. Opt-in: absent wakeUrl => connector never pokes. |
||
|
Ben Barclay
|
40fddc9e4c |
feat(relay): Phase 5 §5.3 going-idle / buffered-flip primitive (gateway side) (#51572)
The gateway half of the going-idle/buffered-flip primitive (scale-to-zero PRIMITIVE, not the behaviour). Integrates with the EXISTING drain transition: - ws_transport: `go_idle()` sends `going_idle` + awaits the connector's `going_idle_ack` (connector-authoritative flip-then-ack, Q-5.3c — stays serving until the ack so nothing is lost in the flip window); acks a buffered inbound (bufferId present) via `inbound_ack` after the handler runs (drain-without-dup on the delivery leg); NET-NEW reconnect loop re-dials + re-handshakes after an unexpected close (off by default, on in production). - adapter: emits `going_idle` from its existing `disconnect()` drain seam before tearing down the socket; best-effort + guarded (never blocks shutdown). - transport Protocol + contract doc §3.2 document the 3 new frames. +6 relay tests (124 pass). NOT in scope: the autonomous idle timer / machine suspend / NAS health model (deferred behaviour). Ben's relay-adapter solo lane. |
||
|
Ben
|
2196584161 |
fix(slack): transcribe in-app voice messages (audio/mp4) instead of failing
Slack in-app voice clips ("record a clip") arrive as MP4/AAC containers
(mimetype audio/mp4, filename audio_message*.mp4), and Slack sometimes
labels them video/mp4. The inbound audio handler derived the cache
extension from the mimetype and fell back to ".ogg" for anything not in
{.ogg,.mp3,.wav,.webm,.m4a} — so audio/mp4 voice messages were cached as
.ogg. OpenAI STT (whisper-1, gpt-4o-transcribe) sniffs the container from
the FILENAME extension, so it received MP4 bytes named .ogg and rejected
them. WhatsApp .ogg and uploaded .m4a worked only because their extension
happened to match the bytes.
Fix:
- _resolve_slack_audio_ext(): pick the cache extension from the real
filename first, then a mimetype map (audio/mp4 -> .m4a), defaulting to
.m4a — never the bogus .ogg fallback. Mirrors the video branch and the
audio map already in gateway/platforms/bluebubbles.py.
- _is_slack_voice_clip(): detect audio-only clips mislabeled video/mp4
via the slack_audio subtype / audio_message* filename, and route them
through the audio path (cached as audio, reported as audio/*) so they
reach STT instead of video understanding. Genuine videos (and
slack_video screen recordings) are left on the video path.
Verified end-to-end against a real audio-only MP4: old path cached it as
.ogg (ffprobe shows MP4 bytes -> container mismatch -> OpenAI rejects);
new path caches it as .mp4 (extension matches bytes -> accepted).
Adds inbound-audio tests (previously none): helper unit tests plus
_handle_slack_message E2E coverage for audio/mp4, video/mp4-mislabeled
voice clips, and a real video staying on the video path. Confirmed the
two voice-message tests fail without the fix (mutation check).
|
||
|
Ben Barclay
|
45bc4fb37f |
feat(relay): declare relevance policy to the connector + document the management plane (#51248)
The gateway half of Phase 6 Unit ζ: project the agent's existing relevance
knobs into the connector's platform-agnostic vocabulary and declare them at boot
over the /relay/policy route, so the SAME mention-gating / free-response /
allow-bots behavior the agent applies directly also governs relay delivery (and
excluded chatter never wakes a scaled-to-zero agent).
- gateway/relay/__init__.py:
- relay_relevance_policy(): project require_mention -> requireAddress,
free_response_channels -> freeResponseScopes, {PLATFORM}_ALLOW_BOTS in
{mentions,all} -> allowOtherBots. Reads the fronted platform's config block
+ bridged top-level keys. Returns None when all-default (the connector's
quiet default already matches) or no concrete platform is fronted.
- send_relay_policy(): POST /relay/policy authenticated with the gateway's own
per-gateway upgrade token (make_upgrade_token — same bearer as the WS
upgrade), so the connector attaches it to the authenticated instance, never
a body-asserted id. Re-declares every boot (self-healing, full replace).
NEVER raises, NEVER blocks boot — relevance is an optimization layered on
the δ/ε authorization gate. Reuses the per-gateway secret + the
/relay/provision host; no new inbound surface, no new credential.
- _policy_url(): ws(s)://…/relay -> http(s)://…/relay/policy.
- gateway/run.py: call send_relay_policy() after register_relay_adapter()
succeeds (the secret is resolved by then).
- docs/relay-connector-contract.md: new §7 documenting per-instance delivery +
the management plane (/manage/* + /relay/policy) + the relevance-declaration
contract; versioning renumbered to §8. Contract conformance test stays green
(§2/§3 tables untouched).
Tests: +12 (projection mapping incl. comma-string + top-level fallback; send
auth/skip/fail-soft/non-200). Full relay suite 118 pass. The connector route is
already E2E-proven (connector repo gateway_policy_driver.py); this adds the real
gateway send-path it pairs with.
This completes Phase 6 (Team Gateway per-user isolation) end to end.
|
||
|
Ben Barclay
|
75a70d98f3 |
feat(relay): forward a stable instance id at self-provision (Phase 6 Unit α) (#50772)
Add relay_instance_id() (env GATEWAY_RELAY_INSTANCE_ID first, then gateway.relay_instance_id in config.yaml, mirroring the other relay readers) and forward it in the /relay/provision body so the connector can bind gatewayId -> instanceId and route inbound per-instance once Phase 6 delivery lands. The value is gateway-asserted but safely scoped: the org/tenant stays NAS-token-verified at the connector, so a dishonest gateway can only bind its OWN tenant's instance — same posture as relay_endpoint(). instanceId is only added to the body when present, so omitting it lets the connector store null (back-compat: self-hosted / pre-Phase-6 gateways simply have no binding yet). For a managed (NAS-hosted) agent the id is NAS's AgentInstance.id, stamped into the container env beside GATEWAY_RELAY_URL. Tests: reader (env/config/absent), self_provision_relay forwards the id (set + absent), and the real _post_provision body includes instanceId ONLY when set. Refs: ~/nous/specs/gateway-gateway plan.md Phase 6 Unit α; decisions.md Q11. |
||
|
Ben Barclay
|
64a507da44 |
feat(relay): handle passthrough_forward over the WS (Phase 5 §5.1, gateway half) (#50702)
The connector half (gateway-gateway) moves the passthrough plane's post-ACK forward off the HTTP gatewayEndpoint onto the gateway's outbound /relay WS via a new passthrough_forward frame. This is the gateway side: the relay adapter now RECEIVES and handles that frame, so a hosted gateway (no public IP) can process forwarded Class-2/3 traffic (Discord interactions, Twilio) over the socket it already holds — closing the "passthrough inbound doesn't work for hosted gateways" gap. - ws_transport.py: decode the passthrough_forward frame; PassthroughForward dataclass + _passthrough_from_wire (base64 body -> exact bytes, byte parity with the connector's toPassthroughForward); set_passthrough_handler mirrors set_interrupt_inbound_handler. - transport.py: PassthroughHandler type + set_passthrough_handler on the RelayTransport protocol. - adapter.py: connect() wires the passthrough handler; _on_passthrough decodes the (already-sanitized, token-free) forward and, for a Discord interaction, converts it to a MessageEvent routed through the normal agent path (handle_message) — the reply egresses over the outbound / token-less follow_up path, so the gateway never holds the interaction credential. Never raises (a bad forward can't kill the read loop). Non-discord forwards (Twilio) are logged + dropped for now. - docs/relay-connector-contract.md: document the passthrough_forward frame + PassthroughForward shape + §3.1. The interaction -> MessageEvent CONVERSION semantics (slash-command vs button UX, option rendering) are the open sub-design flagged in the spec; the TRANSPORT + receive mechanism (this) is settled per Ben's Gate-2 decision: "the relay adapter handles receiving these events over the WS." Tests (tests/gateway/relay/test_relay_passthrough.py): byte-preservation round-trip (+ malformed-body tolerance), connect() wiring, application-command and message-component interactions route through handle_message with correct session source + scope capture, malformed/non-discord forwards dropped cleanly. 100 relay tests green. Pairs with the connector PR (gateway-gateway). |
||
|
Ben Barclay
|
de6b3ae377 |
fix(terminal): bridge docker_extra_args to TERMINAL_DOCKER_EXTRA_ARGS in CLI + gateway (#50631)
terminal.docker_extra_args passes flags verbatim to `docker run` (e.g. --gpus=all, --shm-size=16g). It was wired into DEFAULT_CONFIG, TERMINAL_CONFIG_ENV_MAP (so `hermes config set` bridged it), terminal_tool._get_env_config (reads TERMINAL_DOCKER_EXTRA_ARGS), and DockerEnvironment (applies extra_args) -- but it was MISSING from cli.py's env_mappings and gateway/run.py's _terminal_env_map. Consequence: a user who hand-edits config.yaml (rather than running `hermes config set`) has docker_extra_args silently dropped on the CLI and gateway/desktop startup paths, while docker_image / docker_volumes (which ARE in those maps) bridge correctly -- producing the reported 'Hermes partially reads the Docker config' symptom where --gpus=all and --shm-size=16g never reach docker run. This is the same bridge-coverage bug class that shipped before for docker_run_as_host_user (cli + gateway) and docker_mount_cwd_to_workspace (gateway). Fix by adding the key to both maps, plus a dedicated regression pin in test_terminal_config_env_sync.py mirroring the existing test_docker_*_is_bridged_everywhere guards. |
||
|
Ben Barclayandkyssta-exe
|
6202fdfc35 |
fix(container): detect dashboard role under s6-overlay v3 (#49196) (#50600)
* fix(gateway): walk /proc/*/cmdline to find main-wrapper.sh under s6-overlay v3 (#49196) (cherry picked from commit 3a108c2df0edce4ce0e6f9f3a8eb8db3839a4630) * fix(container): peel s6-v3 rc.init prefix so dashboard role is detected kyssta-exe's preceding commit (#49238) fixed _read_container_argv() to locate the rc.init-launched main-wrapper.sh process under s6-overlay v3, but the skip still never fired: _strip_container_argv_prefix() only peeled a prefix when args[0] was init/main-wrapper.sh/hermes. Under s6 v3 the matched argv is /bin/sh -e /run/s6/basedir/scripts/rc.init top /opt/hermes/docker/main-wrapper.sh dashboard ... so args[0] stayed /bin/sh, _is_dashboard_container() returned False, and the dashboard container reconciled + started its own gateway-default — the exact dual Telegram getUpdates 409 in issue #49196. Fix: strip everything up to and including the main-wrapper.sh token (the stable boundary the image owns), covering both the v2 (/init ...) and v3 (/bin/sh ... rc.init top ...) shapes with one rule, instead of matching launcher tokens positionally. This also repairs _is_legacy_gateway_run_request() under v3, which shares the same strip helper (the issue called this out). Tests: extend the dashboard true/false parametrize sets with the s6-v3 argv shape, and add test_main_skips_reconcile_in_dashboard_container_s6v3 exercising main() end-to-end with the v3 argv. Verified via mutation that both new v3 assertions fail under the old positional strip and pass with the fix. --------- Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com> |
||
|
Ben
|
0ee75469d7 |
feat(dashboard): surface gateway busy/drainable on /api/status
Give an external consumer (NAS) a trustworthy, always-reachable busy/idle
readout it can poll before a disruptive lifecycle action (restart,
migrate, stop, auto-update). The dashboard /api/status is the only HTTP
surface guaranteed up on a hosted agent regardless of which gateway
platforms are enabled, and it already reads gateway_state.json.
Add to /api/status (additive, non-breaking):
- active_agents — in-flight gateway-turn count (now refreshed
per-turn by the companion gateway-side commit)
- gateway_busy — running AND active_agents > 0
- gateway_drainable — running and live (a valid begin-drain target)
- restart_drain_timeout — resolved seconds, so the consumer can size its
poll deadline without out-of-band knowledge
(env HERMES_RESTART_DRAIN_TIMEOUT → config
agent.restart_drain_timeout → default)
The busy/drainable contract is defined once in gateway.status
(derive_gateway_busy / derive_gateway_drainable) and consumed by both
/api/status and /health/detailed so the two surfaces can never disagree.
Liveness keys off gateway_running (a live PID/health probe), NEVER
gateway_updated_at — a healthy idle gateway never advances that timestamp.
All derived fields degrade to safe falsy values when the gateway is down
or the status file is absent/corrupt (never a spurious "busy" that would
wedge the consumer). active_sessions (the 5-min DB recency heuristic the
SPA reads) is left exactly as-is — new signal, new fields.
Tests (behaviour contracts, not snapshots): the pure derivation contract
across every running/state/count/liveness combination; /api/status
integration for busy, idle-drainable, draining, down, stale-busy-file,
corrupt-count, and timeout surfacing; and /health/detailed parity.
|
||
|
Ben
|
51a338a1b6 |
feat(gateway): track active_agents in runtime status on turn boundaries
The gateway only rewrote gateway_state.json on lifecycle transitions
(start/connect/drain/stop), never on turn start/end. Live-verified on a
hosted agent: a confirmed end-to-end turn ran while gateway_updated_at
stayed frozen at boot and active_agents was absent — so any active_agents
read from the file between transitions is stale. That makes it unusable
as a busy/idle signal for an external consumer (NAS deciding whether it's
safe to restart/migrate/auto-update an agent mid-turn).
Add _persist_active_agents(), called at every turn boundary:
- turn start: both running-agent sentinel-claim sites (normal inbound
message path + startup-resume path)
- turn end: the central _release_running_agent_state() choke point
(covers normal completion, /stop, /reset, sentinel cleanup,
stale-eviction — every path that ends a running turn)
It passes ONLY active_agents to write_runtime_status, leaving
gateway_state (and every other field) _UNSET so the read-merge-write
preserves the current lifecycle state. Passing gateway_state=None would
clobber it — hence a dedicated helper rather than reusing
_update_runtime_status. The write is the same cheap JSON write done on
lifecycle transitions today; best-effort (a failed status write never
disrupts a turn).
Behaviour-contract test: an active_agents-only write preserves both
running and draining gateway_state, and the count clamps non-negative.
|
||
|
Ben
|
1928aa0443 |
fix(managed-scope): honor managed scope in config→env bridges too
Manual verification surfaced a second bypass class beyond the standalone config loaders: several code paths bridge config.yaml values into os.environ (HERMES_TIMEZONE, HERMES_REDACT_SECRETS, HERMES_MAX_ITERATIONS, TERMINAL_*, network.force_ipv4, ...) by reading the raw user YAML, so the env the whole process reads carried the USER's value even when an administrator pinned it — e.g. a managed timezone was overridden because gateway/run.py wrote the user's timezone into HERMES_TIMEZONE, and _resolve_timezone_name() checks the env var first. Wired the shared apply_managed_overlay() into every config→env bridge: - gateway/run.py module-level startup bridge (timezone, redact_secrets, max_turns, terminal, display, gateway.strict, ...) - gateway/run.py _reload_runtime_env_preserving_config_authority (the per-turn re-bridge that keeps config authoritative over reloaded .env — must keep MANAGED authoritative on every turn, not just startup) - hermes_cli/main.py early security.redact_secrets / network.force_ipv4 bridge (runs before load_config is usable, at import time) - hermes_cli/send_cmd.py top-level scalar config→env bridge Verified end-to-end against a writable managed dir (12/12 checks incl. timezone, logging, model, skin, gateway settings, write-guard) and in a clean process the gateway per-turn bridge writes HERMES_TIMEZONE=<managed>. Adds an order-independent regression test for the bridge overlay. |
||
|
Ben
|
b0e47a98f9 |
fix(managed-scope): honor managed scope in all standalone config loaders
The skin bug was one instance of a class: several subsystems build their
config dict directly from config.yaml instead of routing through
hermes_cli.config.load_config (which carries the managed merge), so they
silently ignored administrator-pinned values. Audited every config.yaml
reader and fixed the behavioral-read bypasses:
- gateway/config.py load_gateway_config (messaging gateway: session_reset,
quick_commands, stt, model, ...)
- gateway/run.py _load_gateway_config (its read_raw_config fast path also
skipped the merge — read_raw_config returns raw user YAML)
- tui_gateway/server.py _load_cfg (new TUI + desktop backend: skin,
reasoning_effort, service_tier, provider_routing)
- cron/scheduler.py (scheduled-job model/reasoning/toolsets/provider_routing)
- hermes_logging.py (logging.level/max_size_mb/backup_count)
- hermes_time.py (timezone)
- hermes_cli/doctor.py (memory-provider diagnostic reads effective config)
All route through a new shared managed_scope.apply_managed_overlay() helper
that mirrors _load_config_impl (env-only expansion so a user ${VAR} can't
shadow a managed literal, root-model-string normalization, leaf-merge) and is
fail-open. cli.py's earlier inline fix is refactored onto the same helper.
Write-back paths (slash_commands, telegram/yuanbao dm_topics, profile
distribution) are deliberately left reading raw user YAML — overlaying managed
values there would persist them into the user file. The dashboard
(web_server.py) already routes through load_config and needed no change.
TUI loader caches the RAW config so _save_cfg never writes managed values to
disk. Adds test_managed_scope_overlay.py (helper) and
test_managed_scope_loaders.py (per-surface integration); mutation-checked.
|
||
|
Ben
|
732293cf87 |
fix(managed-scope): apply managed layer in cli.py's standalone config loader
cli.py's load_cli_config() builds CLI_CONFIG independently of
hermes_cli.config._load_config_impl (it reads config.yaml directly and merges
into hardcoded defaults), so the Phase 2 managed merge never reached the
interactive CLI/TUI surface. Symptom: a managed display.skin (and any other
display/CLI pref read from CLI_CONFIG) was silently ignored by the TUI while
`hermes config`/`doctor`/write-guards — which go through load_config — correctly
honored it. Found via manual testing: the skin engine kept using 'default'.
Fix: overlay the managed config last in load_cli_config(), mirroring
_load_config_impl — expand against the process env only (so a user ${VAR} can't
shadow a managed literal), normalize the root model key so a managed
`model: x/y` string can't clobber the dict shape callers expect, then
leaf-merge. Fail-open so managed scope can never block CLI startup.
Adds tests/hermes_cli/test_managed_scope_cli_config.py locking that CLI_CONFIG
honors managed values, preserves user siblings, and is inert with no scope.
|