feat(gateway): opt-in compression progress notices via compression.progress_notices (#52995) (#70457)

Routine automatic compression stays silent-by-design on chat platforms
(default unchanged, byte-identical). New opt-in config key
compression.progress_notices (bool, default false) opens a gate on the
gateway noise filter (_prepare_gateway_status_message) that lets ROUTINE
compression progress statuses through to chat surfaces.

- Membership is derived from the #69550 status template constants in
  agent/conversation_compression.py (compiled to a literal-escaped regex,
  never re-inlined wording), so unrelated noisy statuses (aux failures,
  provider retry/rate-limit chatter) stay suppressed even when enabled.
- The compaction completion notice (COMPACTION_DONE_STATUS, #69546
  lifecycle 'compacted' edge) already flows through the status path and
  passes the filter — no new emit site needed.
- Config wired everywhere: hermes_cli/config.py DEFAULT_CONFIG,
  cli-config.yaml.example, gateway raw-YAML read (live, mtime-cached),
  gateway hot-reload cache-busting key list, website configuration docs.
- Failure notices and manual /compress feedback remain always-visible;
  VISIBLE_COMPRESSION_MESSAGES and emit sites untouched.

Design by @havok-training (issue #52995).

cli-config.yaml.example

@@ -407,6 +407,18 @@ compression:
   # Enable automatic context compression (default: true)
   # Set to false if you prefer to manage context manually or want errors on overflow
   enabled: true
+
+  # Opt-in compression progress notices on chat platforms (default: false).
+  # By design, routine automatic compression is SILENT on human-facing chat
+  # gateways (Telegram, Discord, Slack, ...) — it happens in the background
+  # with server-side logging only. Set true to also deliver the routine
+  # progress statuses (compacting started, preflight/pre-API compression,
+  # idle compaction, retry progress, and the compaction-complete notice) to
+  # chat platforms. Unrelated operational noise (auxiliary model failures,
+  # provider retry chatter) stays suppressed either way, and compression
+  # FAILURE notices + manual /compress feedback are always visible
+  # regardless of this setting. (#52995)
+  progress_notices: false
   
   # Trigger compression at this % of model's context limit (default: 0.50 = 50%)
   # Lower values = more aggressive compression, higher values = compress later

gateway/run.py

@@ -54,6 +54,16 @@ from typing import Awaitable, Callable, Dict, Optional, Any, List, Union, cast
 # preserving the established test-patch surface.
 from agent.account_usage import fetch_account_usage, render_account_usage_lines
 from agent.async_utils import consume_detached_task_result, safe_schedule_threadsafe
+from agent.conversation_compression import (
+    COMPACTION_STATUS,
+    COMPRESSION_RETRY_CONTEXT_REDUCED_STATUS_TEMPLATE,
+    COMPRESSION_RETRY_MESSAGES_STATUS_TEMPLATE,
+    COMPRESSION_RETRY_TOKENS_STATUS_TEMPLATE,
+    COMPRESSION_RETRY_TOO_LARGE_STATUS_TEMPLATE,
+    IDLE_COMPACTION_STATUS_TEMPLATE,
+    PRE_API_COMPRESSION_STATUS_TEMPLATE,
+    PREFLIGHT_COMPRESSION_STATUS_TEMPLATE,
+)
 from agent.conversation_loop import INTERRUPT_WAITING_FOR_MODEL_PREFIX
 from agent.i18n import t
 from hermes_cli.config import cfg_get
@@ -106,6 +116,70 @@ _TELEGRAM_NOISY_STATUS_RE = re.compile(
     re.IGNORECASE | re.DOTALL,
 )
 
+
+def _status_template_to_regex(template: str) -> str:
+    """Compile a compression status template constant into a regex source.
+
+    Literal text is escaped verbatim (so wording drift in
+    agent/conversation_compression.py cannot silently diverge from this
+    matcher — the constants ARE the wording) and each ``{field}`` format
+    placeholder is replaced with a numeric-ish pattern covering every value
+    the emit sites format in (ints, ``{:,}`` thousands separators).
+    """
+    parts = re.split(r"\{[^{}]*\}", template)
+    return r"[\d,]+".join(re.escape(part) for part in parts)
+
+
+# ROUTINE compression progress statuses, derived from the SAME template
+# constants the emit sites format (agent/conversation_compression.py, #69550)
+# — never re-inlined wording. Used ONLY by the opt-in
+# ``compression.progress_notices`` gate below (#52995) to decide which of the
+# noisy statuses matched by _TELEGRAM_NOISY_STATUS_RE are compression
+# progress (deliverable when the user opted in) versus unrelated aux/retry
+# chatter (always suppressed on chat surfaces). Failure notices and manual
+# /compress feedback never match _TELEGRAM_NOISY_STATUS_RE in the first
+# place, so they are unaffected by this gate.
+_COMPRESSION_PROGRESS_STATUS_RE = re.compile(
+    "|".join(
+        _status_template_to_regex(_template)
+        for _template in (
+            COMPACTION_STATUS,
+            PRE_API_COMPRESSION_STATUS_TEMPLATE,
+            PREFLIGHT_COMPRESSION_STATUS_TEMPLATE,
+            IDLE_COMPACTION_STATUS_TEMPLATE,
+            COMPRESSION_RETRY_TOO_LARGE_STATUS_TEMPLATE,
+            COMPRESSION_RETRY_MESSAGES_STATUS_TEMPLATE,
+            COMPRESSION_RETRY_TOKENS_STATUS_TEMPLATE,
+            COMPRESSION_RETRY_CONTEXT_REDUCED_STATUS_TEMPLATE,
+        )
+    ),
+    re.IGNORECASE,
+)
+
+
+def _gateway_compression_progress_notices_enabled() -> bool:
+    """True when the user opted into routine compression progress notices.
+
+    Reads ``compression.progress_notices`` from the gateway's raw YAML config
+    (#52995). Default False — routine compression stays silent-by-design on
+    chat platforms unless explicitly enabled. Read live (mtime-cached) so a
+    config edit on a running gateway takes effect on the next status.
+    Fail-closed: any config read error keeps the silent default.
+    """
+    try:
+        config = _load_gateway_config()
+        compression_cfg = config.get("compression") if isinstance(config, dict) else None
+        if isinstance(compression_cfg, dict):
+            return str(compression_cfg.get("progress_notices", False)).strip().lower() in {
+                "true",
+                "1",
+                "yes",
+                "on",
+            }
+    except Exception:
+        pass
+    return False
+
 # Surfaces that consume gateway text programmatically (CLI/TUI "local"
 # diagnostics, API JSON, webhook payloads) and therefore must keep RAW
 # status/error text. EVERY other platform is a human-facing chat surface
@@ -495,7 +569,17 @@ def _prepare_gateway_status_message(platform: Any, event_type: str, message: str
 
     text = _redact_gateway_user_facing_secrets(text)
     if _TELEGRAM_NOISY_STATUS_RE.search(text):
-        return None
+        # Opt-in #52995: `compression.progress_notices: true` lets ROUTINE
+        # compression progress statuses through to chat platforms. The
+        # membership check is derived from the #69550 template constants, so
+        # non-compression noise (aux failures, provider retry chatter, ...)
+        # stays suppressed even when the gate is open. Default False keeps
+        # the silent-by-design behavior byte-identical.
+        if not (
+            _gateway_compression_progress_notices_enabled()
+            and _COMPRESSION_PROGRESS_STATUS_RE.search(text)
+        ):
+            return None
     if _looks_like_gateway_provider_error(text):
         return _gateway_provider_error_reply(text)
     return text
@@ -18166,6 +18250,7 @@ class GatewayRunner(GatewayAuthorizationMixin, GatewayKanbanWatchersMixin, Gatew
         ("model", "context_length"),
         ("model", "max_tokens"),
         ("compression", "enabled"),
+        ("compression", "progress_notices"),
         ("compression", "threshold"),
         ("compression", "model_thresholds"),
         ("compression", "threshold_tokens"),

hermes_cli/config.py

@@ -1398,6 +1398,15 @@ DEFAULT_CONFIG = {
 
     "compression": {
         "enabled": True,
+        "progress_notices": False,    # opt-in (#52995): when True, routine compression
+                                      # progress statuses (compacting/preflight/pre-API/
+                                      # idle/retry) are delivered to chat gateway
+                                      # platforms instead of being suppressed by the
+                                      # gateway noise filter. Default False keeps
+                                      # routine compression silent-by-design on chat
+                                      # surfaces (server-side logging only). Failure
+                                      # notices and manual /compress feedback are
+                                      # always visible regardless of this setting.
         "threshold": 0.50,            # compress when context usage exceeds this ratio.
                                       # Models with context windows below 512K are
                                       # floored at 0.75 (raise-only) so compaction

tests/gateway/test_compression_progress_notices.py

@@ -0,0 +1,172 @@
+"""Opt-in compression progress notices on chat gateways (#52995).
+
+Routine automatic compression is silent-by-design on human-facing chat
+platforms: the gateway noise filter (`_TELEGRAM_NOISY_STATUS_RE` via
+`_prepare_gateway_status_message`) swallows every ROUTINE compression status.
+`compression.progress_notices: true` (default: false) opens an opt-in gate
+that lets those ROUTINE compression statuses through — scoped strictly to
+the #69550 compression status template constants, so unrelated operational
+noise (auxiliary failures, provider retry/rate-limit chatter) stays
+suppressed even when the gate is enabled.
+
+The default-OFF path must stay byte-identical to silent-by-design main:
+tests/gateway/test_telegram_noise_filter.py pins that suite unchanged.
+"""
+
+import pytest
+
+import gateway.run as gateway_run
+from agent.conversation_compression import (
+    COMPACTION_DONE_STATUS,
+    ROUTINE_COMPRESSION_STATUS_SAMPLES,
+)
+from gateway.run import _prepare_gateway_status_message
+
+# Chat surfaces the opt-in must deliver to (subset of the noise-filter
+# suite's CHAT_PLATFORMS; telegram + discord are the required anchors).
+CHAT_PLATFORMS = ["telegram", "discord", "slack", "whatsapp"]
+
+# Noisy statuses that are NOT routine compression progress — they must stay
+# suppressed on chat platforms even when progress_notices is enabled.
+NON_COMPRESSION_NOISE = [
+    "⚠ Auxiliary title generation failed: HTTP 400: Operation contains cybersecurity risk",
+    "⏳ Retrying in 4.2s (attempt 1/3)...",
+    "⏱️ Rate limited. Waiting 30.0s (attempt 2/3)...",
+    "⚠️ Max retries (3) exhausted — trying fallback...",
+    "⚠ Compression summary failed: upstream error. Inserted a fallback context marker.",
+    (
+        "⚠ Configured auxiliary compression provider 'openai' is unavailable — "
+        "context compression will drop middle turns without a summary. Check "
+        "auxiliary.compression in config.yaml and reauthenticate that provider."
+    ),
+    (
+        "⚠ Skipping concurrent compression — another path is already "
+        "compressing this session. Will retry after it finishes."
+    ),
+]
+
+
+@pytest.fixture
+def progress_notices_enabled(monkeypatch):
+    """Gateway config with compression.progress_notices: true."""
+    monkeypatch.setattr(
+        gateway_run,
+        "_load_gateway_config",
+        lambda: {"compression": {"progress_notices": True}},
+    )
+
+
+@pytest.fixture
+def progress_notices_default(monkeypatch):
+    """Gateway config without the key — the silent-by-design default."""
+    monkeypatch.setattr(gateway_run, "_load_gateway_config", lambda: {})
+
+
+@pytest.mark.parametrize("platform", CHAT_PLATFORMS)
+@pytest.mark.parametrize(
+    "message", ROUTINE_COMPRESSION_STATUS_SAMPLES, ids=lambda m: m[:32]
+)
+def test_enabled_delivers_routine_compression_statuses(
+    progress_notices_enabled, platform, message
+):
+    """Opt-in ON: every ROUTINE compression status reaches chat platforms.
+
+    Iterates the sample strings formatted from the SAME template constants
+    the emit sites use, so wording drift at an emit site cannot silently
+    detach the opt-in gate from the real messages.
+    """
+    assert _prepare_gateway_status_message(platform, "lifecycle", message) == message
+
+
+@pytest.mark.parametrize("platform", CHAT_PLATFORMS)
+@pytest.mark.parametrize(
+    "message", ROUTINE_COMPRESSION_STATUS_SAMPLES, ids=lambda m: m[:32]
+)
+def test_default_stays_silent(progress_notices_default, platform, message):
+    """Default (key absent): routine compression statuses stay suppressed."""
+    assert _prepare_gateway_status_message(platform, "lifecycle", message) is None
+
+
+@pytest.mark.parametrize("platform", CHAT_PLATFORMS)
+def test_explicit_false_stays_silent(monkeypatch, platform):
+    """compression.progress_notices: false behaves exactly like the default."""
+    monkeypatch.setattr(
+        gateway_run,
+        "_load_gateway_config",
+        lambda: {"compression": {"progress_notices": False}},
+    )
+    for message in ROUTINE_COMPRESSION_STATUS_SAMPLES:
+        assert _prepare_gateway_status_message(platform, "lifecycle", message) is None
+
+
+@pytest.mark.parametrize("platform", CHAT_PLATFORMS)
+@pytest.mark.parametrize("message", NON_COMPRESSION_NOISE, ids=lambda m: m[:32])
+def test_enabled_still_suppresses_non_compression_noise(
+    progress_notices_enabled, platform, message
+):
+    """The gate is scoped to compression progress statuses ONLY.
+
+    Aux-model failures, provider retry/rate-limit chatter, and other noisy
+    statuses that are not #69550 compression progress templates must stay
+    suppressed on chat surfaces even when progress_notices is enabled.
+    """
+    assert _prepare_gateway_status_message(platform, "warn", message) is None
+
+
+@pytest.mark.parametrize("enabled", [True, False], ids=["enabled", "default"])
+@pytest.mark.parametrize("platform", CHAT_PLATFORMS)
+def test_compaction_completion_notice_reaches_chat(monkeypatch, platform, enabled):
+    """The #69546 'compacted' lifecycle edge is deliverable on chat surfaces.
+
+    COMPACTION_DONE_STATUS already flows through the status callback on
+    compaction completion and is not matched by the noise regex — the opt-in
+    gate must not change that in either mode, so users who enable
+    progress_notices see the completion stat notice paired with the start.
+    """
+    monkeypatch.setattr(
+        gateway_run,
+        "_load_gateway_config",
+        lambda: {"compression": {"progress_notices": enabled}},
+    )
+    assert (
+        _prepare_gateway_status_message(platform, "compacted", COMPACTION_DONE_STATUS)
+        == COMPACTION_DONE_STATUS
+    )
+
+
+def test_config_read_errors_fail_closed(monkeypatch):
+    """A broken config read keeps the silent-by-design default."""
+    def _boom():
+        raise RuntimeError("config unreadable")
+
+    monkeypatch.setattr(gateway_run, "_load_gateway_config", _boom)
+    message = ROUTINE_COMPRESSION_STATUS_SAMPLES[0]
+    assert _prepare_gateway_status_message("telegram", "lifecycle", message) is None
+
+
+def test_enabled_gate_does_not_leak_to_raw_platforms(progress_notices_enabled):
+    """Programmatic surfaces keep raw text regardless of the gate."""
+    message = ROUTINE_COMPRESSION_STATUS_SAMPLES[0]
+    for platform in ("local", "api_server", "webhook", "msgraph_webhook"):
+        assert (
+            _prepare_gateway_status_message(platform, "lifecycle", message) == message
+        )
+
+
+def test_progress_notices_is_a_hot_reload_cache_busting_key():
+    """Editing compression.progress_notices on a running gateway must take
+    effect like every other compression.* key (hot-reload key list)."""
+    assert ("compression", "progress_notices") in gateway_run.GatewayRunner._CACHE_BUSTING_CONFIG_KEYS
+
+
+def test_progress_regex_covers_every_routine_sample():
+    """The template-derived membership regex matches every ROUTINE sample.
+
+    Guards the coupling: a new #69550 template constant added to
+    ROUTINE_COMPRESSION_STATUS_SAMPLES without being added to the gateway's
+    _COMPRESSION_PROGRESS_STATUS_RE alternatives fails here.
+    """
+    for message in ROUTINE_COMPRESSION_STATUS_SAMPLES:
+        assert gateway_run._COMPRESSION_PROGRESS_STATUS_RE.search(message), (
+            f"routine compression sample not covered by the opt-in gate: {message!r}"
+        )

website/docs/user-guide/configuration.md

@@ -743,6 +743,7 @@ All compression settings live in `config.yaml` (no environment variables).
 ```yaml
 compression:
   enabled: true                                     # Toggle compression on/off
+  progress_notices: false                           # Opt-in: deliver routine compression progress notices to chat platforms — see below
   threshold: 0.50                                   # Compress at this % of context limit
   threshold_tokens: null                            # Absolute token cap (optional) — takes lower of ratio vs absolute
   target_ratio: 0.20                                # Fraction of threshold to preserve as recent tail
@@ -768,6 +769,8 @@ auxiliary:
 Older configs with `compression.summary_model`, `compression.summary_provider`, and `compression.summary_base_url` are automatically migrated to `auxiliary.compression.*` on first load (config version 17). No manual action needed.
 :::
 
+`progress_notices` (default `false`) controls whether **routine** compression progress statuses reach chat platforms (Telegram, Discord, Slack, etc.). By design, automatic compression is silent on chat surfaces — it runs in the background with server-side logging only. Set `progress_notices: true` to opt into seeing the routine lifecycle on chat platforms: the "Compacting context…" start notice, preflight/pre-API compression triggers, idle compaction, retry progress ("Compressed 30 → 12 messages, retrying…"), and the "Context compaction complete" notice. The gate is scoped to compression statuses only — unrelated operational noise (auxiliary model failures, provider rate-limit/retry chatter) stays suppressed either way. Compression **failure** notices and manual `/compress` feedback are always visible regardless of this setting. Editing this value on a running gateway takes effect on the next message.
+
 `hygiene_hard_message_limit` is a gateway-only **pre-compression safety valve**. It exists to break a death spiral: when API calls keep disconnecting on an oversized session, the gateway never receives token-usage data, so the token-based threshold can't fire, so the transcript keeps growing and disconnects get worse. This count-based floor fires on message count alone (always known, regardless of API failures) to force compression and recover the session. Default `5000` — far above any normal session, including large-context (1M+) models doing thousands of short turns, which compress on the token threshold long before this. Raise it further for unusual platforms, lower it to force more aggressive compression. Editing this value on a running gateway takes effect on the next message (see below).
 
 `hygiene_timeout_seconds` caps how long the gateway waits for this pre-agent compression pass. If the auxiliary compression backend is down or very slow, the gateway warns the user, continues the incoming message without compression, and records a temporary per-session failure cooldown instead of appearing stuck.