openworker/deploy/reference/UPGRADES.md

Version-aware deployment and rollback

The unit of compatibility is a session's execution identity, not a pod name, image tag or marketing release number. Coworker content, runner, policies and native tool registry/host binary contribute to that identity. Do not rewrite it in PostgreSQL, including on an approval, to make an upgrade succeed.

Deploy side by side

1. Back up and run the new server's explicit migrate command. This change adds an independent versioned operational catalog; session schema v5 is unchanged. Earlier workers can still use their original session tables. This is not a guarantee for future incompatible schema changes.
2. Keep the old image, coworker, runner and configuration immutable. Start a separate new release (for example ow-v2) against the same database, environment deployment and artifact store. New release, new worker pool.
3. Wait for new worker readiness and run a canary using the new API's creation template. Verify actual approval, execution and result; readiness is not a substitute for that canary.
4. Switch the application's new-session creation traffic to the new API. Existing session reads, submissions and controls can use either API; the exact-match scheduler routes their work using the durable session identity. This assumes compatible HTTP/storage protocols and retained credentials.
5. Leave old workers running for old sessions, including sessions waiting for a human and idle/completed sessions that may receive another message.

With Helm, use separate release names and immutable images/configs. The chart's ordinary rolling update is safe only within an execution-compatible cohort. maxUnavailable: 0 and readiness do not authorize replacing the last worker for a different execution identity. The chart intentionally does not infer this from image tags or automatically delete old releases.

Inspect the remaining work

openworker-server-reference --config /private/reference.json worker-pools --limit 50
# If `next` is non-null, pass it as --after on the next read.

This local administrator command uses the trusted database connection, not an application bearer token. There is no new public HTTP endpoint or agent tool. It returns a timestamped page per execution digest:

Field	Meaning
open_sessions	Includes idle and completed-but-open sessions
active_sessions	Work not parked/terminal
queued_turns, pending_controls	Durable outstanding submissions/decisions
approvals, questions	Unanswered interactions on the current parked turn
sessions_without_advertised_worker	No live, non-draining advertisement matches the full scheduling selector
workers	Node, full selector, capacity and draining status; bounded to 100

Selectors include coworker, execution digest, backend, scripted/live mode, fallback model, environment provider, deployment identity and profile. A matching digest alone is not enough. This is scheduling coverage, not model quota, free slots, all provider connections, sandbox availability or a per-tenant guarantee. Inspect workers_truncated; pages may change between reads. Counts contain no prompts, arguments, credentials, workspace paths or session identifiers.

Each server worker advertises using its own store incarnation. Advertisements expire after 20 seconds using database time, refresh every five seconds, and are withdrawn on orderly exit. Stale rows older than one day are cleaned in bounded batches. Advertisements are not session ownership leases and cannot authorize execution. Pre-catalog workers are unobserved, not necessarily absent. Session claims and per-use environment/approval checks remain authoritative.

Readiness versus liveness

worker --health-listen 127.0.0.1:8081 exposes a status-only /readyz (204/503). Non-loopback listeners are rejected. Helm probes it inside the container.

Before intake, the worker loads/hashes its coworker/runner and validates its selector, checks the PostgreSQL catalog, and makes a read-only authenticated gateway/version check. Ongoing checks refresh readiness; stale checks or an immediate drain request make it unready. Gateway/database failure interrupts the worker and exits through normal durable recovery, rather than reopening a poisoned connection and retrying effects. Liveness only checks process presence. No extra sandbox is created by readiness, and no model request is made.

The gateway check verifies the qualified OpenShell version/driver and mTLS connection, not every sandbox or available cluster capacity. A session's exact environment is still inspected when used. The existing 300-second drain plus 45-second cleanup budget applies; termination grace must include both.

Retirement and rollback

An empty active-work count is not permission to retire a pool. First stop all creation paths using its old template. Explicitly close sessions through the existing trusted lifecycle after resolving pending work, or retain the old pool. The current HTTP adapter does not expose an administrative bulk-close API. Recheck all pages for zero open sessions, drain the pool, and verify termination. The report is an advisory snapshot, not an atomic creation fence or automatic retirement controller. Application routing/creation cutoff belongs to the host.

For rollback, switch new creation back to the old API/template. Retain both worker cohorts: sessions already created under the new version still require it. An API-only image rollback must not accidentally replace the new workers too. Restore failed compatible workers from their exact pinned artifacts. Do not claim transparent session migration or silently substitute the old binary for new-version sessions. Migration/reapproval needs a separate explicit design.

Local qualification

deploy/helm/version-drill.mjs takes an explicit private JSON configuration: kubeconfig, context, helm, imageA, imageB, runnerImage, certificates, gateway. Image refs must be immutable and already loaded into kind. Both images must include the new server, a compatible runner and an assistant-example coworker; their coworker contents must differ. The current drill targets the documented local kind context and existing openshell test gateway, not arbitrary clusters.

The script creates a random owned namespace and synthetic PostgreSQL database, fresh credentials, two Helm releases and one explicitly shared test sandbox. Its model is scripted; MCP/file/shell operations run in real OpenShell. It does not access Task Desk, AWS or a model key. Raw API forwarding is loopback-only; this is not a new TLS/load qualification. Network policy is disabled only for this isolated fixture. Reports and ownership metadata live under private .ow-stack/ow-version-*/. On failure it retains exact resources for inspection; the operator must remove that sandbox and namespace after recording evidence.