openworker/deploy/reference/DRAINING.md

Planned worker shutdown

The harness owns draining; Kubernetes or another supervisor owns replacement. This concerns server workers, not the Desktop agent loop or evacuating a host that also runs OpenShell sandboxes and their volumes.

Contract

First SIGTERM/Ctrl-C requests a drain. Stop new session claims, queued follow-up turns and approval/question resumes. Admissions already underway may finish before acknowledgement: an exclusive barrier waits for their shared admission guards. The CLI prints acknowledgement after that boundary; this does not mean all tools have finished.

Already-running turns continue their model/tool steps under existing policy until they finish or park. Leases keep renewing and active cancellation still takes priority. Pending interactions and accepted responses stay durable for another compatible worker. Drain does not wait for a human or an endlessly refilled queue.

After execution settles, normal cleanup releases ownership. On drain timeout, use existing interruption/unknown-effect recovery and exit unsuccessfully with an explicit diagnostic. A second signal requests immediate interruption. DB, lease and runtime failures also interrupt without waiting for the drain deadline. Never retry an uncertain tool just because the old worker exited.

openworker-server-reference --config /private/reference.json worker \
  --node worker-a --slots 2 \
  --drain-timeout-seconds 300 --shutdown-timeout-seconds 45

Drain bounds: 1–3600s; cleanup: 1–120s. Defaults are 300s and 45s. The drain timer includes waiting for admission acknowledgement. An uncooperative native tool cannot be forcibly stopped safely in-process; the supervisor remains the final termination boundary. Durable recovery is still required for forced kills.

Trusted Rust server hosts use openworker_store_postgres::worker::Drain and the additive Worker::run_with_drain(drain, abort_token, timeout). The one-shot handle has request, is_requested, acknowledged and is_acknowledged. Race the acknowledgement against the worker result: a failure may exit without acknowledging. Existing Worker::run(shutdown_token) retains immediate-stop semantics. No remote drain route or high-level Harness SDK protocol change is introduced.

Telemetry: advertised new-work capacity becomes zero on request; occupied slots stay visible and openworker.worker.draining=1 marks acknowledgement. Heartbeats continue until exit (5s reporting interval). A planned loss of capacity can still legitimately alert if no replacement exists.

Supervisors

Helm passes explicit timeouts and requires a termination grace of at least drain + cleanup + 15s, default 360s. RollingUpdate uses maxUnavailable: 0, maxSurge: 1. The local reference launcher uses the same 300/45/15 budget, replacing its previous 35s worker cutoff. APIs retain separate HTTP/SSE shutdown; clients reconnect and replay durable events. Configure other supervisors with a compatible budget.

Pod readiness is not proof of compatible execution capacity. The chart has API readiness but no comprehensive worker/model/OpenShell readiness probe. Verify a compatible replacement pool with a canary and capacity before retiring an old pool. RollingUpdate limits do not promise uninterrupted agent execution. This feature does not drain Kubernetes nodes or move OpenShell workspaces/PVCs.

Version-pinned sessions

Execution identity pins coworker bundle, policy and runner. Native registry sessions also pin the host executable. A new native binary can intentionally be incompatible with an existing session even when the DB schema/protocol matches.

Keep an old compatible worker pool for old sessions, and send new sessions to the new identity. Do not remove all old workers while old queued work/approvals remain. Drain a replica only when another compatible replica can handle remaining work. Restoring an old pool can resume its old session, but does not make it compatible with new sessions. Never rewrite identity/binding digests to force it.

Transparent migration/re-approval needs a separate design. This slice does not weaken pins, migrate storage schemas, promise arbitrary binary rollback, or certify a complete Kubernetes upgrade.

Verification

For side-by-side releases, retirement diagnostics and creation rollback, see Version-aware deployment. Draining alone does not migrate sessions.

Five real Linux/PG process cases cover current execution completing across lease renewals; queued follow-up preservation; approval-response deferral versus active cancellation; deadline/unknown effects without replay; DB loss during drain; and mixed native identities with a retained baseline pool. Two barrier units cover in-flight admission and timeout before acknowledgement. Separate HTTP/OpenShell qualification exercises actual server SIGTERM, parked approvals, API/worker replacement, remote tools, SSE and artifacts.

# harness/; OW_PG_TEST_URL must identify an isolated local test database.
cargo test --locked -p openworker-store-postgres --features test-support \
  --test worker_recovery draining:: -- --include-ignored --test-threads=1

For the mixed-version case, set OW_BASELINE_WORKER to the retained 2a5c208 test executable; without it, that case returns early and is not version evidence. Set OW_CONCURRENCY_REPORT_DIR to an absolute private evidence directory. Native probes are scripted and explicitly unsandboxed, distinct from the real OpenShell transport test. Only owned test resources are cleaned up; Task Desk is unchanged.