openworker/harness/server-reference/README.md

Two API / two worker reference

For the opt-in versioned service API and Rust/TypeScript Harness.connect(), see Service M1 quickstart. For release deployment, RuntimeProfiles and automatic OpenShell provisioning, see Service M2 quickstart. For optional diagnostic recording, including independent execution/observability database connections, see the observability spec and recording adapter guide. M1 stores existing instrumentation; complete usage accounting and management UI queries follow later.

A deliberately small native Rust server adapter and Node client, using the same runtime as Desktop/stdio/SDK. No second agent loop and no Python dependency. This demonstrates the production boundary, not a complete multi-tenant product.

Client ── REST + SSE ── API A / API B
                              │
                       PostgreSQL v5
                              │
                       Worker A / Worker B
                              │ verified mTLS
                       OpenShell gateway
                              │
                    original sandbox + PVC

APIs commit commands and stream durable events. They do not forward a live worker socket or route users to a particular worker. Workers independently claim compatible sessions, renew leases and prioritize approvals/cancellation. An API may disappear without losing a session, pending decision or event cursor. Polling PostgreSQL every 250ms is an implementation detail between the API and store; clients receive SSE. Notifications can optimize this later, not replace durable replay. API, worker and tool execution capacity are separate.

Prerequisites and configuration

• Rust 1.96, Node 20+, PostgreSQL, and a qualified OpenShell 0.1.3 Kubernetes gateway with persistent workspace storage and verified mTLS. See ../../deploy/local/README.md for the isolated local cluster profile.
• A pinned Linux runner image containing /opt/openworker/openworker. The runner configuration is the corresponding Linux ELF identity input, not a host command to execute. Keep replicas within each execution cohort on the same verified build/bundle. Retain old cohorts during upgrades; see version-aware pools and rollback.
• Copy config.example.json outside the repo and replace paths/digest. The placeholder image is intentionally invalid. state_root is controller-local; it is not the remote agent workspace. Give each process a distinct node ID.
• Put a random 32–256-character URL-safe bearer token in a private file. Do not put tokens or database passwords in command-line flags, source or screenshots. Supply the PostgreSQL URL via the named environment variable. Verified database TLS is default; database_loopback_plaintext: true is evaluation-only and rejects non-loopback endpoints. Never point qualification tests at application databases.

The sample uses scripted: true for deterministic protocol qualification. This still executes real MCP file/shell tools inside the sandbox. For live-model use, set scripted: false and an available model ID; supply OPENAI_API_KEY only to workers using a deployment secret. Do not send it to tools, the sandbox or this HTTP API. This reference does not expose model-connection management or configure an auto-approver; the underlying runtime APIs remain available to a richer host.

Run locally

From harness/, build cargo build --locked -p openworker-server-reference. The following commands use target/debug/openworker-server-reference; substitute your build output directory if configured differently. Use a new development schema: v5 rejects v1–v4, without migration/reset.

target/debug/openworker-server-reference --config /private/reference.json migrate
target/debug/openworker-server-reference --config /private/reference.json auth-bootstrap
target/debug/openworker-server-reference --config /private/reference.json environment-create --request-id setup-001 --shared

Save the returned environment id. Reusing that request ID reads the same receipt; it never silently recreates an unknown environment. Provisioning timeout requires inspection of the original object, not a fresh request ID. --shared means all attached sessions share filesystem/network authority; omit it for exclusive use. Do not use shared environments across mutually untrusted tenants.

Start these four commands in separate terminals/processes:

target/debug/openworker-server-reference --config /private/reference.json api --node api-a --listen 127.0.0.1:8081
target/debug/openworker-server-reference --config /private/reference.json api --node api-b --listen 127.0.0.1:8082
target/debug/openworker-server-reference --config /private/reference.json worker --node worker-a --slots 2
target/debug/openworker-server-reference --config /private/reference.json worker --node worker-b --slots 2

The API is loopback-only, role/session-authorized, and rejects browser Origin requests. All endpoints require an active PostgreSQL-registered key, including health. /healthz is liveness; /readyz checks this API's PostgreSQL connection. A poisoned connection remains unavailable until its API process is restarted. See authorization for observer/operator/admin keys, session and environment grants, rotation/revocation and audit. This is not physical tenant isolation. A production app supplies TLS ingress, quotas and admission policy. Do not expose this example directly to an untrusted network. Worker exit is fail-stop; supervise restarts and alert on failures rather than hiding them with mutation retries.

Planned worker termination now drains current execution before interruption; first SIGTERM/Ctrl-C drains, second interrupts. Defaults are 300s drain plus 45s cleanup. See the drain contract for flags, supervisor budgets, pending approvals and the version-pinned session boundary.

export OW_API_TOKEN_FILE=/private/api-token
export OW_API_URLS=http://127.0.0.1:8081,http://127.0.0.1:8082
node server-reference/client.mjs --environment ENV_ID --request-id demo-001 --fixture

Choose Allow Once, Allow for this Session, or Deny at the prompt. For a live worker use --text 'Create a short packing checklist in checklist.txt' instead of --fixture. To reconnect without submitting another task: node server-reference/client.mjs --session SESSION_ID. The client prints request/session IDs and reconnects with its last cursor. Reuse the printed --request-id after response loss; a new ID means a new task.

Reconcile uncertain environment provisioning

These are trusted host/database administration commands, not browser endpoints. After a provisioning timeout, retrieve the original receipt with the same environment-create --request-id (no new object is created on that exact retry), then inspect it and explicitly reconcile the original remote name:

target/debug/openworker-server-reference --config /private/reference.json environment-inspect --environment-id ENV_ID
target/debug/openworker-server-reference --config /private/reference.json environment-reconcile --environment-id ENV_ID --remote-name ORIGINAL_SANDBOX_NAME

Reconciliation only reads the remote object and records a verified binding. It requires the original ownership label, matching provider/image/policy profile, and ready/admitted policy. It never creates, replaces, starts or deletes a sandbox. Missing/changed/not-ready objects fail explicitly. A cloud cold start may exceed the 120-second initial wait; a timeout is not proof of failure or permission to create a second object. Keep the original receipt and inspect the original state.

Managed PostgreSQL CA files may be multi-root PEM bundles; all certificates are validated and loaded (at most 256 KiB / 32 roots). Empty/malformed/non-certificate PEM material is rejected. TLS and hostname verification are never disabled.

HTTP contract (reference v0)

For restore inspection, set recovery_read_only: true: authenticated reads and artifact downloads work without gateway access; HTTP mutations and worker/ environment execution commands refuse. Default false; /readyz identifies the mode. See backup and recovery for the local restore procedure and storage-identity, credential-rollback and effect limits.

For a repeatable local stack and optional read-only console/Grafana dashboards, see the deployment reference.

Optional durable artifacts: the storage and API guide describes filesystem/S3 configuration, explicit upload/download/reconciliation, and artifact-client.mjs. Artifacts stay retrievable when their worker exits; workspace uploads are never automatic. Existing session APIs are unchanged.

Optional, off-by-default OpenTelemetry: add "telemetry":{"endpoint": "http://127.0.0.1:4318"} to the configuration. The customer-selected collector receives content-free traces, metrics and lifecycle logs. Execution never waits on collector I/O. See telemetry setup, bounds and limitations.

Request	Purpose
POST /sessions	{request_id, parameters}; same actor/payload returns same session; changed payload is 409. Parameters use the existing runtime shape plus environment_id.
GET /sessions	Bounded operational directory, limit 1–100 (default 50), optional after session ID; newest creation first. No history/prompt payloads.
GET /sessions/:id	Coherent snapshot and event cursor, pending interactions, bounded messages/attempts.
POST /sessions/:id/turns	{submission_id, text}; returns a durable receipt with stable turn_id, initially queued.
GET /sessions/:id/turns/:submission_id	Inspect admission/completion after response loss.
POST /sessions/:id/controls	{request_id, turn_id, action}; separate priority channel, not behind a waiting turn.
GET /sessions/:id/controls/:request_id	pending, applied or rejected; pending is not applied permission.
GET /sessions/:id/events	SSE progress frames; Last-Event-ID (preferred) or ?after=N supplies cursor. Existing event JSON uses type, not kind.

Approval/question action is {kind:"response", interaction_id, response, binding:{bundle_digest,schema_digest,policy_digest,resource_policy_digest}}. Copy identities from the pending snapshot. Approval response is once, session or deny; a question uses an offered answer. The API stamps its configured principal, not a client-supplied actor. Cancellation is {kind:"cancel"} and targets the current turn. It cannot undo an already-admitted external effect.

Control errors are explicit: malformed request 400, missing receipt 404, changed idempotency payload or target conflict 409, closed session 409, and full queue 429. New controls on closed sessions reject; exact retries still retrieve the original receipt. Trusted runtime/store close must wait for already-admitted controls to settle (controls_pending) rather than stranding them. Unknown storage failures return 503: reconcile the original durable ID before retrying.

SSE IDs are contiguous per session, not globally ordered. On stream_error, reconnect/reconcile using a snapshot and the last received ID. A snapshot's cursor and state come from the same transaction; follow events strictly after that cursor. No push is required for correctness. Only the cursor is a replay marker; queue position, turn ID, owner epoch and canonical revision are different values.

Qualification and limits

The opt-in test cargo test --locked -p openworker-server-reference --features openshell-test --test replicas -- --ignored --nocapture needs the same OW_PG_TEST_URL and OW_PROBE_* variables as the store's real OpenShell test. It launches two actual APIs and two worker processes, exercises racing creation, approval identity after replacing APIs/workers, cursor replay on the other API, real file effects and concurrent sandbox tool calls with cross-API cancellation. It deletes its successful disposable environment; failures print exact IDs for explicit cleanup. No live model credentials are used by the test. Set OW_PROBE_NODE=node to additionally run the shipped client against the real API/worker/tool (the selected test session uses explicit bypass to avoid an interactive prompt in automation).

This is multiple independent processes on a local host plus a real Kubernetes execution environment—not multi-AZ, physical-node or cloud failover evidence. Kubernetes scheduling/PVC recovery follows the provider's guarantees. OpenShell disconnect may leave effects running, and its delete API has no expected-UID precondition; prohibit out-of-band delete/recreate races during managed lifecycle.

Still outside this example: full REST coverage (session close/history pages/model connections/grant revocation), shared credential service, tenant authz, gateway provisioning automation, artifact/object storage, full OTel, HA database operations, TLS ingress and AWS/Azure deployment automation. Manage final session/environment closure through the trusted runtime/store APIs; do not delete shared environments as a side effect of client disconnect. Desktop and the local SDK keep their existing transports and default SQLite behavior.