mirror of
https://github.com/andrewyng/openworker.git
synced 2026-11-20 20:37:44 +00:00
Allow developers to easily build AI apps without worrying about the harness part. Bundles the Rust agent runtime as a SDK (Rust and TypeScript) for building desktop and web applications. Applications can manage a local harness or connect to an authenticated harness service on k8s with separately scaled workers and sandbox execution. The implementation is primarily contained in harness/, with deployment tooling in deploy/, Markdown guides in docs/harness/guides/, and the static documentation site in docs-site/. It does not replace the existing Python harness or migrate OpenWorker Desktop. Co-authored-by: Devika Verma <devikaverma11@gmail.com>
217 lines
13 KiB
Markdown
217 lines
13 KiB
Markdown
# Two API / two worker reference
|
||
|
||
For service placement and RuntimeProfiles, see
|
||
[deployment options](../../docs/harness/guides/deployment.md). For Rust/TypeScript
|
||
`Harness.connect()`, see the SDK READMEs. For diagnostic recording and independent
|
||
execution/observability database connections, see the
|
||
[recording adapter guide](../observability/README.md).
|
||
|
||
A deliberately small native Rust server adapter and Node client, using the same
|
||
runtime as Desktop/stdio/SDK. No second agent loop and no Python dependency.
|
||
This demonstrates the production boundary, not a complete multi-tenant product.
|
||
|
||
```text
|
||
Client ── REST + SSE ── API A / API B
|
||
│
|
||
PostgreSQL v5
|
||
│
|
||
Worker A / Worker B
|
||
│ verified mTLS
|
||
OpenShell gateway
|
||
│
|
||
original sandbox + PVC
|
||
```
|
||
|
||
APIs commit commands and stream durable events. They do not forward a live
|
||
worker socket or route users to a particular worker. Workers independently claim
|
||
compatible sessions, renew leases and prioritize approvals/cancellation. An API
|
||
may disappear without losing a session, pending decision or event cursor.
|
||
Polling PostgreSQL every 250ms is an implementation detail between the API and
|
||
store; clients receive SSE. Notifications can optimize this later, not replace
|
||
durable replay. API, worker and tool execution capacity are separate.
|
||
|
||
## Prerequisites and configuration
|
||
|
||
- Rust 1.96, Node 20+, PostgreSQL, and a qualified OpenShell **0.1.3 Kubernetes**
|
||
gateway with persistent workspace storage and verified mTLS. See
|
||
`../../deploy/local/README.md` for the isolated local cluster profile.
|
||
- A pinned Linux runner image containing `/opt/openworker/openworker`. The
|
||
`runner` configuration is the corresponding Linux ELF identity input, not a
|
||
host command to execute. Keep replicas within each execution cohort on the same
|
||
verified build/bundle. Retain old cohorts during upgrades; see
|
||
[version-aware pools and rollback](../../deploy/reference/UPGRADES.md).
|
||
- Copy `config.example.json` outside the repo and replace paths/digest. The
|
||
placeholder image is intentionally invalid. `state_root` is controller-local;
|
||
it is **not** the remote agent workspace. Give each process a distinct node ID.
|
||
- Put a random 32–256-character URL-safe bearer token in a private file. Do not
|
||
put tokens or database passwords in command-line flags, source or screenshots.
|
||
Supply the PostgreSQL URL via the named environment variable. Verified database
|
||
TLS is default; `database_loopback_plaintext: true` is evaluation-only and rejects
|
||
non-loopback endpoints. Never point qualification tests at application databases.
|
||
|
||
The sample uses `scripted: true` for deterministic protocol qualification. This
|
||
still executes real MCP file/shell tools inside the sandbox. For live-model use,
|
||
set `scripted: false` and an available model ID; supply `OPENAI_API_KEY` only to
|
||
workers using a deployment secret. Do not send it to tools, the sandbox or this
|
||
HTTP API. This reference does not expose model-connection management or configure
|
||
an auto-approver; the underlying runtime APIs remain available to a richer host.
|
||
|
||
## Run locally
|
||
|
||
From `harness/`, build `cargo build --locked -p openworker-server-reference`.
|
||
The following commands use `target/debug/openworker-server-reference`; substitute
|
||
your build output directory if configured differently. Use a **new development
|
||
schema**: v5 rejects v1–v4, without migration/reset.
|
||
|
||
```sh
|
||
target/debug/openworker-server-reference --config /private/reference.json migrate
|
||
target/debug/openworker-server-reference --config /private/reference.json auth-bootstrap
|
||
target/debug/openworker-server-reference --config /private/reference.json environment-create --request-id setup-001 --shared
|
||
```
|
||
|
||
Save the returned environment `id`. Reusing that request ID reads the same
|
||
receipt; it never silently recreates an unknown environment. Provisioning timeout
|
||
requires inspection of the original object, not a fresh request ID. `--shared`
|
||
means all attached sessions share filesystem/network authority; omit it for
|
||
exclusive use. Do not use shared environments across mutually untrusted tenants.
|
||
|
||
Start these four commands in separate terminals/processes:
|
||
|
||
```sh
|
||
target/debug/openworker-server-reference --config /private/reference.json api --node api-a --listen 127.0.0.1:8081
|
||
target/debug/openworker-server-reference --config /private/reference.json api --node api-b --listen 127.0.0.1:8082
|
||
target/debug/openworker-server-reference --config /private/reference.json worker --node worker-a --slots 2
|
||
target/debug/openworker-server-reference --config /private/reference.json worker --node worker-b --slots 2
|
||
```
|
||
|
||
The API is loopback-only, role/session-authorized, and rejects browser `Origin`
|
||
requests. All endpoints require an active PostgreSQL-registered key, including health.
|
||
`/healthz` is liveness; `/readyz` checks this API's PostgreSQL connection. A
|
||
poisoned connection remains unavailable until its API process is restarted.
|
||
See [authorization](AUTHORIZATION.md) for observer/operator/admin keys, session
|
||
and environment grants, rotation/revocation and audit. This is not physical tenant
|
||
isolation. A production app supplies TLS ingress, quotas and admission policy. Do not expose this
|
||
example directly to an untrusted network. Worker exit is fail-stop; supervise
|
||
restarts and alert on failures rather than hiding them with mutation retries.
|
||
|
||
Planned worker termination now drains current execution before interruption;
|
||
first SIGTERM/Ctrl-C drains, second interrupts. Defaults are 300s drain plus 45s
|
||
cleanup. See [the drain contract](../../deploy/reference/DRAINING.md) for flags,
|
||
supervisor budgets, pending approvals and the version-pinned session boundary.
|
||
|
||
```sh
|
||
export OW_API_TOKEN_FILE=/private/api-token
|
||
export OW_API_URLS=http://127.0.0.1:8081,http://127.0.0.1:8082
|
||
node server-reference/client.mjs --environment ENV_ID --request-id demo-001 --fixture
|
||
```
|
||
|
||
Choose Allow Once, Allow for this Session, or Deny at the prompt. For a live
|
||
worker use `--text 'Create a short packing checklist in checklist.txt'` instead
|
||
of `--fixture`. To reconnect without submitting another task:
|
||
`node server-reference/client.mjs --session SESSION_ID`.
|
||
The client prints request/session IDs and reconnects with its last cursor. Reuse
|
||
the printed `--request-id` after response loss; a new ID means a new task.
|
||
|
||
## Reconcile uncertain environment provisioning
|
||
|
||
These are trusted host/database administration commands, not browser endpoints.
|
||
After a provisioning timeout, retrieve the original receipt with the same
|
||
`environment-create --request-id` (no new object is created on that exact retry),
|
||
then inspect it and explicitly reconcile the original remote name:
|
||
|
||
```sh
|
||
target/debug/openworker-server-reference --config /private/reference.json environment-inspect --environment-id ENV_ID
|
||
target/debug/openworker-server-reference --config /private/reference.json environment-reconcile --environment-id ENV_ID --remote-name ORIGINAL_SANDBOX_NAME
|
||
```
|
||
|
||
Reconciliation only reads the remote object and records a verified binding. It
|
||
requires the original ownership label, matching provider/image/policy profile,
|
||
and ready/admitted policy. It never creates, replaces, starts or deletes a sandbox.
|
||
Missing/changed/not-ready objects fail explicitly. A cloud cold start may exceed
|
||
the 120-second initial wait; a timeout is not proof of failure or permission to
|
||
create a second object. Keep the original receipt and inspect the original state.
|
||
|
||
Managed PostgreSQL CA files may be multi-root PEM bundles; all certificates are
|
||
validated and loaded (at most 256 KiB / 32 roots). Empty/malformed/non-certificate
|
||
PEM material is rejected. TLS and hostname verification are never disabled.
|
||
|
||
## HTTP contract (reference v0)
|
||
|
||
For restore inspection, set `recovery_read_only: true`: authenticated reads and
|
||
artifact downloads work without gateway access; HTTP mutations and worker/
|
||
environment execution commands refuse. Default false; `/readyz` identifies the
|
||
mode. See [backup and recovery](../../deploy/recovery/README.md) for the local
|
||
restore procedure and storage-identity, credential-rollback and effect limits.
|
||
|
||
For a repeatable local stack and optional read-only console/Grafana dashboards,
|
||
see [the deployment reference](../../deploy/reference/README.md).
|
||
|
||
Optional durable artifacts: the [storage and API guide](../artifacts/README.md)
|
||
describes filesystem/S3 configuration, explicit upload/download/reconciliation,
|
||
and `artifact-client.mjs`. Artifacts stay retrievable when their worker exits;
|
||
workspace uploads are never automatic. Existing session APIs are unchanged.
|
||
|
||
Optional, off-by-default OpenTelemetry: add `"telemetry":{"endpoint":
|
||
"http://127.0.0.1:4318"}` to the configuration. The customer-selected collector
|
||
receives content-free traces, metrics and lifecycle logs. Execution never waits
|
||
on collector I/O. See [telemetry setup, bounds and limitations](../telemetry/README.md).
|
||
|
||
| Request | Purpose |
|
||
|---|---|
|
||
| `POST /sessions` | `{request_id, parameters}`; same actor/payload returns same session; changed payload is 409. Parameters use the existing runtime shape plus `environment_id`. |
|
||
| `GET /sessions` | Bounded operational directory, `limit` 1–100 (default 50), optional `after` session ID; newest creation first. No history/prompt payloads. |
|
||
| `GET /sessions/:id` | Coherent snapshot and event cursor, pending interactions, bounded messages/attempts. |
|
||
| `POST /sessions/:id/turns` | `{submission_id, text}`; returns a durable receipt with stable `turn_id`, initially queued. |
|
||
| `GET /sessions/:id/turns/:submission_id` | Inspect admission/completion after response loss. |
|
||
| `POST /sessions/:id/controls` | `{request_id, turn_id, action}`; separate priority channel, not behind a waiting turn. |
|
||
| `GET /sessions/:id/controls/:request_id` | `pending`, `applied` or `rejected`; pending is not applied permission. |
|
||
| `GET /sessions/:id/events` | SSE `progress` frames; `Last-Event-ID` (preferred) or `?after=N` supplies cursor. Existing event JSON uses `type`, not `kind`. |
|
||
|
||
Approval/question `action` is `{kind:"response", interaction_id, response,
|
||
binding:{bundle_digest,schema_digest,policy_digest,resource_policy_digest}}`.
|
||
Copy identities from the pending snapshot. Approval response is `once`, `session`
|
||
or `deny`; a question uses an offered answer. The API stamps its configured
|
||
principal, not a client-supplied actor. Cancellation is `{kind:"cancel"}` and
|
||
targets the current turn. It cannot undo an already-admitted external effect.
|
||
|
||
Control errors are explicit: malformed request 400, missing receipt 404,
|
||
changed idempotency payload or target conflict 409, closed session 409, and full
|
||
queue 429. New controls on closed sessions reject; exact retries still retrieve
|
||
the original receipt. Trusted runtime/store close must wait for already-admitted
|
||
controls to settle (`controls_pending`) rather than stranding them. Unknown
|
||
storage failures return 503: reconcile the original durable ID before retrying.
|
||
|
||
SSE IDs are contiguous per session, not globally ordered. On `stream_error`,
|
||
reconnect/reconcile using a snapshot and the last received ID. A snapshot's cursor
|
||
and state come from the same transaction; follow events strictly after that
|
||
cursor. No push is required for correctness. Only the cursor is a replay marker;
|
||
queue position, turn ID, owner epoch and canonical revision are different values.
|
||
|
||
## Qualification and limits
|
||
|
||
The opt-in test `cargo test --locked -p openworker-server-reference --features
|
||
openshell-test --test replicas -- --ignored --nocapture` needs the same
|
||
`OW_PG_TEST_URL` and `OW_PROBE_*` variables as the store's real OpenShell test.
|
||
It launches two actual APIs and two worker processes, exercises racing creation,
|
||
approval identity after replacing APIs/workers, cursor replay on the other API,
|
||
real file effects and concurrent sandbox tool calls with cross-API cancellation.
|
||
It deletes its successful disposable environment; failures print exact IDs for
|
||
explicit cleanup. No live model credentials are used by the test.
|
||
Set `OW_PROBE_NODE=node` to additionally run the shipped client against the real
|
||
API/worker/tool (the selected test session uses explicit bypass to avoid an
|
||
interactive prompt in automation).
|
||
|
||
This is multiple independent processes on a local host plus a real Kubernetes
|
||
execution environment—not multi-AZ, physical-node or cloud failover evidence.
|
||
Kubernetes scheduling/PVC recovery follows the provider's guarantees. OpenShell
|
||
disconnect may leave effects running, and its delete API has no expected-UID
|
||
precondition; prohibit out-of-band delete/recreate races during managed lifecycle.
|
||
|
||
Still outside this example: full REST coverage (session close/history pages/model
|
||
connections/grant revocation), shared credential service, tenant authz, gateway
|
||
provisioning automation, artifact/object storage, full OTel, HA database operations,
|
||
TLS ingress and AWS/Azure deployment automation. Manage final session/environment
|
||
closure through the trusted runtime/store APIs; do not delete shared environments
|
||
as a side effect of client disconnect. Desktop and the local SDK keep their
|
||
existing transports and default SQLite behavior.
|