Kubernetes and OpenShift
Preview. P6 on Kubernetes is implemented but not yet declared Stable; interfaces and limits may change. Known limits: What this does not claim. It is the first target for Stable: what Stable requires and where this path stands is in Maturity and lifecycle.
The P6 (process-manager) runtime as ordinary Kubernetes workloads: the supervisor and its forked workers, the sandbox
relay, Redis, the MU1 control plane and remote-worker container sandboxes. It is milestone P6.1,
slice 1 of epic rossoctl/moca#426 (issue rossoctl/moca#423). The design is
docs/specs/2026-10-02-p6-on-kubernetes-slice1-design.md.
1. What this isโ
This is the sibling of Single VM (systemd and podman on one VM) and Docker Compose (the same processes under Docker Compose). All three run the same processes, the same environment and the same wire. Only the process manager, the addressing and the way isolation is enforced differ. Here, isolation comes from namespaces, NetworkPolicy, Secrets and pod lifecycle.
This directory uses GrpcRelayTransport and records-based discovery exactly as deploy/vm/ does โ
the Knative/KEDA deployment model that once shared this name space was removed entirely
(ADR-0041). ADR-0037 records the
Kubernetes-substrate distinction.
| Workload | Namespace | Ports | Reached by |
|---|---|---|---|
Deployment moca-supervisor | moca | 8080 data; 8081 admin; 8443 ghostunnel and 8453 its status (OCP) | users: port-forward on Kind, the passthrough Route on OCP; kubelet on 8081 (and 8453 on OCP) |
Deployment sandbox-relay | moca | 9443 Attach; 9444 SandboxExec | sandboxes on 9443 only; supervisor pods on 9444 only |
StatefulSet redis | moca | 6379 | supervisor, relay, control plane |
Deployment moca-control-plane | moca | 8080 | users (port-forward on Kind, an edge Route on OCP); the supervisor's workers |
StatefulSet moca-sandbox | moca-sandbox | none | nothing: each sandbox dials out to the relay |
| per-user credentials (Secrets) | moca-credentials | none | the control plane, through the kube API (K8sSecretStore) |
The relay has two Services, sandbox-relay-attach (9443) and sandbox-relay-exec (9444), so the
address a sandbox is given has no route to the exec port. The supervisor's Service is
moca-supervisor (8080, 8081), and on OCP moca-supervisor-tls (8443).
deploy/k8s/
โโโ base/ namespaces, workloads, Services, NetworkPolicy, RBAC
โโโ overlays/kind/ local images (dev.local/...:local), port-forward access
โโโ overlays/kind-ci/ kind + a loopback mock model in the supervisor pod (CI)
โโโ overlays/ocp/ ghostunnel sidecar, Routes, router and openshift-dns rules
โโโ setup.sh bring it up; idempotent
โโโ smoke.sh the live smoke (K8S_LIVE_SMOKE=1)
โโโ tests/setup.test.sh setup.sh with every external command mocked
The manifest tests are in packages/supervisor/test/k8s/
and run with make test.
2. Try it on Kindโ
You need:
- kind 0.24.0 or newer. It is the first release whose default CNI (kindnet, through
kube-network-policies) enforces NetworkPolicy, and this deployment's isolation is
NetworkPolicy.
setup.shrefuses an olderkindand says why. - Docker, or podman's
dockerCLI. kubectl,jq,openssl, andsha256sumorshasum.- For
--build, a checkout with its submodules (git submodule update --init --recursive). The image builds pi-fork itself.
SH_GITHUB_CLIENT_ID=<client id> deploy/k8s/setup.sh --target kind --build
setup.sh creates the kind cluster moca if it is missing, builds both images and loads them,
creates the namespaces, generates the Secrets once, applies the overlay, and waits until every
workload is rolled out and every sandbox is attached to the relay. Every kube call is pinned to the
context kind-moca, never the shell's current one.
kind create clusterswitches your current context. Whensetup.shcreates the cluster, kind sets the current kube context tokind-moca, as it always does.setup.shitself never uses the current context, but your next barekubectlwill talk to kind until you switch back (kubectl config use-context <previous>).
Image choice, for --target kind:
--buildbuilds both images from this checkout.- With neither flag, it pulls
ghcr.io/rossoctl/moca:latestandghcr.io/rossoctl/moca-remote-worker:latest(or the refs given with--imageand--sandbox-image), and builds only if the pull fails. Until this slice is published tolatest, the published harness image has no/readyz, and the supervisor never becomes ready: use--build. --skip-buildassumes both images are already loaded, which makes a re-run take seconds.
Each run stamps the loaded images' IDs on the pod templates (moca.dev/image-id). A rebuilt image
keeps its :local tag, so without that the apply would see no change and the pods would keep
running the old image. With it, a new image rolls the workloads that run it, and an unchanged image
rolls nothing.
Other settings come from the environment: SH_ADMIN_SUBJECTS, SH_ALLOW_OPERATOR_FALLBACK
(default false), SH_SANDBOX_COUNT (default 2, at most 4 digits; 0 runs no container sandboxes) and
SH_WAIT_SECONDS (default 120, the wait for sandboxes to attach). --help prints the synopsis.
When it finishes, it prints how to reach the stack:
P6 is up on kind (context kind-moca). Reach it with two port-forwards:
kubectl --context kind-moca -n moca port-forward svc/moca-supervisor 8080:8080
kubectl --context kind-moca -n moca port-forward svc/moca-control-plane 8090:8080
then: mocactl --control-plane-url http://127.0.0.1:8090 login
Run the two port-forwards in their own terminals, then log in. Export the URL in the terminal you
run mocactl from, so every later command uses the same one (mocactl keys its cached login on
the control-plane URL):
export SH_CONTROL_PLANE_URL=http://127.0.0.1:8090
mocactl login
The control plane advertises the harness as http://127.0.0.1:8080 by default, so the supervisor's
forward must use local port 8080. If 8080 conflicts with another local service, override it with
SH_PUBLIC_HARNESS_URL at setup time (see ยง2's sticky settings). mocactl runs from a checkout
(docs/harness/how-to/set-up-mocactl.md). The GitHub OAuth app
needs Enable Device Flow ticked (Single VM, "The GitHub OAuth
app").
Without SH_GITHUB_CLIENT_ID, the control plane is installed with 0 replicas, because nobody
can log in without one, and setup.sh says so. It is rendered that way, not scaled down later, so
it never starts into a crash-loop. To fix it, re-run with the client id set:
SH_GITHUB_CLIENT_ID=<client id> deploy/k8s/setup.sh --target kind --skip-build
The settings change rolls the control plane, and the supervisor with it: setup.sh stamps a hash
of its settings on both pod templates (moca.dev/settings-hash), so the apply itself restarts them
exactly when they change, even if an earlier run failed after writing them. The supervisor is in it
because it reads the sandbox tiers from the same ConfigMap (ยง11.1); it drains on the roll, with a
120 s grace.
The first apply of a P6.3 release disrupts every session, so run it at a quiet time. It does more than roll the supervisor and the control plane once:
- It recreates the relay.
base/relay.yamlgainsSH_SANDBOX_AFFINITY_TTL_SECONDS, and the relay isstrategy: Recreate. While it is down, every sandbox and every P4 host is detached, and in-flight turns fail and are retried. - It rolls every
moca-sandboxpod, even on an untiered stack:base/sandbox.yamlgainsSANDBOX_TIER. A container sandbox's workspace is local to its pod, so every container session's workspace is reset. This happens silently: the pod names do not change, so affinity returns each session to the same sandbox id, and noworkspace resetnotice is shown.
A re-run with nothing changed converges without restarting any pod or touching any Secret. To
remove everything: kind delete cluster --name moca.
Inputs are sticky. A re-run keeps every input it is not given, so you change one input by re-running with just that one:
SH_GITHUB_CLIENT_ID,SH_ADMIN_SUBJECTS,SH_ALLOW_OPERATOR_FALLBACK,SH_API_TOKEN_TTL_SECONDSandSH_SESSION_TOKEN_TTL_SECONDSlive in the ConfigMapmoca-settings. A variable that is unset keeps the stored value. A variable set to empty clears it:SH_ADMIN_SUBJECTS= deploy/k8s/setup.sh ...removes every admin,SH_GITHUB_CLIENT_ID=takes the control plane back to 0 replicas, and an empty token lifetime is the control plane's default (900 seconds for the API token a login gets, 300 for a session token). A lifetime is a whole number of seconds; a change rolls the control plane. The one exception to stickiness iskind-ci's placeholder client id, described under the CI variant below.SH_MODEL_CATALOG_HOSTSlives inmoca-settingstoo: the comma-separated bare hostnames the control plane's model catalog may contact besidesapi.anthropic.comand the default inference endpoint -- add each gateway your users' inference credentials name (docs/reference/configuration.md; without it, listing and choosing models per session degrades tomodel_catalog_unavailable). Empty, the default, allows only those two. A scheme, port or path in an entry is refused here, matching the control plane's own boot check. A change rolls the control plane. A pre-0.8 stack may carry the allow-list as a hand-patchedSH_MODEL_CATALOG_HOSTSenv on the control plane Deployment (the manifests did not wire it then); setup.sh carries that value intomoca-settingsโ a value given for the run or one already stored wins over it โ and strips the env before its own apply, without which server-side apply would merge the literal with the ConfigMap ref above and the API server would reject the Deployment. The strip rolls the control plane once on its own, just before the apply rolls it again: a one-time cost of the upgrade.SH_WORKSPACE_IDLElives inmoca-settingstoo: how long a container sandbox keeps a session's workspace after the session's last command, as hours, minutes and seconds in that order (8h,90m,1h30m). Empty, the default, is8h, the valuedeploy/microvmwrites; the worker's own default is 30 minutes, which resets a human-in-the-loop session that waits longer than that. A value without a unit (30), a sign, a fraction or zero is refused, as the worker would refuse it at boot. The sandboxes run inmoca-sandboxand cannot readmoca-settings, so setup.sh also writes the value to ConfigMapmoca-sandbox-settingsthere, and stamps it on the sandbox pod template (moca.dev/workspace-idle): a change rolls everymoca-sandboxpod, which resets every container session's workspace, so change it at a quiet time. A workload's sandboxes read the same key when their pods next start. Disk is not admission-controlled:/workspaceis the pod's writable layer (anemptyDirwith nosizeLimitonocp-single), on the node's disk, and no quota or limit bounds it. What bounds it is the idle sweep, which deletes a workspace at most a minute after it has been idle this long, so a longer value holds more idle sessions' files at once. Under node disk pressure the kubelet evicts a sandbox pod, which loses all its workspaces. The reset when the sweep fires is still silent (#338).SH_PUBLIC_HARNESS_URL(kind and--target ocp-singlewithout Routes) overrides the defaulthttp://127.0.0.1:8080the control plane advertises viaGET /v1/discovery; set it when 8080 conflicts with another local service. The port in the URL also becomes the printedkubectl port-forward svc/moca-supervisorlocal port. Must behttp://HOST:PORT(nohttps, no path). Sticky like the other inputs here: a plain re-run keeps the stored value;SH_PUBLIC_HARNESS_URL=returns tohttp://127.0.0.1:8080. Refused when Routes are on (SH_ROUTE_DOMAINnon-empty), where the Route host is authoritative.SH_SANDBOX_COUNT, and on OpenShift--imageand--sandbox-image, live in the ConfigMapmoca-setup(namespacemoca, nothing secret in it). A given value replaces the stored one. To go back to a default, pass it explicitly (SH_SANDBOX_COUNT=2,--image ghcr.io/rossoctl/moca:latest), or delete the stored key:kubectl -n moca patch configmap moca-setup --type=json -p '[{"op":"remove","path":"/data/IMAGE"}]'. On Kind the images are never stored: the stack always runs the locally loadeddev.localtags.SH_SANDBOX_EGRESS_EXCEPT(ยง6) lives inmoca-setupon every target. On OpenShift,SH_P4_SANDBOX_IDSandSH_SANDBOX_DEFAULT_TIER(ยง11) live there too, and on--target ocp-singleso doSH_ROUTE_DOMAINand--tls-secret(ยง12.5). For the variables, as for the settings, one set to empty clears the stored value.
kubectl -n moca get configmap moca-settings moca-setup -o yaml shows what the next run will reuse.
A failed read of either ConfigMap aborts the run rather than resetting the inputs.
The CI variant is --target kind-ci. It adds a scripted mock model to the supervisor pod and a
placeholder client id, so the smoke can drive real turns with no model key and no login:
deploy/k8s/setup.sh --target kind-ci --build
K8S_LIVE_SMOKE=1 deploy/k8s/smoke.sh --target kind-ci
That placeholder is the one setting that is not sticky, because both kind targets share one
cluster and so one moca-settings: a later --target kind run discards it and goes back to 0
control-plane replicas, rather than coming up with an id nobody can log in with. Give that run a
real SH_GITHUB_CLIENT_ID to log in.
3. The front door is L4โ
Nothing that speaks HTTP may sit in front of the supervisor's port 8080. That rules out an Ingress, an OpenShift edge or re-encrypt Route, and a service-mesh sidecar.
The reason is that the supervisor works per connection. It accepts a TCP connection, reads the
request head, picks a worker (leastInFlight or stickyBySession), admits or refuses the
connection against that worker's capacity, and hands the socket itself to the worker (ADR-0034). An
L7 proxy terminates the client's connection and pools its own upstream connections, so many users'
turns would arrive over a few long-lived connections. Routing, admission and stickiness would then
apply to the proxy's connections, not the users', and they would silently stop meaning anything.
Nothing would fail visibly.
On Kind, users reach 8080 by kubectl port-forward, which is a byte stream to the pod.
On OpenShift, TLS is terminated inside the supervisor pod by a ghostunnel sidecar, a
byte-for-byte TCP proxy on 8443 that forwards to 127.0.0.1:8080. The Route moca is
passthrough, so the OpenShift router routes on SNI and never decrypts. Each client connection
becomes exactly one supervisor connection. The control plane is ordinary HTTP, so it gets an edge
Route; connection reuse there is harmless.
The image is docker.io/ghostunnel/ghostunnel:v1.11.3, pinned by digest in
overlays/ocp/kustomization.yaml. Docker Hub is the project's official channel; it does not
publish to GHCR.
Each ghostunnel sidecar's readiness probe is GET /_metrics on its own status port (--status;
8453 beside the supervisor, 8454 and 8455 beside the relay's attach and exec sidecars), never a TCP
connect to its TLS port, which logged a handshake error every 5 s. /_metrics dials nothing
(/_status would open a connection to the target on every probe), and no NetworkPolicy admits the
status ports, so only the kubelet's probes reach them. A cluster that needs the node-CIDR fallback
for probes (section 10) must admit the status ports there too.
The manifest tests fail on any Ingress or serving.knative.dev object, and on an OCP supervisor
Route that is not passthrough to the sidecar's port. Replacing the sidecar with an HTTP proxy
would need an ADR superseding point 3 of ADR-0037.
4. Scalingโ
W, the worker count, is pinned, not inferred. base/supervisor.yaml sets three values together:
| Setting | Value | Why |
|---|---|---|
SH_WORKERS | 2 | W |
resources.limits.cpu | "2" | one CPU per worker |
resources.requests.memory | 768Mi (limit 1Gi) | at least W ร 256Mi (the per-process tsx footprint), plus 256Mi headroom |
SH_TURNS_PER_WORKER (trial) | 4, as in Compose | a trial value; its real value is an output of E8 |
Change them together. The manifest tests fail when the memory request is below W ร 256Mi. Leaving
SH_WORKERS unset would follow the CPU limit (#341), but then a CPU change would silently change
the memory needed.
The supervisor and its whole pool are one container. The workers are fork()ed and receive
sockets over IPC, which cannot cross containers. Never split the workers into their own containers
or Deployments; the manifest tests check that only the supervisor container runs the supervisor.
More supervisor pods are allowed. Set replicas in an overlay. With leastInFlight (the
default SH_ROUTING_POLICY), the workers already share nothing but Redis, so more pods add no new
kind of concurrency. With stickyBySession, affinity holds only within one pod: the Service spreads
connections across pods without looking at the session.
Async fan-out is replicas ร SH_WORKERS wide, and nothing autoscales it. With
SH_ASYNC_DRAIN=1 (section 14) every worker also runs a drain loop, and a loop takes one queued
leaf at a time: the default W=2 on one pod runs two async leaves at once however deep the queue
gets, and the rest wait. SH_TURNS_PER_WORKER does not widen that โ it only gates whether a worker
claims at all.
No autoscaler ships, by design. KEDA's queue-depth scaling went with the Knative path
(ADR-0041) and nothing replaced it.
The drain shares one admission budget with interactive turns, so queue depth is the wrong signal to
size the front door by: a scale-up on a deep queue adds interactive capacity nobody asked for, and a
scale-down evicts pods holding leaves mid-run. Instead, watch async_queue_depth and
async_oldest_age_ms on the admin port (section 14 โ depth alone cannot tell a deep fast queue from
a stalled shallow one), alert on them, and set replicas yourself.
The relay stays at one replica (strategy: Recreate). It is presence-only: each Attach stream
is held by one process, so two replicas would split the sandboxes between them and an exec could
reach the wrong one.
Sandboxes: SH_SANDBOX_COUNT for setup.sh sets the StatefulSet's replicas. Each pod's
SANDBOX_ID is its name (moca-sandbox-0, -1, ...), so replicas never collide on one
sh:sandbox:records entry.
5. Secretsโ
setup.sh generates every Secret on its first run and never rotates one. A re-run keeps every
existing value, and fills in only a key that is missing. It reads each Secret with
--ignore-not-found, so "absent" is the only state that counts as missing: any other API error
(a timeout, an RBAC denial, an expired login) aborts the run instead of looking like a missing
Secret and replacing it. No value is ever put on a command line. Values travel through pipes and
through the environment of the one jq that writes each Secret, and are applied on stdin.
| Secret | Namespace | Keys | Consumed by |
|---|---|---|---|
moca-relay | moca | SH_RELAY_TOKEN, MOCA_RELAY_EXEC_TOKEN | relay (both, env); supervisor (MOCA_RELAY_EXEC_TOKEN only, env) |
moca-redis | moca | REDIS_PASSWORD, REDIS_URL, redis.conf | Redis (redis.conf as a file; REDIS_PASSWORD as REDISCLI_AUTH for probes); supervisor, relay, control plane (REDIS_URL) |
moca-redis-provisioner | moca | REDIS_PASSWORD, REDIS_URL (only with SH_WORKLOAD_SANDBOXES=1; kept when off) | provisioner (REDIS_URL: its own Redis user, GET on sh:wl:* only) |
moca-mu1 | moca | SH_SESSION_TOKEN_PRIVATE_KEY, SH_SESSION_TOKEN_PUBLIC_KEYS, SH_CREDENTIAL_KEK, SH_EXCHANGE_TOKEN | control plane (private key, KEK, exchange token as files); supervisor (exchange token as a file, public keyset as env) |
moca-test-users | moca | SH_TEST_USER_SECRET (only while SH_TEST_USERS > 0; deleted when off) | control plane (as a file, from an optional projected source; ADR-0045, ยง13) |
moca-relay-attach | moca-sandbox | SH_RELAY_TOKEN (copied from moca-relay) | sandboxes |
moca-supervisor-tls | moca (OCP) | tls.crt, tls.key | ghostunnel sidecar |
- The tokens and the Redis passwords are
openssl rand -hex 32. REDIS_URLandredis.confare re-derived from the Redis passwords on every run (REDIS_PASSWORD, and with workload sandboxes on,moca-redis-provisioner's too, for its user line), so they never disagree.- With workload sandboxes on,
moca-provisionerreads Redis as a user of its own, never withREDIS_PASSWORD(#626). Its orphan sweep only reads workload records, so the user may runGETonsh:wl:*and nothing else: no write, no other key, no Pub/Sub.redis.confdefines it besiderequirepass, holding the SHA-256 of its password, not the password itself. The password is inmoca-redis-provisioner, kept once made, and itsREDIS_URLisredis://moca-provisioner:<password>@redis.<ns>.svc:6379. Redis readsredis.confonly at start, sosetup.shputs a hash of the user's line on Redis's pod template: adding the user, changing its password or turning the feature off rollsredis-0. - The MU1 keys come from the control plane's own generator, run once in the harness image as the
pod
moca-genkeys. They are validated withdeploy/compose/install.sh's regexes before anything is written, so a garbled generator never leaves half a set. See Troubleshooting for where those values sit while the pod runs. moca-supervisor-tlsis replaced only when--tls-certis given (section 9).
To rotate on purpose, delete the Secret, re-run setup.sh, then restart what consumes it. The
re-run needs no other input: settings, the sandbox count and (on OpenShift) the images are sticky
(section 2), so it does not reset them. For example, the relay tokens on Kind:
kubectl --context kind-moca -n moca delete secret moca-relay
deploy/k8s/setup.sh --target kind --skip-build
kubectl --context kind-moca -n moca rollout restart deployment/sandbox-relay deployment/moca-supervisor
kubectl --context kind-moca -n moca-sandbox rollout restart statefulset/moca-sandbox
The consumers are in the table. For moca-redis, restart redis as well. For
moca-redis-provisioner, the re-run rolls redis itself; restart deployment/moca-provisioner
(it reads REDIS_URL at start). For moca-mu1,
restart the control plane and the supervisor.
Rotating
moca-mu1logs everyone out: a new signing key invalidates every API and session token. If the KEK changes, every stored credential becomes unreadable: the per-user credentials inmoca-credentialsare sealed underSH_CREDENTIAL_KEK, and there is no way back without the old KEK. A half keypair (a private key without its public keyset, or the reverse) is refused: delete both keys to regenerate the pair.
6. Isolationโ
Each namespace has a default-deny NetworkPolicy, both directions. Each workload's own policy then
allows exactly its edges. This replaces the VM's podman isolate=strict and nftables.
| Pod | Ingress allowed | Egress allowed |
|---|---|---|
| supervisor | 8080: nothing from the pod network (port-forward and the OCP sidecar's loopback are not filtered). 8443 (OCP): the router namespace. 8081: pods in moca; kubelet probes are admitted by the CNI (section 10). | DNS; redis 6379; sandbox-relay 9444; moca-control-plane 8080; 0.0.0.0/0 TCP 443 (model API) |
| relay | 9443 from pods in moca-sandbox; 9444 from moca-supervisor pods only | DNS; redis 6379 |
| Redis | 6379 from supervisor, relay, control plane | none |
| control plane | 8080 from moca-supervisor pods; the router namespace (OCP edge Route) | DNS; redis 6379; 0.0.0.0/0 TCP 443 and 6443 (kube API, GitHub) |
| sandbox | none | DNS; sandbox-relay 9443; 0.0.0.0/0 except 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, 169.254.0.0/16 (all ports) |
DNS is kube-system port 53; the OCP overlay adds openshift-dns port 5353.
The sandbox rule lets research turns reach the internet while blocking every cluster-internal address: Pod and Service CIDRs sit inside the private ranges on Kind and OCP. It also blocks cloud instance metadata, which the VM path leaves open (#357).
A publicly routable node or infrastructure network is not covered. The rule assumes everything
that is not the internet uses one of those ranges. Some clusters put their nodes on a publicly
routable range, and there the rule admits them: a sandbox can reach its node's kubelet (10250) and
SSH (22), and other hosts on that network (ยง12.3, #446). Add those ranges with
SH_SANDBOX_EGRESS_EXCEPT, a comma-separated list of IPv4 CIDRs appended to the except list on
every target:
SH_SANDBOX_EGRESS_EXCEPT=203.0.113.0/24,198.51.100.0/24 deploy/k8s/setup.sh --target ocp ...
It is empty by default, so the rule is the one in the table. Each entry must be a canonical CIDR
(network address, prefix 1-32) not already in the built-in list; setup.sh refuses anything else
before it touches the cluster. It is sticky (section 2): a re-run keeps the list, and
SH_SANDBOX_EGRESS_EXCEPT= clears it. Find the node network with
kubectl get nodes -o wide (the INTERNAL-IP column) when you have node read access, or ask the
cluster administrator when you do not (--target ocp-single).
The only Secret any moca object references in the sandbox namespace is the attach token (OpenShift adds its own per-ServiceAccount pull Secrets there): the exec token, the Redis password and the MU1 keys never exist there. The manifest tests check that nothing in it references any other Secret.
What the smoke's claim 7 proves. From inside moca-sandbox-0, TCP connects to
redis.moca.svc:6379, sandbox-relay-exec.moca.svc:9444 and 169.254.169.254:80 are blocked, and
sandbox-relay-attach.moca.svc:9443 is open. On --target ocp it also requires
kubernetes.default.svc:443 to be blocked. On Kind it only prints that probe's result as a note:
single-node kindnet does not filter traffic to its own node (section 10).
A CNI that enforces NetworkPolicy is a hard requirement. On a CNI that ignores NetworkPolicy, every row above is open, and nothing else stands in. kind โฅ 0.24.0 and OpenShift's OVN-Kubernetes enforce it.
7. What this does not claimโ
These are the container tier's non-claims, the same as on the VM
(docs/demos/vm-multi-user-demo.md, "Notes and limits").
- No sandbox isolation between users. Every user's turns lease the same sandbox pods, which
share
/workspace, the Unix user and the process list (remote-worker/internal/exec/runner.goignoresworkspace_key, #408). The fix is MI1 S5, owner binding. - Direct credential mode. With no injector, a user's real inference secret reaches the shared harness worker for the length of a turn, which puts it in the model request's headers. The sandbox never receives it: an Exec carries only the command, stdin, timeout and workspace key. So the agent's bash cannot read a user's key, but one harness process holds every user's key during their turns. MI1 S2's grants replace this.
- Open egress from sandboxes, to the whole internet. Unlike the VM, not to instance metadata
and not to cluster ranges (section 6). Egress control is MI1 S5's
moca-egress.
8. CIโ
.github/workflows/k8s-e2e.yml, job k8s-kind-e2e, runs on
pushes to main and on pull requests touching deploy/k8s/**, the mock model, packages/**,
harness/**, remote-worker/**, pi-fork, the Dockerfile or the lockfile. It installs kind
v0.33.0 (checked by sha256), runs SH_ASYNC_DRAIN=1 setup.sh --target kind-ci --build โ the drain is
opt-in (section 14) and the smoke's async claims skip themselves without it โ then
K8S_LIVE_SMOKE=1 smoke.sh --target kind-ci under pipefail, so a failed claim fails the job. On
failure it uploads the artifact k8s-kind-e2e-logs: pods, describe, every container's log,
events, the smoke's output and its kept log directory, with every token header file deleted first.
It is not a required check until it has passed ten consecutive runs on main. Until then, a
red run is investigated, not waved through, but it does not block a merge.
9. Demo on OpenShiftโ
This is the slice's acceptance run: the container-tier acts of
docs/demos/vm-multi-user-demo.md, against
--target ocp. It is performed by a person and recorded on rossoctl/moca#423. Every outward step
(pushing an image, creating objects on a shared cluster, posting on the issue) needs that person's
go-ahead.
You need:
- an OpenShift 4.x cluster where you can grant SCCs (cluster-admin or equivalent), and
oc logindone; - a harness image built from this branch, pushed to a registry the cluster can pull from (the
published
latestpredates/readyz); - a GitHub OAuth app with Enable Device Flow ticked;
- two GitHub accounts, two laptops (or one laptop with two
XDG_CONFIG_HOMEdirectories and a private browser window, as the VM demo's 1a trap describes), and two inference credentials; - optionally, a certificate and key valid for the supervisor Route's host,
moca-moca.<apps domain>.
The cluster pulls ghostunnel and valkey/valkey:9-alpine from Docker Hub, so it needs access to
docker.io.
9.1 Bring it upโ
Without a certificate of your own (setup.sh generates a self-signed one, see below):
export LOG_DIR=/tmp/kagenti/tdd/moca; mkdir -p "$LOG_DIR"
SH_GITHUB_CLIENT_ID=<client id> SH_ADMIN_SUBJECTS= \
deploy/k8s/setup.sh --target ocp --image <registry>/moca:<branch-tag> \
>"$LOG_DIR/t19-setup.log" 2>&1; echo "EXIT:$?"
tail -6 "$LOG_DIR/t19-setup.log"
With a certificate and key valid for moca-moca.<apps domain>:
export LOG_DIR=/tmp/kagenti/tdd/moca; mkdir -p "$LOG_DIR"
SH_GITHUB_CLIENT_ID=<client id> SH_ADMIN_SUBJECTS= \
deploy/k8s/setup.sh --target ocp --image <registry>/moca:<branch-tag> \
--tls-cert <fullchain.pem> --tls-key <key.pem> >"$LOG_DIR/t19-setup.log" 2>&1; echo "EXIT:$?"
tail -6 "$LOG_DIR/t19-setup.log"
setup.sh --target ocp checks oc whoami and prints the target context. It reads the apps domain
from ingresses.config/cluster, grants nonroot-v2 to each ServiceAccount before the apply, and
sets both Route hosts: moca-moca.<apps domain> (supervisor, passthrough) and
moca-control-plane-moca.<apps domain> (control plane, edge). --sandbox-image overrides the
sandbox image the same way --image overrides the harness.
Without --tls-cert, it generates a self-signed certificate for the supervisor host (825 days)
and prints a warning. Every run then writes the file mocactl users trust, and prints the line
they need:
export NODE_EXTRA_CA_CERTS=<checkout>/deploy/k8s/.generated/ocp/moca-ca.crt
moca-ca.crt holds the supervisor's certificate when it is self-issued, and the cluster's default
ingress CA (ca-bundle.crt of openshift-config-managed/default-ingress-cert), which signs the
control plane's edge Route and is usually not publicly trusted either. Both are read back from the
cluster on every run, so a re-run, or a run from a second checkout, prints the line and rebuilds the
file too. If the ingress CA cannot be read, the run warns and the file holds the supervisor's
certificate alone. With an issued --tls-cert, the file holds only the ingress CA.
Copy that file to both laptops and export it there before running mocactl.
Look for: EXIT:0, N sandbox(es) attached to the relay, and the two Route URLs.
If redis-0 is never created, check for the SCC:
kubectl -n moca get events | grep -i 'security context constraint'
unable to validate against any security context constraint means the explicit Redis UID 999 (or
its fsGroup 1000) is refused, even with nonroot-v2. The fallback (spec ยง10) is a patch in
overlays/ocp that drops runAsUser, runAsGroup and fsGroup from the Redis pod's
securityContext, so restricted-v2 assigns both the UID and the fsGroup from the namespace's
range:
# overlays/ocp/kustomization.yaml, under patches:
- target: { kind: StatefulSet, name: redis }
patch: |-
- { op: remove, path: /spec/template/spec/securityContext/runAsUser }
- { op: remove, path: /spec/template/spec/securityContext/runAsGroup }
- { op: remove, path: /spec/template/spec/securityContext/fsGroup }
Keeping fsGroup: 1000 does not work: restricted-v2 refuses it (its fsGroup must lie in the
namespace's range), so nonroot-v2 admits the pod with no UID at all, and the image runs as root.
The symptom is redis-0 in CreateContainerConfigError, with
container has runAsNonRoot and image will run as root in kubectl -n moca describe pod redis-0.
Commit it, re-run setup.sh, and record that it was needed.
9.2 Live smoke on OpenShiftโ
With a real model. Put the token in the environment from a password manager, never typed on a command line (the smoke reads it from the environment only):
export SMOKE_MODEL_TOKEN="$(<your password manager's CLI>)"
K8S_LIVE_SMOKE=1 SMOKE_MODEL_URL=https://api.anthropic.com SMOKE_MODEL_KIND=api-key \
deploy/k8s/smoke.sh --target ocp 2>&1 | tee "$LOG_DIR/t19-smoke.log"
unset SMOKE_MODEL_TOKEN
SMOKE_MODEL_KIND is api-key for a raw Anthropic key and bearer (the default) for a gateway
token. The smoke reaches everything by port-forward, as on Kind; the Route path is the demo's.
Look for: PASS=12 FAIL=0. On OCP, claim 7 must also report kube API ... BLOCKED. Run the
smoke before the acts: claim 9 deletes the supervisor pod and claim 10 deletes redis-0.
Re-check two things that Kind could not prove, because OVN-Kubernetes behaves differently from kindnet:
-
Probes. On Kind, kubelet probes are admitted because kindnet accepts all traffic the node's root originates. If the supervisor or control plane is not Ready under OVN-Kubernetes, and
kubectl -n moca describe podshows probe timeouts, apply the node-CIDR fallback (section 10). -
The kube API and the kubelet from a sandbox. Claim 7 probes
kubernetes.default.svc:443. Also probe the API server's endpoint address and a node's kubelet port directly, which is how the Kind spike found the gap:kubectl get endpoints kubernetes -n default -o jsonpath='{.subsets[0].addresses[0].ip}'; echokubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'; echofor t in <apiserver ip>:6443 <node ip>:10250; dokubectl -n moca-sandbox exec moca-sandbox-0 -c sandbox -- bash -c \'timeout 3 bash -c "</dev/tcp/${0%:*}/${0#*:}" 2>/dev/null && echo "$0 OPEN" || echo "$0 BLOCKED"' "$t"doneBoth must be
BLOCKED.
9.3 The actsโ
One user per laptop. On each, mocactl runs from a checkout
(docs/harness/how-to/set-up-mocactl.md):
alias mocactl="node $PWD/packages/mocactl/bin/mocactl.mjs"
export NODE_EXTRA_CA_CERTS=<the CA file>
export SH_CONTROL_PLANE_URL=https://moca-control-plane-moca.<apps domain>
mocactl renews its API token by itself, for 30 days after its last use (90 at most), so one
login before the acts is enough. The token lasts 15 minutes unless setup.sh was run with
SH_API_TOKEN_TTL_SECONDS (ยง2's sticky settings). A longer value buys nothing with a renewing
login and costs revocation latency: mocactl logout stops new tokens at once, but a token already
minted stays valid for its whole lifetime. MOCA_TENANCY is unset, as on the VM demo; see its note
on tenancy.
L4 baseline (operator), before act 1. Record the supervisor's counters:
kubectl -n moca exec deploy/moca-supervisor -c supervisor -- wget -qO- 127.0.0.1:8081/metrics |
jq '.counters'
Act 1. Log in, and doctor. Both users:
mocactl login
mocactl doctor
Look for: logged in as <GitHub name> (the device flow approves for whichever account the browser
is signed into), and every doctor check green. --control-plane-url URL on each command does the
same as SH_CONTROL_PLANE_URL.
Act 2. Each user's own credential. Each user stores theirs, under a name of their own (the secret is read from stdin), then runs a turn:
mocactl credentials add anthropic-user1 --host api.anthropic.com --kind api-key \
--endpoint https://api.anthropic.com </path/to/a/0600/file/holding/the/key
mocactl run "Run uname -a in your sandbox and reply with its output only."
A gateway token is --kind bearer with its own host and endpoint. Then the operator shows whose
credential each turn spent:
kubectl -n moca exec redis-0 -- sh -c 'redis-cli XREVRANGE sh:cp:audit + - COUNT 6'
Look for: for each user's session, session_created and credential_issued entries with that
user's subject and their own credential name. The audit records names, never values.
Act 3. Disjoint sessions, and another user's session does not exist. Each user:
mocactl sessions
Each list holds only that user's sessions. User 1 sends a session id to user 2, who tries it, then an id that never existed:
mocactl run "what did you find?" --session <user 1's session id>; echo "exit $?"
mocactl run "what did you find?" --session sess-does-not-exist; echo "exit $?"
Look for the same answer both times, that session no longer exists, or is not yours and exit 1.
User 1 then resumes it, to show it was real.
Act 4. Research from the sandbox. Each user runs the VM demo's 2a prompt, with
D=research-user1 or D=research-user2, and then its 2b check on any machine
(git ls-remote https://github.com/rossoctl/moca HEAD | cut -c1-12 and the nodejs.org index).
Look for the agent's COMMIT=, NODE_VERSION= and NODE_DATE= lines matching the world. Then the
honest beat, where the two research directories landed:
for i in 0 1; do echo "moca-sandbox-$i:"; kubectl -n moca-sandbox exec moca-sandbox-$i -c sandbox -- ls /workspace; done
Both users' turns lease the same pods (section 7).
L4 check (operator), after the acts. This is spec ยง10's third risk: the passthrough path must
keep connections 1:1. Read the counters again (as in the baseline), and while two users' turns are
running at once, count the established connections inside the supervisor pod, on 8443 (from the
router) and on 8080 (from the sidecar). Read both socket tables: ghostunnel is a Go program, and
its 0.0.0.0:8443 listener is a dual-stack IPv6 socket, so the router's connections appear in
/proc/net/tcp6 (as ::ffff: addresses), not in /proc/net/tcp:
kubectl -n moca exec deploy/moca-supervisor -c supervisor -- awk \
'$4 == "01" { split($2, a, ":"); if (a[2] == "20FB") s++; if (a[2] == "1F90") d++ }
END { print "8443:", s + 0, "8080:", d + 0 }' /proc/net/tcp /proc/net/tcp6
Look for: the two numbers equal, and as many as the clients connected (two concurrent turns from
two laptops give at least 2). A router that pooled would show fewer 8443 connections than clients.
Take two readings a few seconds apart: the sidecar's tcpSocket readiness probe and keep-alive
connections come and go. Stop any port-forward to the supervisor first, since its connections
count on 8080 too.
/metrics has no per-connection hand-off counter, so record handoff_retries, handoff_failures
and over_admission before and after as context only. over_admission is not a pooling signal on
its own: in the Kind spike it rose once per turn, with no proxy in front at all. If the connections
are not 1:1, the fallback is an L4 LoadBalancer Service for the sidecar's port.
9.4 Recordโ
Post on rossoctl/moca#423, with the go-ahead: the commit and image, the setup tail, the smoke summary, the probe and kube-API re-checks, each act's result, the connection counts and counters, and any fallback applied (the Redis SCC patch, the node-CIDR policy). Then tick slice 1 in epic #426.
9.5 Cleanupโ
The smoke's residue. smoke.sh stores a credential smoke-inference and creates sessions,
all under the subject smoke:1, and deletes none of them. Deleting the namespaces (below) removes
them. To keep the stack, delete them through the control plane with a token minted the way the
smoke mints it, inside the control-plane pod:
kubectl -n moca port-forward svc/moca-control-plane 18090:8080 >/dev/null 2>&1 &
PF=$!; sleep 3
HDR="$(mktemp)"
kubectl -n moca exec deploy/moca-control-plane -c control-plane -- \
node --import tsx --input-type=module -e \
"import { readFileSync } from 'node:fs'; import { makeSigner } from './src/token.ts';
const s = makeSigner(readFileSync('/run/credentials/SH_SESSION_TOKEN_PRIVATE_KEY', 'utf8'));
process.stdout.write(s.mint({ sub: 'smoke:1', tenant: 'smoke:1', roles: [], scope: ['api'], ttlSeconds: 900 }));" |
sed 's/^/Authorization: Bearer /' >"$HDR"
curl -s -o /dev/null -w '%{http_code}\n' -X DELETE -H @"$HDR" http://127.0.0.1:18090/v1/credentials/smoke-inference
for id in $(curl -s -H @"$HDR" http://127.0.0.1:18090/v1/sessions | jq -r '.sessions[].sessionId'); do
curl -s -o /dev/null -w "$id %{http_code}\n" -X DELETE -H @"$HDR" "http://127.0.0.1:18090/v1/sessions/$id"
done
rm -f "$HDR"; kill "$PF"
Each delete answers 204.
On each laptop, delete the run's sessions (mocactl sessions delete ID) and credentials
(mocactl credentials delete NAME), and run
mocactl logout (it revokes the login on the control plane and deletes auth.json). On the cluster, delete the
three namespaces (moca, moca-sandbox, moca-credentials); deleting moca deletes the Redis
PVC with it. The nonroot-v2 grants are cluster objects:
oc adm policy remove-scc-from-user nonroot-v2 -z <sa> -n <ns> for each ServiceAccount.
10. Troubleshootingโ
The commands here are written for Kind (--context kind-moca, as setup.sh and smoke.sh always
pin it). On OpenShift, drop --context kind-moca.
Kind on macOS with Podman. Both workarounds were needed on the Task 16 run (macOS arm64, Podman
5.7.1 behind the docker CLI). Neither is expected on Linux with Docker.
-
A proxy in your shell.
kind createcopiesHTTP_PROXY/HTTPS_PROXY/NO_PROXYinto the node's containerd. A proxy on127.0.0.1then points at the node itself, and in-cluster pulls (valkey/valkey:9-alpine) fail withproxyconnect tcp: dial tcp 127.0.0.1:.... Create the cluster without them, then runsetup.shas usual (it reuses an existingmocacluster):env -u HTTPS_PROXY -u https_proxy -u HTTP_PROXY -u http_proxy -u NO_PROXY -u no_proxy \kind create cluster --name mocaA cluster already created with the proxy has to be deleted and recreated.
-
Flaky DNS from pods. The node's
/etc/resolv.conffrom the Podman network lists an IPv6 nameserver first that is unreachable, and the IPv4 one timed out under load. Research turns then fail withCould not resolve host. Point CoreDNS at public resolvers:kubectl --context kind-moca -n kube-system get configmap coredns -o yaml |sed 's#forward . /etc/resolv.conf#forward . 1.1.1.1 8.8.8.8#' |kubectl --context kind-moca apply -f -kubectl --context kind-moca -n kube-system rollout restart deployment/corednsBefore suspecting the egress policy, look at
kubectl --context kind-moca -n kube-system logs -l k8s-app=kube-dnsfor upstream errors.
Where the smoke keeps its logs. On any failed claim, smoke.sh keeps its working directory
and prints logs kept in <dir> (a mktemp -d directory, mode 0700). It holds the port-forward
logs and each turn's SSE stream; the files holding tokens are deleted first. A run that passes
removes it. A smoke that stops before the claims (port-forward to the supervisor never came up)
prints the port-forward log itself and keeps nothing. In CI, the directory is in the
k8s-kind-e2e-logs artifact.
The generated MU1 keys sit in the moca-genkeys pod's log until --rm deletes it. The
generator writes the four values to its stdout, which setup.sh reads over the attach, so for the
pod's few seconds of life they are also in its container log on the node, readable by anyone with
pods/log in moca. --rm deletes the pod and its log when the run completes. If setup.sh is
killed in that window, delete it yourself:
kubectl --context kind-moca -n moca delete pod moca-genkeys --ignore-not-found (setup.sh also deletes a leftover one before it starts).
On single-node Kind, a sandbox can reach the kube API and its own node's kubelet.
kubernetes.default.svc:443 and the node's port 10250 are open from a sandbox, although the
egress rule excepts 10.0.0.0/8. kindnet enforces NetworkPolicy only in the forwarding path, and
pod-to-own-node traffic never passes it. The exposure is anonymous only: sandboxes mount no
ServiceAccount token, so /version answers and secrets is forbidden. No manifest change can fix
it on kindnet, so the smoke reports it as a note on Kind and enforces it on --target ocp
(section 9.2). Kind CI therefore does not prove the kube-API carve-out; the OpenShift run does.
kindnet also accepts all UDP/53, so the DNS policies are not exercised on Kind either.
Kubelet probes and the node-CIDR fallback. If NetworkPolicy filtered the kubelet's probes, the
supervisor (8081) and control plane (8080) would never become Ready, with probe timeouts in
kubectl describe pod. The fallback (spec ยง10) is a per-overlay policy allowing ingress from the
node CIDR to those two ports:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: probes-from-nodes, namespace: moca }
spec:
podSelector:
{
matchExpressions: [{ key: app, operator: In, values: [moca-supervisor, moca-control-plane] }],
}
policyTypes: [Ingress]
ingress:
- from: [{ ipBlock: { cidr: <node CIDR> } }]
ports: [{ protocol: TCP, port: 8081 }, { protocol: TCP, port: 8080 }]
On OpenShift, the ghostunnel sidecars are probed too, on their status ports: 8453 on the
supervisor pods, 8454 and 8455 on the relay pods (section 3). A cluster that needs this fallback
adds sandbox-relay to the selector's values and { protocol: TCP, port: 8453 },
{ protocol: TCP, port: 8454 } and { protocol: TCP, port: 8455 } to its ports, or the sidecars
stay NotReady.
It was not needed on Kind: kindnet accepts traffic originated by the node's root user before any policy, which covers the kubelet. It is still to be checked on OVN-Kubernetes (section 9.2).
The control plane stays 0/1. It reconnects to Redis forever, with a bounded backoff, and its
/readyz fails fast while Redis is down, so it recovers by itself once Redis is up (the common case
on a first apply, since everything is applied at once). If it stays unready, its log shows
[control-plane] redis client error: ...; check redis-0. Two smokes run back to back can catch
claim 5 inside that 5โ10 s reconnect window after the previous run's Redis restart: wait for
/readyz, or re-run setup.sh, in between.
only 0 of N sandboxes attached to the relay. The relay retries each sandbox's presence write
until it lands or the sandbox detaches, so a late Redis delays the records but no longer loses
them. If the count stays short, the message names the two logs to read:
kubectl --context kind-moca -n moca logs deployment/sandbox-relay (presence put failed ...)
and kubectl --context kind-moca -n moca-sandbox logs statefulset/moca-sandbox.
redis-0 stays Pending. Its PVC binds only through a default StorageClass. setup.sh warns
when the cluster has none: mark one default (storageclass.kubernetes.io/is-default-class=true).
A container restarted. Smoke claim 11 fails on any restart in either namespace. The likeliest is
the supervisor OOM-killed at its 1Gi limit (kubectl --context kind-moca -n moca describe pod shows OOMKilled):
raise the memory request and limit together (section 4). A worker that dies inside the pod is not a
container restart: look for worker_exit in the supervisor's log.
Claim 12 notes could not tell affinity from luck. Its four first turns run at once so the
least-loaded choice spreads them over the sandboxes; when they all land on one, every second turn
matches by luck too. The claim still passes; re-run the smoke for a run that proves affinity. A
claim 12 FAIL names the session that moved and both pod names.
The supervisor never becomes Ready on a pulled image. The published latest may predate
/readyz, so the startup probe fails. Use --build on Kind, or a branch image with --image on
OCP.
port-forward is filtered. Not seen on Kind (port 8080 has no ingress rule at all, yet answers
through port-forward). On a CNI that does filter it, smoke claims 1, 3 and 5 fail; the fallback is
an allow from the node CIDR on that overlay, as for the probes.
11. P4 on Kubernetes: microVM hosts outside the clusterโ
P6 runs in the cluster; only the P4 microVM tier needs a KVM host (bare metal or a nested-virt VM).
Each such host runs microvm-worker and attaches to the in-cluster relay over TLS, through an L4
sidecar (ghostunnel) behind the passthrough Route moca-relay. The cluster issues each host's
token; the relay reads it from a mounted Secret on every attach, so adding a host restarts nothing.
Design:
docs/specs/2026-10-04-p6-on-kubernetes-slice2-design.md.
OpenShift only: setup.sh refuses SH_P4_SANDBOX_IDS on kind and on ocp-single. A stack
may run P4 hosts beside its container sandboxes, and several P4 hosts: since P6.3 (#425) a session
returns to the sandbox that holds its workspace
(docs/specs/2026-10-04-p6-on-kubernetes-slice3-design.md).
11.1 Container sandboxes and P4 together, or P4 onlyโ
# A mixed stack: two container sandboxes and one P4 host.
SH_SANDBOX_COUNT=2 SH_P4_SANDBOX_IDS=moca_microvm_0 deploy/k8s/setup.sh --target ocp
# P4 only.
SH_SANDBOX_COUNT=0 SH_P4_SANDBOX_IDS=moca_microvm_0 deploy/k8s/setup.sh --target ocp
A mixed stack is tiered (SH_SANDBOX_COUNT above 0 and SH_P4_SANDBOX_IDS not empty):
setup.sh writes SH_SANDBOX_TIERS=container,microvm to moca-settings, which the supervisor and
the control plane both read, and logs two sandbox tiers. A session is created in a tier and every
turn of it runs there. A user picks the tier at creation, in mocactl's New Session overlay or with
mocactl run --new --option sandboxTier=microvm; a session that names none gets the default tier.
SH_SANDBOX_DEFAULT_TIER(containerormicrovm, defaultcontainer) sets that default. It is sticky, stored inmoca-setup;SH_SANDBOX_DEFAULT_TIER=clears it back tocontainer. Any other value is refused before anything is stored. Kind is never tiered and stores none, but it refuses a bad value too.- A single-tier stack stays untiered (containers only, or P4 only): both tier settings are
empty, and a stored
SH_SANDBOX_DEFAULT_TIERis kept but unused. Untiered on purpose: a worker older than P6.3 advertises no tier, and a tiered supervisor would exclude it. - On a mixed stack every worker must be P6.3 or later (the container sandboxes' image and each
P4 host's
microvm-worker): an older one advertises no tier and receives no sessions. - Any settings change now rolls the supervisor too (section 2): a change of client id or admins
restarts both Deployments, and the first run of a P6.3
setup.shon an existing stack rolls both once, because the settings it hashes gained the two tier keys. - That first run also recreates the relay and rolls the container sandboxes (section 2), on
every stack, tiered or not. Every sandbox and P4 host detaches while the relay is down, and every
container session's workspace is reset with no
workspace resetnotice, since each session returns to a sandbox of the same name. Run it at a quiet time.
The rest applies to every stack with P4 hosts:
- IDs match
^[A-Za-z_][A-Za-z0-9_]*$(no dashes: the relay looks each up by name), comma-separated, each listed once. DIRis reserved and refused:SH_RELAY_TOKEN_DIRis the relay's token-directory setting, so the relay refuses that ID before any lookup.- The list is sticky like every other input;
SH_P4_SANDBOX_IDS=(empty) clears it (see 11.4 for the objects that clearing leaves behind). --relay-tls-cert FILE --relay-tls-key FILEinstalls an operator certificate formoca-relay-moca.<apps domain>. Without them a self-signed one is generated once and kept.- Every bundle carries the relay certificate as
relay-ca.crtwhenever it is self-issued (setup.sh's own, or a self-signed one you supplied). A certificate issued by another CA is not shipped: that CA must be one the hosts already trust system-wide.
setup.sh ends by printing, per ID, the bundle path and the commands to install it:
P4 hosts attach to https://moca-relay-moca.<apps domain> (the relay, TLS passthrough). One bundle each -- it holds
that host's relay token, so copy it straight to the host, install it, then delete the copy:
moca_microvm_0: <checkout>/deploy/k8s/.generated/ocp/p4/moca_microvm_0
ssh <kvm-host> rm -r moca-p4-moca_microvm_0 (an earlier copy, if any: scp -r would nest into it)
scp -r <checkout>/deploy/k8s/.generated/ocp/p4/moca_microvm_0 <kvm-host>:moca-p4-moca_microvm_0
on <kvm-host>, from a moca checkout: sudo deploy/microvm/setup-microvm.sh --remote ~/moca-p4-moca_microvm_0
The bundle (worker.env and relay-ca.crt, mode 0600, in a 0700 directory) holds that host's relay
token. Copy it straight to its host and delete the copy once installed.
Re-copying a bundle (after a new certificate, or an ID removed and added again): first remove
~/moca-p4-<id> on the host, as the printed ssh <kvm-host> rm -r moca-p4-<id> does.
scp -r <dir> <kvm-host>:moca-p4-<id> into an existing directory nests the copy as
moca-p4-<id>/<id>, and --remote would then install the stale bundle left at the top. On a first
copy the rm only reports that nothing is there.
11.2 Install a hostโ
The host needs what MicroVM tier (P4) lists for P4 (/dev/kvm, Firecracker and jailer,
a golden snapshot, Go or MICROVM_BIN) and outbound TCP 443 to the relay Route host. Check that
first with curl -v https://moca-relay-moca.<apps domain>: a completed TLS handshake proves it is
reachable, even when curl then stops at a certificate verification error (as it does with the
default self-signed certificate); a timeout or Connection refused does not. It does not need
P6 installed.
setup-microvm.sh --remote <bundle> installs the worker with the bundle's address, TLS, ID and
token (the cluster's token replaces any local one), installs relay-ca.crt as
/etc/serverless-harness/microvm-relay-ca.crt, and swaps the worker's drop-in to
50-moca-remote.conf. Of a local P6 it touches only the tier files: on a host an earlier mixed
local run tiered, it removes microvm-tiers.env and both drop-ins
(sh-supervisor.service.d/50-microvm-tiers.conf, sh-control-plane.service.d/50-microvm-tiers.conf)
and try-restarts the installed sh-supervisor and sh-control-plane, because the host's own P6 is
container-only from then on (P6.3 spec ยง7).
It finishes only once the worker's current process has logged attached, serving execs and the
attach has then held for MICROVM_ATTACH_SETTLE seconds (default 5, at least 1) with no reconnect,
all within MICROVM_ATTACH_TIMEOUT (default 60; a hold that starts before then may finish). The
hold matters because the worker logs attached when its stream opens, before the relay checks the
token; a refused token ends the stream milliseconds later. Otherwise it dies naming that process's
journal (journalctl -u microvm-worker _SYSTEMD_INVOCATION_ID=โฆ), which carries the TLS or token
error.
Running setup-microvm.sh again without --remote switches the host back to its local P6
relay, restoring its local token.
11.3 Verifyโ
K8S_LIVE_SMOKE=1 SMOKE_MODEL_URL=โฆ SMOKE_MODEL_TOKEN=โฆ deploy/k8s/smoke.sh --target ocp --tier p4
Claims 1, 5 and 6 are the container tier's. P2: every P4 ID is attached. On a P4-only (untiered)
stack no container sandbox may be; on a tiered stack container sandboxes may be attached beside
them, and each P4 ID's record must advertise the microvm tier (moca.dev/tier). P3: a turn runs
on a kernel that is no node's and writes a file. P4: a second turn reads it back. P6: through the relay Route, a wrong attach token is refused at the relay, and SandboxExec is
not served even with the exec token. P7 (only with SMOKE_P4_ADD_ID=<scratch id>): setup.sh adds
that ID and removes it again, both runs keeping the sticky SH_SANDBOX_COUNT (so a mixed stack
keeps its container sandboxes), and the relay pod is the same one, with the same start time and no
restart. Claims 7โ12 do not run on this tier. On a tiered stack every smoke session asks for its tier
(microvm here, container without --tier p4) and the smoke prints a note naming the stack's
tiers, so both smokes test the right tier whatever the default; on an untiered stack sessions name
none. A moca-settings the smoke cannot read is a FAIL, not "untiered".
11.4 Add, revoke, and their timingโ
-
Add an ID to the list and re-run
setup.sh: the new token reaches the relay's directory within the kubelet's Secret sync (about a minute; P7 prints the measured time), and the host's next attach succeeds. Attached hosts stay attached. -
Revoke by removing the ID: its token and bundle are deleted, and once the removal reaches the relay's directory (the same kubelet Secret sync, about a minute) its next attach is refused. A host already attached stays attached until its stream drops; to cut it off at once, restart the relay (
kubectl -n moca rollout restart deployment/sandbox-relay), which drops every attached sandbox. -
Clear every ID (
SH_P4_SANDBOX_IDS=; on a P4-only stack, withSH_SANDBOX_COUNTback above 0 to return to container sandboxes; a mixed stack becomes untiered): the relay pod loses its TLS sidecar, so nothing listens on 8444 any more.setup.shapplies with a non-pruningkubectl apply -k, though, so four objects of the p4-relay component stay behind. Delete them by hand:kubectl -n moca delete route/moca-relay service/sandbox-relay-tls \networkpolicy/sandbox-relay-from-router secret/moca-relay-tlsSecret
moca-relay-sandbox-tokensstays, emptied, on purpose: an emptied Secret revokes every host deterministically. Deletingmoca-relay-tlsmeans a later switch back to P4 generates a new self-signed certificate (unless--relay-tls-certis given), so every bundle is then re-copied.
11.5 Troubleshootingโ
The relay pod has two containers here, and the tls sidecar is listed first. The pod's
kubectl.kubernetes.io/default-container: sandbox-relay annotation sends plain kubectl logs and
kubectl exec to the relay; name the sidecar to read it:
kubectl -n moca logs deployment/sandbox-relay -c tls.
setup-microvm.sh --remote dies with did not stay attached. The message names the worker
journal to read. A wrong or revoked token shows there as attached, serving execs followed at once
by stream ended (โฆ); reconnecting: the bundle is stale, or its ID is no longer listed. Re-run
setup.sh, copy the fresh bundle and install it again.
The worker journal says missing selected ALPN property. The relay sidecar is not negotiating
h2 (--alpn h2 in overlays/ocp/p4-relay); grpc-go 1.67+ requires it.
x509: certificate signed by unknown authority. The bundle's relay-ca.crt was not installed
(re-run --remote with the full bundle), or an operator certificate does not chain to a CA the host
trusts.
The worker attaches, then drops every few minutes. The router's idle timeout is shorter than the
stream's quiet periods. The Route carries haproxy.router.openshift.io/timeout: 5m, above the
worker's 30 s keepalive; check the annotation survived.
P2 fails with a P4 ID missing. The host's worker is not attached; read its journal on the
host (journalctl -u microvm-worker).
12. Single-namespace dev/test target (--target ocp-single)โ
For a developer or CI tenant on a shared OpenShift cluster where you have edit rights in one
namespace and nothing else: no cluster-scope RBAC, no SCC grants, no permission to read
cluster-scoped objects. The whole stack -- supervisor, relay, control plane, Redis, the container
sandbox pool, every Secret, the credential Role -- runs in that one namespace, reached by
port-forward as on Kind -- or, when your tenant has a custom domain, by opt-in Routes (ยง12.5).
export LOG_DIR=/tmp/kagenti/tdd/moca; mkdir -p "$LOG_DIR"
SH_GITHUB_CLIENT_ID=<client id> SH_SINGLE_NAMESPACE=<your-namespace> \
deploy/k8s/setup.sh --target ocp-single >"$LOG_DIR/t-single-setup.log" 2>&1; echo "EXIT:$?"
tail -4 "$LOG_DIR/t-single-setup.log"
The namespace must already exist (creating one needs cluster scope, which this target assumes you
lack); SH_SINGLE_NAMESPACE defaults to moca-single. A ResourceQuota of 8 CPU / 32 Gi covers the
default stack several times over (ยง4). The one requirement the cluster must meet anyway: a default
StorageClass, for the Redis PVC.
12.1 How it stays within the namespaceโ
Everything the ocp target does outside the namespace is skipped or replaced:
The ocp target | ocp-single |
|---|---|
creates the moca, moca-sandbox, moca-credentials namespaces | uses your existing namespace (existence checked, never created) |
grants nonroot-v2 to each ServiceAccount (oc adm) | no SCC: every explicit UID/GID is stripped, restricted-v2 assigns from the namespace's range |
Routes, ingresses.config/cluster read, TLS secrets | no Routes by default (kubectl port-forward, like Kind); opt-in Routes with SH_ROUTE_DOMAIN (ยง12.5), which needs no cluster-scope read because you give the domain |
requires oc | kubectl only |
The isolation the base draws from the namespace boundary moves to pod-label NetworkPolicies: the default-deny, the sandbox's egress (relay attach port and the public internet only), the relay's exec-port rule (supervisor only) are all enforced per pod label. The sandbox's egress policy is the one that carries the isolation: a tenant's own namespace policies can widen ingress (ยง12.3). The sandboxes mount no ServiceAccount token, so the kube API refuses them even where the cluster leaves it reachable (ยง12.3).
Three consequences of the collapse, accepted for a dev/test tenant:
- The secrets the base keeps out of
moca-sandbox(the exec token, the Redis password, the MU1 keys) share the namespace with the sandbox pods -- anyone who can edit the namespace can read them. - The control plane's Secret Role (
moca-control-plane-credentials: get, create, update, patch, delete; no list) is scoped tomoca-credentialsin the base, so it reaches user credentials only. In one namespace it reaches every Secret: the relay attach and exec tokens, the Redis password, the MU1 keys. A compromised control plane can read and overwrite platform secrets by name, not just user credentials. - The sandbox workspace and home are
emptyDirs, so they are lost when a sandbox pod restarts (they are container-local in the base too).
12.2 Smokeโ
K8S_LIVE_SMOKE=1 SMOKE_MODEL_URL=<endpoint> SMOKE_MODEL_TOKEN=<token> \
deploy/k8s/smoke.sh --target ocp-single >"$LOG_DIR/t-single-smoke.log" 2>&1; echo "EXIT:$?"
There is no in-pod mock model on this target (the kind-ci one rides the dev.local image), so the
smoke needs a real model credential, like --target ocp. The claims are the container tier's:
pods up, sandboxes attached, authenticated turns in a sandbox, isolation (redis, relay exec,
metadata BLOCKED; relay attach OPEN), drain, Redis-restart persistence, no restarts, and a session
staying on its sandbox (claim 12). The kube API
probe is reported as a note, not a claim (ยง12.3). For a custom namespace, export the same
SH_SINGLE_NAMESPACE you installed with; without it, smoke reads the moca-setup ConfigMap in the
kubeconfig's current namespace, so it finds the stack only when the context is already set to it.
12.3 Verifiedโ
Brought up and smoked (11/11) on a shared OpenShift 4.x tenant with edit-only namespace rights, an
8 CPU / 32 Gi quota, RHACS image admission active, and no SCC grants: restricted-v2 assigned
UID/GID/fsGroup 1002610000 from the namespace's range, the emptyDirs over /workspace and
/home/sandbox were owned by the assigned fsGroup, and the cluster's CNI (Calico, on IBM Cloud
ROKS) enforced the pod-label policies, with the exceptions below.
The kube-API Service VIP was reachable from the sandbox. The VIP (172.21.0.1:443) is
DNAT'd to ROKS's node-local apiserver proxy, 172.20.0.1:2040, and the sandbox reaches that
address directly too, although it is inside the excepted 172.16.0.0/12; other ports on it, other
private IPs, and the DNS VIP were blocked. The namespace's own policies do not explain it, so a
cluster-level allowance does (not readable with namespace rights). The sandboxes mount no
ServiceAccount token, so the API refuses them; treat it as defence-in-depth lost, not as an
escape.
The node network is outside the egress except list. That tenant's nodes sit on a
publicly routable (non-RFC 1918) range, so the sandbox's internet rule admits it: the sandbox
reached its node's kubelet (10250) and SSH (22). The except list covers RFC 1918, CGNAT and
link-local only; on a cluster whose node or infrastructure network is publicly routable, add
those CIDRs with SH_SANDBOX_EGRESS_EXCEPT (ยง6, #446).
A tenant allow-same-namespace policy (ingress from any pod in the namespace, to every pod)
was preinstalled. Policies are additive, so the overlay's ingress rules (the relay's
supervisor-only exec port, Redis) admit any pod in the namespace there. The sandboxes stay
isolated because their own egress policy is enforced: Redis and the relay exec port were BLOCKED
from a sandbox.
12.4 What does not work hereโ
P4 microVM hosts cannot attach: their path is the relay's OpenShift Route (ยง11), which this
target does not render. Container sandboxes only, so this target is never tiered; a given
SH_SANDBOX_DEFAULT_TIER is stored in moca-setup but unused.
Renaming on re-run: SH_SINGLE_NAMESPACE is not sticky -- it is validated and applied fresh
every run. Pointing it at a different namespace installs a second stack there; the first one stays
until its namespace is deleted.
12.5 Routes, opt-in (SH_ROUTE_DOMAIN)โ
The port-forwards are the default because the target cannot learn a Route hostname on its own:
reading the cluster's apps domain (ingresses.config/cluster) is a cluster-scoped read, which this
target assumes you lack. But a tenant whose operator set up a custom domain -- a wildcard DNS
record and certificate for *.<namespace>.<domain>, plus an ingress controller -- can be reached
properly. Opt in with the domain:
SH_GITHUB_CLIENT_ID=<client id> SH_SINGLE_NAMESPACE=<ns> SH_ROUTE_DOMAIN=<domain> \
deploy/k8s/setup.sh --target ocp-single [--tls-secret NAME]
Two Routes render, in the namespace: moca.<domain> (the supervisor, TLS passthrough to a
ghostunnel L4 sidecar, exactly ยง11's shape -- an edge Route there would silently break the
supervisor's per-connection routing) and moca-control-plane.<domain> (the control plane, ordinary
edge). moca-settings's SH_PUBLIC_HARNESS_URL becomes the supervisor's Route, so /v1/discovery
advertises it and mocactl needs no port-forward:
mocactl --control-plane-url https://moca-control-plane.<domain> login
The supervisor's certificate comes from one of three sources, in this order:
| Source | What setup.sh does |
|---|---|
--tls-secret NAME | references a preinstalled kubernetes.io/tls Secret by name; never copies it |
--tls-cert FILE --tls-key FILE | installs the pair as the moca-supervisor-tls Secret |
| neither | generates a self-signed one, and says so (NODE_EXTRA_CA_CERTS, as on --target ocp) |
The edge Route embeds no certificate: the ingress controller serves its own (on a custom-domain tenant, that is the operator's wildcard; on a standard cluster, the router's default for the apps domain).
SH_ROUTE_DOMAIN and --tls-secret are sticky like every input: a re-run without them keeps the
Routes (and the certificate) it finds in moca-setup; SH_ROUTE_DOMAIN= set-but-empty turns the
Routes off and returns to port-forward access. The three certificate sources are mutually
exclusive per run (--tls-secret with --tls-cert is refused), and --tls-cert/--tls-key on a
re-run replaces a saved --tls-secret rather than being silently beaten by it -- the saved name
is dropped from moca-setup, so a later re-run with no certificate input cannot resurrect it over
what the pair installed. The smoke still reaches the stack by port-forward, Routes or not.
The ยง12.3 tenant verified all of it: the operator's Let's Encrypt wildcard for
*.moca-ns.vpc-int.res.ibm.com (served by --tls-secret, and also covering the edge Route as the
controller's default certificate), a namespace-scoped ingress controller admitted by the
policy-group.network.openshift.io/ingress selector, and mocactl logins and turns over the Routes.
13. Test users, for unattended multi-user testing (SH_TEST_USERS)โ
Preview. Test-user logins are implemented but not yet declared Stable; interfaces and limits may change. Known limits: dev and test only, container tier only, and not available on the sandbox profile (see below and ADR-0045).
Every login is GitHub's device flow, which needs a person and a browser. SH_TEST_USERS=N adds N
synthetic subjects test:1 โฆ test:N that log in with a shared secret instead, so a script can
exercise the multi-user claims โ each subject spending only its own credential, and one user being
refused another's session. A test user is configured exactly like a GitHub user: the control plane
issues the same refresh family and the same token lifetimes, so rotation, roles
(SH_ADMIN_SUBJECTS), the ownership index and the audit trail behave identically.
It is off unless asked for, and gated by a secret, because the endpoint issues a real login for
an arbitrary subject: without one, anyone who can reach the control plane is every test user.
POST /v1/auth/test-token answers 404 while SH_TEST_USERS is unset, the route is not mounted on
the sandbox profile, and the control plane refuses to boot if the two are combined.
13.1 Bring it upโ
SH_TEST_USERS is sticky like every other setting (section 2); SH_TEST_USERS= turns it off and
deletes the secret with it. On --target ocp -- the shared-cluster target -- it additionally needs
SH_TEST_USERS_ACK=i-know-this-is-not-for-production, so a stale export cannot quietly leave a
password-gated login path on a cluster other people use.
export LOG_DIR=/tmp/kagenti/tdd/moca; mkdir -p "$LOG_DIR"
SH_TEST_USERS=10 SH_GITHUB_CLIENT_ID=<client id> SH_SINGLE_NAMESPACE=<ns> \
deploy/k8s/setup.sh --target ocp-single --skip-build \
--image <registry>/moca:<tag> --sandbox-image <registry>/moca-remote-worker:<tag> \
>"$LOG_DIR/t-users-setup.log" 2>&1; echo "EXIT:$?"
Look for: WARNING: SH_TEST_USERS=10: test:1..test:10 can log in with the shared secret, and
either generated SH_TEST_USER_SECRET or nothing (a re-run keeps the stored one and never rotates
it). The secret lives in a Secret of its own, moca-test-users, and reaches the control plane as a
credential file under CREDENTIALS_DIRECTORY, never an environment line. Its own Secret
because the credential loader treats an empty file as an error and only an absent one as "unset":
setup.sh therefore writes it only while test users are on and deletes it when they go off, and
control-plane.yaml projects it from an optional source so a stack without it still boots. Give your own instead of a generated one
with SH_TEST_USER_SECRET=<at least 16 characters>.
kubectl -n <ns> get secret moca-test-users -o jsonpath='{.data.SH_TEST_USER_SECRET}' | base64 -d
13.2 Check the gate firstโ
Before trusting the feature, prove it is invisible without it. On a stack where SH_TEST_USERS is
unset this must answer 404:
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: application/json' \
-d '{"index":1}' http://127.0.0.1:8090/v1/auth/test-token
13.3 One user, by handโ
The secret travels in MOCA_TEST_USER_SECRET, never a flag โ a flag would land in the shell history
and in every ps listing on the machine.
export MOCA_TEST_USER_SECRET=<the secret>
mocactl --control-plane-url http://127.0.0.1:8090 login --test-user 3
# logged in as Test User 3 (test:3)
13.4 N users, and the cross-user caseโ
deploy/k8s/test-users.py drives N users (default 10, --users N) through real mocactl
commands, then asserts the cross-user claims. --api runs the same claims straight against the
HTTP API, which separates a client bug from a server one.
export MOCA_TEST_USER_SECRET=<the secret>
./deploy/k8s/test-users.py --users 10 --control-plane-url http://127.0.0.1:8090
./deploy/k8s/test-users.py --users 10 --api --json
Look for: users 10/10 fully ok; cross-user 4/4 and RESULT: PASS.
Each user gets its own XDG_CONFIG_HOME, because mocactl keeps one login in one auth.json
(packages/mocactl/src/config.ts): a second login in the same directory overwrites the first, so
one process in one directory cannot hold ten logins. That is also what makes --jobs safe.
The operator's check, that each turn spent its own user's credential โ the audit records credential names, never values:
kubectl -n <ns> exec redis-0 -- sh -c \
'redis-cli --no-auth-warning -a "$REDISCLI_AUTH" XREVRANGE sh:cp:audit + - COUNT 300' |
grep -A1 '^subject$' | grep '^test:' | sort -u # ten distinct subjects
Stop any port-forward on the harness port (8080) and on 18090 before running smoke.sh:
it binds 18090 itself and reaches the harness at the URL the control plane advertises, so a
leftover forward makes every /turn answer 404. Section 9.3 gives the same warning for its
connection count.
Tested on OpenShift 4.x (IBM Cloud ROKS, --target ocp-single, edit-only namespace rights),
2026-10-26: N=10 passed in both CLI and --api modes, with ten distinct test:N subjects and ten
credential names in the audit stream.
14. Async fan-out, opt-in (SH_ASYNC_DRAIN)โ
An orchestrator that wants to dispatch many leaves and collect them later submits them to the
control plane and polls; it does not hold a connection per leaf. This is archetype A
(dispatch archetypes), the KEDA ScaledObject's
replacement, decided in ADR-0047.
Status: single-tenant in v0.10.0. Async fan-out under
MOCA_TENANCY=multi(grants minted at dequeue, #388) is being worked on, onmain, and is targeted for v0.12.0. Until it ships, run the drain only underMOCA_TENANCY=single; themultinotes in this section describe work in progress.
The consumer is not a new workload. No Deployment, no --role=leaf pod, no autoscaler: with the
knob on, each of the supervisor's ordinary --role=turn workers also runs a drain loop that claims
one queued turn at a time and runs it through the same runLeaf a synchronous /v1/runs uses. Both
things it needs are already on the pod โ SH_CONTROL_PLANE_URL and the SH_EXCHANGE_TOKEN file
(section 5) โ so turning it on takes no other wiring.
14.1 Turn it onโ
SH_ASYNC_DRAIN is a sticky setup.sh input, like the token lifetimes (section 2). Set it on a
re-run; moca-settings keeps it afterwards, and the settings hash on the pod template rolls the
supervisor as part of the same apply, so nothing needs restarting by hand:
SH_ASYNC_DRAIN=1 deploy/k8s/setup.sh --target kind --skip-build
Only 1 and empty are accepted โ SH_ASYNC_DRAIN=true is refused rather than read as off, because
a typo that silently costs you the whole feature is worse than one re-run. To turn it off again,
pass it set but empty (SH_ASYNC_DRAIN= deploy/k8s/setup.sh โฆ); leaving it unset keeps whatever
the ConfigMap holds.
Check that it armed, on the admin port:
kubectl --context kind-moca -n moca port-forward svc/moca-supervisor 18081:8081 &
curl -s http://127.0.0.1:18081/metrics | jq '{async_queue_depth, async_oldest_age_ms}'
A number (even 0) means worker slot 0's drain loop is running and reporting. "NaN" means it
never reported โ the knob is off, or a prerequisite was missing and the worker said so and stayed
off rather than failing: kubectl -n moca logs deploy/moca-supervisor | grep '\[async\]'. The drain
arms under either tenancy: under multi the dequeue hands it grants instead of a key.
14.2 Submit and collectโ
All three routes are on the control plane and take an api token โ the one a mocactl login
returns โ not a session token:
# 202 {"sessionId":"โฆ","turnId":"t-โฆ"} -- the queue carries the turn id and nothing else.
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"async":true,"kind":"prompt","prompt":"summarise README.md","idempotencyKey":"job-1"}' \
"$CP/v1/sessions/$SID/turns"
# {"turnId","state","attempts"} always; "result" only once state is terminal, because a result
# written by a worker that then died is withheld until the reaper has ruled on it.
curl -s -H "Authorization: Bearer $TOKEN" "$CP/v1/sessions/$SID/turns/$TURN_ID"
# Cancel one that is still queued (409 turn_not_cancellable once a worker has it).
curl -s -X DELETE -H "Authorization: Bearer $TOKEN" "$CP/v1/sessions/$SID/turns/$TURN_ID"
idempotencyKey defaults to the session id, so N async turns on one session collapse into the
first one's turn. Fanning out N leaves means N sessions, or N distinct keys.
GET /runs/status on the harness never reports an async turn, and does not fail either: it reads
the session-wide result key, which an async turn never writes, so it answers queued forever โ or,
if that session also ran a synchronous /v1/runs leaf, returns that leaf's result, which looks
like an answer. Poll the turn route above.
14.3 Watchโ
On the admin port's /metrics, under counters (counts, so missing and zero mean the same thing)
and as two top-level gauges:
| Reading | Means |
|---|---|
async_claims | leaves claimed by this pod's workers since they started |
async_reclaims | of those, the ones redelivered: a worker died holding one, OR it was deferred |
async_dead_letters | of those, the ones that exhausted their attempts |
async_claim_denied_full | claims this pod's workers refused because their admission cap was full |
async_deferred_subject | entries handed back unrun because their owner was at SH_TURNS_PER_SUBJECT |
async_queue_depth | queued turns, from slot 0 only |
async_oldest_age_ms | age of the oldest queued turn, from slot 0 only |
async_deferred_subject rising while slots sit idle is the per-owner cap pacing one submitter -- the
fairness working, not a fault. Rising with no second owner submitting means SH_TURNS_PER_SUBJECT is
too low for this deployment: a deferred entry waits out the 90 s reclaim window before any worker may
take it again, so each deferral costs that turn up to 90 s. That window is also why it inflates
async_reclaims: the entry comes back by the same XAUTOCLAIM a dead worker's entry does, so
subtract async_deferred_subject before reading a reclaim rate as workers dying mid-leaf.
A flat zero is not proof of fairness: the
cap counts sync and async together and a worker runs one queued leaf at a time, so an async-only
backlog never reaches it and is drained FIFO -- one owner's deep queue ahead of another's single
turn. At replicas > 1 that is what MOCA_ASYNC_RUNNING_PER_OWNER is for (#391): it caps an owner's
RUNNING leaves across every pod, counted in Redis, and is the only one of the two caps a pod-local
counter cannot provide. Unset by default; add it to moca-settings to turn it on, and give every pod
the same value. Then roll the supervisor yourself --
kubectl -n moca rollout restart deployment/moca-supervisor. The pods read the key as environment at
start, and their moca.dev/settings-hash annotation covers only the keys setup.sh writes, so a key
added by hand changes no annotation and the running pods keep their old environment indefinitely. Its
defers land in this same counter, so a rise after the restart is the new cap working.
The five counters are one pod's, not the deployment's: /metrics answers from whichever pod you
reached, each worker counts in its own memory, and a worker that respawns starts its own count at
zero. At replicas > 1 (section 4) add the pods up yourself. The two gauges describe the one shared
queue, so every pod's slot 0 reports the same numbers.
Read the last two as a pair. Depth alone cannot tell a deep queue that is draining fast from a
shallow one that is stalled; a rising async_oldest_age_ms at a flat depth is the stall.
async_queue_depth is also the signal to size replicas by โ section 4 says why nothing does that
for you.
14.4 Limitsโ
- Under
multi, inference is isolated and egress is not yet.multiworks: the dequeue returns MI1 ยง6.1 grants and the worker spends the inference one at the control plane's inference listener. It costs three control-plane settings โMOCA_GRANT_SIGNING_KEY,MOCA_INFERENCE_ADDR, andMOCA_INFERENCE_URLwhen the bind is a wildcard โ and the boot refuses without them. These manifests wire none of them: no ConfigMap key and no Service port ship for the listener, so turningmultion here is a manifest change first. The residual: the SANDBOX grant has no redeemer, so an async turn gets inference isolation and not egress isolation. - Two leaves per pod at the default W. The loop is sequential; see section 4 for the width rule and why no autoscaler ships.
- Caps: both of them, enforced by the worker.
turn_expโ anchored at the dequeue and clamped tomin(caps.wallClock, MOCA_TURN_MAX_S)โ is the deadline the worker stops the leaf at, and undermultiit also bounds how long the turn's grants are usable.caps.tokensis enforced inside the run. Either one spent ends the turnfailedwithresult.reason: "budget"(#391).caps.costUsdis refused at submission. - The scheduled on-ramp has no equivalent. Archetype C went with the KEDA path and has no P6 design (#536).
14.5 Smokeโ
smoke.sh claims 13 and 14 cover this path, and skip with a note on a stack whose
moca-settings has the knob off. Claim 13 fans five leaves out over a stack that can run two at
once and requires all five to reach a result; claim 14 deletes the supervisor pod under a running
leaf and requires that leaf to finish on a second delivery (attempts โฅ 2), which is the
redelivery acceptance of #389 on this path. CI brings
its Kind stack up with SH_ASYNC_DRAIN=1, so both run on every k8s-kind-e2e (section 8).
Claim 14 assumes one supervisor pod, as claim 9 does: it picks the pod before submitting the
leaf, and cannot learn which pod's drain then claimed it. Above replicas: 1 claims 9 and 14 skip
with a note rather than delete a pod that may be holding nothing. If you raised replicas
(section 4) and want the redelivery claim back, scale the supervisor to one for the run.