---
title: Running N replicas
---

Every Loom process is a full replica. Run as many as you need on one
ClickHouse service behind one URL, with no sticky sessions: each one serves
every op, verifies every token and writes to the shared service. The work that
must happen once runs on whichever process holds the ClickHouse lease, and
moves when that process stops:

| Once per deployment                       | Where it runs                                                                                              |
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Schema migrations                         | the first process to boot a build with pending migrations, under the migrate lease; the others wait for it |
| Background workers                        | the holder of the `workers_supervisor` lease ([deployment](/operations/deployment))                                 |
| Publishing the deployment's configuration | the same lease holder (below)                                                                              |

Each process keeps its own git mirrors (a history read clones a missing one;
`LOOM_CODE_HISTORY_CLONE_ON_READ`), its own caches, and its own rate-limit
buckets.

## One env for all of them

Start every replica from the same env, with `LOOM_DEPLOYMENT=1`:

```bash
LOOM_DEPLOYMENT=1
LOOM_CH_HOST=...                 # the one ClickHouse service
LOOM_CH_PORT=8443
LOOM_CH_USER=...
LOOM_CH_PASSWORD=...
LOOM_JWT_SECRET=...              # the same root secret on every replica
LOOM_EMBEDDING_MODEL=...         # the same model and width on every replica
LOOM_EMBEDDING_DIM=...
OPENAI_API_KEY=...               # each replica makes its own provider calls
LOOM_TRUSTED_PROXIES=...         # the load balancer, so per-client limits see the client
```

`LOOM_JWT_SECRET` signs as kid `default` and seals every other signing key and
every stored credential, so with one value every token works on every replica.
Signing keys, revocations, grants and service accounts are rows in the shared
service; a rotation or a revoke reaches every replica within
`LOOM_JWT_KEYRING_REFRESH_SECONDS` and `LOOM_JWT_REVOCATION_CACHE_TTL_SECONDS`.

## Configuration drift

Each process writes a heartbeat row to `loom.instances` every
`LOOM_INSTANCE_HEARTBEAT_SECONDS` (10 s by default): its build, a fingerprint
of its MUST_MATCH settings (the embedding space: `LOOM_EMBEDDING_PROVIDER`, `_MODEL`,
`_DIM`, `_BASE_URL` and `_SEND_DIMENSIONS`), a
fingerprint of its root secret and its start time. A row not renewed for three
intervals, or one its process marked stopped at shutdown, is not live.

The lease holder publishes its configuration to `loom.deployment_config`: the
MUST_MATCH settings and an HMAC fingerprint of its root secret. Every process
compares itself with that snapshot at boot and on every heartbeat. A different root
secret (its tokens and stored credentials would not open elsewhere) or a
different MUST_MATCH value (its vectors would be in another space) is drift:
the process is not ready, so the load balancer sends it nothing, and
`/v1/health` reports the `config_drift` problem with what differs. Not-ready
only stops new connections, so the process also answers every API, MCP and
sign-in call with `503 CONFIG_DRIFT` and closes the connection: a client's
retry lands on a replica that agrees. The values
compared are the env's: a model switch from the dashboard is the deployment's
(every replica reads it from ClickHouse), so it is no drift, and a replica
started with the same env after it is ready.
`/v1/health` `deployment_config` shows the last check and the live hosts.

The lease holder never publishes over a snapshot it drifts from: a replica
with the wrong root secret or embedding model that wins the lease stays not
ready, and the replicas on the published env stay ready. A snapshot it agrees
with is rewritten only once no other live replica runs the stored one, so the
differently shaped snapshots of N-1 and N during a rollout never flip back and
forth.

**Changing a MUST_MATCH setting or `LOOM_JWT_SECRET`** cannot roll: replicas on
the new value are not ready while replicas on the old one serve. Stop every
replica, change the env, run `loom deployment publish --yes` with the new env
(it lists what changes), then start them again (a full re-embed follows an
embedding change, `loom reembed`). After a secret change re-mint tokens and
store the credentials again.

Processes without `LOOM_DEPLOYMENT` (a branch or A/B server on the same
service) heartbeat too, but never publish, are never checked and do not count
as replicas.

## Rate limits

The configured caps are deployment-wide. Each replica's share is the cap
divided by the hosts with a live `loom.instances` row, so adding or removing a
replica resizes every bucket within a heartbeat. `LOOM_REPLICA_COUNT` is the
floor of that count, and the count while the rows cannot be read. Web workers of
one host share one bucket table ([deployment](/operations/deployment)).

## Code repositories

Every process re-reads `loom.code_repos` every `LOOM_CODE_REPOS_REFRESH_SECONDS`
(30 s by default), so a repository added or restricted on one replica applies on
every replica within that time.

A host that should serve no git history (an edge with no disk for mirrors) sets
`LOOM_GIT_HISTORY=none`: blame, commit checks and PR history then answer without
it.

## Stopping a replica

Stop it gracefully (`docker stop`, SIGTERM). It first closes admission permanently,
stops its producers, and waits for actual request, worker and registered descendant
completion. It retires only after buffered, carried and server-side writes settle.
A canceled task that keeps executing or an exhausted shutdown budget leaves the
participant nonretired. A stopped or expired instance heartbeat is diagnostic.

A trusted process on the same host and pid namespace can prove the exact old PID
exited or was reused, then retire it after all cross-replica queries settle:

```bash
loom participant retire --participant <participant_id>
```

The command verifies retained boot, PID namespace and process-start identity; it
refuses remote or unknown identity evidence and live/uncountable queries. A
changed container namespace or Kubernetes API object is insufficient. An operator
assertion does not override these checks. For an inaccessible host, positively
terminate or fence the exact runtime before the documented
[quiescent lifecycle recovery](/architecture/cells#participant-lifecycle-baseline).
