# Remote workers

> Running some of Orca's pipelines on a second machine: what a remote worker is, a worker token, a placement, the worker container that needs only outbound HTTPS, the census as the example, and what doesn't run on workers yet.

[magmalake](https://magmalake.dev/) [Orca](https://magmalake.dev/s/orca)

A remote worker runs a pipeline's runs on a second machine. It needs no database access and no inbound ports, only outbound HTTPS to the Orca UI and to the bucket.

## What it is

The scheduler stays the only brain: it fires, senses, queues and retries, and keeps all state in its Postgres. A pipeline registered with `placement = "worker:<pool>"` is never started by the scheduler. Its runs wait, queued, until a worker of that pool claims one. The worker polls the UI's HTTPS API (`/api/worker/*`), runs the pipeline's binary on its own machine, and the run records its state through the same API instead of Postgres. Data and logs go from the worker straight to the warehouse and logs buckets.

That is the reason to use one: a second machine with spare CPU, memory or disk without exposing Postgres to it, and the way to scale out when the first machine is full. A worker holds a token scoped to its pool and nothing else. A worker that dies has its runs queued again.

The only path from the worker to the server is HTTPS to the UI (claims, state calls, heartbeats). Postgres is never reachable from the worker.

## Setting one up

Deploy the server first. Then four steps: a token, a placement, the worker, and a look at Ops.

### 1\. A token, on the server

```
orca worker-token create --pool bgent --label bgent   # prints the token, once
orca worker-token list
orca worker-token revoke 3
```

The token is scoped to one pool and expires (by default after a year at most). The server keeps only its hash. Copy it to the worker's machine as a file.

### 2\. A placement, in orca.toml

```
[pipelines.mojo_history]
placement = "worker:bgent"
env = { DISTA_WORKER_SLOTS = "3" }
```

Or `orca register --placement worker:bgent`. Register again after the change. The pipeline's page then says its runs are waiting for a worker of the pool.

### 3\. The worker

The simplest worker is the server's own image at the same tag, so the pipeline binaries are at the paths they were registered with. It needs the pool, the UI's URL, the token file and the bucket's settings:

```
docker run -d --name orca-worker --restart always \
  --env-file s3.env -v orca-worker-secrets:/etc/orca/w:ro \
  -e ORCA_WAREHOUSE=s3://<bucket>/warehouse -e ORCA_LOGS=s3://<bucket>/logs \
  -e AWS_REGION=<location> -e AWS_ENDPOINT_URL_S3=https://<location>.your-objectstorage.com \
  ghcr.io/magmalake/orca:<tag> worker --pool bgent --max-runs 3 \
    --remote https://orca.example.com --token-file /etc/orca/w/token
```

`--max-runs` is how many runs the worker takes at once. A worker with `ORCA_DATABASE_URL` set refuses to start. Since a worker admits runs against its own host, its log says what it can grant: `grants up to 8 CPUs and 11421 MB`, less `ORCA_RESERVE_CPUS` and `ORCA_RESERVE_MEMORY_MB`.

### 4\. Check Ops

Ops, then Workers, (`/api/workers` as JSON) lists each token with its pool, the worker's name and image, its last heartbeat, its current runs, CPUs and memory. A run's page says which worker ran it. `docker logs orca-worker` shows each run it starts and how it exited.

## The census

[The census](https://github.com/magmalake/mojo-census) runs on a Hetzner CAX21: the scheduler, the UI, Postgres, the site, and gharchive's fill, mojo\_repos and mojo\_stats. The git-heavy pipelines, mojo\_history, mojo\_versions and mojo\_activity, run on a second machine, bgent, an Apple-silicon Mac running Docker. It is arm64 like the CAX21, so it runs the same image. Each of the three is registered like this:

```
[pipelines.mojo_history]
placement = "worker:bgent"
env = { DISTA_WORKER_SLOTS = "3" }
```

The worker is the container above, with the census's bucket and one more setting, `MOJO_HISTORY_WORKDIR=/work/clones` on a volume, for the clones. The [census README](https://github.com/magmalake/mojo-census#a-second-machine) has the exact commands.

## Limits

- **Fills.** `fills = "all"` doesn't run on worker pools yet ([orca #177](https://github.com/magmalake/orca.mojo/issues/177)). Use `orca backfill` for a pool's partitions.
- **The same tag.** The worker runs the code of its own image. Deploy the server first, then restart the worker on the same tag.
- **Secrets.** A pipeline with secrets needs `ORCA_SECRET_FETCH` on the worker, to read them there. The spec names the secrets and never carries their values.
- **The UI's port.** By default the worker API shares the UI's port, so a slow page can delay a worker. From orca 0.15.0, `ORCA_WORKER_ADDR` serves it on a listener of its own.

## Docker Desktop and the token

Docker Desktop runs containers in a virtual machine, which may not see a file bind-mounted from under `~/.config`: the mount shows up as an empty directory, and the worker finds no token. Put the token in a Docker volume instead:

```
docker run --rm -i --entrypoint sh -v orca-worker-secrets:/s \
  ghcr.io/magmalake/orca:<tag> -c 'cat > /s/token && chmod 600 /s/token' < worker-token
```

The worker container above mounts that volume read-only at `/etc/orca/w`.

## More

[workers.md](https://github.com/magmalake/orca.mojo/blob/main/docs/workers.md) in the Orca repository has the flags, how a dead worker's runs are retried, the state calls and the network details. [Scheduling and running](https://magmalake.dev/s/orca/docs/run#placement) says where placement fits among the runners.

[A magmalake project](https://magmalake.dev/)

Apache 2.0 · © 2026 Marius Seritan