magmalake Orca

Docs

Remote workers

A remote worker runs a pipeline's runs on a second machine. It needs no database access and no inbound ports, only outbound HTTPS to the Orca UI and to the bucket.

What it is

The scheduler stays the only brain: it fires, senses, queues and retries, and keeps all state in its Postgres. A pipeline registered with placement = "worker:<pool>" is never started by the scheduler. Its runs wait, queued, until a worker of that pool claims one. The worker polls the UI's HTTPS API (/api/worker/*), runs the pipeline's binary on its own machine, and the run records its state through the same API instead of Postgres. Data and logs go from the worker straight to the warehouse and logs buckets.

That is the reason to use one: a second machine with spare CPU, memory or disk without exposing Postgres to it, and the way to scale out when the first machine is full. A worker holds a token scoped to its pool and nothing else. A worker that dies has its runs queued again.

A remote worker and the server The worker polls the UI over HTTPS to claim runs and record their state; the UI and scheduler use Postgres. Server and worker both read and write the bucket directly. server scheduler orca ui Postgres worker orca worker --pool bgent runs the pipeline's binary HTTPS bucket: warehouse and logs
The only path from the worker to the server is HTTPS to the UI (claims, state calls, heartbeats). Postgres is never reachable from the worker.

Setting one up

Deploy the server first. Then four steps: a token, a placement, the worker, and a look at Ops.

1. A token, on the server

orca worker-token create --pool bgent --label bgent   # prints the token, once
orca worker-token list
orca worker-token revoke 3

The token is scoped to one pool and expires (by default after a year at most). The server keeps only its hash. Copy it to the worker's machine as a file.

2. A placement, in orca.toml

[pipelines.mojo_history]
placement = "worker:bgent"
env = { DISTA_WORKER_SLOTS = "3" }

Or orca register --placement worker:bgent. Register again after the change. The pipeline's page then says its runs are waiting for a worker of the pool.

3. The worker

The simplest worker is the server's own image at the same tag, so the pipeline binaries are at the paths they were registered with. It needs the pool, the UI's URL, the token file and the bucket's settings: 

docker run -d --name orca-worker --restart always \
  --env-file s3.env -v orca-worker-secrets:/etc/orca/w:ro \
  -e ORCA_WAREHOUSE=s3://<bucket>/warehouse -e ORCA_LOGS=s3://<bucket>/logs \
  -e AWS_REGION=<location> -e AWS_ENDPOINT_URL_S3=https://<location>.your-objectstorage.com \
  ghcr.io/magmalake/orca:<tag> worker --pool bgent --max-runs 3 \
    --remote https://orca.example.com --token-file /etc/orca/w/token

--max-runs is how many runs the worker takes at once. A worker with ORCA_DATABASE_URL set refuses to start. Since a worker admits runs against its own host, its log says what it can grant: grants up to 8 CPUs and 11421 MB, less ORCA_RESERVE_CPUS and ORCA_RESERVE_MEMORY_MB.

4. Check Ops

Ops, then Workers, (/api/workers as JSON) lists each token with its pool, the worker's name and image, its last heartbeat, its current runs, CPUs and memory. A run's page says which worker ran it. docker logs orca-worker shows each run it starts and how it exited.

The census

The census runs on a Hetzner CAX21: the scheduler, the UI, Postgres, the site, and gharchive's fill, mojo_repos and mojo_stats. The git-heavy pipelines, mojo_history, mojo_versions and mojo_activity, run on a second machine, bgent, an Apple-silicon Mac running Docker. It is arm64 like the CAX21, so it runs the same image. Each of the three is registered like this: 

[pipelines.mojo_history]
placement = "worker:bgent"
env = { DISTA_WORKER_SLOTS = "3" }

The worker is the container above, with the census's bucket and one more setting, MOJO_HISTORY_WORKDIR=/work/clones on a volume, for the clones. The census README has the exact commands.

Limits

  • Fills. fills = "all" doesn't run on worker pools yet (orca #177). Use orca backfill for a pool's partitions.
  • The same tag. The worker runs the code of its own image. Deploy the server first, then restart the worker on the same tag.
  • Secrets. A pipeline with secrets needs ORCA_SECRET_FETCH on the worker, to read them there. The spec names the secrets and never carries their values.
  • The UI's port. By default the worker API shares the UI's port, so a slow page can delay a worker. From orca 0.15.0, ORCA_WORKER_ADDR serves it on a listener of its own.

Docker Desktop and the token

Docker Desktop runs containers in a virtual machine, which may not see a file bind-mounted from under ~/.config: the mount shows up as an empty directory, and the worker finds no token. Put the token in a Docker volume instead: 

docker run --rm -i --entrypoint sh -v orca-worker-secrets:/s \
  ghcr.io/magmalake/orca:<tag> -c 'cat > /s/token && chmod 600 /s/token' < worker-token

The worker container above mounts that volume read-only at /etc/orca/w.

More

workers.md in the Orca repository has the flags, how a dead worker's runs are retried, the state calls and the network details. Scheduling and running says where placement fits among the runners.