Getting started

Maintain and release

A runbook for the person who owns the stack: the pull-request routine, releasing tins and the shelf CLI, operating Orca and the census, and where things are. Every command was checked against the source on 2026-11-06 (shelf 0.7.0); where something is a known caveat, it says so. For laying out the checkouts and running the tests, see Develop locally.

Day-to-day pull requests

main is protected in every magmalake repository: changes arrive by pull request only, with no direct push and no force push. Cut a branch from a pushed main. A squash-merge folds local commits that were never pushed into your PR under your subject, so check first:

git checkout main && git pull --ff-only
git log origin/main..main          # must print nothing
git checkout -b my-change

Open the PR with gh pr create and watch it with gh pr checks --watch. Each repository's ci.yml sorts a PR before it spends machine time: a docs-only PR (prose and similar files that no test, build or check reads) skips the code jobs, and a release-only PR (only versions, CHANGELOG.md, revs and locks change) takes a short path that skips the nightly leg and the heavy tests. The final ci job accepts "skipped" for those, so a green check there is the one to merge on.

Known flaky tests

A red job is real unless it is one of these three, which fail on a loaded runner:

• Port collisions in the stand-in EC2 tests: "Address already in use" or "stand-in EC2 didn't start".
• tests/fleet.sh failing on "by CPU, highest first", when the machine is busy.
• kind tests that time out inside the vCPU budget.

Rerun only the failed jobs, once:

gh run list -R magmalake/orca.mojo --branch my-change -L 3
gh run rerun <run-id> --failed -R magmalake/orca.mojo

If it passes, merge. If the same job fails twice, or the message is anything else, it is a real failure: read the log (gh run view <run-id> --log-failed) and fix it. Do not rerun a third time.

CHANGELOG.md conflicts

A repository that has a changes/ directory next to its CHANGELOG.md keeps a changelog fragment per PR instead of editing the changelog, so two open PRs never conflict there. Add yours with shelf changes new <slug> (--bump minor for a new feature); it writes changes/<slug>.md, which you fill in with the entry as it should read. The level is the file name's suffix (155-fills.minor.md) or a first line bump: minor: patch, minor, major or breaking; no level is a patch, and a ### Breaking or ### Removed heading makes it breaking. In PR CI, shelf changes check fails a PR that changes code without adding or editing a fragment. PRs that change only prose pass, and so do PRs labelled no-changelog or release.

shelf changes new fills --bump minor     # writes changes/fills.minor.md
shelf changes check                      # what CI runs; compares HEAD with origin/main

Two fragments never conflict, since each PR adds its own file. The repositories that have not moved yet (orca.mojo among them) still use ## Unreleased: every change adds a line there, so two open PRs conflict. Resolve by keeping both entries. If a release landed in between, the heading is now the version (## 0.10.0): put your entry under a fresh ## Unreleased above it, not under the released version. The level of bump shelf release picks comes from these headings, so keep them spelled as the file already does. The design is in mojoshelf #36; it needs shelf 0.8.0 or later.

Orca's database migrations

_migrations() in src/orca/state.mojo is a list, and the schema version is a migration's index plus one, so append only; never insert, edit or reorder. tests/test_state.mojo asserts the latest number (30 on 2026-11-06), so a PR that adds a migration bumps that number in the same commit. When two open PRs each add one, they collide in both files: renumber in merge order, the second PR rebasing onto main, moving its migration after the first's and bumping the asserted number by one. A binary refuses a database whose version is newer than its own ("the database is at schema version N, newer than this binary's M"), so deploy the scheduler and UI together.

Releasing tins with shelf release

Tins pin each other by git revision, so releasing one means releasing its dependents too. shelf release plans that cascade, cuts a release branch in each repository, tests them together, opens the PRs and (with --merge) merges and publishes parents first. It needs gh logged in with push rights.

Before you start

Run it from the directory that holds all the checkouts, with each directory named after its repository (iceberg.mojo, orca.mojo, ...). The release builds from the local checkouts, so a stale one silently leaves merged work out of the release. Pull every involved checkout first:

export TMPDIR=/Volumes/ExtNVMe/tmp
cd /Volumes/ExtNVMe/dev/magmalake
for r in dista.mojo parquet.mojo iceberg.mojo orca.mojo; do git -C $r pull --ff-only; done
shelf --version                                  # 0.8.0 or later

Tin names are the registry's, which differ from the directory names. Check one with curl -s https://mojoshelf.org/api/tins/<name>: it lists each version and its commit_sha. The ones you will use: parquet-mojo (and parquet-full-mojo, released together with it from the same repository), iceberg-mojo, orca-mojo, dista-mojo; the rest are listed on the tins page.

Plan, then apply

plan changes nothing: it prints the tins to release, in order, with their versions and every pin that moves, and for each tin the changelog fragments it will release with their levels. Read it. A tin you did not name appears when it depends on one you did. A tin's bump is the highest of its fragments' levels (patch when there are none); a repository still on ## Unreleased gets it from that section's headings instead. Override per tin:

shelf release plan orca-mojo --bump orca-mojo=minor
shelf release plan --changed                    # every checked-out tin with commits since its release

When the plan is right, apply it with the same arguments:

shelf release apply orca-mojo --bump orca-mojo=minor --wait --merge

That cuts the branches, builds the release heads together (preflight), opens the PRs, polls their required checks (--wait), and merges and publishes each one once its pin on its parent is the parent's current release head (--merge). In each release commit the fragments are written into CHANGELOG.md under ## X.Y.Z - date and deleted. State is kept in .shelf-release.json. Other flags worth knowing:

• --resume: after an interruption (laptop slept, a check failed and was fixed or rerun). It re-pins what sits below a branch that moved and carries on from the step that stopped.
• --publish: when you merged the PRs by hand, publish each merged release.
• --waves: one level of the graph at a time, each published before the next is cut. Slower, easier to follow when the cascade is large.
• --test: run each tin's test task in preflight. --skip-preflight skips the combined build; avoid it.
• --without <tin>: leave a tin out, its dependents keep its published version. --repo name=path: a checkout somewhere else.
• --allow-mismatch (new in 0.7.0): release even when a dependent pins, or checks out in CI, a tin outside the cascade at a different commit than its parents pin it at. A mismatch means two tins would be built against two versions of a third. Use the flag only after you have looked at which tin and why and decided the difference is harmless; otherwise add that tin to the cascade or re-pin it.

The rule: a package dependency's rev must equal the commit the registry published for that version. A rev that was never published builds on your machine and breaks for everyone else.

After a release

curl -s https://mojoshelf.org/api/tins/orca-mojo | head -c 600       # newest version, commit_sha
for r in dista.mojo parquet.mojo iceberg.mojo orca.mojo; do
  echo $r; gh run list -R magmalake/$r --branch main -L 1
done

Each released repository's main must be green (a release merge triggers a full CI run on main). Then re-pin the consumers. mojo-census/pixi.toml pins dista-mojo, parquet-full-mojo, iceberg-mojo, orca-mojo and the rest by rev; set each to the published commit, regenerate the lock, and open a PR as in the first section:

cd mojo-census && git checkout -b repin
$EDITOR pixi.toml            # each rev = commit_sha from the registry
pixi lock

When a release goes wrong

• CI breaks on a sibling after the release. On 2026-11-06 orca's CI checked parquet out at a stale sha, because the workflow pins its sibling repositories separately from pixi.toml. shelf 0.7.0 moves those refs in a release. If CI is red anyway, open .github/workflows/ci.yml and compare every sibling actions/checkout ref: with the registry's commit for that tin; fix it in a PR.
• A PR check fails mid-cascade. Fix or rerun it (see flaky tests), then shelf release apply ... --wait --merge --resume with the same arguments.
• Pins stop being its parent's head. Something merged to a parent while you waited. --resume re-pins; if it still refuses, start again from plan after pulling.
• Merged by hand. shelf release apply <tins> --publish.

Releasing the shelf CLI

The CLI lives in mojoshelf, with its own procedure. Work in a clean checkout.

1. Bump the version in all four places: crates/mojoshelf/Cargo.toml, Cargo.lock (cargo update -p mojoshelf), pixi.toml and recipe.yaml. recipe.yaml was missed once and the package built under the old version.
2. Do not commit a stray pixi.lock that a local build produced.
3. Open a PR titled "Release CLI X.Y.Z", wait for green, merge it.
4. Tag the merge commit and push the tag:
   Copy

   git checkout main && git pull --ff-only
   git tag cli-v0.8.1 && git push origin cli-v0.8.1

5. The release-cli workflow builds the package for three platforms, reindexes the channel under crates/shelf-worker/public/channel and commits it to main (gh run watch). Pull main afterwards.
6. The workflow's "Deploy Worker" step is skipped unless the repository has a CLOUDFLARE_API_TOKEN secret (not set yet: mojoshelf #39). Until it is, deploy by hand from an up-to-date main, which has the channel commit, with wrangler logged in to the account that owns mojoshelf.org. If the release added a D1 migration, apply it first, because wrangler deploy does not (see OPERATIONS.md):
   Copy

   cd crates/shelf-worker
   npx wrangler d1 migrations list mojoshelf --remote     # what is pending
   npx wrangler d1 migrations apply mojoshelf --remote    # only if something is
   npx wrangler deploy

7. Verify that the channel serves it:
   Copy

   pixi global update mojoshelf && shelf --version

Orca and the census in production

Where it runs

The AWS project was paused twice by its spend limit in November 2026, so the census and Orca are moving to one Hetzner Cloud ARM server with a fixed monthly cost (orca #166). The runbook for it is docs/hetzner.md in orca.mojo; it lands with PR #167, so that link is a 404 until that PR merges. Until then the old AWS stack and orca aws still describe what is running. In short, the design is:

• one server (CAX21: 4 ARM vCPU, 8 GB), rescaled CPU and RAM only to a CAX41 for the initial import and any full rebuild, then back;
• the local runner, with Dista threaded workers (DISTA_WORKER_SLOTS), so Orca launches no instances and spend cannot run away;
• the warehouse and logs on Hetzner Object Storage (S3-compatible, through AWS_ENDPOINT_URL_S3), Postgres in a container on a Volume with a nightly pg_dump to Object Storage;
• no open ports but SSH: a Cloudflare Tunnel serves orca.mlake.dev and census.mlake.dev; secrets come from files through ORCA_SECRET_FETCH.

Cutover checklist

From #166; the issue is the source if it has changed since.

1. Prerequisites merged and released: orca #165, iceberg-mojo 0.11.1 (the fix for #164), then orca-mojo 0.10.0; mojo-census re-pinned to those commits (the release steps); the mojoshelf Worker deployed so shelf 0.7.0 installs (the CLI steps).
2. Accounts and credentials, yours to create: a Hetzner Cloud project and read/write API token (for hcloud on the laptop only), an Object Storage bucket with S3 keys, a Cloudflare Tunnel and token with routes for orca.mlake.dev and census.mlake.dev, and a GitHub token for the census's mojo_repos.
3. deploy/hetzner/up.sh on the small server; the two hostnames move to the tunnel.
4. orca user add <name> (it reads the password from stdin), then register the census pipelines from its orca.toml.
5. rescale.sh up, let the fill run (the pipeline page's Fill line shows progress), then rescale.sh down.
6. Check the census console and the row counts per day.
7. Tear down the AWS stack only when AWS support has answered and you have decided what to do with the old S3 data. It may hold duplicate or missing hours (#164), so it is not migrated.

Census settings for the re-import

The census registers from its orca.toml. Until iceberg 0.11.1 is pinned everywhere, the fill is limited so that concurrent commits cannot claim the same snapshot (#164). For each pipeline that fills:

fills = "all"
max_runs = 1
fill_max_runs = 1
# and, in the scheduler's environment (the env block or the env file):
# DISTA_WORKER_SLOTS=<slots per worker>

Check the exact keys and where each sits with orca config show (what is registered and where each value came from) and Orca's orca-toml reference. Change a fill's limits in orca.toml and orca register again; raise them back only once 0.11.1 is pinned. Register by hand with orca register --config orca.toml <pipeline binary> ....

Operating

The UI is at orca.mlake.dev: Ops > Control for the scheduler host, Ops > Fleet for launched instances (empty on the local runner, and its cost rules are off there), and the watchdog page for the rules, their values and active trips. Operations go through the UI or the remote CLI, which keep audit rows. Log in once, then pass --remote:

orca login https://orca.mlake.dev [--user NAME] [--days N] [--label L]
orca --remote https://orca.mlake.dev status
orca --remote https://orca.mlake.dev watchdog status
orca --remote https://orca.mlake.dev watchdog snooze <rule> 2h        # 30m, 1d; "off" ends it
orca --remote https://orca.mlake.dev pause gharchive
orca --remote https://orca.mlake.dev unpause gharchive
orca --remote https://orca.mlake.dev backfill --preview gharchive 2024-01-01..2024-01-31
orca --remote https://orca.mlake.dev backfill gharchive 2024-01-01..2024-01-31
orca --remote https://orca.mlake.dev backfill --rerun gharchive 2024-01-05   # done keys too
orca --remote https://orca.mlake.dev cancel gharchive 2024-01-01..2024-01-31  # or: cancel <run>
orca --remote https://orca.mlake.dev resume <run>                      # rerun <run> <step> starts from a step

Without --remote the same commands act on the database named by ORCA_DATABASE_URL (or set ORCA_REMOTE). backfill prints "queued N runs"; keys skipped, and why, go to stderr. --preview queues nothing. A rerun of a step that appends is refused unless you add --allow-append (it would duplicate rows; see the README's "Reruns that replace a partition"). A mistyped command prints the full usage text. Running docker exec ... orca ... on the server directly is for debugging only.

Caveat: the pipeline name above is an example; list real ones with orca status.

Where things are

The other tins (objectstore, threads, postgres, zstd and so on) are repositories in the same organisation, named like their tin; the registry lists them all. Issues are tracked per repository, in each one's Issues tab. There are no priority labels, so read the open issues list as it is, and the release label where a repository has one. Pull requests are the other queue: gh pr list -R magmalake/orca.mojo.