# Maintain and release > Run the magmalake stack without an assistant: the pull-request routine and the known flaky tests, releasing tins with shelf release, releasing the shelf CLI, operating Orca and the census in production, and where everything lives. A runbook for the person who owns the stack: the pull-request routine, releasing tins and the `shelf` CLI, operating Orca and the census, and where things are. Every command was checked against the source on 2026-11-06 (shelf 0.7.0); where something is a known caveat, it says so. For laying out the checkouts and running the tests, see [Develop locally](https://magmalake.dev/getting-started/develop). ## Day-to-day pull requests `main` is protected in every magmalake repository: changes arrive by pull request only, with no direct push and no force push. Cut a branch from a _pushed_ main. A squash-merge folds local commits that were never pushed into your PR under your subject, so check first: ``` git checkout main && git pull --ff-only git log origin/main..main # must print nothing git checkout -b my-change ``` Open the PR with `gh pr create` and watch it with `gh pr checks --watch`. Each repository's `ci.yml` sorts a PR before it spends machine time: a docs-only PR (prose and similar files that no test, build or check reads) skips the code jobs, and a release-only PR (only versions, `CHANGELOG.md`, revs and locks change) takes a short path that skips the nightly leg and the heavy tests. The final `ci` job accepts "skipped" for those, so a green check there is the one to merge on. ### Known flaky tests A red job is real unless it is one of these three, which fail on a loaded runner: - Port collisions in the stand-in EC2 tests: "Address already in use" or "stand-in EC2 didn't start". - `tests/fleet.sh` failing on "by CPU, highest first", when the machine is busy. - `kind` tests that time out inside the vCPU budget. Rerun only the failed jobs, once: ``` gh run list -R magmalake/orca.mojo --branch my-change -L 3 gh run rerun --failed -R magmalake/orca.mojo ``` If it passes, merge. If the same job fails twice, or the message is anything else, it is a real failure: read the log (`gh run view --log-failed`) and fix it. Do not rerun a third time. ### CHANGELOG.md conflicts A repository that has a `changes/` directory next to its CHANGELOG.md keeps a changelog _fragment_ per PR instead of editing the changelog, so two open PRs never conflict there. Add yours with `shelf changes new ` (`--bump minor` for a new feature); it writes `changes/.md`, which you fill in with the entry as it should read. The level is the file name's suffix (`155-fills.minor.md`) or a first line `bump: minor`: `patch`, `minor`, `major` or `breaking`; no level is a patch, and a `### Breaking` or `### Removed` heading makes it breaking. In PR CI, `shelf changes check` fails a PR that changes code without adding or editing a fragment. PRs that change only prose pass, and so do PRs labelled `no-changelog` or `release`. ``` shelf changes new fills --bump minor # writes changes/fills.minor.md shelf changes check # what CI runs; compares HEAD with origin/main ``` Two fragments never conflict, since each PR adds its own file. The repositories that have not moved yet (orca.mojo among them) still use `## Unreleased`: every change adds a line there, so two open PRs conflict. Resolve by keeping both entries. If a release landed in between, the heading is now the version (`## 0.10.0`): put your entry under a fresh `## Unreleased` above it, not under the released version. The level of bump `shelf release` picks comes from these headings, so keep them spelled as the file already does. The design is in [mojoshelf #36](https://github.com/mojoshelf/mojoshelf/issues/36); it needs `shelf` 0.8.0 or later. ### Orca's database migrations `_migrations()` in `src/orca/state.mojo` is a list, and the schema version is a migration's _index plus one_, so append only; never insert, edit or reorder. `tests/test_state.mojo` asserts the latest number (30 on 2026-11-06), so a PR that adds a migration bumps that number in the same commit. When two open PRs each add one, they collide in both files: renumber in merge order, the second PR rebasing onto main, moving its migration after the first's and bumping the asserted number by one. A binary refuses a database whose version is newer than its own ("the database is at schema version N, newer than this binary's M"), so deploy the scheduler and UI together. ## Releasing tins with shelf release Tins pin each other by git revision, so releasing one means releasing its dependents too. `shelf release` plans that cascade, cuts a release branch in each repository, tests them together, opens the PRs and (with `--merge`) merges and publishes parents first. It needs `gh` logged in with push rights. ### Before you start Run it from the directory that holds all the checkouts, with each directory named after its repository (`iceberg.mojo`, `orca.mojo`, ...). The release builds from the **local** checkouts, so a stale one silently leaves merged work out of the release. Pull every involved checkout first: ``` export TMPDIR=/Volumes/ExtNVMe/tmp cd /Volumes/ExtNVMe/dev/magmalake for r in dista.mojo parquet.mojo iceberg.mojo orca.mojo; do git -C $r pull --ff-only; done shelf --version # 0.8.0 or later ``` Tin names are the registry's, which differ from the directory names. Check one with `curl -s https://mojoshelf.org/api/tins/`: it lists each version and its `commit_sha`. The ones you will use: `parquet-mojo` (and `parquet-full-mojo`, released together with it from the same repository), `iceberg-mojo`, `orca-mojo`, `dista-mojo`; the rest are listed on [the tins page](https://magmalake.dev/tins). ### Plan, then apply `plan` changes nothing: it prints the tins to release, in order, with their versions and every pin that moves, and for each tin the changelog fragments it will release with their levels. Read it. A tin you did not name appears when it depends on one you did. A tin's bump is the highest of its fragments' levels (patch when there are none); a repository still on `## Unreleased` gets it from that section's headings instead. Override per tin: ``` shelf release plan orca-mojo --bump orca-mojo=minor shelf release plan --changed # every checked-out tin with commits since its release ``` When the plan is right, apply it with the same arguments: ``` shelf release apply orca-mojo --bump orca-mojo=minor --wait --merge ``` That cuts the branches, builds the release heads together (preflight), opens the PRs, polls their required checks (`--wait`), and merges and publishes each one once its pin on its parent is the parent's current release head (`--merge`). In each release commit the fragments are written into CHANGELOG.md under `## X.Y.Z - date` and deleted. State is kept in `.shelf-release.json`. Other flags worth knowing: - `--resume`: after an interruption (laptop slept, a check failed and was fixed or rerun). It re-pins what sits below a branch that moved and carries on from the step that stopped. - `--publish`: when you merged the PRs by hand, publish each merged release. - `--waves`: one level of the graph at a time, each published before the next is cut. Slower, easier to follow when the cascade is large. - `--test`: run each tin's `test` task in preflight. `--skip-preflight` skips the combined build; avoid it. - `--without `: leave a tin out, its dependents keep its published version. `--repo name=path`: a checkout somewhere else. - `--allow-mismatch` (new in 0.7.0): release even when a dependent pins, or checks out in CI, a tin _outside_ the cascade at a different commit than its parents pin it at. A mismatch means two tins would be built against two versions of a third. Use the flag only after you have looked at which tin and why and decided the difference is harmless; otherwise add that tin to the cascade or re-pin it. **The rule:** a package dependency's `rev` must equal the commit the registry published for that version. A rev that was never published builds on your machine and breaks for everyone else. ### After a release ``` curl -s https://mojoshelf.org/api/tins/orca-mojo | head -c 600 # newest version, commit_sha for r in dista.mojo parquet.mojo iceberg.mojo orca.mojo; do echo $r; gh run list -R magmalake/$r --branch main -L 1 done ``` Each released repository's main must be green (a release merge triggers a full CI run on main). Then re-pin the consumers. `mojo-census/pixi.toml` pins `dista-mojo`, `parquet-full-mojo`, `iceberg-mojo`, `orca-mojo` and the rest by `rev`; set each to the published commit, regenerate the lock, and open a PR as in the first section: ``` cd mojo-census && git checkout -b repin $EDITOR pixi.toml # each rev = commit_sha from the registry pixi lock ``` ### When a release goes wrong - **CI breaks on a sibling after the release.** On 2026-11-06 orca's CI checked parquet out at a stale sha, because the workflow pins its sibling repositories separately from `pixi.toml`. shelf 0.7.0 moves those refs in a release. If CI is red anyway, open `.github/workflows/ci.yml` and compare every sibling `actions/checkout` `ref:` with the registry's commit for that tin; fix it in a PR. - **A PR check fails mid-cascade.** Fix or rerun it (see flaky tests), then `shelf release apply ... --wait --merge --resume` with the same arguments. - **Pins stop being its parent's head.** Something merged to a parent while you waited. `--resume` re-pins; if it still refuses, start again from `plan` after pulling. - **Merged by hand.** `shelf release apply --publish`. ## Releasing the shelf CLI The CLI lives in [mojoshelf](https://github.com/mojoshelf/mojoshelf), with its own procedure. Work in a clean checkout. 1. Bump the version in **all four** places: `crates/mojoshelf/Cargo.toml`, `Cargo.lock` (`cargo update -p mojoshelf`), `pixi.toml` and `recipe.yaml`. recipe.yaml was missed once and the package built under the old version. 2. Do not commit a stray `pixi.lock` that a local build produced. 3. Open a PR titled "Release CLI X.Y.Z", wait for green, merge it. 4. Tag the merge commit and push the tag: ``` git checkout main && git pull --ff-only git tag cli-v0.8.1 && git push origin cli-v0.8.1 ``` 5. The `release-cli` workflow builds the package for three platforms, reindexes the channel under `crates/shelf-worker/public/channel` and commits it to main (`gh run watch`). Pull main afterwards. 6. The workflow's "Deploy Worker" step is **skipped** unless the repository has a `CLOUDFLARE_API_TOKEN` secret (not set yet: [mojoshelf #39](https://github.com/mojoshelf/mojoshelf/issues/39)). Until it is, deploy by hand from an up-to-date main, which has the channel commit, with `wrangler` logged in to the account that owns mojoshelf.org. If the release added a D1 migration, apply it first, because `wrangler deploy` does not (see `OPERATIONS.md`): ``` cd crates/shelf-worker npx wrangler d1 migrations list mojoshelf --remote # what is pending npx wrangler d1 migrations apply mojoshelf --remote # only if something is npx wrangler deploy ``` 7. Verify that the channel serves it: ``` pixi global update mojoshelf && shelf --version ``` ## Orca and the census in production ### Where it runs The AWS project was paused twice by its spend limit in November 2026, so the census and Orca are moving to one Hetzner Cloud ARM server with a fixed monthly cost ([orca #166](https://github.com/magmalake/orca.mojo/issues/166)). The runbook for it is [docs/hetzner.md](https://github.com/magmalake/orca.mojo/blob/main/docs/hetzner.md) in orca.mojo; it lands with [PR #167](https://github.com/magmalake/orca.mojo/pull/167), so that link is a 404 until that PR merges. Until then the old AWS stack and [orca aws](https://magmalake.dev/getting-started/step-by-step#aws) still describe what is running. In short, the design is: - one server (CAX21: 4 ARM vCPU, 8 GB), rescaled CPU and RAM only to a CAX41 for the initial import and any full rebuild, then back; - the `local` runner, with Dista threaded workers (`DISTA_WORKER_SLOTS`), so Orca launches no instances and spend cannot run away; - the warehouse and logs on Hetzner Object Storage (S3-compatible, through `AWS_ENDPOINT_URL_S3`), Postgres in a container on a Volume with a nightly `pg_dump` to Object Storage; - no open ports but SSH: a Cloudflare Tunnel serves orca.mlake.dev and census.mlake.dev; secrets come from files through `ORCA_SECRET_FETCH`. ### Cutover checklist From #166; the issue is the source if it has changed since. 1. Prerequisites merged and released: orca #165, iceberg-mojo 0.11.1 (the fix for [#164](https://github.com/magmalake/orca.mojo/issues/164)), then orca-mojo 0.10.0; mojo-census re-pinned to those commits (the release steps); the mojoshelf Worker deployed so shelf 0.7.0 installs (the CLI steps). 2. Accounts and credentials, yours to create: a Hetzner Cloud project and read/write API token (for `hcloud` on the laptop only), an Object Storage bucket with S3 keys, a Cloudflare Tunnel and token with routes for orca.mlake.dev and census.mlake.dev, and a GitHub token for the census's mojo\_repos. 3. `deploy/hetzner/up.sh` on the small server; the two hostnames move to the tunnel. 4. `orca user add ` (it reads the password from stdin), then register the census pipelines from its `orca.toml`. 5. `rescale.sh up`, let the fill run (the pipeline page's Fill line shows progress), then `rescale.sh down`. 6. Check the census console and the row counts per day. 7. Tear down the AWS stack only when AWS support has answered and you have decided what to do with the old S3 data. It may hold duplicate or missing hours (#164), so it is not migrated. ### Census settings for the re-import The census registers from its `orca.toml`. Until iceberg 0.11.1 is pinned everywhere, the fill is limited so that concurrent commits cannot claim the same snapshot (#164). For each pipeline that fills: ``` fills = "all" max_runs = 1 fill_max_runs = 1 # and, in the scheduler's environment (the env block or the env file): # DISTA_WORKER_SLOTS= ``` Check the exact keys and where each sits with `orca config show` (what is registered and where each value came from) and Orca's [orca-toml reference](https://github.com/magmalake/orca.mojo/blob/main/docs/orca-toml.md). Change a fill's limits in `orca.toml` and `orca register` again; raise them back only once 0.11.1 is pinned. Register by hand with `orca register --config orca.toml ...`. ### Operating The UI is at [orca.mlake.dev](https://orca.mlake.dev): **Ops > Control** for the scheduler host, **Ops > Fleet** for launched instances (empty on the `local` runner, and its cost rules are off there), and the **watchdog** page for the rules, their values and active trips. Operations go through the UI or the remote CLI, which keep audit rows. Log in once, then pass `--remote`: ``` orca login https://orca.mlake.dev [--user NAME] [--days N] [--label L] orca --remote https://orca.mlake.dev status orca --remote https://orca.mlake.dev watchdog status orca --remote https://orca.mlake.dev watchdog snooze 2h # 30m, 1d; "off" ends it orca --remote https://orca.mlake.dev pause gharchive orca --remote https://orca.mlake.dev unpause gharchive orca --remote https://orca.mlake.dev backfill --preview gharchive 2024-01-01..2024-01-31 orca --remote https://orca.mlake.dev backfill gharchive 2024-01-01..2024-01-31 orca --remote https://orca.mlake.dev backfill --rerun gharchive 2024-01-05 # done keys too orca --remote https://orca.mlake.dev cancel gharchive 2024-01-01..2024-01-31 # or: cancel orca --remote https://orca.mlake.dev resume # rerun starts from a step ``` Without `--remote` the same commands act on the database named by `ORCA_DATABASE_URL` (or set `ORCA_REMOTE`). `backfill` prints "queued N runs"; keys skipped, and why, go to stderr. `--preview` queues nothing. A rerun of a step that appends is refused unless you add `--allow-append` (it would duplicate rows; see the README's "Reruns that replace a partition"). A mistyped command prints the full usage text. Running `docker exec ... orca ...` on the server directly is for debugging only. Caveat: the pipeline name above is an example; list real ones with `orca status`. ## Where things are | Repository | What it is | | --- | --- | | [magmalake/orca.mojo](https://github.com/magmalake/orca.mojo) | The orchestrator: scheduler, UI, CLI, orca aws. Tin orca-mojo. Database migrations live here. | | [magmalake/mojo-census](https://github.com/magmalake/mojo-census) | The example and production workload: three pipelines and the console. Pins every library by commit in pixi.toml; not a tin. | | [magmalake/iceberg.mojo](https://github.com/magmalake/iceberg.mojo) | Iceberg tables, reads and writes. Tin iceberg-mojo. | | [magmalake/parquet.mojo](https://github.com/magmalake/parquet.mojo) | Parquet reader and writer; full/ adds the compression codecs. Tins parquet-mojo, parquet-full-mojo. | | [magmalake/dista.mojo](https://github.com/magmalake/dista.mojo) | Distributed tasks and actors. Tin dista-mojo. | | [mojoshelf/mojoshelf](https://github.com/mojoshelf/mojoshelf) | The registry (a Cloudflare Worker plus D1), the shelf CLI and the conda channel. OPERATIONS.md has the Worker's secrets, deploy and database steps. It is in the _mojoshelf_ organisation, not magmalake. | | [magmalake/magmalake.org](https://github.com/magmalake/magmalake.org) | This site. | The other tins (objectstore, threads, postgres, zstd and so on) are repositories in the same organisation, named like their tin; the registry lists them all. Issues are tracked per repository, in each one's Issues tab. There are no priority labels, so read the open issues list as it is, and the `release` label where a repository has one. Pull requests are the other queue: `gh pr list -R magmalake/orca.mojo`.