# Develop locally > Change Orca, Dista, Iceberg, Parquet or the census and run it on a laptop before any release or deploy: the checkouts to lay out side by side, Orca's examples and test suites, the census on a local warehouse, testing a library change inside the census, and the caveats that bite locally. For changing [Orca](https://github.com/magmalake/orca.mojo), Dista, Iceberg, Parquet or the [census](https://github.com/magmalake/mojo-census) and running the change on a laptop, macOS on Apple silicon or Linux, before anything is released or deployed. Nothing on this page touches AWS. Every command was run on a Mac at the revisions CI pinned on 2026-11-06; timings are from that run and will differ on yours. ## Lay out the checkouts You need [pixi](https://pixi.sh) and [mojoshelf's CLI](https://mojoshelf.org/getting-started), which gives `pixi shelf` and `shelf release`. Point `TMPDIR` at a disk with room: builds, and the throwaway Postgres and MinIO servers the tests start, write there. ``` curl -fsSL https://pixi.sh/install.sh | sh pixi global install --channel https://mojoshelf.org/channel mojoshelf export TMPDIR=/Volumes/Big/tmp # somewhere with tens of GB; the default is fine on a roomy disk ``` Clone the repositories side by side in one directory. Orca's tests put its siblings on the include path as `-I ../dista.mojo/src` and so on (`DEP_INC` in its `pixi.toml`), so the directory names matter. Which siblings an environment needs: | Environment | Siblings, next to orca.mojo | | --- | --- | | default (stable, Mojo 1.2.0): examples, test | dista.mojo, k8s.mojo | | data: Iceberg tables, lineage, sensors, MinIO | those two, and memory-region.mojo, arrow-mlake.mojo, parquet.mojo (with its full/), iceberg.mojo, threads.mojo, thrift.mojo, hashes.mojo, snappy.mojo, deflate.mojo, avro.mojo, roaring.mojo, objectstore.mojo, zstd.mojo, lz4.mojo, brotli.mojo, sqlite.mojo, postgres.mojo | | nightly: the same tests on the nightly compiler | dista, k8s, memory-region, arrow-mlake, objectstore, postgres, threads, sqlite, and flare\_mlake and json\_mlake | CI checks each out at a pinned commit, and the tests mean what CI means only at the same ones. The `data` job of orca's [.github/workflows/ci.yml](https://github.com/magmalake/orca.mojo/blob/main/.github/workflows/ci.yml) lists every sibling that environment needs with its commit, so read the list from the file instead of copying it from here, where it would go stale. In 16 seconds: ``` mkdir lake && cd lake git clone https://github.com/magmalake/orca.mojo sed -n '/^ data:/,/^ ci:/p' orca.mojo/.github/workflows/ci.yml \ | grep -o 'repository: [^,]*, ref: [0-9a-f]*, path: [^}]*' \ | while IFS=', ' read -r _ repo _ ref _ dir; do git clone -q --no-checkout "https://github.com/$repo" "$dir" && git -C "$dir" checkout -q "$ref" done git clone https://github.com/magmalake/mojo-census ``` The loop calls its directory `dir` because zsh ties `path` to `PATH`: `read ... path` empties the command search path, and every later command is "not found". For the nightly environment, also clone flare\_mlake and json\_mlake at the commits of the nightly job's steps (`grep -A3 'magmalake/flare_mlake' orca.mojo/.github/workflows/ci.yml`). The census needs none of this: it takes Orca and every library as pixi packages, pinned by commit in its `pixi.toml`, and builds on its own (until you test a library change in it). Some of Orca's tasks are defined in more than one environment, which makes a task that depends on one of them ambiguous: `pixi run test` stops with "the task 'build-shared' is ambiguous". Name the environment, as CI does: `pixi run -e default test`. Two things about the size. A checkout of a tin that the environment also installs as a package is shadowed by the package: in the `data` environment an edit to iceberg.mojo's source is compiled in, and one to zstd.mojo's, lz4's, brotli's, sqlite's, postgres's or objectstore's is silently ignored. Test a change to those with a pin, as below. And pixi-build leaves its build trees in `.pixi/bld`: after the first `pixi run`, 5.7 GB in Orca's default environment, 9.5 GB once `data` exists, and 18 GB in the census, beside 14 GB in pixi's cache (`~/Library/Caches/rattler` on macOS). The environments themselves are about 1 GB each. ## Orca on its own ### The examples The first `pixi run` installs the environment: a minute. Each example then builds in 10 to 15 seconds, and the `orca` command in 45\. A pipeline runs as a plain process and prints its run, with no database or scheduler: ``` cd orca.mojo pixi run toy && ./build/toy run --partitions 2024-01..2024-03 pixi run fanout && ./build/fanout run --partitions 2024-01 # steps calling Dista tasks pixi run daily && ./build/daily run --partitions 2024-02-27..2024-03-02 ``` With no `ORCA_DATABASE_URL` a run is kept in memory. Give it a SQLite file and the run is recorded and resumable, with no server: ``` export ORCA_DATABASE_URL=sqlite:build/runs.db ./build/toy run --partitions 2024-01..2024-03 ./build/toy status unset ORCA_DATABASE_URL ``` For Postgres, `scripts/with-postgres.sh` starts a throwaway server on a Unix socket under `$TMPDIR`, exports `ORCA_DATABASE_URL` for the command you give it, and removes the server when the command ends. The server comes from conda-forge's `postgresql`, so run it through pixi: ``` pixi run scripts/with-postgres.sh sh -c './build/toy run --partitions 2024-01..2024-03 && ./build/toy status' ``` ### The playground To look at the UI before running anything real, the playground builds the examples, starts a throwaway Postgres, makes runs that succeed and fail, and runs two fanout runs against a stand-in EC2 that is a local Python process. It is ready in about a minute and a half. Log in to the UI's actions as `demo` / `demo-password`. Ctrl-C ends it and removes everything: ``` ORCA_UI_ADDR=127.0.0.1:8275 pixi run playground # http://127.0.0.1:8275 ``` ### The scheduler and the UI The scheduler fires schedules, backfills and sensors, and `orca ui` serves the runs. Both need Postgres. This one lasts between sessions, in `build/pg` with its socket in `build/`: ``` pixi run orca pixi run initdb -D build/pg --auth=trust --username=postgres -E UTF8 --locale=C pixi run pg_ctl -D build/pg -l build/pg.log -o "-k $PWD/build -c listen_addresses=" -w start pixi run createdb -h "$PWD/build" -U postgres orca export ORCA_DATABASE_URL="postgresql:///orca?host=$PWD/build&user=postgres" export ORCA_UI_ADDR=127.0.0.1:8275 ./build/orca register ./build/toy ./build/daily ./build/orca scheduler > build/scheduler.out 2>&1 & echo $! > build/scheduler.pid ./build/orca ui > build/ui.out 2>&1 & echo $! > build/ui.pid ./build/orca backfill daily 2024-02-27..2024-03-02 ./build/orca status daily # five runs queued, done in a few seconds ``` The watchdog is on by default. `orca watchdog status` prints each rule with its value and limit (launches an hour, cost, idle CPU, failures, stuck runs, the control instance's load), and `orca watchdog snooze` silences one. Nothing local trips it, since there are no instances to launch. Per-pipeline placement (`orca register --placement control`, [#152](https://github.com/magmalake/orca.mojo/pull/152)) is on main and ships in the next orca release. `orca login` and `--remote` work against the local UI exactly as against a deployed one, and plain http is allowed only to this machine. The password is read from stdin: ``` printf 'a password\n' | ./build/orca user add alice printf 'a password\n' | ./build/orca login http://127.0.0.1:8275 --user alice ./build/orca --remote http://127.0.0.1:8275 pause toy ./build/orca --remote http://127.0.0.1:8275 unpause toy ./build/orca --remote http://127.0.0.1:8275 backfill --preview daily 2024-03-03 ./build/orca logout http://127.0.0.1:8275 ``` Stop exactly what you started: the two PIDs you saved, then the server. `pkill orca` would also stop every other checkout's UI on the machine. ``` kill $(cat build/scheduler.pid) $(cat build/ui.pid) pixi run pg_ctl -D build/pg stop ``` ### The test suites Every test is a pixi task. Most start their own throwaway Postgres, and each picks its ports from its PID, so suites can run side by side. A task's binaries are cached under `build/.mojo-build` by the content of their sources, so a second run of a test builds nothing. ``` pixi run test-steps # one suite: seconds pixi run test-cli pixi run test-pause # a scheduler suite, on a throwaway Postgres pixi run -e default test-executor # CI's first shard: 8 minutes pixi run -e default test-scheduler-ui # its second: 3 minutes pixi run -e default test # both pixi run -e nightly test-steps # the nightly compiler (needs flare_mlake, json_mlake) pixi run -e fmt format-check ``` The `data` environment adds `orca.tables`, which reads and commits Iceberg tables, and is the one that needs the siblings of the table above. Its tests are in three groups, as CI runs them, and individually: ``` pixi run -e data test-tables # pinned reads, commits: 31 s pixi run -e data test-data-tables # tables, append-files, replace, chaos, lineage, expire, taxi pixi run -e data test-data-sensors # sensors and rebuilds; lineage and sensors on MinIO pixi run -e data test-data-keyed # partitions named by a column, 20,000 keys pixi run -e data test-lineage-s3 # one suite over MinIO: 111 s including the install ``` The tests over object storage run against MinIO, which `scripts/with-minio.sh` starts on a free port with a bucket `orca` and the `AWS_*` variables pointing at it, the way `with-postgres.sh` does for Postgres. MinIO's open-source repository is archived and `dl.min.io` serves no binaries, so the `data` environment installs conda-forge's `minio-server` (macOS arm64 and linux-64 only). To use it in your own run: ``` pixi run -e data scripts/with-minio.sh sh -c 'echo $ORCA_S3_BUCKET $AWS_ENDPOINT_URL' ``` ## The census locally The [census](https://github.com/magmalake/mojo-census) runs entirely on a laptop, against a slice of the archive. It supports macOS arm64 and linux-aarch64; there is no linux-64 build in its `pixi.toml`. The first build installs the environment and compiles every dependency, about nine minutes; `pixi run test` then takes 72 seconds and uses no network. ``` cd ../mojo-census pixi run build # build/orca, the pipelines and build/console pixi run test ``` ### The pipelines on their own No database. `GHARCHIVE_HOURS` keeps the download to two hourly files of about 100 MB each, and `MOJO_HISTORY_MAX_KB` skips repositories over 100 MB, which includes modular/modular. On 2024-06-12, hours 15 and 17, the run below takes about 40 seconds and leaves 165 MB under `build/warehouse`: ``` export ORCA_WAREHOUSE=$PWD/build/warehouse GHARCHIVE_HOURS=15,17 pixi run ./build/gharchive run --partitions 2024-06-12 # 6 s pixi run ./build/mojo_repos run --partitions 2024-06-12 # 12 s MOJO_HISTORY_MAX_KB=100000 pixi run ./build/mojo_history run --partitions @all # 6 s pixi run ./build/mojo_activity run --partitions 2024-06-12 pixi run ./build/mojo_versions run --partitions @all # 6 s pixi run ./build/mojo_stats run --partitions $(date +%F) PORT=8080 pixi run ./build/console # http://127.0.0.1:8080 ``` The result is 476,227 events, 23 candidate repositories, 4 of them Mojo (NuMojo, Mojonet, mojo-flx and modular/modular, which is skipped for its size), 3 histories with 372 commits and 14 tags, and the console answering with its overview, repository list and JSON under `/api/`. mojo\_stats takes today's date, not 2024's: its partitions start on 2026-11-01\. The census's one-off pipelines (`gharchive_index`, `mojo_retool`) are for repairing a lake that already exists, and a fresh warehouse has no use for them. ### The Orca way This runs the pipelines as production does, with the scheduler, the sensors and the UI. Start a Postgres under `build/`, the same one as above. Then `scripts/env.sh` points `ORCA_DATABASE_URL` at it, puts the warehouse, logs and clones under `build/` and sets the small slice of the archive. A sensor counts only snapshots made after it was registered, so start from an empty warehouse (delete the `build/warehouse` of the run above), register every pipeline first, then backfill one day: ``` pixi run initdb -D build/pg --auth=trust --username=postgres -E UTF8 --locale=C pixi run pg_ctl -D build/pg -l build/pg.log -o "-k $PWD/build -c listen_addresses=" -w start pixi run createdb -h "$PWD/build" -U postgres orca pixi shell . scripts/env.sh export ORCA_UI_ADDR=127.0.0.1:8275 # 8270 is the default; use another if one is running for p in gharchive mojo_repos mojo_history mojo_activity mojo_stats; do ./build/orca locate build/$p --root .; done ./build/orca register ./build/gharchive ./build/mojo_repos ./build/mojo_history ./build/mojo_activity ./build/mojo_stats ./build/orca scheduler > build/scheduler.out 2>&1 & echo $! > build/scheduler.pid ./build/orca ui > build/ui.out 2>&1 & echo $! > build/ui.pid ./build/orca backfill gharchive 2024-06-12 ``` gharchive loads the day, which starts mojo\_repos for it, whose new repositories start mojo\_history and mojo\_activity. On the run above all of that took 25 seconds, and `./build/orca lineage mojo.commits` traced the histories back through `mojo.repos` and `gharchive.candidates` to the gharchive run. The UI shows the same, and `build/logs///` has the logs. The console from above reads the warehouse these runs wrote. Stop with the two PIDs and `pg_ctl -D build/pg stop`, as in Orca's section. ### What needs the network - **GH Archive** (data.gharchive.org): gharchive downloads about 100 MB an hour. Nothing else reads it; mojo\_activity works from the tables gharchive wrote. - **The GitHub API**: mojo\_repos asks about each candidate and mojo\_stats about each repository, 60 requests an hour without a token. Unauthenticated, mojo\_repos checks 25 candidates a run and records the rest as unchecked for the next run; with `GITHUB_TOKEN` set (a token with no scopes will do) the budget is 1000 and the limit 5000 an hour. To check the rest of a day after the limit resets: `./build/orca backfill --rerun mojo_repos 2024-06-12`. - **github.com**: mojo\_history and mojo\_versions clone each repository. - **Nothing**: building and testing once the environment is installed, mojo\_activity, the console on a local warehouse, the scheduler, the UI and Postgres. `pixi run test-daily` compares gharchive's commits of a day (one snapshot of each index table) with a build from before that change: it needs `build/gharchive-main`, built from the commit before `8922605`, and would fail against today's main, which already commits daily. It downloads three hours and takes about 50 seconds, and is for changes to how gharchive commits. ## A library change in the census To see a change to Orca, Iceberg or Parquet work in the census before releasing it, build the census against your checkout. The census pins every tin by commit in `[dependencies]`, and `orca-mojo`, which pixi-build compiles from source, pins the tins it is built from in its own `[package.host-dependencies]` and `[package.run-dependencies]`. pixi solves them as one environment, so a tin must have one source, the same URL and commit, everywhere in that closure. Two commits of one tin fail the solve: ``` failed to solve the environment encountered duplicate records for iceberg-mojo-0.10.0-h60d57d3_0_source.conda ``` That is why a release of one tin is a release of every tin that pins it, and why the same rule applies to a trial. Three cases, easiest first. **A change in Orca alone.** Point the census at the checkout, as long as the checkout pins the same commit of every other tin as the census does (a checkout of Orca's main does at the time of writing). Locking takes 2 seconds, and building takes a minute, since pixi compiles orca-mojo from the checkout: ``` cd mojo-census sed -i.bak 's|^orca-mojo = .*|orca-mojo = { path = "../orca.mojo" }|' pixi.toml pixi lock pixi run build-orca && ./build/orca --help | grep watchdog # a command main has and 0.8.0 lacks ``` **A branch pushed to GitHub**, for a change someone else will test or that CI builds. Pin its head commit with `rev`, not `branch`: a git dependency without a `rev` stays at whatever `pixi.lock` first resolved, and a branch moves under it. ``` sed -i.bak "s|^orca-mojo = .*|orca-mojo = { git = \"https://github.com/magmalake/orca.mojo.git\", rev = \"$(git -C ../orca.mojo rev-parse HEAD)\" }|" pixi.toml pixi run build-orca # 24 s ``` **A change in a library Orca depends on.** Every place that names the tin must agree: the census's `pixi.toml`, Orca's two package tables and any other tin in the closure that pins it. Find them with `grep -n 'iceberg' */pixi.toml` in the directory that holds the checkouts, change each to the same pin (a pushed commit, or `path = "../iceberg.mojo"` written relative to the file it is in), and run `pixi lock`. A tin left on its old commit anywhere gives the error above. A sibling checkout on the include path is easy to miss, so lock after every change. With the paths in place, a change to the checkout's source is rebuilt by the next `pixi run` (22 seconds for Iceberg and Orca). To go back to the released pins, restore the files and the lock; the lock is authoritative, so nothing else changes: ``` git restore pixi.toml pixi.lock && rm -f pixi.toml.bak pixi install --locked ``` What a release costs is a question for `shelf release plan`, run from the directory that holds the checkouts. It changes nothing and reads the registry, which is not on AWS. A tin's dependents must be checked out beside it, and the plan names any that are not. For Iceberg, with the checkouts above, it says that Orca moves too: ``` cd .. shelf release plan iceberg-mojo # release plan, parents first (2 repositories, 2 tins): # 1. iceberg.mojo release/iceberg-mojo-0.10.1 iceberg-mojo 0.10.0 -> 0.10.1 named # 2. orca.mojo release/orca-mojo-0.8.1 orca-mojo 0.8.0 -> 0.8.1 pins iceberg-mojo # pixi.toml [package.host-dependencies] iceberg-mojo ab62e59fbf3f -> release/iceberg-mojo-0.10.1 head # pixi.toml [package.run-dependencies] iceberg-mojo ab62e59fbf3f -> release/iceberg-mojo-0.10.1 head # .github/workflows/ci.yml:312 ab62e59fbf3f -> release/iceberg-mojo-0.10.1 head ``` The census is an application, not a tin, so it follows a release with a pull request that moves its pins to the commits the registry published. The rules, and `shelf release apply` that cuts the whole cascade, are in the [mojoshelf-publish skill](https://github.com/mojoshelf/mojoshelf/blob/main/skills/mojoshelf-publish/SKILL.md), under "Releasing tins that pin each other". ## Caveats - **macOS, local mode and an S3 warehouse crash.** A pipeline run on its own on macOS with `ORCA_WAREHOUSE=s3://...` fails with "no workers left: every local worker went away": the driver used libcurl before forking its Dista workers, and macOS aborts a forked child of a process that initialised it ([orca #135](https://github.com/magmalake/orca.mojo/issues/135)). It reproduced against a local MinIO, and the `OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES` stopgap the issue suggests did not get around it. Use a directory for the warehouse. Linux is not affected, and `test-lineage-s3` passes on macOS. - **Ports.** The UI listens on 127.0.0.1:8270 and the console on `PORT`. Another checkout's UI may hold 8270, so set `ORCA_UI_ADDR` and, when a page looks wrong, check whose UI you are looking at. - **Stopping everything.** Save the PID of every background process you start and kill those; stop a server you started with `pg_ctl -D build/pg stop`. A throwaway server stops itself with its command. If a test is interrupted, look for `orcapg.*` and `orcaminio.*` directories under `$TMPDIR`. - **Socket paths.** macOS limits a Unix socket's path to 104 bytes. A checkout deep in a long path, or a long default `TMPDIR`, can break Postgres's socket; keep the paths short. - **Disk.** See the sizes above. `.pixi/bld` is build scratch and can be deleted once the environment is installed (`pixi run` does not need it again); `build/warehouse` is the lake you made, and the census's two-hour slice is 165 MB. A day of the archive is about 15 GB to download. - **A busy machine.** `test-fleet` sorts hosts by CPU, and the control instance it lists is your machine. With builds running beside it, the test failed once with "by CPU, highest first"; run alone, the shard passed. ## From local to AWS When it works here, release what changed with `shelf release` (the cascade above), move the census's pins to the published commits, and deploy with `orca aws`: [the deploy steps](https://magmalake.dev/getting-started/step-by-step#aws), [the command's reference](https://magmalake.dev/s/orca/docs/cli#aws), [local or AWS?](https://magmalake.dev/s/orca/docs/cli#where) and [the runners](https://magmalake.dev/s/orca/docs/run#runners) in Orca's docs, and the [magmalake-apps skill's AWS reference](https://github.com/magmalake/skills/blob/main/magmalake-apps/references/aws.md). A stack's template changes should reach a running stack by replacing its control instance from a clean boot, which [orca #154](https://github.com/magmalake/orca.mojo/issues/154) is building as `orca aws replace-control`, not by editing the live instance by hand. Until it lands, rehearse a stack change on a separate stack first. The release and operations routine is in [Maintain and release](https://magmalake.dev/getting-started/maintain).