Docs

Debugging failures

A run that fails leaves an error on each step that didn't succeed, a log for each attempt, and its process's output as driver.log. This page says where each of them is, what they can and can't tell you, and how to run the failing partition again, here or on the cluster.

Where failures show

In the UI, /runs?status=failed lists the failed runs, newest first (the Status filter on the Runs page), and ?pipeline= narrows it to one pipeline. A run's page has the run's own Error when it ended without a step saying why, such as interrupted: its process stopped, and its Steps table has an Error for each step and partition, with a link to its latest attempt, whose page shows its log (the last MiB of it) and its error. The run's page links its driver.log, and on EC2 each instance's log. Every page is JSON too, so the same is a curl away:

curl -s 'https://orca.example.com/runs?status=failed&format=json'
curl -s 'https://orca.example.com/runs/73?format=json'      # its steps, with "error" and "log"
curl -s https://orca.example.com/runs/73/driver.log

On the command line, <binary> status <run> prints the run's steps, their attempts, and where each attempt's log is. The logs themselves are under ORCA_LOGS, a directory or a prefix in object storage; on the stack orca aws up makes it is s3://orca-<account>-<region>/logs, whose objects expire after 30 days. Each run's are under <pipeline>/<run>/:

aws s3 ls s3://orca-123456789012-us-east-2/logs/mojo_history/73/ --recursive
aws s3 cp s3://orca-123456789012-us-east-2/logs/mojo_history/73/driver.log -

An attempt's log is uploaded when the attempt ends, so an attempt whose process died may have none; its lines went to the process's output as well, tagged [step partition], and so are in driver.log. A run whose machine ran out of memory or disk shows it on its page, in the Machines section.

Raised errors

A step fails when it raises. Orca records the Error's message as the attempt's error, against the step, the partition and the attempt's number, retries it up to the step's retries, and skips what depends on it. The message is all there is: Mojo's Error carries no stack trace today (its get_stack_trace() is empty even with MOJO_ENABLE_STACK_TRACE_ON_ERROR set), so a bare HTTP 404 or unexpected end of input from deep in a library says nothing about where it came from.

So add what you were doing where you catch it, and raise it again. Name the key and the operation, not the step or the partition, which Orca records already, and keep the original message last:

def hour_task(day: String, hour: Int) raises -> DataFiles:
    try:
        return fetch_hour(day, hour, load_dataset(EVENTS))
    except e:
        raise Error(day, " hour ", hour, ": ", e)

That gives 2025-01-01 hour 7: GET https://data.gharchive.org/2025-01-01-7.json.gz: HTTP 404. Do it at a step's boundaries rather than around every call: where one step handles many keys (a repository, a file, an hour), where it calls a library whose errors don't say what they were reading, and where a task's error comes back from a Dista worker. Orca does the same at its own: a table that can't be loaded, written or committed names its dataset (orca: can't commit to lake.events (9 tries): ...). Leave a defer out of it: Orca reads the wait it asks for from the start of its message, so a defer wrapped in another message is just a failure. A step that handles many keys and shouldn't fail for one bad key can log the key's error with ctx.log and carry on, as mojo_history does for a repository it can't clone.

Crashes

A segfault, an abort() or a failed bounds check doesn't raise: it ends the process, and every attempt it was running with it. Mojo's runtime prints a stack dump to standard error first, which goes into driver.log. The runner that sees the process end (the scheduler for a local run, the head instance's script on EC2, the scheduler keeping the driver pod's log on Kubernetes) finds the dump there and fails the attempts the process left running with its first frames, so the step's Error on the run's page says where it crashed:

crashed: /src/project/pipelines/gharchive.mojo:52:14: the input is bad; at load_day
/src/project/pipelines/gharchive.mojo:52:14, orca::step::Step::run src/orca/step.mojo:399:23,
orca::cli::_resume src/orca/cli.mojo:381:23, orca_main src/orca/cli.mojo:133:27

The part after crashed: is what abort or the failed debug_assert said, when it said anything; a segfault has only the frames. The run then goes the way a run whose process died does: a scheduled one is started again within its retries, and one you started is failed with interrupted: its process stopped, ready to resume. A process killed with no dump, by kill -9 or the kernel's out-of-memory killer, leaves interrupted: the process running it stopped instead; its machine's memory is on the run's page.

The dump names functions and lines only when two things hold. The runtime has to find llvm-symbolizer, through LLVM_SYMBOLIZER_PATH or on PATH; without it, it says Stack dump without symbol names and prints addresses. And the binary has to have line tables, from mojo build -g1; without them, the frames inlined into main all come out as main. The image orca aws push builds does both: it builds every binary with -g1 and compresses the line tables with objcopy, copies the pixi environment's llvm-symbolizer to /opt/orca/libexec, and sets LLVM_SYMBOLIZER_PATH to it. Compressed, the line tables make a binary about half as big again (mojo-census's gharchive goes from 4.2 MB to 6.5 MB; 25 MB uncompressed), and it runs as fast as before. A project with a Dockerfile.orca of its own should do the same:

# build stage: line tables, and the symbolizer with what it links
RUN pixi run mojo build -g1 -I src pipelines/gharchive.mojo -o /out/pipelines/gharchive     && objcopy --compress-debug-sections=zlib /out/pipelines/gharchive
RUN mkdir -p /out/libexec && cp -L .pixi/envs/default/bin/llvm-symbolizer /out/libexec/
# runtime stage
COPY --from=build /out/libexec /opt/orca/libexec
ENV LLVM_SYMBOLIZER_PATH=/opt/orca/libexec/llvm-symbolizer

llvm-symbolizer links the pixi environment's libstdc++, which the image's /opt/orca/lib has already for the pipelines. Locally, pixi run puts it on PATH, so a binary built with -g1 crashes with its frames named; for a debugger, build with -g, the full debug info.

Reproducing it locally

A pipeline binary runs any partition by itself, with no scheduler, no database and no cluster: without ORCA_DATABASE_URL it records the run in memory, and it forks its Dista workers on this machine. Run the partition that failed, with ORCA_LOGS for the attempts' logs and a warehouse of your own so nothing shared is written:

pixi run mojo build -g1 -I src pipelines/gharchive.mojo -o build/gharchive
ORCA_WAREHOUSE=build/warehouse ORCA_LOGS=build/logs \
    pixi run ./build/gharchive run --partitions 2025-01-01
cat build/logs/gharchive/0/load_day/2025-01-01/1.log

A keyed pipeline takes the key's name, --partitions owner/name. To rule out what differs on Linux, such as a library that won't load or a missing command, run the same in the image the run used, which is on its page:

docker run --rm --entrypoint sh -e ORCA_WAREHOUSE=/tmp/w <image> \
    -c 'gharchive run --partitions 2025-01-01'

Resuming and rerunning

Once it's fixed, a failed run is resumed rather than started over: the steps that succeeded are read back, and only the rest run, with their recorded inputs. The run's Resume (or Retry, in a runs table) queues it for the scheduler; <binary> resume <run> runs it here. A fix is new code, which resume refuses unless told (--allow-new-code, or the UI's confirmation), so register the new build first and the pipeline's queued runs move to it. Rerun from here, on an attempt's page, makes a new run from that step on, the steps before it copied from the old run. To redo partitions that succeeded, orca backfill --rerun queues them again. Scheduling and running says what the scheduler retries on its own.

./build/orca register ./build/gharchive         # the fixed build
./build/gharchive resume 73 --allow-new-code     # here, now
./build/orca backfill --rerun gharchive 2025-01-01