# Debugging failures

> Debugging a failed Orca run: where its errors and logs are, in the UI and in S3, what a raised error carries and how to add context to it, crash stack dumps and how the image symbolizes them, reproducing a partition locally, and resuming or rerunning it.

[magmalake](https://magmalake.dev/) [Orca](https://magmalake.dev/s/orca)

A run that fails leaves an error on each step that didn't succeed, a log for each attempt, and its process's output as `driver.log`. This page says where each of them is, what they can and can't tell you, and how to run the failing partition again, here or on the cluster.

## Where failures show

In [the UI](https://magmalake.dev/s/orca/docs/ui), `/runs?status=failed` lists the failed runs, newest first (the Status filter on the Runs page), and `?pipeline=` narrows it to one pipeline. A run's page has the run's own Error when it ended without a step saying why, such as `interrupted: its process stopped`, and its Steps table has an Error for each step and partition, with a link to its latest attempt, whose page shows its log (the last MiB of it) and its error. The run's page links its `driver.log`, and on EC2 each instance's log. Every page is JSON too, so the same is a `curl` away:

```
curl -s 'https://orca.example.com/runs?status=failed&format=json'
curl -s 'https://orca.example.com/runs/73?format=json'      # its steps, with "error" and "log"
curl -s https://orca.example.com/runs/73/driver.log
```

On the command line, `<binary> status <run>` prints the run's steps, their attempts, and where each attempt's log is. The logs themselves are under `ORCA_LOGS`, a directory or a prefix in object storage; on the stack `orca aws up` makes it is `s3://orca-<account>-<region>/logs`, whose objects expire after 30 days. Each run's are under `<pipeline>/<run>/`:

| File | What's in it |
| --- | --- |
| driver.log | everything the run's process printed, each start after a --- start <n> line: the step lines, the run's status, a crash's stack dump |
| <step>/<partition>/<attempt>.log | one attempt's start, end, error and ctx.log lines, with UTC timestamps; the partition is \_run for a gather step, and a key's / is written %2F |
| cluster/<role>-<instance>.log | an EC2 run's head or worker output, uploaded every ORCA\_CLUSTER\_LOG\_SECONDS while it runs, so a Dista task that crashed on a worker is here |
| metrics/<instance>.jsonl | the machine's CPU, memory, disk and network while the run went on |
| timeline.json | the run's Dista tasks, for Perfetto |

```
aws s3 ls s3://orca-123456789012-us-east-2/logs/mojo_history/73/ --recursive
aws s3 cp s3://orca-123456789012-us-east-2/logs/mojo_history/73/driver.log -
```

An attempt's log is uploaded when the attempt ends, so an attempt whose process died may have none; its lines went to the process's output as well, tagged `[step partition]`, and so are in `driver.log`. A run whose machine ran out of memory or disk shows it on its page, in the Machines section.

## Raised errors

A step fails when it raises. Orca records the `Error`'s message as the attempt's error, against the step, the partition and the attempt's number, retries it up to the step's `retries`, and skips what depends on it. The message is all there is: Mojo's `Error` carries no stack trace today (its `get_stack_trace()` is empty even with `MOJO_ENABLE_STACK_TRACE_ON_ERROR` set), so a bare `HTTP 404` or `unexpected end of input` from deep in a library says nothing about where it came from.

So add what you were doing where you catch it, and raise it again. Name the key and the operation, not the step or the partition, which Orca records already, and keep the original message last:

```
def hour_task(day: String, hour: Int) raises -> DataFiles:
    try:
        return fetch_hour(day, hour, load_dataset(EVENTS))
    except e:
        raise Error(day, " hour ", hour, ": ", e)
```

That gives `2025-01-01 hour 7: GET https://data.gharchive.org/2025-01-01-7.json.gz: HTTP 404`. Do it at a step's boundaries rather than around every call: where one step handles many keys (a repository, a file, an hour), where it calls a library whose errors don't say what they were reading, and where a task's error comes back from a Dista worker. Orca does the same at its own: a table that can't be loaded, written or committed names its dataset (`orca: can't commit to lake.events (9 tries): ...`). Leave a `defer` out of it: Orca reads the wait it asks for from the start of its message, so a `defer` wrapped in another message is just a failure. A step that handles many keys and shouldn't fail for one bad key can log the key's error with `ctx.log` and carry on, as `mojo_history` does for a repository it can't clone.

## Crashes

A segfault, an `abort()` or a failed bounds check doesn't raise: it ends the process, and every attempt it was running with it. Mojo's runtime prints a stack dump to standard error first, which goes into `driver.log`. The runner that sees the process end (the scheduler for a local run, the head instance's script on EC2, the scheduler keeping the driver pod's log on Kubernetes) finds the dump there and fails the attempts the process left running with its first frames, so the step's Error on the run's page says where it crashed:

```
crashed: /src/project/pipelines/gharchive.mojo:52:14: the input is bad; at load_day
/src/project/pipelines/gharchive.mojo:52:14, orca::step::Step::run src/orca/step.mojo:399:23,
orca::cli::_resume src/orca/cli.mojo:381:23, orca_main src/orca/cli.mojo:133:27
```

The part after `crashed:` is what `abort` or the failed `debug_assert` said, when it said anything; a segfault has only the frames. The run then goes the way a run whose process died does: a scheduled one is started again within its retries, and one you started is failed with `interrupted: its process stopped`, ready to resume. A process killed with no dump, by `kill -9` or the kernel's out-of-memory killer, leaves `interrupted: the process running it stopped` instead; its machine's memory is on the run's page.

The dump names functions and lines only when two things hold. The runtime has to find `llvm-symbolizer`, through `LLVM_SYMBOLIZER_PATH` or on `PATH`; without it, it says `Stack dump without symbol names` and prints addresses. And the binary has to have line tables, from `mojo build -g1`; without them, the frames inlined into `main` all come out as `main`. The image `orca aws push` builds does both: it builds every binary with `-g1` and compresses the line tables with `objcopy`, copies the pixi environment's `llvm-symbolizer` to `/opt/orca/libexec`, and sets `LLVM_SYMBOLIZER_PATH` to it. Compressed, the line tables make a binary about half as big again (mojo-census's `gharchive` goes from 4.2 MB to 6.5 MB; 25 MB uncompressed), and it runs as fast as before. A project with a `Dockerfile.orca` of its own should do the same:

```
# build stage: line tables, and the symbolizer with what it links
RUN pixi run mojo build -g1 -I src pipelines/gharchive.mojo -o /out/pipelines/gharchive     && objcopy --compress-debug-sections=zlib /out/pipelines/gharchive
RUN mkdir -p /out/libexec && cp -L .pixi/envs/default/bin/llvm-symbolizer /out/libexec/
# runtime stage
COPY --from=build /out/libexec /opt/orca/libexec
ENV LLVM_SYMBOLIZER_PATH=/opt/orca/libexec/llvm-symbolizer
```

`llvm-symbolizer` links the pixi environment's `libstdc++`, which the image's `/opt/orca/lib` has already for the pipelines. Locally, `pixi run` puts it on `PATH`, so a binary built with `-g1` crashes with its frames named; for a debugger, build with `-g`, the full debug info.

## Reproducing it locally

A pipeline binary runs any partition by itself, with no scheduler, no database and no cluster: without `ORCA_DATABASE_URL` it records the run in memory, and it forks its Dista workers on this machine. Run the partition that failed, with `ORCA_LOGS` for the attempts' logs and a warehouse of your own so nothing shared is written:

```
pixi run mojo build -g1 -I src pipelines/gharchive.mojo -o build/gharchive
ORCA_WAREHOUSE=build/warehouse ORCA_LOGS=build/logs \
    pixi run ./build/gharchive run --partitions 2025-01-01
cat build/logs/gharchive/0/load_day/2025-01-01/1.log
```

A keyed pipeline takes the key's name, `--partitions owner/name`. To rule out what differs on Linux, such as a library that won't load or a missing command, run the same in the image the run used, which is on its page:

```
docker run --rm --entrypoint sh -e ORCA_WAREHOUSE=/tmp/w <image> \
    -c 'gharchive run --partitions 2025-01-01'
```

## Resuming and rerunning

Once it's fixed, a failed run is resumed rather than started over: the steps that succeeded are read back, and only the rest run, with their recorded inputs. The run's Resume (or Retry, in a runs table) queues it for the scheduler; `<binary> resume <run>` runs it here. A fix is new code, which `resume` refuses unless told (`--allow-new-code`, or the UI's confirmation), so register the new build first and the pipeline's queued runs move to it. Rerun from here, on an attempt's page, makes a new run from that step on, the steps before it copied from the old run. To redo partitions that succeeded, `orca backfill --rerun` queues them again. [Scheduling and running](https://magmalake.dev/s/orca/docs/run#failures) says what the scheduler retries on its own.

```
./build/orca register ./build/gharchive         # the fixed build
./build/gharchive resume 73 --allow-new-code     # here, now
./build/orca backfill --rerun gharchive 2025-01-01
```

[A magmalake project](https://magmalake.dev/)

Apache 2.0 · © 2026 Marius Seritan