# Scheduling and running > Scheduling and running Orca pipelines: the scheduler and its fires, retries and limits, runs on this machine, on Kubernetes or on EC2, runs recorded and resumed in Postgres, logs and machine metrics, lineage and rebuilds, every environment variable, and what happens when things fail. [magmalake](https://magmalake.dev/) [Orca](https://magmalake.dev/s/orca) The scheduler starts runs when they're due, retries the ones that fail, and starts each on this machine or on a cluster of its own. Every run is recorded in Postgres, so any of them can be resumed, and its tables say what made them. ## The scheduler `orca scheduler` links no pipeline. It knows one by the `describe` that `orca register` recorded, and starts its runs as ` resume `, so a run records its own progress exactly as a run started by hand does. It needs `ORCA_DATABASE_URL`; [the command line](https://magmalake.dev/s/orca/docs/cli#orca) lists its commands. ``` ./build/orca register ./build/gharchive ./build/orca backfill gharchive 2025-01-01..2025-01-31 ./build/orca scheduler ``` Each fire is a row with a unique (pipeline, time), inserted with its queued run in one transaction, so two schedulers ticking at once, or one killed mid-tick, never fire twice. Fires missed while it was down are made on its next tick, once each. A queued run starts only while fewer than `max_runs` of the pipeline's runs are live, counted under a lock on the pipeline's row, and none start while it's paused. A scheduled, backfilled or sensed run that fails is started again after `ORCA_RETRY_SECONDS`, doubled with each start, plus up to a tenth as jitter, up to `run_retries` times; one whose process died is started again as soon as its heartbeat is stale. A run you started by hand, or from the UI, isn't retried. The scheduler holds nothing in memory, so it can be stopped and restarted at any time. Registering a new build moves the pipeline's queued runs to the new code. ## On Kubernetes and EC2 By default the scheduler starts a run as a process on its own machine, and the run forks its Dista workers there. `ORCA_RUNNER` starts each run on a cluster of its own instead, from the image the pipeline was registered with: ``` ./build/orca register --image 123456789012.dkr.ecr.us-east-2.amazonaws.com/orca:data \ --head-type c7g.large --worker-type c7g.2xlarge --workers 3 ./build/taxi ORCA_RUNNER=ec2 ./build/orca scheduler ``` **`ORCA_RUNNER=ec2`** starts a head instance, on demand, running the Dista head and the driver (` resume `), and `--workers` worker instances pointed at it; `--spot` makes the workers Spot instances, where the account allows them, and `--workers MIN..MAX` starts `MIN` and adds or drains workers up to `MAX` as the head's demand changes. Each instance is tagged with its run, told to terminate when it shuts down, and recorded with its type, when it started and ended, and why it ended (it shut itself down, it was terminated, or a Spot interruption). **`ORCA_RUNNER=kubernetes`** creates a driver Job, a head Pod and Service and the worker Pods in `ORCA_RUNS_NAMESPACE`, all labelled with the run and owned by the Job. A pipeline without Dista tasks gets only its driver. Both clean up in layers. A run's instances or pods end when it does; every tick the scheduler removes those of runs that aren't running any more and those older than `ORCA_RUN_DEADLINE`, so nothing a killed scheduler left behind keeps running. A run whose head goes is started again on new instances. Before making a cluster, the scheduler adds up the vCPUs of every live run's pods or instances, and a run that would take them past `ORCA_VCPU_BUDGET` waits. `orca aws up` makes all of this on AWS as one stack: a control instance running the scheduler with `ORCA_RUNNER=ec2`, Postgres and the UI, and what run instances need ([orca aws](https://magmalake.dev/s/orca/docs/cli#aws), [local or AWS?](https://magmalake.dev/s/orca/docs/cli#where)). ## Runs in Postgres With `ORCA_DATABASE_URL` set, the binary creates and migrates an `orca` schema at start-up and records every run, every step of every partition, and every attempt, with the inputs and outputs of each. Each change of state is one transaction. ``` export ORCA_DATABASE_URL=postgresql:///orca ./build/gharchive run --partitions 2025-01-01..2025-01-03 ./build/gharchive status 7 # run 7's steps and attempts ./build/gharchive resume 7 ``` `resume` reruns every step that hasn't succeeded, in order, with its recorded input; steps that succeeded are read back, not run again. A run started by a binary with a different registry hash is refused unless `--allow-new-code` is passed. One process owns a run at a time. It holds the run with a heartbeat, written from a thread with its own connection, and `resume` refuses a run whose heartbeat is younger than `ORCA_STALE_SECONDS`. A run whose process was killed goes stale and can be resumed; a run that ended can be resumed at once. Each attempt also checks that its process still owns the run, so a process that lost its run stops at its next step rather than racing the new owner. `tests/kill.sh` kills a run at a random step with `kill -9`, resumes it, and checks every step ran to success exactly once. Postgres must keep `synchronous_commit` on: with it off, an immediate restart of the server can lose commits it already acknowledged. The chaos test from [PR #6](https://github.com/magmalake/orca.mojo/pull/6) found this when a run's row vanished. ## Logs With `ORCA_LOGS` set to a directory, each attempt writes `////.log`: its start, its end, its error, and its `ctx.log` lines with UTC timestamps. The path is recorded with the attempt and shown by `status `. A run with Dista tasks writes the runtime's timeline beside them as `timeline.json`, in Chrome's trace format, which Perfetto opens. The scheduler keeps a run's driver output as `driver.log`, and an EC2 run's head and workers upload theirs to `cluster/-.log` every `ORCA_CLUSTER_LOG_SECONDS` while they run, as an instance can go without notice. `ORCA_LOGS` and `ORCA_WAREHOUSE` can be locations in object storage, `s3://bucket/prefix`, configured from the `AWS_*` environment or the instance's role. An object can't be appended to, so a log is written to a file under `ORCA_LOG_STAGE` and uploaded when it's complete: an attempt's when the attempt ends, a timeline when the run does. The UI reads them back from wherever they are. ## Machine metrics Beside every process doing a run's work, the runner starts ` sample`, which writes a JSON line of the machine's CPU, memory, disk, disk I/O and network every `ORCA_SAMPLE_SECONDS` to `//metrics/.jsonl` under `ORCA_LOGS`. A run's page draws them as its Machines section, on the same time axis as its steps, and marks the stretches where memory use or a disk was over 90% or CPU steal over 20%, which explain most slow or failed runs. ## Lineage Every read and commit a step makes is recorded with its [dataset](https://magmalake.dev/s/orca/docs/build#datasets), so lineage is two walks over them. Back from a snapshot: the run, step, partition and code that made it, the snapshots that attempt read, what made each of those, and so on, to sources no Orca run made. Forward: everything made from it. Both stop after 20 steps. ``` $ ./build/orca lineage lake.report lake.report@3141592653 made from: 0 lake.report@3141592653 by report run 9 report code 0x5e1f... from lake.clean@2718281828 1 lake.clean@2718281828 by clean run 8 clean 2024-02 code 0x9a0c... from lake.raw@1618033988 2 lake.raw@1618033988 by ingest run 6 load 2024-02 code 0x77d2... made from it: nothing ``` ## Rebuilds ``` $ ./build/orca rebuild raw.events 2024-03 rebuild 1 level 0 clean 2024-03 level 1 report 2024-03 ``` A rebuild runs every pipeline downstream of a dataset, a [sensor](https://magmalake.dev/s/orca/docs/build#sensors) or not, over the partitions given. Each pipeline is placed at its longest distance from the dataset, so one that reads two rebuilt datasets runs after both, and the scheduler queues a level once every run before it has succeeded. A run that fails stops the rebuild; resuming it from the UI carries on. The rebuild's own commits don't also start the sensors it reruns, so each pipeline runs once. ## Environment | Variable | What it sets | Default | | --- | --- | --- | | ORCA\_DATABASE\_URL | the Postgres database runs are recorded in; unset, a pipeline's run runs in memory | unset | | ORCA\_STALE\_SECONDS | how old a run's heartbeat must be before another process may resume it | 30 | | ORCA\_WORKERS | the local Dista workers a run forks | 4 | | ORCA\_LOGS | where attempts' logs, timelines, driver logs, instance logs and metrics go: a directory or s3://bucket/prefix | unset | | ORCA\_LOG\_STAGE | where a log bound for object storage is written until it's complete | $TMPDIR/orca-logs | | ORCA\_WAREHOUSE | the root of the filesystem catalog datasets are found in, a directory or s3://; ns.table is /ns/table | unset | | ORCA\_RETRY\_SECONDS | the scheduler's first wait before retrying a failed run, doubled with each start | 60 | | ORCA\_SENSE\_SECONDS | how often the scheduler asks a sensor's binary for its datasets' current snapshots | 60 | | ORCA\_SAMPLE\_SECONDS | how often a run's sampler writes a line of machine metrics; 0 turns it off | 5 | | ORCA\_RUNNER | where the scheduler starts runs: local, kubernetes or ec2 | local | | ORCA\_VCPU\_BUDGET | the vCPUs all live runs' pods or instances may use together; a run that would go over waits | no limit | | ORCA\_RUN\_ENV | the scheduler's variables, by name and comma-separated, copied to every run's pods or instances | unset | | ORCA\_RUN\_DEADLINE | seconds after which a run's pods or instances are removed whatever their state | 86400 | | ORCA\_IMAGE | the image a run's binary is in, recorded with its version; the runners set it | unset | | ORCA\_EC2\_HEAD\_TYPE | the head instance's type, unless registration gave one | c7g.large | | ORCA\_EC2\_WORKER\_TYPE | the worker instances' type, unless registration gave one | c7g.xlarge | | ORCA\_EC2\_AMI | the image run instances boot | unset | | ORCA\_EC2\_SUBNET, ORCA\_EC2\_SECURITY\_GROUP, ORCA\_EC2\_PROFILE | where run instances go, their security group, and their instance profile | unset | | ORCA\_EC2\_DATABASE\_URL | the database URL run instances use, such as the control instance's private address | ORCA\_DATABASE\_URL | | ORCA\_EC2\_PREAMBLE | shell run on each instance before the pipeline, such as an ECR login | unset | | ORCA\_EC2\_MODE | docker runs the image on the instance; direct runs the binary, for the tests' stand-in EC2 | docker | | ORCA\_CLUSTER\_LOG\_SECONDS | how often a run instance uploads its head's or worker's log | 30 | | ORCA\_RUNS\_NAMESPACE | the Kubernetes namespace runs' pods go in | orca-runs | | ORCA\_UI\_ADDR | host:port orca ui listens on | 127.0.0.1:8270 | | ORCA\_UI\_LOGIN | required makes the UI's actions need a login even before any user exists | on once a user exists | | ORCA\_UI\_SESSION\_HOURS | how long a login lasts | 12 | | ORCA\_UI\_SECURE\_COOKIES | 1 marks the session cookie Secure, for a UI behind HTTPS | unset | | ORCA\_UI\_TRUST\_PROXY | 1 takes a request's address from X-Forwarded-For, behind a proxy you run | unset | | ORCA\_UI\_URL | the UI's public address, for the Open in Perfetto link | unset | | ORCA\_DOCS\_URL | where the UI's Docs links point | https://mlake.dev/s/orca/docs/ | | ORCA\_NOW | fixes the clock of fires and the scheduler, as 2024-03-10T07:00Z or epoch seconds, for tests | the clock | | ORCA\_ZONEINFO | where time zones' TZif files are read from | /usr/share/zoneinfo | | DISTA\_ROLE | head or worker: the binary runs as that Dista process instead | unset | | DISTA\_HEAD | host:port of a Dista head to join instead of forking workers | local mode | ## When things fail **A step that raises** is retried up to its `retries`, then fails. The steps downstream of it are skipped, while other partitions' steps that don't depend on it carry on, so one bad month doesn't stop the others; the run ends failed and can be resumed. **A process that dies** leaves its run running with a stale heartbeat; `resume` marks the interrupted attempt failed and continues from there. One that crashed has its attempts failed with its stack dump's first frames as soon as its runner sees it end ([Debugging failures](https://magmalake.dev/s/orca/docs/debug#crashes)). **A commit that landed before the crash** is found by the retry rather than made again. Fuzzing, and a chaos test that kills runs, restarts Postgres and races two resumes over many seeds, run in [CI](https://github.com/magmalake/orca.mojo/pull/6). [A magmalake project](https://magmalake.dev/) Apache 2.0 · © 2026 Marius Seritan