Orca · API

orca.actions

What orca ui does to runs: each action is rows in Postgres that the scheduler, or the run's own process, acts on.

The UI never starts a pipeline's binary. Starting, resuming, rerunning and backfilling leave a queued run, which the scheduler starts on its next tick as it starts any other, within max_runs; cancelling sets cancel_requested, which a running run sees before its next attempt (an attempt already going finishes). So the UI works the same wherever the scheduler runs, and retries and limits stay in one place.

Each action returns an Outcome: the run it made or acted on, or why it refused, which is something the person can fix (a bad range, a run that's still going) rather than an error.

Structs

Outcome

struct Outcome

Implements Movable

Fields
run: Int: The run made or acted on; 0 when refused, or for a backfill.
queued: Int: How many runs a backfill queued.
refused: String: Why the action wasn't done, or empty.
confirm: Bool: Refused only until confirmed: a resume onto new code.

done

def done(run: Int, queued: Int = Int(0)) -> Self

no

def no(var why: String, confirm: Bool = False) -> Self

Functions

stale_seconds

def stale_seconds() -> Float64

ORCA_STALE_SECONDS (default 30): how old a running run's heartbeat is when its process is taken to have died.

stalled_sql

def stalled_sql() -> String

True for a run that's running with no live process: its heartbeat, or its start when it has none yet, is older than the stale limit.

start_run

def start_run(mut db: State, pipeline: String, keys: String) -> Outcome

A queued run over the partitions in keys, trigger ui.

backfill_runs

def backfill_runs(mut db: State, pipeline: String, keys: String, rerun: Bool = False) -> Outcome

What orca backfill does: queued runs, keys_per_run keys each, of the keys in keys that aren't done and aren't in a live run; for time keys, by default a run for each partition in keys that has none queued, running or succeeded. With rerun, done keys are queued too.

resume_run

def resume_run(mut db: State, run: Int, allow_new_code: Bool) -> Outcome

Queues a failed, cancelled or stalled run again, so the scheduler resumes it at what's unfinished. A run whose pipeline's code has changed since is refused until allow_new_code, which the scheduler passes on as --allow-new-code.

rerun_from

def rerun_from(mut db: State, run: Int, step: String) -> Outcome

A new queued run over the same partitions, trigger rerun, with the steps before step that succeeded copied from run (their outputs, and copied_from naming the step they came from, whose attempts, reads and commits stay with it). Resuming it runs step and what follows. run is left as it was.

cancel_run

def cancel_run(mut db: State, run: Int) -> Outcome

Cancels a run: a queued or failed one at once (a failed one isn't retried), a running one before its next attempt.

rebuild_downstream

def rebuild_downstream(mut db: State, dataset: String, keys: String) -> Outcome

What orca rebuild does: plans runs of every pipeline downstream of dataset over keys, which the scheduler queues a level at a time. The outcome's run is the rebuild's ID.