magmalake Orca

Orca · API

orca.ui

orca ui: pipelines and runs as web pages, and the same as JSON.

ORCA_DATABASE_URL=postgresql:///orca ./build/orca ui     # 127.0.0.1:8270
Path
/ pipelines, with their schedules, next runs and last runs
/pipelines/<name> a pipeline's steps, partitions, schedule, next runs and runs; a keyed one's keys, done, live and failed
/pipelines/<name>/keys/<key> one key of a keyed pipeline: its steps in every run it was in
/runs runs, newest first; ?pipeline=, ?status=, ?trigger=
/runs/<id> a run's DAG, steps and partitions, attempts, errors, reads and commits
/runs/<id>/attempts/<attempt> an attempt's log, with its error
/runs/<id>/timeline.json the run's Dista timeline, for Perfetto
/runs/<id>/driver.log the run's driver output, as the scheduler kept it
/datasets datasets, with their writers, readers and latest snapshots, and the graph of pipelines and datasets
/datasets/<name> a dataset's place in that graph and its snapshots
/datasets/<name>/snapshots/<id> a snapshot's lineage, back and forward (orca.lineage)
/rebuilds/<id> a rebuild's pipelines by level, with their runs (orca.sensors)

And the actions, each a POST of a form (see orca.actions):

POST
/pipelines/<name>/runs start a run over partitions
/pipelines/<name>/backfill a run per partition in partitions that has none; rerun=1 for done ones too
/runs/<id>/resume queue a failed, cancelled or stalled run again; allow_new_code=1 confirms new code
/runs/<id>/rerun a new run from step on, the steps before it copied
/runs/<id>/cancel cancel a run, before its next attempt if it's running
/datasets/<name>/rebuild rebuild every pipeline downstream over partitions
/login, /logout a session for the actions, as a cookie; name, password, next

A page of something still going on (a queued or running run, a running attempt) refreshes its live part every two seconds from a fragment path, /runs/<id>/steps or /runs/<id>/attempts/<attempt>/log, which answers 286 once there's nothing more to show, and the page stops asking. That's the only JavaScript.

Every path answers HTML, or JSON to Accept: application/json or ?format=json, built from the same rows, so a script and a person see the same thing. An action answers 303 See Other to the page of what it made, or that page's JSON; one refused answers 409 saying why. The UI holds no state: its actions are rows the scheduler acts on, it connects to Postgres per request, and it can run beside the scheduler or anywhere else that reaches the database. ORCA_UI_ADDR (host:port) moves it; it binds to localhost unless told otherwise, and a POST whose Origin (or Referer) isn't the UI's own is refused, so another site's page can't post to it through a browser. Reading needs no login; once login is on (ORCA_UI_LOGIN=required, or any user added with orca user add), every action needs a session from /login, and each one is recorded as its user's (orca.auth), which a run's page shows.

Structs

PipelineRow

struct PipelineRow

Implements Copyable, Movable

Fields
name: String
registry_hash: String
binary: String
kind: String
start: String
cron: String
tz: String
fills: String
max_runs: Int
run_retries: Int
running: Int
queued: Int
last_run: Int: 0 when it has none.
last_status: String
last_partitions: String
next_fire: String
next_partition: String

RunRow

struct RunRow

Implements Copyable, Movable

Fields
id: Int
pipeline: String
partitions: String
trigger: String
status: String
starts: Int
created: String
started: String
ended: String
cancel_requested: Bool
stalled: Bool: running with no live process (orca.actions.stalled_sql).

VersionRow

struct VersionRow

A version of a pipeline's code (orca.source).

Implements Copyable, Movable

Fields
id: Int
registry_hash: String
source: String
commit: String
dirty: Bool
image: String
first_seen: String
last_seen: String
runs: Int
locations: String: Where its functions are (orca.locate), as JSON, or empty.

InstanceRow

struct InstanceRow

An instance a run ran on (orca.instances).

Implements Copyable, Movable

Fields
id: String
start: Int
role: String
instance_type: String
market: String
launched: String
ended: String
why: String
from_start: Float64: Seconds from the run's start to its launch.
to_end: Float64: Seconds from the run's start to its end, or to now.

StepDef

struct StepDef

Implements Copyable, Movable

Fields
name: String
kind: String
after: String
reads: List[String]
writes: List[String]

Fire

struct Fire

Implements Copyable, Movable

Fields
at: String
local: String
partition: String

StepRow

struct StepRow

Implements Copyable, Movable

Fields
step: String
partition: String
status: String
attempts: Int
error: String
log: String
started: String
ended: String
attempt_id: Int: The latest attempt's ID, 0 before the first.
from_run: Int: The run a rerun copied this step from, or 0.

Touch

struct Touch

A table a step read or committed to.

Implements Copyable, Movable

Fields
step: String
partition: String
kind: String: read or commit.
table: String
snapshot: String
dataset: String: Empty for a table outside the warehouse.

CodeLinks

struct CodeLinks

A version's code on its host: the pipeline's declaration, each step's function and each dista task, from orca.locate's map.

Implements Movable

Fields
pipeline: String
steps: Dict[String, String]: Step name to its function's line.
tasks: List[Tuple[String, String]]: A task's function name and its line.

__init__

def __init__(out self)

step

def step(self, name: String) -> String

Page

struct Page

Implements Movable

Fields
status: Int
json: String
html: String
content_type: String: Set for what isn't a page or its JSON (a fragment, a file): html is sent as this, whatever was asked for.
location: String: Set for an action that was done: a browser is sent there with 303, and a script gets json with 200.

__init__

def __init__(out self, status: Int, var json: String, var html: String, content_type: String = "", location: String = "")

KeyCounts

struct KeyCounts

A keyed pipeline's keys and how far it is through them.

Implements Copyable, Movable

Fields
dataset: String
snapshot: Int64
total: Int
done: Int
live: Int
failed: List[String]: Keys not done or in a live run, with a failed or skipped step.
error: String: Why its keys couldn't be read, or empty.

AttemptRow

struct AttemptRow

Implements Copyable, Movable

Fields
id: Int
run: Int
pipeline: String
step: String
partition: String
attempt: Int
status: String
host: String
started: String
ended: String
error: String
log: String

Ui

struct Ui

The pages, over the database at url.

Implements Copyable, Handler, Movable

Fields
url: String
comptime BodyType = InlineBody

The Body impl this handler authors its response with.

__init__

def __init__(out self, url: String)

serve

def serve(self, req: Request) -> Response

Functions

home

def home(mut db: State) -> Page

key_page

def key_page(mut db: State, name: String, key: String) -> Page

One key of a keyed pipeline: its steps in every run it was in.

pipeline

def pipeline(mut db: State, name: String) -> Page

runs

def runs(mut db: State, pipeline: String, status: String, trigger: String) -> Page

run

def run(mut db: State, id: Int) -> Page

run_steps

def run_steps(mut db: State, id: Int) -> Page

The live part of a run's page, as a fragment; 286 once it's ended.

timeline

def timeline(mut db: State, id: Int) -> Page

driver_log

def driver_log(mut db: State, id: Int) -> Page

The run's driver log, as the scheduler kept it under ORCA_LOGS.

cluster_log

def cluster_log(mut db: State, id: Int, name: String) -> Page

An instance's log, as its script kept it under ORCA_LOGS: only the names of the run's own instances are served.

attempt

def attempt(mut db: State, run: Int, id: Int) -> Page

attempt_log

def attempt_log(mut db: State, run: Int, id: Int) -> Page

An attempt's log, as a fragment; 286 once the attempt has ended.

not_found

def not_found(what: String) -> Page

datasets_page

def datasets_page(mut db: State) -> Page

dataset_page

def dataset_page(mut db: State, name: String) -> Page

rebuild_page

def rebuild_page(mut db: State, id: Int) -> Page

snapshot_page

def snapshot_page(mut db: State, name: String, id: Int64) -> Page

login_page

def login_page(next: String, why: String, status: Int = Int(200)) -> Page

serve_ui

def serve_ui(url: String, addr: String)

Serves the pages on addr until the process is stopped.

Aliases

comptime DEFAULT_ADDR = "127.0.0.1:8270"