magmalake Orca

Tins · Orca

API reference

The orca package, from Orca 0.3.2, as published on mojoshelf (commit 1db5de9aab67), by mojo doc from Mojo 1.2.0. Source.

orca: orchestration for Mojo data pipelines.

Declare steps and the pipeline they make, and give the binary orca's command line:

from dista import Seq
from orca import Ctx, Gather, Mapped, Month, Pipeline, Step, orca_main

def size(mut ctx: Ctx, m: Month) raises -> Int:
    return m.month * 100

def total(mut ctx: Ctx, sizes: Seq[Int]) raises -> Int:
    var t = 0
    for i in range(len(sizes)):
        t += sizes[i]
    return t

comptime Size = Step["size", Month, Int, size, Mapped]
comptime Total = Step["total", Seq[Int], Int, total, Gather, after=Size]
comptime Sizes = Pipeline["sizes", Size, Total]

def main() raises:
    orca_main[Sizes]()

See DESIGN.md for how it works and where it's going.

Modules

  • orca.actions What orca ui does to runs: each action is rows in Postgres that the scheduler, or the run's own process, acts on. 1 struct · 8 functions
  • orca.auth Who may act in orca ui: users with passwords, their sessions, and a record of what each did. 10 functions
  • orca.aws orca aws: a project's orca on AWS, with the aws and docker commands its user already has (aws login, or AWS_PROFILE for a named profile). 1 function
  • orca.aws_assets The stack template and the project Dockerfile orca aws writes out: deploy/aws/orca.yaml and deploy/project.Dockerfile, embedded by scripts/embed-aws.py. Edit those, then run it; pixi run test-aws-assets fails while this is out of date.
  • orca.awsec2 The few EC2 calls the ec2 runner makes (orca.ec2), over objectstore's SigV4 signing and HTTP. 2 structs · 1 function
  • orca.chart Runs per day, drawn as SVG on the server for the home page: one bar a day, stacked by how the runs ended. 1 struct · 1 function
  • orca.civil Calendar arithmetic: dates to days since 1970-01-01 and back. 1 struct · 4 functions
  • orca.cli orca_main: the command line every pipeline binary gets. 3 functions
  • orca.command The orca command: the scheduler, the UI and registration, in a binary that links no pipeline. A project builds its own from two lines: 1 function
  • orca.cron Cron expressions, and when they next fire in a time zone. 1 struct
  • orca.ctx Ctx: what a step gets besides its input. 3 structs
  • orca.dag A pipeline's DAG drawn as SVG, on the server. 1 struct · 2 functions
  • orca.day Day: the partition key of a daily pipeline. 1 struct
  • orca.ec2 Runs on EC2: a head instance and worker instances per run, started and ended by the scheduler (DESIGN.md, phase 5, PR 5). 2 structs · 1 function
  • orca.graph A directed graph drawn as SVG, on the server: the pipelines and datasets that feed each other, and a snapshot's lineage. 2 structs · 1 function
  • orca.heartbeat A run's heartbeat, kept fresh from a thread while the run's steps work. 1 struct · 2 functions
  • orca.html Writing HTML: every value escaped, markup only where it's said to be. 1 struct · 3 functions
  • orca.json The little JSON orca writes: quoted strings. 1 function
  • orca.key Key: what a partition key is, whatever it counts. 2 traits · 1 function
  • orca.keyed Keyed partitions: keys that are names, read from a column of a dataset. 1 struct
  • orca.kube Runs on Kubernetes: a dista cluster per run, made and removed by the scheduler (K8S.md). 2 structs · 1 function
  • orca.lineage Lineage: which snapshots a dataset's snapshot was made from, and what was made from it, from the reads and commits every attempt records. 5 structs · 6 functions
  • orca.locate Where a pipeline's step functions and dista tasks are defined in its source, so orca ui can link each one to its line at the commit a run ran. 3 functions
  • orca.logs Where attempts' logs, drivers' logs and runs' timelines go. 12 functions
  • orca.month Month: the partition key of a monthly pipeline. 1 struct · 2 functions
  • orca.name Name: the partition key of a keyed pipeline (orca.keyed). 1 struct · 3 functions
  • orca.partitions Partition sets: which keys a pipeline is partitioned by, from when. 2 structs · 1 trait · 1 function
  • orca.pipeline Pipelines: a named list of steps, checked and run in order. 3 structs
  • orca.proc Starting processes: what the scheduler needs to run pipeline binaries. 4 functions
  • orca.schedule Schedules: when a pipeline runs by itself, and which partition it fills. 4 structs · 1 trait
  • orca.scheduler The scheduler: fires schedules, senses datasets, queues backfills and rebuilds, and starts runs. 2 structs · 9 functions
  • orca.secrets Secrets a pipeline's runs get: orca register --secret NAME=ID. 3 functions
  • orca.sensors Runs started by datasets: sensors, and rebuilds downstream of a dataset. 2 structs · 8 functions
  • orca.source Where a pipeline binary's code came from: its git repository and commit, given at build time. 6 functions
  • orca.state orca's state in Postgres: pipelines, runs, steps and attempts. 4 structs · 1 function
  • orca.step Steps: the functions a pipeline runs, and the edges between them. 3 structs · 1 trait
  • orca.store Where the executor records what it does. 1 struct · 1 trait
  • orca.tables Iceberg tables from steps: pinned reads, and commits a retry recognises. 9 functions
  • orca.tasks The dista tasks a pipeline's steps call, and the runtime a run gets. 2 structs · 1 trait · 3 functions
  • orca.timeline A run's Dista timeline and its workers over time, drawn as SVG on the server for the run's page. 2 structs · 3 functions
  • orca.tz Time zones, from the system's TZif files (RFC 8536). 2 structs
  • orca.ui orca ui: pipelines and runs as web pages, and the same as JSON. 13 structs · 18 functions
  • orca.ui_style The stylesheet orca ui serves at /static/orca.css: magmalake.dev's colours and type, light and dark, and system fonts (nothing is fetched).
  • orca.warehouse A dataset's current snapshot, read from its table's metadata file without iceberg, for <binary> sense. 1 struct · 2 functions