Roadmap
Five phases, each ending at a gate
A gate is something that works, not a feature list: a phase is done when its gate passes. Phases 1 to 4 are done; running on Kubernetes is next. Phase 5 is planned, not built; what's written here about it is the design, and it changes as the earlier phases teach something. The detail is in the repository's VISION.md and DESIGN.md. As of 2 November 2026.
-
Declare steps and the datasets between them; run a DAG once, locally, in dependency order, with a Dista runtime per run; record runs and steps in Postgres.
Gate: The taxi load runs as a DAG (download, per-month load, compaction), and a run killed halfway resumes at the step that failed.
- The 24 months of 2023–2024 load to 79,478,796 rows, and every column's sum equals taxibench's table.
- A run killed with kill -9 while loading 2023-10, then resumed, loads each month once: 25 commits, one per month and the compaction.
# Pull request Its gate State 1 The design Agreed. merged 2 Steps and pipelines: compile-time edges, describe, an in-memory run A three-step pipeline runs over three partitions in order; the compile-fail tests are rejected. merged 3 State in Postgres: runs, steps and attempts, heartbeats, run, resume and status tests/kill.sh on the toy pipeline: killed at a random step, resumed, every step ran to success exactly once. merged 4 Pinned reads and recognised commits over Iceberg tables A step killed right after its commit, then resumed, leaves one snapshot, not two; a resumed step reads its pinned snapshot after another commit lands. merged 5 Dista inside steps, per-attempt logs and the run's timeline A step fans 100 Dista tasks over 4 workers; each attempt's log is where status says. merged 6 The taxi pipeline: download, load and compact each month of NYC taxi trips 2023–2024 loads to 79,478,796 rows, the same as taxibench's table, and a run killed during the load resumes at the failed month. merged -
Cron schedules, monthly and daily partitions, a scheduler, backfills, retries with backoff.
Gate: A schedule fills each month's partition as it falls due, and a backfill over 2011–2024 runs with a bounded number at once.
- orca backfill taxi 2011-01..2024-12: all 168 months loaded on their first start, never more than 4 at once, in 11 minutes on one 10-core Mac, to 1,438,321,685 rows, one file per month.
- With the clock set to 2 February 2025, a tick at 05:59 fired nothing; at 06:00 it fired once, for January, which downloaded from the TLC and loaded its 3,475,204 rows.
- It found a bug: iceberg never seeds its random snapshot IDs, so concurrent pipeline processes drew the same ones and overwrote each other's manifest lists. Orca now seeds each process, and iceberg's fix takes them from the OS.
# Pull request Its gate State 1 Partition keys and sets: Month and Day, Monthly and Daily, partitions= on the pipeline A daily pipeline runs over a range of days; a pipeline whose first step takes a Month under Daily doesn't compile. merged 2 Cron and time zones: next fires, a TZif reader with POSIX TZ rules, schedule= on the pipeline Next fires match a table of known ones across both daylight-saving transitions in New York and London; the parsers are fuzzed. merged 3 The scheduler: register, tick, scheduler and backfill; max_runs, retries with backoff, missed fires With the clock stepped month by month, partitions fill once each; racing ticks fire nothing twice; a backfill never has more than max_runs running. merged 4 The taxi backfill: every month from 2011, on a monthly schedule orca backfill taxi 2011-01..2024-12 with max_runs=4 loads every month once, and the schedule then fills the next month when it falls due. merged -
Pipelines, runs, steps, logs and the Dista timeline, served by flare; start, rerun and backfill from the browser.
Gate: A failed step is found, its logs read and the run resumed, without a terminal.
- A toy run fails a step; the test finds it on /runs, reads the failed attempt's log and posts resume, and with the scheduler running the run succeeds, the failed step on its second attempt and nothing else run again.
- A rerun from measure copies days from the old run and runs measure and total; a cancelled running run stops before its next step and ends cancelled; a resume onto new code waits for confirmation.
- A POST from another site's page, or with neither Origin nor Referer, is refused and changes nothing.
# Pull request Its gate State 1 Read-only pages and their JSON: orca ui on flare, an escaping HTML writer, pipelines and runs Every page answers HTML and JSON; a run's JSON agrees with status; a step's error with <script> in it comes out as text. merged 2 Logs, the DAG and live updates: attempt pages, the DAG as SVG, polled fragments, the Dista timeline A running run's page follows it to succeeded and stops polling; the DAG has a box per step and an edge per after. merged 3 Actions: start, resume, rerun from a step, backfill and cancel, as rows the scheduler acts on Phase 3's gate, over HTTP only: a failed step found, its log read, the run resumed and succeeding under the scheduler. merged -
Datasets declared on steps; queries and a graph over the reads and commits Orca records; runs started by a new upstream snapshot, and rebuilds downstream of a table.
Gate: Committing a new snapshot to a source table rebuilds exactly the tables downstream of it.
- With the scheduler running, a program that isn't an Orca pipeline commits to raw.events: the sensor clean runs for the partition the snapshot holds, its commit starts the sensor report, and a sensor over an unrelated dataset never runs.
- Each snapshot starts one run, however many ticks see it, three of them racing the scheduler; what the tables held when the sensors were registered starts nothing.
- orca rebuild raw.events 2024-03 reruns clean, then report once clean has ended, and its commits start no sensor: six runs in all.
- Testing found two bugs: two writers creating a table at once could livelock retrying their commits, now retried with jittered backoff, and fuzzing the metadata reader found a read past the end of the text.
# Pull request Its gate State 1 Datasets on steps: reads= and writes=, checked when a step touches a table, and the cycle check at registration A step that scans a dataset it didn't declare fails naming both; registering a pipeline that closes a cycle is refused. merged 2 Lineage: back from a snapshot and forward from it, orca lineage, and the dataset pages with their graph Over ingest, clean and report, lineage back from a report snapshot names its run, its code and its inputs, and agrees with the snapshots' own summaries. merged 3 Sensors and rebuild: sensor=True, <binary> sense, one run per snapshot, orca rebuild and the UI's form Phase 4's gate: a snapshot committed outside Orca starts clean, whose commit starts report, and nothing else; a rebuild reruns both in order. merged -
The scheduler as a Deployment; a Dista cluster per run, created and deleted through the Kubernetes API.
Gate: The taxi pipeline runs on a schedule on EC2, each run on its own cluster.
- A head, a worker set and a driver Job per run, labelled with the run ID, deleted when it ends
- Logs and timelines in S3
- Cluster start time measured; a warm pool only if it dominates short runs
Measured so far: phase 1's gate, the 2023–2024 NYC taxi load, 79,478,796 rows, checked column by column against taxibench's table, and phase 2's, every month since 2011, 1,438,321,685 rows, backfilled four runs at a time in 11 minutes. Those are the same rows Dista's benchmarks query.