`orca.lineage`

Lineage: which snapshots a dataset's snapshot was made from, and what was made from it, from the reads and commits every attempt records.

orca lineage lake.report            the latest snapshot's, back and forward
orca lineage lake.report 4417...    one snapshot's

A snapshot that an orca attempt committed is made by that attempt: its run, pipeline, step, partition and code, and the snapshots it read, its inputs. The walks go over those edges in SQL (recursive CTEs):

back from a snapshot: the attempt that made it, then the attempts that made each of its inputs, and so on, to inputs no orca attempt made (sources: a table written by something else, or a read of a dataset before it had any snapshot, which isn't listed).
forward from a snapshot: every snapshot made by an attempt that read it, then what was made from those, and so on.

Both stop at depth steps (default 20), so a deep graph is cut rather than slow. A snapshot recorded by two attempts (a retry that recognised its earlier attempt's commit) is listed once, as the nearer and earlier. Only datasets take part: a table outside the warehouse has no name to follow. The same edges are in the snapshots' own summaries, as orca.reads.

Structs

`SnapshotRef`

struct SnapshotRef

Implements Copyable, Equatable, Movable, Writable

Fields: dataset: String; snapshot: Int64

`eq`

def __eq__(self, other: Self) -> Bool

`write_to`

def write_to(self, mut writer: T)

`Made`

struct Made

A snapshot, the attempt that made it, and what that attempt read.

Implements Copyable, Movable

Fields: at: SnapshotRef; depth: Int: Steps from the snapshot the walk started at: 0 for it, 1 for its inputs' (back) or for what was made from it (forward).; pipeline: String; run: Int; step: String; partition: String; attempt: Int; code: String; ended: String; inputs: List[SnapshotRef]

`Dataset`

struct Dataset

Implements Copyable, Movable

Fields: name: String; writers: List[String]: The pipelines whose steps declare it in writes=.; readers: List[String]; commits: Int; latest: Int64: The newest snapshot an orca attempt committed, or -1.; latest_run: Int

`Flow`

struct Flow

An edge of the graph of pipelines and datasets: a pipeline reads a dataset, or writes one.

Implements Copyable, Movable

Fields: pipeline: String; dataset: String; writes: Bool

`Commit`

struct Commit

A snapshot of a dataset an orca attempt committed.

Implements Copyable, Movable

Fields: snapshot: Int64; pipeline: String; run: Int; step: String; partition: String; ended: String

Functions

`back`

def back(mut db: State, start: SnapshotRef) -> List[Made]

The snapshot, if an orca attempt made it, and everything it was made from: one Made per snapshot an orca attempt made.

`forward`

def forward(mut db: State, start: SnapshotRef) -> List[Made]

Every snapshot made from start, directly or not.

`latest`

def latest(mut db: State, dataset: String) -> Int64

The dataset's newest snapshot orca knows of, committed or read, or -1 if it knows of none.

`datasets`

def datasets(mut db: State, only: String = "") -> List[Dataset]

Every dataset a pipeline declares or an attempt touched, or the one named, by name.

`flows`

def flows(mut db: State) -> List[Flow]

Every pipeline's declared reads and writes.

`commits`

def commits(mut db: State, dataset: String, limit: Int = Int(50)) -> List[Commit]

The dataset's snapshots orca committed, newest first.

Aliases

comptime DEPTH = 20