`orca.partitions`

Partition sets: which keys a pipeline is partitioned by, from when.

comptime Taxi = Pipeline[
    "taxi", Download, Load, Compact, partitions=Monthly["2011-01"]
]
comptime Weather = Pipeline["weather", Fetch, partitions=Daily["2020-01-01"]]
comptime Labels = Pipeline["labels", Label, partitions=Keyed["av2.logs", "log_id"]]

A pipeline's first step takes its key (Month for Monthly, Day for Daily, Name for orca.keyed.Keyed), or a Seq of them for a first gather step. start is the first partition: keys before it are refused, and phase 2's scheduler and backfills count from it. A pipeline without partitions= is Monthly[], with no start.

Each set reads --partitions itself (parse): time keys take keys and ranges, keyed partitions lists of names. Only time keys have a calendar, so only they fill a schedule's partition, and only keyed partitions have current keys, read from their dataset.

Structs

`Monthly`

struct Monthly[first: StringSpan[ImmStaticOrigin] = StringSpan("")]

A partition per month, from first (YYYY-MM) if given.

Implements Movable, Partitions

comptime K = Month

`kind`

def kind() -> StaticString

`start`

def start() -> StaticString

`dataset`

def dataset() -> StaticString

`column`

def column() -> StaticString

`parse`

def parse(s: StringSpan, check: Bool) -> List[Month]

`fill`

def fill(year: Int, month: Int, day: Int, previous: Bool) -> Month

`keys`

def keys() -> Tuple[Int64, List[String]]

`Daily`

struct Daily[first: StringSpan[ImmStaticOrigin] = StringSpan("")]

A partition per day, from first (YYYY-MM-DD) if given.

Implements Movable, Partitions

comptime K = Day

`kind`

def kind() -> StaticString

`start`

def start() -> StaticString

`dataset`

def dataset() -> StaticString

`column`

def column() -> StaticString

`parse`

def parse(s: StringSpan, check: Bool) -> List[Day]

`fill`

def fill(year: Int, month: Int, day: Int, previous: Bool) -> Day

`keys`

def keys() -> Tuple[Int64, List[String]]

Traits

`Partitions`

comptime K

The partition key.

`kind`

def kind() -> StaticString

monthly, daily or keyed, as describe writes it.

`start`

def start() -> StaticString

The first partition, as text; empty for none.

`dataset`

def dataset() -> StaticString

The dataset keyed partitions' keys come from; empty for time.

`column`

def column() -> StaticString

`parse`

def parse(s: StringSpan, check: Bool) -> List[Self.K]

The keys s names, as run --partitions takes them. With check, keyed partitions refuse a name their dataset doesn't have now; a resume doesn't check, as the run's keys were checked when it was made.

`fill`

def fill(year: Int, month: Int, day: Int, previous: Bool) -> Self.K

The key a date falls in, or the one before it; raises for keys that aren't dates.

`keys`

def keys() -> Tuple[Int64, List[String]]

The keys dataset's current snapshot and its keys, sorted; raises for time keys.

Functions

`parse_partitions`

def parse_partitions[P: Partitions](s: StringSpan) -> List[P.K]

The keys of P that s names, as run --partitions takes them: for time keys a key, A..B, or a comma-separated list of either; for keyed partitions a list of names, @all or @file:<path>.