Tins · iceberg

API reference

The iceberg package, from iceberg 0.8.6, as published on mojoshelf (commit ec4f4e07605b), by mojo doc from Mojo 1.2.0. Source.

Native Apache Iceberg: metadata, planning, and rows — the iceberg package of iceberg.mojo.

Part of magmalake: data lake building blocks in Mojo.

from iceberg.catalog.filesystem import Table

def main() raises:
    var t = Table.load_local("/warehouse/db/orders")
    var rows = t.scan().filter('[">","id",2]').to_table()
    print(rows.to_csv())

Reads format versions 1, 2 and 3, and tolerates the parts of v4 the spec already tells readers to tolerate: metadata, snapshots, partition transforms, expressions, manifests, scan planning, Puffin deletion vectors, and the data files themselves — over local files, S3, GCS, Azure or HTTP, from a filesystem layout or a live REST catalog.

Writes v2 and v3: create_table, append, delete_where (copy-on-write, or merge-on-read with deletion vectors on v3 and position delete files on v2), delete_by_equality, overwrite, dynamic_partition_overwrite and expire_snapshots, each committed optimistically against a filesystem layout or a REST catalog — and schema changes (UpdateSchema: add, rename, make optional, widen) through the same commit paths.

Modules

iceberg.append Fast append: new data files, a new manifest, a new snapshot. 1 struct · 4 functions
iceberg.batch Building an Arrow RecordBatch from Mojo values. 2 structs · 8 functions
iceberg.catalog Catalogs: how a table name becomes a TableMetadata.
iceberg.catalog.filesystem Filesystem tables: a table is a directory, no catalog service involved. 3 structs · 5 functions
iceberg.catalog.rest Iceberg REST catalog — URLs, headers, responses, and the socket. 4 structs · 4 functions
iceberg.catalog.sql A JDBC-style SQL catalog, over sqlite.mojo or postgres.mojo — the schema and semantics PyIceberg's SqlCatalog uses. 2 structs
iceberg.commit A snapshot that adds and removes files, with the manifest maintenance that implies. 1 struct · 1 function
iceberg.delete Row-level delete and overwrite: which rows go, and how they are recorded. 1 struct · 14 functions
iceberg.expressions Row filters: unbound and bound predicates, binding, projection, evaluators. 7 structs · 9 functions
iceberg.io File access: locations in, bytes out — over local files, S3, GCS, Azure or plain HTTP. 1 struct · 5 functions
iceberg.json A small, dependency-free JSON reader/writer for Iceberg metadata. 2 structs · 7 functions
iceberg.kernels Columnar kernels: the Arrow arrays a scan produces, without a Datum in sight. 33 functions
iceberg.maintain Expiring snapshots, and deleting the files that nothing points at any more. 1 struct · 3 functions
iceberg.manifest Manifest lists and manifests, read through avro.mojo. 5 structs · 12 functions
iceberg.manifest_write Writing manifests and manifest lists. 2 structs · 13 functions
iceberg.metadata TableMetadata — the whole of an Iceberg *.metadata.json, v1 through v3. 9 structs
iceberg.nested Nested columns — struct, list and map — as Arrow trees. 2 structs · 17 functions
iceberg.puffin Puffin files: the sidecar format Iceberg keeps deletion vectors and sketches in. 3 structs · 4 functions
iceberg.read Reading rows: TableScan.to_table() and to_batches(). 5 structs · 9 functions
iceberg.scan Scan planning: which data files a query must read, and which deletes apply. 3 structs
iceberg.schema Schema — an Iceberg struct type with a schema id and identifier fields. 2 structs
iceberg.transforms Partition transforms and partition specs. 5 structs · 6 functions
iceberg.types Iceberg data types — primitives, nested types, and their JSON forms. 3 structs · 3 functions
iceberg.update_schema Schema evolution: add columns, rename them, make them optional, widen them. 2 structs · 2 functions
iceberg.util The three things a writer needs from outside itself: the wall clock, a random 64-bit number, and a UUID. 5 functions
iceberg.values Typed single values (Datum) and Iceberg's two single-value encodings. 1 struct · 22 functions
iceberg.write Writing data files: Arrow in, Parquet out, a DataFile record back. 1 struct · 11 functions