Tins · Tables & catalogs
iceberg
Native Apache Iceberg. Read a table's metadata.json, pick a snapshot, decode its manifests, plan a scan and read the rows — then create, append, delete, overwrite and expire. No JVM, no Python, no Rust in the path.
What it does
Works today
- Format v1–v3 reads: position and equality deletes, v3 deletion vectors, schema evolution, nested types
- Writes: create, fast-append, delete (copy-on-write and merge-on-read), overwrite, dynamic partition overwrite, expire_snapshots
- Schema evolution as a write: add optional columns, rename, make optional, widen types and union_by_name, committed through every catalog
- REST, filesystem and SQL catalogs — the SQL one on SQLite or PostgreSQL, PyIceberg's schema; local, HTTP, S3, GCS and Azure IO
Not yet
- Compaction — rewrite_manifests
- Encryption
- Predicates inside list and map elements
The spec's v4 drafts — relative paths and content_stats — are tracked but not implemented. What is proposed but unbuilt lives in the issue tracker. The last spec there, a PostgreSQL tin, shipped as postgres.mojo and put the SQL catalog on a server.
Correctness
Row-sets cell-exact against PyIceberg 0.11.1 and DuckDB 1.5.5 across 48 of 48 filter cases, in both directions, including deletes, deletion vectors and schema evolution.
Tables magmalake writes are read back cell-exact by both — and PyIceberg can append to a table magmalake created.
Building it found bugs upstream
Holding three implementations against each other surfaces disagreements, and not all of them turned out to be my fault. I cut each one below down to a standalone script or unit test with no magmalake code in it, then either reported it upstream or — where someone had got there first — matched it to the report that already existed.
- iceberg-rust apache/iceberg-rust#3118 reported An In predicate whose literals straddle a file's bounds is never pruned: the lower and upper bounds are each tested against the whole literal set instead of narrowing it, as Java and PyIceberg do.
- PyIceberg apache/iceberg-python#3690 reproduced v3 manifests and manifest lists are read with the v2 projection, so first_row_id, referenced_data_file, content_offset and content_size_in_bytes are dropped on the way in.
- PyIceberg apache/iceberg-python#3620 reproduced The mirror image on the write path: the manifest writer builds its record schema at DEFAULT_READ_VERSION, so those same v3 fields would serialise as null.
- PyIceberg apache/iceberg-python#3833 reproduced A null list<struct> or map<K, struct> is rebuilt as an empty one — on the write path too, so the null is gone at rest and no reader can recover it.
Also in Tables & catalogs