magmalake iceberg

iceberg · API

iceberg.batch

Building an Arrow RecordBatch from Mojo values.

The write path takes Arrow, which is the right input when the data already is Arrow — a scan of another table, a Parquet file, anything speaking the C Data Interface. When it is not, this is the bridge: a ColumnBuilder per column, a Datum per cell, and batch_of to tie them together.

var ids = ColumnBuilder(String("id"), 1, P_LONG)
var region = ColumnBuilder(String("region"), 2, P_STRING)
for k in range(n):
    ids.add(Datum.long_(Int64(k)))
    region.add(Datum.string_(names[k]))
var batch = batch_of([ids^, region^])

This is deliberately the slow side of the library — one tagged value per cell, which is exactly what iceberg.kernels exists to avoid on the read path. It is here because the alternative is asking a caller to lay out validity bitmaps and offset buffers by hand, and a batch built once and written once is not where the time goes.

Structs

ColumnBuilder

struct ColumnBuilder

One column's values, before they become an Arrow array.

Implements Copyable, Movable, Sized

Fields
name: String
field_id: Int
kind: UInt8
precision: Int
scale: Int
length: Int
nullable: Bool
values: List[Datum]

__init__

def __init__(out self, var name: String, field_id: Int, kind: UInt8, precision: Int = Int(0), scale: Int = Int(0), length: Int = Int(0), nullable: Bool = True)

of

def of(schema: Schema, field_id: Int) -> Self

A builder typed by one of the table's columns.

add

def add(mut self, var value: Datum)

add_null

def add_null(mut self)

__len__

def __len__(self) -> Int

build_tree

def build_tree(self) -> ColumnTree

The same array, ready to sit beside a nested one.

build

def build(self) -> ArrayData

NestedBuilder

struct NestedBuilder

A struct, list or map column, one JSON value per row.

A nested cell has no scalar to carry it, so the input here is text: the same JSON shape ScanResult.cell prints, which makes a round trip through this library one string per cell.

var addr = NestedBuilder.of(schema, 4)
addr.add('{"city":"eu","zip":10}')
addr.add_null()
var batch = batch_of_columns([ids^.build_tree(), addr^.build()])

Structs are objects keyed by field name, lists are arrays, and a map is either {"keys": [...], "values": [...]} — the spec's own shape — or a plain object when its keys are strings.

Implements Copyable, Movable, Sized

Fields
name: String
field_id: Int
ctype: ColumnType
nullable: Bool
rows: List[String]

__init__

def __init__(out self, var name: String, field_id: Int, var ctype: ColumnType, nullable: Bool = True)

of

def of(schema: Schema, field_id: Int) -> Self

A builder typed by one of the table's columns.

add

def add(mut self, var json: String)

add_null

def add_null(mut self)

__len__

def __len__(self) -> Int

build

def build(self) -> ColumnTree

Functions

array_from_datums

def array_from_datums(name: String, field_id: Int, kind: UInt8, precision: Int, scale: Int, length: Int, nullable: Bool, values: List[Datum]) -> ArrayData

One Arrow array from a column of tagged values.

is_var_width_array

def is_var_width_array(a: ArrayData) -> Bool

append_datum

def append_datum(mut a: ArrayData, d: Datum, kind: UInt8)

Append one value to a primitive array, growing every buffer it has.

finish_array

def finish_array(mut a: ArrayData)

Pad the buffers a partly filled last byte leaves short.

batch_of

def batch_of(columns: List[ColumnBuilder]) -> RecordBatch

One RecordBatch from a set of equally long column builders.

push_json

def push_json(mut arena: ArrayArena, node: Int, store: TypeStore, ti: Int, doc: Json, at: Int)

Append one JSON value to a (possibly nested) Arrow array.

finish_tree

def finish_tree(mut arena: ArrayArena, node: Int)

Pad every buffer in a freshly built tree.

batch_of_columns

def batch_of_columns(var columns: List[ColumnTree]) -> RecordBatch

One RecordBatch from column trees, nested or not.