magmalake arrow-mlake

arrow-mlake · API

arrow_mlake.flatbuf

FlatBuffers, as much as Arrow IPC metadata needs: a builder and a reader.

The builder came from flight.mojo, where it wrote Schema and RecordBatch messages. Writing means we choose the layout, so it only has to be self-consistent and spec-legal. Buffers are built back to front: _head starts at the end of the storage and walks downwards, so the root table ends up nearest the front and every child it points at is already written behind it. offset() is "bytes written so far", measured from the end, and every stored offset is a backwards distance from the field to its target, which is why _ref subtracts.

A table is written as: field data, then a vtable listing where each field landed, then a signed backwards offset from the table to that vtable. Absent fields are simply not written. There is no vtable deduplication; Arrow messages are small.

The reader takes whatever layout a producer chose, so it trusts nothing: every offset is bounds-checked before it is followed, and a field the vtable doesn't reach reads as absent. It copies the message into a FlatReader rather than borrowing it, because IPC metadata is a few hundred bytes and an owned buffer keeps origins out of the API.

Structs

FlatBufferBuilder

struct FlatBufferBuilder

Builds one FlatBuffer, back to front.

Not reusable: finish leaves the builder holding the completed buffer, and the intended lifecycle is one builder per message.

Implements Movable

__init__

def __init__(out self, initial: Int = Int(1024))

offset

def offset(self) -> Int

Bytes written so far, which is also the position of the next write measured from the end of the buffer.

prepend_u8

def prepend_u8(mut self, v: UInt8)

prepend_bool

def prepend_bool(mut self, v: Bool)

prepend_i16

def prepend_i16(mut self, v: Int)

prepend_i32

def prepend_i32(mut self, v: Int)

prepend_i64

def prepend_i64(mut self, v: Int)

start_table

def start_table(mut self, num_slots: Int)

add_bool

def add_bool(mut self, slot: Int, v: Bool, default: Bool = False)

Add a field, unless it already holds the schema default.

Omitting defaults is not just a size trick: a reader that asks for an absent field gets the default back, so writing it would be identical but larger.

add_i16

def add_i16(mut self, slot: Int, v: Int, default: Int = Int(0))

add_i32

def add_i32(mut self, slot: Int, v: Int, default: Int = Int(0))

add_i64

def add_i64(mut self, slot: Int, v: Int, default: Int = Int(0))

add_u8

def add_u8(mut self, slot: Int, v: UInt8, default: UInt8 = UInt8(0))

add_offset

def add_offset(mut self, slot: Int, target: Int)

Add a reference field (table, string or vector). 0 means absent.

end_table

def end_table(mut self) -> Int

Finish the table and return its offset.

Writes the vtable, then the table's own backwards pointer to it. The vtable is trimmed to the last field actually set, which is what lets a reader of a longer schema see the extra fields as absent rather than as garbage.

create_string

def create_string(mut self, s: String) -> Int

Write a length-prefixed, NUL-terminated UTF-8 string.

start_vector

def start_vector(mut self, elem_size: Int, count: Int, align: Int)

Begin a vector; elements are prepended in reverse order.

end_vector

def end_vector(mut self) -> Int

create_offset_vector

def create_offset_vector(mut self, offsets: List[Int]) -> Int

A vector of references, written back to front.

finish

def finish(mut self, root: Int) -> List[UInt8]

Write the root offset and return the finished buffer.

FlatReader

struct FlatReader

One FlatBuffer, read through absolute byte positions.

Tables, vectors and strings are named by where they start in the buffer. A position of 0 means absent: no table can start there, because the buffer starts with the root offset.

Implements Movable

Fields
buf: List[UInt8]

__init__

def __init__(out self, var buf: List[UInt8])

u8

def u8(self, pos: Int) -> Int

u16

def u16(self, pos: Int) -> Int

u32

def u32(self, pos: Int) -> Int

i16

def i16(self, pos: Int) -> Int

i32

def i32(self, pos: Int) -> Int

i64

def i64(self, pos: Int) -> Int

root

def root(self) -> Int

Where the root table starts.

field

def field(self, table: Int, slot: Int) -> Int

Where field slot of table is stored, or 0 if it's absent.

get_bool

def get_bool(self, table: Int, slot: Int, default: Bool = False) -> Bool

get_u8

def get_u8(self, table: Int, slot: Int, default: Int = Int(0)) -> Int

get_i16

def get_i16(self, table: Int, slot: Int, default: Int = Int(0)) -> Int

get_i32

def get_i32(self, table: Int, slot: Int, default: Int = Int(0)) -> Int

get_i64

def get_i64(self, table: Int, slot: Int, default: Int = Int(0)) -> Int

get_ref

def get_ref(self, table: Int, slot: Int) -> Int

Follow a table, vector or string field; 0 if it's absent.

get_string

def get_string(self, table: Int, slot: Int) -> String

A string field, or "" if it's absent.

vector_len

def vector_len(self, vector: Int) -> Int

vector_table

def vector_table(self, vector: Int, i: Int) -> Int

Element i of a vector of tables.

vector_struct

def vector_struct(self, vector: Int, i: Int, size: Int) -> Int

Where element i of a vector of size-byte structs starts.