magmalake iceberg

iceberg · API

iceberg.values

Typed single values (Datum) and Iceberg's two single-value encodings.

A Datum is one primitive value tagged with its Iceberg type. It is what predicates compare against, what a partition transform consumes and produces, and what a manifest's lower_bound / upper_bound bytes decode into.

Two serializations matter for reading:

  • Appendix D, binary — lower_bounds, upper_bounds and partition field summaries. datum_from_bytes / datum_to_bytes.
  • Appendix D, JSON — initial-default, write-default and the JSON forms of partition values. datum_from_json.

Integers, dates, times and timestamps all ride in Datum.i (Int64); floats and doubles in Datum.f; strings in Datum.s; everything byte-shaped (binary, fixed, uuid, and a decimal's unscaled two's-complement) in Datum.b.

Decimals carry both forms: b holds the minimal big-endian two's-complement unscaled value (what bucket hashing and the binary encoding need) and i holds the same value as an Int64 when precision ≤ 18 (what truncation needs). Comparison always goes through the bytes, so decimals wider than 64 bits still compare correctly even though this build cannot do arithmetic on them.

Structs

Datum

struct Datum

One primitive Iceberg value, tagged with its type.

Implements Copyable, Movable, Writable

Fields
kind: UInt8
i: Int64
f: Float64
s: String
b: List[UInt8]
precision: Int
scale: Int
length: Int
valid: Bool: False for "no value" — an absent bound, a null partition value.

none

def none() -> Self

bool_

def bool_(v: Bool) -> Self

int_

def int_(v: Int64) -> Self

long_

def long_(v: Int64) -> Self

integral

def integral(kind: UInt8, v: Int64) -> Self

int/long/date/time/timestamp[_ns][tz] — all carried as Int64.

float_

def float_(v: Float64) -> Self

double_

def double_(v: Float64) -> Self

string_

def string_(var v: String) -> Self

binary_

def binary_(var v: List[UInt8]) -> Self

fixed_

def fixed_(var v: List[UInt8]) -> Self

uuid_

def uuid_(var v: List[UInt8]) -> Self

decimal_

def decimal_(var unscaled_be: List[UInt8], precision: Int, scale: Int) -> Self

decimal_int

def decimal_int(unscaled: Int64, precision: Int, scale: Int) -> Self

is_nan

def is_nan(self) -> Bool

write_to

def write_to(self, mut writer: T)

repr_

def repr_(self) -> String

to_json

def to_json(self) -> String

Appendix D, JSON half.

Functions

compare

def compare(a: Datum, b: Datum) -> Int

Iceberg's natural order: -1 / 0 / 1. Both values must be comparable.

cmp_unsigned

def cmp_unsigned(a: List[UInt8], b: List[UInt8]) -> Int

Lexicographic unsigned-byte order (uuid, fixed, binary).

cmp_be_twos

def cmp_be_twos(a: List[UInt8], b: List[UInt8]) -> Int

Signed compare of two minimal big-endian two's-complement integers.

be_twos_to_int64

def be_twos_to_int64(b: List[UInt8]) -> Int64

Big-endian two's complement → Int64. Wider-than-64-bit input saturates.

int64_to_be_twos

def int64_to_be_twos(v: Int64) -> List[UInt8]

Int64 → the minimal big-endian two's-complement byte string.

hex_text

def hex_text(b: List[UInt8]) -> String

hex_bytes

def hex_bytes(s: String) -> List[UInt8]

uuid_text

def uuid_text(b: List[UInt8]) -> String

uuid_bytes

def uuid_bytes(s: String) -> List[UInt8]

decimal_text

def decimal_text(unscaled_be: List[UInt8], scale: Int) -> String

Render a decimal from its unscaled two's-complement value and scale.

decimal_from_text

def decimal_from_text(s: String, precision: Int, scale: Int) -> Datum

Parse decimal text at the column's scale, to full 38-digit width.

floor_div

def floor_div(a: Int64, b: Int64) -> Int64

Division that rounds toward negative infinity.

Mojo's // on integers already floors (Python semantics, verified on both stable 1.0.0 and the nightly: -1 // 86400000000 == -1, -7 // 2 == -4), so this is a rename that documents the intent. Every date and timestamp calculation in Iceberg needs floor, not truncation, or pre-epoch values land a day early.

floor_mod

def floor_mod(a: Int64, b: Int64) -> Int64

Remainder with the sign of the divisor — % already does this in Mojo.

civil_from_days

def civil_from_days(z_in: Int64) -> List[Int64]

Days since 1970-01-01 → [year, month, day]. Howard Hinnant's algorithm.

days_from_civil

def days_from_civil(y_in: Int64, m: Int64, d: Int64) -> Int64

iso_text

def iso_text(kind: UInt8, v: Int64) -> String

ISO-8601 rendering used by the JSON single-value form.

parse_iso

def parse_iso(kind: UInt8, text: String) -> Int64

ISO-8601 → the integer Iceberg stores for kind.

datum_from_bytes

def datum_from_bytes(store: TypeStore, type_idx: Int, b: List[UInt8]) -> Datum

Decode Appendix-D binary bytes as the type at type_idx.

Byte length disambiguates promoted columns: a long column whose bounds were written while it was still an int carries 4 bytes, a double promoted from float carries 4, and a timestamp promoted from date carries 4. The spec requires readers to infer the original type from the length rather than mis-decode.

datum_from_bytes_prim

def datum_from_bytes_prim(prim: UInt8, precision: Int, scale: Int, length: Int, b: List[UInt8]) -> Datum

datum_to_bytes

def datum_to_bytes(d: Datum) -> List[UInt8]

Encode a Datum in Appendix-D binary form.

datum_from_json

def datum_from_json(store: TypeStore, type_idx: Int, doc: Json, i: Int) -> Datum

datum_from_json_prim

def datum_from_json_prim(prim: UInt8, precision: Int, scale: Int, length: Int, doc: Json, i: Int) -> Datum

Aliases

comptime MICROS_PER_DAY = Int64(86400000000)
comptime MICROS_PER_HOUR = Int64(3600000000)
comptime NANOS_PER_MICRO = Int64(1000)