# iceberg.kernels > A module of iceberg's API. Columnar kernels: the Arrow arrays a scan produces, without a `Datum` in sight. Everything here operates on parquet.mojo's `ArrayData` — one Arrow array, its validity bitmap and its buffers — and produces another one. A scan uses them in this order: 1. **cast** — the file's physical column is retagged or widened to the type the table's *current* schema says the column has (`int` -> `long`, `float` -> `double`, a decimal whose precision grew, a timestamp whose unit differs, `large_utf8` -> `utf8`); 2. **constant** — a column the file does not have becomes an array of one repeated value: an identity partition value, an `initial-default`, a metadata column like `_file`, or all-null; 3. **select** — deletes and the residual predicate produce one `List[Bool]` per batch, and `filter_array` applies it to every output column in one pass; 4. **concat** — batches are stitched together when a caller wants one table instead of a stream. None of this materialises a per-cell tagged value, which is the whole point: the previous implementation built a `Datum` for every cell of every column it read, and that cost about 70x. ## Functions ### `is_binary_type` ```mojo def is_binary_type(t: ArrowType) -> Bool ``` Variable-width with 32-bit offsets. ### `is_large_binary_type` ```mojo def is_large_binary_type(t: ArrowType) -> Bool ``` Variable-width with 64-bit offsets. ### `is_var_width` ```mojo def is_var_width(t: ArrowType) -> Bool ``` ### `compare_class` ```mojo def compare_class(t: ArrowType) -> Int ``` Which comparison kernel, if any, serves this Arrow type. ### `arrow_type_for` ```mojo def arrow_type_for(kind: UInt8, precision: Int, scale: Int, length: Int) -> ArrowType ``` ### `empty_array` ```mojo def empty_array(name: String, field_id: Int, var t: ArrowType) -> ArrayData ``` ### `constant_array` ```mojo def constant_array(name: String, field_id: Int, kind: UInt8, precision: Int, scale: Int, length: Int, d: Datum, n: Int) -> ArrayData ``` `n` copies of one value — a partition constant, a default, or all-null. ### `encode_cell` ```mojo def encode_cell(d: Datum, t: ArrowType, kind: UInt8) -> List[UInt8] ``` One value as the bytes Arrow stores for it. ### `decimal_le16` ```mojo def decimal_le16(d: Datum) -> List[UInt8] ``` A decimal's unscaled value as Arrow's little-endian 16-byte two's complement. ### `int64_array` ```mojo def int64_array(name: String, values: List[Int64]) -> ArrayData ``` A non-null `int64` column — `_pos`, `_row_id` and friends. ### `cast_array` ```mojo def cast_array(var a: ArrayData, target: ArrowType, name: String, field_id: Int) -> ArrayData ``` The file's column, produced at the table's current type. Iceberg only ever widens: `int` -> `long`, `float` -> `double`, and a decimal whose precision grew (which Arrow stores at a fixed 16 bytes either way, so only the tag changes). Everything else here is a retag — the same bytes under a different Arrow type — which is what makes a full scan of an unevolved table copy nothing at all. ### `int_at` ```mojo def int_at(a: ArrayData, i: Int) -> Int64 ``` The integral value at `i`, whatever fixed width it was stored at. ### `float_at` ```mojo def float_at(a: ArrayData, i: Int) -> Float64 ``` ### `value_extent` ```mojo def value_extent(a: ArrayData, i: Int) -> Tuple[Int, Int] ``` `(start, end)` of the raw bytes of element `i` in `a.values`. ### `bytes_at` ```mojo def bytes_at(a: ArrayData, i: Int) -> List[UInt8] ``` ### `extract_datum` ```mojo def extract_datum(a: ArrayData, i: Int, kind: UInt8, precision: Int, scale: Int, length: Int) -> Datum ``` Element `i`, typed as the table's current schema says it is. This is the only place a scan builds a per-cell tagged value, and it is built on demand: `ScanResult.value`, CSV and JSON output, the partition transforms a write applies, and the handful of predicate shapes no vectorised kernel covers. ### `sel_uninit` ```mojo def sel_uninit(n: Int) -> List[Bool] ``` A selection vector of `n` rows, every one of them uninitialised. Zeroing a vector that the kernel about to run overwrites in full is a second pass over the same memory for nothing: at 0.23 ns/row it cost more than the comparison it preceded, and a three-predicate scan of the taxi table paid it three times. **Every caller must write all `n` elements** — the leaf paths in `read.mojo` do, on both the vectorised branch and the `Datum` fallback that follows a refusal. ### `sel_count` ```mojo def sel_count(keep: List[Bool]) -> Int ``` How many rows a selection vector keeps. A branch per row cost 0.60 ns/row — 47 ms on every 79.5M-row scan of the taxi table, filtered or not — for a number the loop that produced the vector could have carried. Summing the low bits sixteen at a time costs 0.033 ns/row instead. The per-lane accumulator is 32-bit, which cannot overflow: it sees one batch, and a batch would need 34 billion rows. ### `sel_any` ```mojo def sel_any(keep: List[Bool]) -> Bool ``` Whether any row survives. Stops at the first one that does. ### `sel_and` ```mojo def sel_and(mut left: List[Bool], right: List[Bool]) ``` `left &= right`, row-wise. ### `sel_or` ```mojo def sel_or(mut left: List[Bool], right: List[Bool]) ``` `left |= right`, row-wise. ### `sel_not` ```mojo def sel_not(mut keep: List[Bool]) ``` `keep = not keep`, row-wise. Only the low bit of each byte carries. ### `sel_next_true` ```mojo def sel_next_true(keep: List[Bool], start: Int) -> Int ``` The first kept row at or after `start`, or `len(keep)`. Sixteen rows are tested at a time so that a highly selective filter walks its own gaps at memory speed rather than one branch per row. On the taxi suite's most selective query that turned a 79.5M-iteration scan per output column into 5M vector tests. ### `sel_next_false` ```mojo def sel_next_false(keep: List[Bool], start: Int) -> Int ``` The first dropped row at or after `start`, or `len(keep)`. ### `sel_compare` ```mojo def sel_compare[store: DType, compare: DType](values: Span[UInt8], n: Int, lit: Scalar[compare], lt: Bool, eq: Bool, gt: Bool, mut out: List[Bool]) ``` `out[r] = accept(cmp(values[r], lit))` over a fixed-width column. `store` is how the column is laid out and `compare` is the width Iceberg orders it at: a narrow integer widens to `int64` and a `float` widens to `double`, which is what makes the comparison exact rather than an approximation of the literal in the column's own width. The widening is a register move, and the loop stays bound by the values buffer. Iceberg orders NaN *above* every number, where IEEE makes it unordered, so a NaN lane fails all three machine comparisons and is folded into the greater-than case by hand. A NaN *literal* is not handled here — the caller keeps the scalar path for that, because it inverts the rule. Nulls are not this kernel's business: `sel_apply_validity` clears them afterwards, which costs one byte read per eight rows instead of a branch per row. ### `sel_apply_validity` ```mojo def sel_apply_validity(mut out: List[Bool], validity: Span[UInt8], n: Int) ``` Drop the rows a comparison kernel judged without looking at nulls. A null satisfies no comparison, so this is an unconditional clear rather than a re-evaluation. An all-valid byte is skipped and an all-null byte clears eight rows at once, which is why carrying nulls costs nothing on the columns that have none and very little on the columns that do. ### `sel_compact` ```mojo def sel_compact[dtype: DType](values: Span[UInt8], keep: List[Bool], n: Int, n_keep: Int, mut out: List[UInt8]) ``` Copy the kept elements of a fixed-width buffer into `out`, in order. The run-based copy above is the right shape for a range predicate over a sorted column, where a run is thousands of rows and one `extend` moves it. It is the wrong shape for an equality predicate over a scattered column: `payment_type = 2` keeps 15% of the taxi table in runs averaging barely more than one row, and the per-run bookkeeping — a capacity check and a call — then costs about twenty times the eight bytes it moves. So this loop does not look for runs at all. It stores every element unconditionally and advances the write cursor only for the kept ones, which is branch-free, and it skips sixteen rows at a time when none of them survive, which is what keeps a highly selective filter cheap. The speculative store is why `out` is allocated one element longer than the answer: the cursor is written before it is advanced, so the last dropped row would otherwise write one past the end. One element, and not one per input row — sizing the buffer to the input instead cost a 26 MB allocation per column per batch and made the most selective query in the taxi suite 2.5x slower than the run-based copy it replaced. ### `sel_compact_width` ```mojo def sel_compact_width(values: Span[UInt8], keep: List[Bool], n: Int, n_keep: Int, width: Int, mut out: List[UInt8]) -> Bool ``` `sel_compact` for whichever of Arrow's fixed widths this column has. Returns `False` for a width with no specialisation — only `decimal128` and an odd `fixed(n)`, which keep the run-based copy. ### `filter_array` ```mojo def filter_array(a: ArrayData, keep: List[Bool], n_keep: Int) -> ArrayData ``` The rows of `a` for which `keep` is true, in order. Kept rows are copied in **runs**, not one at a time: a filter over a million rows is usually either a scattering of long runs (a range predicate over a sorted column) or a dense sample (an equality predicate over a low-cardinality one), and both come out of the same loop — find how far the run of kept rows extends, then move that whole span with one `extend`. A byte-at-a-time copy of a fixed-width column cost eight bounds checks and eight capacity checks per value. ### `concat_into` ```mojo def concat_into(mut dst: ArrayData, src: ArrayData) ``` Append every row of `src` to `dst`; both must have the same type. The common case — neither side has a null — never touches a validity bitmap: Arrow's "buffer absent" encoding already says every row is valid, and materialising a million all-ones bits only to clear them again was most of what concatenating a scan's batches cost. ### `append_key` ```mojo def append_key(mut key: List[UInt8], a: ArrayData, i: Int) ``` Append element `i` of `a` to a canonical key buffer. A null contributes a single `0` tag, a value a `1` tag and its Arrow bytes. Two rows produce the same key exactly when their values are equal — which is what an equality delete asks — and `null` matches `null`, as the spec requires. ### `hash_key` ```mojo def hash_key(key: List[UInt8]) -> UInt64 ``` FNV-1a, which is enough to bucket a delete set. ### `keys_equal` ```mojo def keys_equal(a: List[UInt8], b: List[UInt8]) -> Bool ``` ## Aliases ### `CMP_NONE` ```mojo comptime CMP_NONE = 0 ``` ### `CMP_INT` ```mojo comptime CMP_INT = 1 ``` ### `CMP_FLOAT` ```mojo comptime CMP_FLOAT = 2 ``` ### `CMP_BYTES` ```mojo comptime CMP_BYTES = 3 ``` ### `SEL_LANES` ```mojo comptime SEL_LANES = 16 ``` How many rows one step of a selection-vector kernel handles. Every loop in this section is bound by the buffer it reads rather than by its arithmetic — a `>` over a `double` column at this width reads 128 bytes per step and runs at about 57 GB/s on an M4 — so the width is not tuned per operation. Sixteen is one 128-bit register of `List[Bool]` bytes, which is what the combine, count and run-scan kernels move. ### `COMPACT_AT_ONE_IN` ```mojo comptime COMPACT_AT_ONE_IN = 32 ``` Above this selectivity `filter_array` compacts; below it, it copies runs. The two shapes cross over between 1% and 5% on the taxi table, measured on the same file set with the same projection: at 5.0% kept the compaction is 57.7 ms against the run copy's 110.8, and at 1.06% kept it is 73.4 against 41.9. What turns over is not the copying — the compaction moves far less — but the scan of the selection vector: the run copy exits its skip loop on the first kept row and is predicted perfectly, while the compaction takes an unpredictable branch once per sixteen rows whether or not anything survives them. One in 32 sits between the two measurements and is deliberately not tuned finer than that, because the curve is flat there.