magmalake parquet

parquet · API

parquet.assemble

Dremel record assembly: definition and repetition levels back into Arrow.

Every leaf of a Parquet file carries, for each of its value slots, the definition level (how deep the path to the value is actually present) and the repetition level (at which level of nesting the value repeats). Those two numbers are enough to rebuild the whole nested structure, and that is what build_field does — one Arrow array per Arrow field, recursively.

The key idea is a presence threshold. A field is asked to build only the slots whose definition level is at least require_def, which its parent sets:

  • the root asks for require_def = 0, so every slot is a row;
  • a LIST or MAP asks its child for require_def = elem_def_level, the definition level at which the repeated group actually has an entry — so the child array comes back already containing exactly the list's elements, with the empty and null containers dropped;
  • a STRUCT passes its own threshold straight through, because a struct does not change granularity.

Levels come from the field's first descendant leaf. Every leaf under a field agrees about the levels at and above that field, so any one of them will do — which is what makes it possible to build a struct's children independently and still have them line up.

Structs

LeafSlice

struct LeafSlice

The slot range of one leaf that a batch covers, and where its values start in the chunk's value buffer.

Implements Copyable, Defaultable, Movable

Fields
s0: Int
s1: Int
v0: Int

__init__

def __init__(out self)

Functions

first_leaf

def first_leaf(s: ParquetSchema, fi: Int) -> Int

The first descendant leaf of Arrow field fi — its level source.

first_included_leaf

def first_included_leaf(s: ParquetSchema, fi: Int, include: List[Bool]) -> Int

first_leaf, restricted to the fields include selects.

A pruned read skips whole sub-trees, so the level source of a struct can no longer be its first child: it has to be the first child actually being built. Every leaf under a field agrees about the levels at and above that field, so any included one will do.

build_field

def build_field(s: ParquetSchema, fi: Int, require_def: Int, chunks: List[ColumnData], slices: List[LeafSlice], mut arena: ArrayArena, include: List[Bool]) -> Int

Build the Arrow array for field fi; return its index in arena.

include, when it is not empty, is one flag per entry of s.fields: a struct builds only the children it selects, so a projection that wants a.b and nothing else decodes exactly the leaves under a.b. A list and a map always build their element (and a map its key), because their offsets mean nothing without one.