`parquet.page`

Page decoding: headers, compression, levels, and the values inside.

read_column_chunk walks every page of one column chunk — the optional dictionary page and then the data pages, v1 or v2 — and returns the whole chunk as three parallel things:

defs — one definition level per slot, absent when the leaf's maximum definition level is 0;
reps — one repetition level per slot, absent when the maximum repetition level is 0;
values — the non-null values, back to back, still physical.

Page CRC32s are verified when the writer stored them. Every length in a page header is checked against the bytes that are actually there, so a truncated or doctored file raises instead of reading past the end of the buffer.

Structs

`PageWindow`

struct PageWindow

Which pages of a column chunk read_column_chunk has to decode.

Default-constructed — no page rows, an empty row range — it means all of them, which is what every read without page pruning asks for and is the path this reader always had.

first_rows is OffsetIndex.page_locations[k].first_row_index, one entry per data page in file order: the number of records of the row group that are complete before page k starts. With first_rows[npages] taken to be group_rows, page k can hold values for rows

[first_rows[k], first_rows[k + 1]]

— closed at the top, because a record can still be in flight when a page ends and then has values in both pages. That one row of overhang is the whole difference between a repeated column and a flat one here, and getting it wrong drops list elements at a page seam. A flat column's pages hold [first_rows[k], first_rows[k + 1]) exactly, one slot per row.

Implements Copyable, Defaultable, Movable

Fields: first_rows: List[Int]: Per data page, the first row of the row group it can hold. Empty means no page index, so nothing can be skipped.; row_lo: Int; row_hi: Int: The rows the caller still wants, as one outer hull. row_hi <= row_lo means no restriction.; group_rows: Int: Rows in the whole row group — the implied first_rows[npages].

`init`

def __init__(out self)

`restricts`

def restricts(self) -> Bool

Is there anything here that could let a page be skipped?

`page_first_row`

def page_first_row(self, k: Int) -> Int

first_rows[k], with the one-past-the-end entry spelled out.

`page_range`

def page_range(self, flat: Bool) -> Tuple[Int, Int]

The half-open range of data pages that can hold the wanted rows.

A page is in it when the rows it can hold meet [row_lo, row_hi). For a flat column that is first_rows[k] < row_hi and first_rows[k + 1] > row_lo; a repeated column widens the second test to >=, which is the in-flight record the struct's own note describes.

`ColumnData`

struct ColumnData

One decoded column chunk: levels and non-null values.

Implements Copyable, Defaultable, Movable

Fields: defs: List[UInt16]; reps: List[UInt16]; values: PhysBuffer; num_slots: Int: Level slots — the number of values including nulls.; first_row: Int: The row group row this chunk's first record is. Non-zero only when page pruning let whole leading pages be skipped.; num_rows: Int: Records this chunk covers, or -1 when it covers the whole row group and the row group's own count is the answer.; pages_read: Int: Data pages actually decompressed and decoded.; all_present: Bool: Every slot is at the leaf's maximum definition level, so defs was never materialised. This is the usual case — a column with no nulls — and skipping the per-slot level array is worth a great deal on a wide read.; page_slot: List[Int]: The first slot of each data page from the first one with a null in it, and one final entry equal to num_slots. Empty until then, and so empty altogether for a chunk with no nulls.; page_value: List[Int]: The first value of each data page page_slot covers, and one final entry equal to the number of values in values.; packed: Bool: This leaf's definition levels are a validity bitmap — max_def == 1 and max_rep == 0, so a level is one bit — and mask, not defs, is where they land. Set by read_column_chunk from the leaf's descriptor before a single page is read, so it is a property of the column and not of what any one page happened to contain.; mask: List[UInt8]: Packed validity when packed: bit i, LSB-first, is 1 when slot i is at max_def. Empty when the chunk turned out to have no nulls at all, which is what all_present then says.

`init`

def __init__(out self)

`masked`

def masked(self) -> Bool

Validity for this chunk is a bitmap, not an array of levels.

`value_at`

def value_at(self, row: Int, max_def: Int) -> Int

How many of the first row slots hold a value.

The answer for a slot that is a page boundary is stored; for anything else it is that answer plus a scan of at most one page — a popcount over the validity mask where there is one. Nothing here is per row, which is the point: the caller wants this at a handful of batch boundaries, not at every row.

`def_at`

def def_at(self, i: Int, max_def: Int) -> Int

The definition level of slot i.

`rep_at`

def rep_at(self, i: Int) -> Int

Functions

`copy_mask_bits`

def copy_mask_bits(src: List[UInt8], at: Int, mut dst: List[UInt8], n: Int) -> Int

Copy n bits from bit at of src to bit 0 of dst, and return how many of them are set.

dst is already sized to (n + 7) // 8 and zeroed. A batch that starts on a byte boundary — every batch whose size is a multiple of 8, which is every default one — makes this a memcpy; otherwise it is one shift per byte. This is how a leaf's Arrow validity buffer gets built when the chunk's definition levels were decoded as a bitmap in the first place: no pass over levels, and the null count comes out of the same popcount.

`chunk_start`

def chunk_start(cm: ColumnMetaData) -> Int

Where a column chunk's pages begin — the dictionary page if it has one.

`read_column_chunk`

def read_column_chunk[Codecs: CodecSet](file: Span[UInt8], cm: ColumnMetaData, leaf: LeafColumn, verify_crc: Bool, window: PageWindow) -> ColumnData

Decode the pages of one column chunk that window still wants.

A default PageWindow — what every read without page pruning passes — is every page, and then this is the whole-chunk decode it always was: nothing below branches on the window until there is one.

With a window, the pages outside it are found from the page index and never decompressed. Their headers are still read, because that is how the next page's offset is known, so what is saved is decompression, level decoding and value decoding — everything after Codecs.decompress — which is where a column chunk's time is.

A record can span a page boundary. The Parquet spec has pages change on record boundaries when there is an OffsetIndex, but nothing in a file proves it, and a window that starts inside a record would drop that record's leading values and silently shorten a list. That is exactly what the first slot's repetition level says: a chunk that begins a record has reps[0] == 0. When it does not, this falls back to decoding the whole chunk rather than trusting the seam — a rare, self-correcting cost, and the only alternative that cannot be wrong.