parquet.reader

Take ownership of the file bytes.

Initialise every field over a buffer owned by someone else.

from_span is the public spelling. This exists so the owning constructor can share it rather than keeping a second copy in sync.

Read a file out of a buffer this reader does not own.

The caller must keep data alive for as long as the reader and must not mutate it; nothing checks either. Use it when the bytes already exist somewhere that outlives the read -- a mapped file, a shared arena, a benchmark timing many reads over one buffer. Prefer the owning constructor otherwise.

Open path, fetching only the bytes columns actually needs.

open slurps the whole file. That is the right default when a caller wants all of it, and pure waste when it does not: a four-column projection out of a nineteen-column file touches about a fifth of the bytes, and on a 60 MiB NYC-taxi file reading the other four fifths costs more than decompressing the fifth that is wanted.

So this reads the footer, applies the projection, asks needed_byte_ranges which chunks that reaches, and reads those. The buffer is still the file's full length — Parquet addresses everything by absolute offset, so a sparse buffer decodes exactly as the whole file does — but the gaps are never written and so are never faulted in. Filling them with zeroes instead would put most of the cost back.

The returned reader is projected already, and may not be re-projected. Selecting a column whose bytes were not fetched reads the gap, which is uninitialised memory: select_columns, select_fields and select_all are not safe on it. Nothing enforces that, which is the price of not reading the file twice.

Anything unexpected about the footer — too short, bad magic, a length that does not fit — falls back to reading the whole file, so this is never wrong, only sometimes no faster.

Write src into this reader's file buffer at absolute offset.

The other half of footer_only_buffer. A footer is enough to build a reader, resolve a projection against it and ask needed_byte_ranges which chunks that reaches — but not to decode them, because those bytes have not been fetched. This is where a caller with its own I/O — an object store, an iceberg.FileIO — puts them.

Legal only on a reader that owns its buffer, and only for bytes it has not read yet. The footer is already parsed into values of the reader's own, so filling a gap changes nothing it has looked at; writing past the end of the file raises rather than leaving a reader that would go on to decode nonsense.

The file bytes a read with this projection would actually touch.

One (offset, length) per column chunk that the current projection and row-group selection reach, in ascending offset order, with neighbours that touch or overlap merged into one. The footer is not in the list: a caller cannot have built this reader without it.

This is what a reader over a network wants — it turns "download the object" into one range request per run of wanted chunks — and it is what open_projected uses for a local path. Row-group pruning is honoured, so calling prune_row_groups first narrows the answer.

Iceberg's data_file.split_offsets: where each row group starts.

The decoded statistics of one column chunk, empty if absent.

Project by Arrow field index, sub-fields included.

A selected field brings its whole sub-tree; every ancestor is kept as a wrapper, so the value still arrives inside the struct it belongs to, and nothing else is decoded. Roots come out in the order they were asked for, which is the contract select_columns already has. select_columns is the special case where every selection is a root.

select_fields, addressing the fields by Parquet field id.

Project by top-level column name.

Project by Parquet field id — what Iceberg schema evolution needs.

Drop row groups whose statistics prove no row can match. Returns the number of row groups left.

Data pages this reader has decoded since the last rewind.

What prune_pages is for: a page the page index puts outside every wanted row range has its header parsed, so that the next page can be found, and is then stepped over — not decompressed, not decoded. This is the count that drops when that happens, and the only way to tell a read that skipped pages from one that read them all, since both give the same values.

The next batch of at most batch_size rows. A batch never spans two row groups, and never crosses a gap left by page pruning.

The rows of row group rg whose pages could satisfy predicates.

Uses the ColumnIndex bounds and the OffsetIndex first-row indices. A column without a page index puts no restriction on the answer, so a file written without one behaves exactly as it did before.

A repeated column's pages do line up with rows — through OffsetIndex.first_row_index, which is the number of records of the row group complete before the page starts. What it does not line up with is record boundaries: a record can still be in flight when a page ends, so page k holds values for rows

[first_row_index[k], first_row_index[k + 1]]

with the top end closed. That last row is the overhang, and this rounds outward to keep it: a page whose bounds match contributes one row more than it completes. Rounding inward instead — treating the range as half-open, as a flat column's is — drops the row whose list straddles the seam, which is a wrong answer that looks like a right one. A flat column has no records to be in flight and keeps the half-open range it always had.

Restrict reading to the pages whose bounds could match. Returns the number of rows left.

Every selected row group, as a list of batches.

This is the one entry point that parallelises across row groups as well as columns, and the reason is a memory contract rather than a performance one. read_table was always going to hold every row group of the selection at once — that is what returning a Table means — so decoding several of them at once asks for no lifetime the caller was not already granting. The streaming path makes the opposite promise, and keeps it: read_batch holds one row group and is untouched.

What it does add is intermediate state. A loaded row group holds decompressed pages, level arrays and per-row indices on top of the Arrow batches built from them, and sequentially only one row group's worth is ever alive. So the window is capped at num_workers row groups: peak decode state grows by at most num_workers - 1 row groups over the sequential path, never with the size of the file, and at num_workers = 1 — the default — this is the same loop it always was, byte for byte and allocation for allocation.

Within a window the batches are cut by the ordinary iterator walk — _plan_window, which is read_batch with the building taken out — and then built by a second fan-out over (batch, top-level field) pairs, because sequential Arrow assembly was the whole serial remainder of a threaded read. They are appended in file order on this thread whatever order the workers finished in, and each field is grafted into its batch at the index it would have had sequentially, so neither row order nor arena layout depends on the worker count.