# parquet.reader > A module of parquet's API. `ParquetReader` — open a file, pick columns and row groups, read batches. ```mojo from parquet import ParquetReader var r = ParquetReader.open("part-0.parquet") r.select_columns(["id", "name"]) var t = r.read_table() print(t.num_rows, "rows") ``` The reader is parametrised on a `CodecSet` (see `parquet.codec`); the default covers `UNCOMPRESSED`, `SNAPPY` and `GZIP` with no FFI, and `parquet_full.AllCodecs` adds `ZSTD` and `LZ4`. Projection is by name, by dotted leaf path, or — for Iceberg — by Parquet **field id**. Row groups can be chosen explicitly, or pruned automatically from their statistics against simple `column op literal` predicates. ## Structs ### `Predicate` ```mojo struct Predicate ``` `column op literal`, for row-group and page pruning. Implements: `Copyable`, `Movable` Fields: - `column: String` - `op: Int` - `value: ScalarValue` ### `ParquetReader` ```mojo struct ParquetReader[Codecs: CodecSet = DefaultCodecs] ``` A Parquet file, projected and batched. Implements: `Movable` Fields: - `data: FileBytes`: The file bytes, however they are held. Points into `_owned` when this reader owns them. - `meta: FileMetaData` - `schema: ParquetSchema` - `batch_size: Int` - `max_batch_value_bytes: Int`: The most `BYTE_ARRAY` bytes one batch may carry, per column. - `verify_crc: Bool` - `num_workers: Int`: How many OS threads turn file bytes into Arrow at once. #### `__init__` ```mojo def __init__(out self, var data: List[UInt8]) ``` Take ownership of the file bytes. ```mojo def __init__(out self, *, unsafe_borrowing: Span[UInt8, ImmUntrackedOrigin]) ``` Initialise every field over a buffer owned by someone else. `from_span` is the public spelling. This exists so the owning constructor can share it rather than keeping a second copy in sync. #### `from_span` ```mojo def from_span(data: Span[UInt8]) -> Self ``` Read a file out of a buffer this reader does **not** own. The caller must keep `data` alive for as long as the reader and must not mutate it; nothing checks either. Use it when the bytes already exist somewhere that outlives the read -- a mapped file, a shared arena, a benchmark timing many reads over one buffer. Prefer the owning constructor otherwise. #### `open` ```mojo def open(path: StringSpan) -> Self ``` #### `open_projected` ```mojo def open_projected(path: StringSpan, columns: List[String]) -> Self ``` Open `path`, fetching only the bytes `columns` actually needs. `open` slurps the whole file. That is the right default when a caller wants all of it, and pure waste when it does not: a four-column projection out of a nineteen-column file touches about a fifth of the bytes, and on a 60 MiB NYC-taxi file reading the other four fifths costs more than decompressing the fifth that is wanted. So this reads the footer, applies the projection, asks `needed_byte_ranges` which chunks that reaches, and reads those. The buffer is still the file's full length — Parquet addresses everything by absolute offset, so a sparse buffer decodes exactly as the whole file does — but the gaps are never written and so are never faulted in. Filling them with zeroes instead would put most of the cost back. **The returned reader is projected already, and may not be re-projected.** Selecting a column whose bytes were not fetched reads the gap, which is uninitialised memory: `select_columns`, `select_fields` and `select_all` are not safe on it. Nothing enforces that, which is the price of not reading the file twice. Anything unexpected about the footer — too short, bad magic, a length that does not fit — falls back to reading the whole file, so this is never wrong, only sometimes no faster. #### `fill_range` ```mojo def fill_range(mut self, offset: Int, src: Span[UInt8]) ``` Write `src` into this reader's file buffer at absolute `offset`. The other half of `footer_only_buffer`. A footer is enough to build a reader, resolve a projection against it and ask `needed_byte_ranges` which chunks that reaches — but not to decode them, because those bytes have not been fetched. This is where a caller with its own I/O — an object store, an `iceberg.FileIO` — puts them. Legal only on a reader that owns its buffer, and only for bytes it has not read yet. The footer is already parsed into values of the reader's own, so filling a gap changes nothing it has looked at; writing past the end of the file raises rather than leaving a reader that would go on to decode nonsense. #### `needed_byte_ranges` ```mojo def needed_byte_ranges(self) -> List[Tuple[Int, Int]] ``` The file bytes a read with this projection would actually touch. One `(offset, length)` per column chunk that the current projection and row-group selection reach, in ascending offset order, with neighbours that touch or overlap merged into one. The footer is not in the list: a caller cannot have built this reader without it. This is what a reader over a network wants — it turns "download the object" into one range request per run of wanted chunks — and it is what `open_projected` uses for a local path. Row-group pruning is honoured, so calling `prune_row_groups` first narrows the answer. #### `num_rows` ```mojo def num_rows(self) -> Int ``` #### `num_row_groups` ```mojo def num_row_groups(self) -> Int ``` #### `created_by` ```mojo def created_by(self) -> String ``` #### `key_value_metadata` ```mojo def key_value_metadata(self) -> List[Tuple[String, String]] ``` #### `split_offsets` ```mojo def split_offsets(self) -> List[Int64] ``` Iceberg's `data_file.split_offsets`: where each row group starts. #### `statistics` ```mojo def statistics(self, rg: Int, leaf: Int) -> TypedStats ``` The decoded statistics of one column chunk, empty if absent. #### `offset_index` ```mojo def offset_index(self, rg: Int, leaf: Int) -> Optional[OffsetIndex] ``` #### `column_index` ```mojo def column_index(self, rg: Int, leaf: Int) -> Optional[ColumnIndex] ``` #### `select_fields` ```mojo def select_fields(mut self, fields: List[Int]) ``` Project by Arrow field index, sub-fields included. A selected field brings its whole sub-tree; every ancestor is kept as a wrapper, so the value still arrives inside the struct it belongs to, and nothing else is decoded. Roots come out in the order they were asked for, which is the contract `select_columns` already has. `select_columns` is the special case where every selection is a root. #### `select_field_ids_deep` ```mojo def select_field_ids_deep(mut self, ids: List[Int32]) ``` `select_fields`, addressing the fields by Parquet field id. #### `select_all` ```mojo def select_all(mut self) ``` #### `select_columns` ```mojo def select_columns(mut self, names: List[String]) ``` Project by top-level column name. #### `select_field_ids` ```mojo def select_field_ids(mut self, ids: List[Int32]) ``` Project by Parquet field id — what Iceberg schema evolution needs. #### `select_row_groups` ```mojo def select_row_groups(mut self, var groups: List[Int]) ``` #### `prune_row_groups` ```mojo def prune_row_groups(mut self, predicates: List[Predicate]) -> Int ``` Drop row groups whose statistics prove no row can match. Returns the number of row groups left. #### `rewind` ```mojo def rewind(mut self) ``` #### `pages_read` ```mojo def pages_read(self) -> Int ``` Data pages this reader has decoded since the last `rewind`. What `prune_pages` is *for*: a page the page index puts outside every wanted row range has its header parsed, so that the next page can be found, and is then stepped over — not decompressed, not decoded. This is the count that drops when that happens, and the only way to tell a read that skipped pages from one that read them all, since both give the same values. #### `has_next` ```mojo def has_next(mut self) -> Bool ``` #### `read_batch` ```mojo def read_batch(mut self) -> RecordBatch ``` The next batch of at most `batch_size` rows. A batch never spans two row groups, and never crosses a gap left by page pruning. #### `page_row_ranges` ```mojo def page_row_ranges(self, rg: Int, predicates: List[Predicate]) -> List[Tuple[Int, Int]] ``` The rows of row group `rg` whose pages could satisfy `predicates`. Uses the `ColumnIndex` bounds and the `OffsetIndex` first-row indices. A column without a page index puts no restriction on the answer, so a file written without one behaves exactly as it did before. **A repeated column's pages do line up with rows** — through `OffsetIndex.first_row_index`, which is the number of records of the row group complete before the page starts. What it does *not* line up with is record boundaries: a record can still be in flight when a page ends, so page `k` holds values for rows [first_row_index[k], first_row_index[k + 1]] with the top end *closed*. That last row is the overhang, and this rounds outward to keep it: a page whose bounds match contributes one row more than it completes. Rounding inward instead — treating the range as half-open, as a flat column's is — drops the row whose list straddles the seam, which is a wrong answer that looks like a right one. A flat column has no records to be in flight and keeps the half-open range it always had. #### `prune_pages` ```mojo def prune_pages(mut self, predicates: List[Predicate]) -> Int ``` Restrict reading to the pages whose bounds could match. Returns the number of rows left. #### `read_table` ```mojo def read_table(mut self) -> Table ``` Every selected row group, as a list of batches. This is the one entry point that parallelises across **row groups** as well as columns, and the reason is a memory contract rather than a performance one. `read_table` was always going to hold every row group of the selection at once — that is what returning a `Table` means — so decoding several of them at once asks for no lifetime the caller was not already granting. The streaming path makes the opposite promise, and keeps it: `read_batch` holds one row group and is untouched. What it does add is intermediate state. A loaded row group holds decompressed pages, level arrays and per-row indices *on top of* the Arrow batches built from them, and sequentially only one row group's worth is ever alive. So the window is capped at `num_workers` row groups: peak decode state grows by at most `num_workers - 1` row groups over the sequential path, never with the size of the file, and at `num_workers = 1` — the default — this is the same loop it always was, byte for byte and allocation for allocation. Within a window the batches are cut by the ordinary iterator walk — `_plan_window`, which is `read_batch` with the building taken out — and then built by a second fan-out over *(batch, top-level field)* pairs, because sequential Arrow assembly was the whole serial remainder of a threaded read. They are appended in file order on this thread whatever order the workers finished in, and each field is grafted into its batch at the index it would have had sequentially, so neither row order nor arena layout depends on the worker count. ### `Table` ```mojo struct Table ``` Every batch of a read, with accessors that run across all of them. Implements: `Copyable`, `Defaultable`, `Movable` Fields: - `batches: List[RecordBatch]` - `num_rows: Int` #### `__init__` ```mojo def __init__(out self) ``` #### `num_columns` ```mojo def num_columns(self) -> Int ``` #### `name` ```mojo def name(self, i: Int) -> String ``` #### `type` ```mojo def type(self, i: Int) -> ArrowType ``` #### `column_i64` ```mojo def column_i64(self, i: Int) -> Tuple[List[Int64], List[Bool]] ``` #### `column_f64` ```mojo def column_f64(self, i: Int) -> Tuple[List[Float64], List[Bool]] ``` #### `column_bool` ```mojo def column_bool(self, i: Int) -> Tuple[List[Bool], List[Bool]] ``` #### `column_str` ```mojo def column_str(self, i: Int) -> Tuple[List[String], List[Bool]] ``` ## Functions ### `footer_start_of` ```mojo def footer_start_of(size: Int, tail: Span[UInt8]) -> Int ``` Where the serialised `FileMetaData` begins, from a file's last 8 bytes. `-1` when `tail` is not a Parquet trailer or the length it declares does not fit, so a caller that reads the tail speculatively can fall back to reading the whole file instead of raising. ### `footer_only_buffer` ```mojo def footer_only_buffer(size: Int, footer_start: Int, footer: Span[UInt8]) -> List[UInt8] ``` A `size`-byte buffer holding nothing but the magic and the footer. `ParquetReader` addresses every page by its absolute file offset, so a buffer of the file's real length with the chunks a projection wants written into it decodes exactly as the whole file does — and the bytes between them are never read. This builds that buffer with the gap left **uninitialised**: fresh pages are not faulted in until something writes them, and zero-filling instead would cost most of what the sparse read saves. Fill the wanted ranges — `needed_byte_ranges` names them — before handing it to a reader, and never read a byte outside them. `footer` is the file from `footer_start` to its end: the metadata, the 4-byte length and the trailing `PAR1`. ### `op_name` ```mojo def op_name(op: Int) -> String ``` ### `range_can_match` ```mojo def range_can_match(op: Int, lo: ScalarValue, hi: ScalarValue, v: ScalarValue) -> Bool ``` Could any value in `[lo, hi]` satisfy `x op v`? ## Aliases ### `OP_EQ` ```mojo comptime OP_EQ = 0 ``` ### `OP_NE` ```mojo comptime OP_NE = 1 ``` ### `OP_LT` ```mojo comptime OP_LT = 2 ``` ### `OP_LE` ```mojo comptime OP_LE = 3 ``` ### `OP_GT` ```mojo comptime OP_GT = 4 ``` ### `OP_GE` ```mojo comptime OP_GE = 5 ``` ### `FileBytes` ```mojo comptime FileBytes = Span[UInt8, ImmUntrackedOrigin] ``` How `ParquetReader` holds the file, whether or not it owns the bytes. The origin is untracked because a borrowing reader has no lifetime relation the compiler can express, and a real origin parameter here would infect every type that holds a reader.