# avro.cursor > A module of avro's API. `RecordCursor` — a schema-compiled, allocation-free Avro reader. `DataFileReader.next()` hands back a `Value`: a fresh node arena behind an `ArcPointer`, every node carrying a `String`, two `List`s and a `List[UInt8]`. That is the right shape for a dynamically typed datum, and the wrong shape for a scan planner reading a million manifest entries — a five-field record costs about sixty heap allocations and every `field("name")` walks a list of `String`s comparing bytes. A `RecordCursor` compiles the schema **once** into a flat program of ops and then runs it per record into pre-sized, reused buffers: ```mojo from avro.cursor import RecordCursor var c = RecordCursor.open("manifest.avro", ["status", "data_file.file_path"]) var path = c.plan.slot_of("data_file.file_path") var status = c.plan.slot_of("status") while c.next(): print(c.get_long(status), c.get_str(path)) ``` Three things make it fast: * **Slots, not names.** Every value the plan decodes lands in a numbered slot. `slot_of` is a one-off name lookup at plan-build time; the hot loop only ever indexes. * **Nothing is allocated per record.** Slot storage is `List[List[SlotVal]]` with a parallel `counts` array; a new record resets the counts to zero and overwrites what is already there, so after the first few records every `List` is already big enough and the steady state does no allocation at all. `tests/test_cursor.mojo` asserts this by counting allocations. * **Strings and bytes are views.** `get_str` and `get_bytes` return a slice of the decompressed block the cursor is holding — no copy. They borrow from `self.reader.block`, so the compiler ties them to the cursor: a view cannot outlive it, and it is only valid until the next `next()`. Selection prunes work but not correctness: an unselected field still has an op, because the decoder has to step over its bytes, but it gets no slot and nothing is stored. Nesting is flattened. A field inside a record is `parent.child`; array elements live under `path.element`, map entries under `path.key` and `path.value`. A slot inside an array holds one value per element, read with the second argument of the accessors, and the array's own slot holds the element count: ```mojo var n = c.array_len(bounds) for k in range(n): print(c.get_long(bound_key, k), c.get_bytes(bound_val, k)) ``` ## Structs ### `PlanOp` ```mojo struct PlanOp ``` One instruction. Plain integers, so a plan is cheap to copy. Implements: `Copyable`, `Movable` Fields: - `op: Int` - `slot: Int`: Where the value lands, or -1 to decode and discard. - `aux: Int` - `kid0: Int`: First child: an index into `DecodePlan.kids`. - `nkid: Int` - `fill: Int`: `OP_UNION` only: where this union's per-branch fill lists start in `DecodePlan.fill_spans`, or -1 when every branch writes the same slots. ### `SlotInfo` ```mojo struct SlotInfo ``` What a slot means. Built once; never touched on the hot path. Implements: `Copyable`, `Movable` Fields: - `path: String`: Dotted path, e.g. `data_file.lower_bounds.element.key`. - `kind: Int`: The reader-visible Avro kind. - `depth: Int`: Repetition depth: 0 is one value per record, 1 is one per element. - `symbols: Int`: Index into `DecodePlan.symbols` for an enum slot, else -1. ### `SlotVal` ```mojo struct SlotVal ``` One decoded value — 24 bytes, so a slot buffer stays cache-friendly. `i` carries integers directly and floats bit-for-bit (`bitcast`, never `SIMD.cast`, which converts by value); `off`/`ln` locate a string or a `bytes` in the block buffer. `off < 0` marks the rare value that lives on the plan instead — a `string` filled in from a schema default. Implements: `Copyable`, `ImplicitlyCopyable`, `Movable` Fields: - `i: Int64` - `off: Int32` - `ln: Int32` - `nul: Bool` #### `none` ```mojo def none() -> Self ``` #### `of_long` ```mojo def of_long(v: Int64) -> Self ``` #### `of_double` ```mojo def of_double(v: Float64) -> Self ``` #### `of_span` ```mojo def of_span(off: Int, ln: Int) -> Self ``` #### `as_double` ```mojo def as_double(self) -> Float64 ``` ### `PlanData` ```mojo struct PlanData ``` The compiled program itself. Reached through `DecodePlan`. This is the expensive thing — a few hundred `String`s and two arenas — and it is immutable once built, which is what lets `DecodePlan` share it rather than copy it. Implements: `Copyable`, `Movable` Fields: - `ops: List[PlanOp]` - `kids: List[Int]`: Child op indices, referenced by `PlanOp.kid0` / `PlanOp.nkid`. - `slots: List[SlotInfo]` - `symbols: List[List[String]]` - `enum_maps: List[List[Int]]`: Writer symbol index -> reader symbol index, for a resolved enum. - `consts: List[SlotVal]` - `const_strings: List[String]`: Text of a `string`/`bytes` resolution default, by slot. - `messages: List[String]` - `fills: List[Int]`: Slot numbers a union branch has to null out — see `_fill_for`. - `fill_spans: List[Int]`: (start, count) into `fills`, two entries per union branch. - `root: Int` - `by_path: Dict[String, Int]` #### `__init__` ```mojo def __init__(out self) ``` ### `DecodePlan` ```mojo struct DecodePlan ``` A schema compiled into a decode program plus its slot table. A plan is a **shared, immutable handle**: copying one bumps a refcount instead of duplicating the slot paths and the path index. That is what makes `avro.cache.PlanCache` worth having — a scan planner reading five hundred manifests written by one writer compiles the schema once and every cursor after that costs an atomic increment. It also settles the lifetime question: a cursor holds its own reference, so a cached plan outlives every cursor built from it no matter what order they are destroyed in. Implements: `Copyable`, `Movable` Fields: - `data: ArcPointer[PlanData]` #### `__init__` ```mojo def __init__(out self) ``` ```mojo def __init__(out self, var data: PlanData) ``` #### `num_slots` ```mojo def num_slots(self) -> Int ``` #### `num_ops` ```mojo def num_ops(self) -> Int ``` #### `root` ```mojo def root(self) -> Int ``` #### `try_slot` ```mojo def try_slot(self, path: StringSpan) -> Int ``` The slot for `path`, or -1 when the schema has no such field. #### `has_slot` ```mojo def has_slot(self, path: StringSpan) -> Bool ``` #### `slot_of` ```mojo def slot_of(self, path: StringSpan) -> Int ``` #### `slot_path` ```mojo def slot_path(self, slot: Int) -> String ``` #### `slot_kind` ```mojo def slot_kind(self, slot: Int) -> Int ``` #### `slot_depth` ```mojo def slot_depth(self, slot: Int) -> Int ``` #### `paths` ```mojo def paths(self) -> List[String] ``` Every slot path, in slot order — handy in tests and REPL work. #### `build` ```mojo def build(schema: Schema, select: List[String] = List(__list_literal__=NoneType(None))) -> Self ``` Compile `schema`, keeping only the selected paths (all if empty). #### `build_resolved` ```mojo def build_resolved(resolved: ResolvedReader, select: List[String] = List(__list_literal__=NoneType(None))) -> Self ``` Compile a `resolve(writer, reader)` plan: the data is decoded as the writer wrote it and lands in the reader's shape. ### `PlanCache` ```mojo struct PlanCache ``` Compiled plans, keyed by the raw `avro.schema` bytes of the file. A scan planner opens hundreds of files written by one writer, and every one of them carries a byte-identical copy of the same schema. Parsing that JSON and compiling it is most of what it costs to open a small Avro file; doing it once and handing out a shared `DecodePlan` is what this is for. **The key is the bytes, before parsing.** Byte equality is sufficient for two schemas to compile to the same plan, it is what the repetition actually looks like in the wild, and establishing it needs no parsing. The hash is only a filter: a hit is confirmed by comparing the bytes, so a collision is slow and never wrong. The selected paths are part of the key too, because they change the plan. The cache is **caller-owned**: there is no global here. Hand one to `RecordCursor.open_cached` for as long as the files belong together — one scan, one directory listing — and let it go afterwards. Plans it handed out stay alive on their own; a cursor holds its own reference. Lookup is a linear scan of the hashes, which is right for the shape this is for: a run of files carrying a handful of distinct schemas between them. A workload with hundreds of *different* schemas in one cache would want an index instead, and does not have one. ```mojo var cache = PlanCache() for k in range(len(paths)): var c = RecordCursor.open_cached(paths[k], cache) while c.next(): ... print(cache.hits, "of", cache.hits + cache.misses) ``` Implements: `Movable`, `Sized` Fields: - `hits: Int` - `misses: Int` - `enabled: Bool`: False makes every lookup a miss and stores nothing — the escape hatch that lets a test run the same code path with and without the cache. - `collide_hashes: Bool`: A test seam: force every key to hash to zero, so the byte compare is the only thing telling two schemas apart. Never set in normal use. #### `__init__` ```mojo def __init__(out self) ``` #### `disabled` ```mojo def disabled() -> Self ``` A cache that never hits and never stores. #### `__len__` ```mojo def __len__(self) -> Int ``` #### `hash_of` ```mojo def hash_of(self, key: Span[UInt8]) -> UInt64 ``` #### `find` ```mojo def find(self, key: Span[UInt8], select: List[String]) -> Int ``` The entry for these schema bytes and this selection, or -1. #### `insert` ```mojo def insert(mut self, key: Span[UInt8], select: List[String], var plan: DecodePlan) -> Int ``` Store `plan` under these schema bytes; returns its entry number, or -1 when the cache is disabled and nothing was stored. #### `plan_at` ```mojo def plan_at(self, i: Int) -> DecodePlan ``` The plan at `i` — a refcount bump, not a copy of the program. ### `RecordCursor` ```mojo struct RecordCursor[C: CodecSet = DefaultCodecs] ``` A schema-compiled reader over an Object Container File. The cursor owns the `DataFileReader`, and therefore the decompressed block that `get_str` and `get_bytes` return views of. That is the whole lifetime story: a view borrows from `self.reader.block`, so it cannot outlive the cursor, and it is only meaningful until the next `next()` overwrites the position — or until a block boundary replaces the buffer. Copy with `get_string` / `get_bytes_copy` if it has to live longer. Implements: `Movable` Fields: - `reader: DataFileReader[C]` - `plan: DecodePlan` - `vals: List[List[SlotVal]]` - `counts: List[Int]` - `index: Int`: How many records `next()` has returned. - `plan_id: Int`: Which `PlanCache` entry the plan came from, or -1 when it was compiled here. Two cursors with the same non-negative id hold the same compiled program, which a caller can use as a cheap stand-in for comparing the schema bytes all over again. #### `__init__` ```mojo def __init__(out self, var reader: DataFileReader[C], var plan: DecodePlan) ``` #### `of` ```mojo def of(var reader: DataFileReader[C], select: List[String] = List(__list_literal__=NoneType(None))) -> Self ``` Take over `reader` and compile its own schema. #### `open` ```mojo def open(path: StringSpan, select: List[String] = List(__list_literal__=NoneType(None))) -> Self ``` #### `from_bytes` ```mojo def from_bytes(data: Span[UInt8], select: List[String] = List(__list_literal__=NoneType(None))) -> Self ``` #### `of_bytes` ```mojo def of_bytes(var data: List[UInt8], select: List[String] = List(__list_literal__=NoneType(None))) -> Self ``` Take ownership of `data` — no copy, unlike `from_bytes`. #### `of_cached` ```mojo def of_cached(var reader: DataFileReader[C], mut cache: PlanCache, select: List[String] = List(__list_literal__=NoneType(None))) -> Self ``` Take over `reader`, compiling its schema only if `cache` cannot already answer for the exact bytes it carries. #### `open_cached` ```mojo def open_cached(path: StringSpan, mut cache: PlanCache, select: List[String] = List(__list_literal__=NoneType(None))) -> Self ``` #### `of_bytes_cached` ```mojo def of_bytes_cached(var data: List[UInt8], mut cache: PlanCache, select: List[String] = List(__list_literal__=NoneType(None))) -> Self ``` Take ownership of `data` — no copy — and use `cache`. #### `resolved` ```mojo def resolved(var reader: DataFileReader[C], var resolution: ResolvedReader, select: List[String] = List(__list_literal__=NoneType(None))) -> Self ``` Read `reader` — whose schema must be `resolution`'s writer — into the reader schema `resolution` was built with. #### `next` ```mojo def next(mut self) -> Bool ``` Decode the next record into the slots; False at end of file. #### `skip` ```mojo def skip(mut self) -> Bool ``` Step over a record without filling any slot. #### `slot_watermark` ```mojo def slot_watermark(self) -> Int ``` Total values the slot buffers have ever held. The only place the hot path can allocate is a slot buffer growing, so a run where this does not change is a run that did not allocate. `tests/test_cursor.mojo` leans on that. #### `count` ```mojo def count(self, slot: Int) -> Int ``` How many values this record put in `slot` — the repetition count. #### `is_null` ```mojo def is_null(self, slot: Int, k: Int = Int(0)) -> Bool ``` True when the field was absent, or a union took its `null` branch. #### `array_len` ```mojo def array_len(self, slot: Int, k: Int = Int(0)) -> Int ``` Elements in the array (or entries in the map) at `slot`. #### `get_bool` ```mojo def get_bool(self, slot: Int, k: Int = Int(0)) -> Bool ``` #### `get_long` ```mojo def get_long(self, slot: Int, k: Int = Int(0)) -> Int64 ``` #### `get_int` ```mojo def get_int(self, slot: Int, k: Int = Int(0)) -> Int32 ``` #### `get_double` ```mojo def get_double(self, slot: Int, k: Int = Int(0)) -> Float64 ``` #### `get_float` ```mojo def get_float(self, slot: Int, k: Int = Int(0)) -> Float32 ``` #### `get_bytes` ```mojo def get_bytes(self, slot: Int, k: Int = Int(0)) -> Span[UInt8, origin_of(self.reader.block)] ``` A view of the block buffer — valid until the next `next()`. #### `get_str` ```mojo def get_str(self, slot: Int, k: Int = Int(0)) -> StringSpan[origin_of(self.reader.block)] ``` The field as text, without copying. Same lifetime as `get_bytes`. #### `get_string` ```mojo def get_string(self, slot: Int, k: Int = Int(0)) -> String ``` An owned copy of the field's text. #### `get_bytes_copy` ```mojo def get_bytes_copy(self, slot: Int, k: Int = Int(0)) -> List[UInt8] ``` An owned copy of the field's bytes. #### `enum_index` ```mojo def enum_index(self, slot: Int, k: Int = Int(0)) -> Int ``` #### `get_symbol` ```mojo def get_symbol(self, slot: Int, k: Int = Int(0)) -> String ``` The enum symbol's name. #### `union_branch` ```mojo def union_branch(self, slot: Int, k: Int = Int(0)) -> Int ``` The wire branch of the union whose `$branch` slot this is. ## Functions ### `schema_hash` ```mojo def schema_hash(data: Span[UInt8]) -> UInt64 ``` A 64-bit digest of raw schema bytes, eight bytes at a time. Only ever a *filter*: `PlanCache` compares the bytes themselves before it trusts a hit, so a collision costs a memcmp and never a wrong answer. ## Aliases ### `OP_NULL` ```mojo comptime OP_NULL = Int(0) ``` Nothing on the wire; the slot records a present-but-null value. ### `OP_BOOL` ```mojo comptime OP_BOOL = Int(1) ``` ### `OP_VARINT` ```mojo comptime OP_VARINT = Int(2) ``` A zigzag varint — `int` and `long`. ### `OP_FLOAT` ```mojo comptime OP_FLOAT = Int(3) ``` ### `OP_DOUBLE` ```mojo comptime OP_DOUBLE = Int(4) ``` ### `OP_BYTES` ```mojo comptime OP_BYTES = Int(5) ``` Length-prefixed bytes — `bytes` and `string`, which share a wire form. ### `OP_FIXED` ```mojo comptime OP_FIXED = Int(6) ``` `aux` bytes with no prefix. ### `OP_ENUM` ```mojo comptime OP_ENUM = Int(7) ``` A varint symbol index; `aux` indexes `DecodePlan.enum_maps`, or -1. ### `OP_RECORD` ```mojo comptime OP_RECORD = Int(8) ``` ### `OP_ARRAY` ```mojo comptime OP_ARRAY = Int(9) ``` ### `OP_MAP` ```mojo comptime OP_MAP = Int(10) ``` `aux` is the key slot. ### `OP_UNION` ```mojo comptime OP_UNION = Int(11) ``` `aux` is the branch-index slot, or -1. ### `OP_CONST` ```mojo comptime OP_CONST = Int(12) ``` No wire bytes: store `DecodePlan.consts[aux]`. A resolution default. ### `OP_ERROR` ```mojo comptime OP_ERROR = Int(13) ``` A writer union branch the reader cannot accept; `aux` indexes `messages`. ### `OP_VARINT_D` ```mojo comptime OP_VARINT_D = Int(14) ``` A varint an `int -> float` / `long -> double` promotion turns into a float. Its own op so the hot `OP_VARINT` never has to consult the slot table.