avro.cursor

One instruction. Plain integers, so a plan is cheap to copy.

What a slot means. Built once; never touched on the hot path.

One decoded value — 24 bytes, so a slot buffer stays cache-friendly.

i carries integers directly and floats bit-for-bit (bitcast, never SIMD.cast, which converts by value); off/ln locate a string or a bytes in the block buffer. off < 0 marks the rare value that lives on the plan instead — a string filled in from a schema default.

The compiled program itself. Reached through DecodePlan.

This is the expensive thing — a few hundred Strings and two arenas — and it is immutable once built, which is what lets DecodePlan share it rather than copy it.

A schema compiled into a decode program plus its slot table.

A plan is a shared, immutable handle: copying one bumps a refcount instead of duplicating the slot paths and the path index. That is what makes avro.cache.PlanCache worth having — a scan planner reading five hundred manifests written by one writer compiles the schema once and every cursor after that costs an atomic increment. It also settles the lifetime question: a cursor holds its own reference, so a cached plan outlives every cursor built from it no matter what order they are destroyed in.

Compiled plans, keyed by the raw avro.schema bytes of the file.

A scan planner opens hundreds of files written by one writer, and every one of them carries a byte-identical copy of the same schema. Parsing that JSON and compiling it is most of what it costs to open a small Avro file; doing it once and handing out a shared DecodePlan is what this is for.

The key is the bytes, before parsing. Byte equality is sufficient for two schemas to compile to the same plan, it is what the repetition actually looks like in the wild, and establishing it needs no parsing. The hash is only a filter: a hit is confirmed by comparing the bytes, so a collision is slow and never wrong. The selected paths are part of the key too, because they change the plan.

The cache is caller-owned: there is no global here. Hand one to RecordCursor.open_cached for as long as the files belong together — one scan, one directory listing — and let it go afterwards. Plans it handed out stay alive on their own; a cursor holds its own reference.

Lookup is a linear scan of the hashes, which is right for the shape this is for: a run of files carrying a handful of distinct schemas between them. A workload with hundreds of different schemas in one cache would want an index instead, and does not have one.

var cache = PlanCache()
for k in range(len(paths)):
    var c = RecordCursor.open_cached(paths[k], cache)
    while c.next():
        ...
print(cache.hits, "of", cache.hits + cache.misses)

A schema-compiled reader over an Object Container File.

The cursor owns the DataFileReader, and therefore the decompressed block that get_str and get_bytes return views of. That is the whole lifetime story: a view borrows from self.reader.block, so it cannot outlive the cursor, and it is only meaningful until the next next() overwrites the position — or until a block boundary replaces the buffer. Copy with get_string / get_bytes_copy if it has to live longer.

Nothing on the wire; the slot records a present-but-null value.

A zigzag varint — int and long.

Length-prefixed bytes — bytes and string, which share a wire form.

aux bytes with no prefix.

A varint symbol index; aux indexes DecodePlan.enum_maps, or -1.

aux is the key slot.

aux is the branch-index slot, or -1.

No wire bytes: store DecodePlan.consts[aux]. A resolution default.

A writer union branch the reader cannot accept; aux indexes messages.

A varint an int -> float / long -> double promotion turns into a float. Its own op so the hot OP_VARINT never has to consult the slot table.