magmalake objectstore

objectstore · API

objectstore.ranges

objectstore.ranges — byte ranges, and the plan that turns many into few.

A Parquet scan does not read a file, it reads a list of spans: the footer, then one span per column chunk of each row group it kept. Over local storage that is a list of seeks and costs nothing. Over HTTP it is a list of requests, and even with connection reuse each one still costs a round trip — so the useful optimisation is not making requests faster, it is making fewer of them.

Column chunks of the same row group are laid out consecutively, which means the spans a scan asks for are usually adjacent or separated by the columns it skipped. Fetching across a small hole and throwing the hole away is cheaper than a second round trip: this module works out where those holes are.

The bound is max_gap (1 MiB by default), applied per join, so the bytes read and discarded are at most max_gap times the number of joins — never more than the caller asked for plus that.

This module deliberately knows nothing about transports; fileio and httpio execute the plan.

Structs

ByteRange

struct ByteRange

length bytes starting at offset.

A negative offset means what it means to read_range — a suffix range, "the last length bytes" — and is never coalesced, because where it lands is not known until the length is.

Implements Copyable, Movable, Writable

Fields
offset: Int
length: Int

end

def end(self) -> Int

write_to

def write_to(self, mut writer: T)

CoalescedPlan

struct CoalescedPlan

How to answer n ranges with len(fetches) requests.

For each input range i: group[i] is the fetch it comes from, or -1 if it has to be read on its own (a suffix or empty range), and group_offset[i] is where inside that fetch its bytes begin.

Implements Copyable, Movable

Fields
fetches: List[ByteRange]
group: List[Int]
group_offset: List[Int]

__init__

def __init__(out self)

requests_saved

def requests_saved(self, asked: Int) -> Int

Traits

RangeReader

The one method coalescing needs. InputFile is a superset.

Implements Copyable, Movable

read_range

def read_range(self, offset: Int, length: Int) -> List[UInt8]

Functions

plan_ranges

def plan_ranges(ranges: List[ByteRange], max_gap: Int = Int(1048576)) -> CoalescedPlan

Groups ranges that are adjacent, overlapping, or max_gap apart.

The input order is preserved in the result — a caller that asked for the footer last still gets the footer last — and ranges are sorted internally only to find the groups.

read_ranges_coalesced

def read_ranges_coalesced[T: RangeReader](reader: T, ranges: List[ByteRange], max_gap: Int = Int(1048576)) -> List[List[UInt8]]

Fetches the plan's groups one by one and slices the caller's ranges back out. Backends that can fetch groups at once do that instead and call assemble.

assemble

def assemble[T: RangeReader](reader: T, plan: CoalescedPlan, ranges: List[ByteRange], var buffers: List[List[UInt8]]) -> List[List[UInt8]]

The caller's ranges, in order, cut from buffers (one per fetch of plan). A range a fetch answers exactly gets that buffer, uncopied; a suffix or empty range, which no fetch covers, is read on its own.

Aliases

comptime DEFAULT_COALESCE_GAP = 1048576

Holes up to this size are read and discarded rather than skipped. A MiB is roughly where a loopback round trip stops being cheaper than the bytes; over a real network the crossover is much higher, so this is the conservative end.