`arrow_mlake.carrow_shared`

Publish a batch into a shared mapping, for a consumer in another process.

carrow.mojo exports over the C Data Interface, which hands a consumer pointers. That is why it cannot cross a process: a pointer means nothing in another address space. This writes the same buffers — the same Arrow memory layout, byte for byte — into a MappedRegion, and describes where each one landed in a small JSON manifest.

A consumer maps the file, adds its own base to every offset, and wraps the buffers where they lie. Nothing is decoded and nothing is copied on the reading side:

base, _ = map_the_file(path)
buf = pa.foreign_buffer(base + off, length, owner)
col = pa.Array.from_buffers(pa.float64(), n, [validity, buf], null_count=k)

What it costs, honestly

One copy, on the producer's side: the decode wrote its buffers somewhere, and this moves them into the mapping. That is the same copy export_c already makes for an in-process handover — so this is a cross-process handover at the price of the in-process one, not a free lunch.

Removing that copy as well means the decode allocating into the region in the first place, which is a change to ArrayData's buffers rather than to this file. memory_region is the half of that which already exists.

Flat columns only, for now

Primitive, utf8 and binary columns: validity, offsets and values. Nested types — struct, list, map — have children, and a manifest that describes a tree is a bigger thing than the one here. export_shared raises on them rather than writing something a consumer would misread.

Functions

`shared_size`

def shared_size(arena: ArrayArena, roots: List[Int]) -> Int

How many bytes publishing these columns will take.

A region cannot grow, so a caller publishing several batches into one has to add this up first. Every buffer is padded to 8 because a consumer casts them where they lie.

`export_shared_into`

def export_shared_into(mut bump: BumpAllocator[MappedRegion], arena: ArrayArena, roots: List[Int], names: List[String]) -> String

Write these columns into a region already open, and describe them.

The offsets in the manifest are from the region's base, not from this batch, so several batches can share one mapping: the producer sizes it for all of them, writes each in turn, and publishes each as it lands. That is what keeps a mapping's create/size/map/unmap cycle from being paid per batch — it was the larger half of publishing, bigger than the copy.

`export_shared`

def export_shared(arena: ArrayArena, roots: List[Int], names: List[String], path: String) -> String

One batch into a mapping of its own; returns the manifest.

The convenience form of export_shared_into for a caller with a single batch to publish. The region is unmapped before returning: the producer is finished with it, and the file carries the bytes.