# arrow_mlake.carrow_shared

> A module of arrow-mlake's API.

Publish a batch into a shared mapping, for a consumer in another process.

`carrow.mojo` exports over the C Data Interface, which hands a consumer
**pointers**. That is why it cannot cross a process: a pointer means nothing
in another address space. This writes the same buffers — the same Arrow memory
layout, byte for byte — into a `MappedRegion`, and describes where each one
landed in a small JSON manifest.

A consumer maps the file, adds its own base to every offset, and wraps the
buffers where they lie. Nothing is decoded and nothing is copied on the
reading side:

```python
base, _ = map_the_file(path)
buf = pa.foreign_buffer(base + off, length, owner)
col = pa.Array.from_buffers(pa.float64(), n, [validity, buf], null_count=k)
```

## What it costs, honestly

One copy, on the producer's side: the decode wrote its buffers somewhere, and
this moves them into the mapping. That is the same copy `export_c` already
makes for an in-process handover — so this is a *cross-process* handover at
the price of the in-process one, not a free lunch.

Removing that copy as well means the decode allocating into the region in the
first place, which is a change to `ArrayData`'s buffers rather than to this
file. `memory_region` is the half of that which already exists.

## Flat columns only, for now

Primitive, `utf8` and `binary` columns: validity, offsets and values. Nested
types — struct, list, map — have children, and a manifest that describes a
tree is a bigger thing than the one here. `export_shared` raises on them
rather than writing something a consumer would misread.

## Functions

### `shared_size`

```mojo
def shared_size(arena: ArrayArena, roots: List[Int]) -> Int
```

How many bytes publishing these columns will take.

A region cannot grow, so a caller publishing several batches into one has
to add this up first. Every buffer is padded to 8 because a consumer casts
them where they lie.

### `export_shared_into`

```mojo
def export_shared_into(mut bump: BumpAllocator[MappedRegion], arena: ArrayArena, roots: List[Int], names: List[String]) -> String
```

Write these columns into a region already open, and describe them.

The offsets in the manifest are from the region's base, not from this
batch, so several batches can share one mapping: the producer sizes it for
all of them, writes each in turn, and publishes each as it lands. That is
what keeps a mapping's create/size/map/unmap cycle from being paid per
batch — it was the larger half of publishing, bigger than the copy.

### `export_shared`

```mojo
def export_shared(arena: ArrayArena, roots: List[Int], names: List[String], path: String) -> String
```

One batch into a mapping of its own; returns the manifest.

The convenience form of `export_shared_into` for a caller with a single
batch to publish. The region is unmapped before returning: the producer is
finished with it, and the file carries the bytes.