# arrow_mlake.ipc

> A module of arrow-mlake's API.

The Arrow IPC stream format: `RecordBatch`es to bytes and back.

The C Data Interface hands buffers to another library in the same process.
IPC is for everything else: a file, a socket, a shared mapping another
process reads. pyarrow's `ipc.open_stream` reads what `IpcWriter` writes, and
`read_ipc_stream` reads what pyarrow's `ipc.new_stream` writes.

```mojo
var w = IpcWriter(batch)      # the schema comes from the first batch
w.write(batch)
var bytes = w^.finish()

var s = read_ipc_stream(Span(bytes))
print(len(s.batches), "batches of", s.schema.num_columns(), "columns")
```

## The framing

Each message in a stream is:

    ffffffff                  continuation marker
    <int32 metadata_size>     little-endian, a multiple of 8
    <flatbuffer Message>      padded out to metadata_size
    <body>                    the buffers, each padded to a multiple of 8

and the stream ends with a continuation marker and a zero length. The first
message is the `Schema`; every later one is a `RecordBatch` whose `nodes` and
`buffers` list each array in pre-order, parent before children. The reader
also accepts the pre-0.15 framing without the continuation marker.

## What is supported

Every type `ArrayData` holds: null, bool, the eight integers, the three
floats, utf8 and binary and their large forms, fixed-size binary,
decimal128, date32, time32/64, timestamps with a unit and zone, list, large
list, struct and map, nested to any depth. Fields carry their nullability
and two metadata keys, `ARROW:extension:name` and `PARQUET:field_id`, which
is where `ArrowType.extension` and `ArrayData.field_id` travel.

Dictionary encoding, compression, unions, run-end encoding, the view types
and the IPC *file* format are not implemented; a stream that uses them is
rejected with an error that names the feature.

A validity bitmap is only written when an array has nulls; otherwise the
buffer is empty, which the format allows. Reading copies each buffer into
the `ArrayData`'s own lists.

## Structs

### `IpcWriter`

```mojo
struct IpcWriter
```

Writes an Arrow IPC stream: the schema, then one message per batch.

Every batch must have the schema's columns, in order, with the same
types; `write` checks the top-level types.

Implements: `Movable`

#### `__init__`

```mojo
def __init__(out self, schema: RecordBatch)
```

Starts a stream with `schema`'s columns. Only the types and names are read; its rows are not written.

#### `write`

```mojo
def write(mut self, batch: RecordBatch)
```

#### `finish`

```mojo
def finish(deinit self) -> List[UInt8]
```

The finished stream, end-of-stream marker included.

### `IpcStream`

```mojo
struct IpcStream
```

What `read_ipc_stream` found: the schema, as a batch of no rows, and every record batch in order.

Implements: `Movable`

Fields:

- `schema: RecordBatch`
- `batches: List[RecordBatch]`

#### `__init__`

```mojo
def __init__(out self, var schema: RecordBatch, var batches: List[RecordBatch])
```

## Functions

### `encode_schema_message`

```mojo
def encode_schema_message(batch: RecordBatch) -> List[UInt8]
```

The flatbuffer `Message` holding `batch`'s schema, unframed.

### `write_ipc_stream`

```mojo
def write_ipc_stream(batches: List[RecordBatch]) -> List[UInt8]
```

`batches` as one stream; the first batch gives the schema.

### `read_ipc_stream`

```mojo
def read_ipc_stream(data: Span[UInt8]) -> IpcStream
```

Reads a whole Arrow IPC stream. Every buffer is copied out of `data`.