Performance
Crossing the boundary
One column of a 79,478,796-row table, read by the same engine — Daft — with nothing changing but how the rows arrive. Every row but one is pyarrow doing the reading, so the ratios are the boundary rather than two readers; the exception says so. Measured 19 September 2026 with pyarrow-flight.example's `pixi run transports`, p50 of five full reads after a discarded warm-up, every leg asserting the same row count.
| Operation | magmalake | Reference |
|---|---|---|
| In this process, no boundary at all the control: pyarrow reading the files and handing Daft the batches, which is exactly what the Flight server does minus the wire | 62 ms | — |
| In this process, Arrow C Data Interface iceberg.mojo scanning an Iceberg table into Daft's buffers, nothing encoded or sent | 89 ms | a different reader, so this is not a boundary cost — there is no boundary. It is what our Iceberg scan costs against pyarrow reading the Parquet files directly. |
| Arrow Flight, TCP on loopback pyarrow's own Flight server over the same Parquet files | 149 ms — 2.4× the in-process read | crossing a process costs 2.4×, not an order of magnitude: a fair price for a crash boundary, a retry boundary, or a credential that should not leave a service |
| Arrow Flight, flight.mojo's own server the same protocol, our implementation | 377 ms — 2.5× pyarrow's server | 12.1 s before four fixes: byte-at-a-time copies in four places, gzip applied to an Arrow payload the client merely said it would accept, a full table scan on every call to learn the schema, and a quadratic in HTTP/2 flow control that copied about 6 GB to send 28 MB |
| Shared memory, between two processes the same Flight plan, but DoGet answers with the name of a mapping and the client maps it — pyarrow on both sides | 228 ms — 1.5× the loopback stream | the consumer's half is nearly free; the producer has to materialise a whole split before the consumer may touch it, and both sides fault every page, where a stream overlaps the two and writes into kernel buffers that are already resident |
| Shared memory, our own producer iceberg.mojo writing Arrow buffers into one mapping per split, publishing each batch as it lands, read by a pool of processes | 115 ms — faster than the same column over Flight | not comparable to the rows above: a decoder 1.4× slower than pyarrow's and a process pool rather than one threaded server, so two things move at once. It was 288 ms when each batch got a mapping of its own — a mapping's create/size/map/unmap/unlink cycle is 4.01 ms per 8 MB batch against 3.05 ms for the copy inside it, so the lifecycle was the larger half all along. |
| Arrow Flight, Unix domain socket pyarrow's server again, gRPC over a socket instead of loopback | 573 ms | the obvious same-machine optimisation, measured twice either side of the TCP run, and four times worse than loopback on macOS |
| The handover alone, no Parquet in it one column already in Arrow: an IPC file memory-mapped, then read into the heap | 24 ms mapped / 54 ms copied | the gap is one copy of 636 MB, and that copy is the whole of what a shared mapping saves — the C Data Interface cannot do it across processes because it hands over pointers |
scroll the table sideways →
Apple M4 — ten cores, four of them performance — macOS, stable Mojo 1.0.0, with nothing else building on the machine. Measured on 5 September 2026; the Iceberg rows on 15 September, the boundary rows on 19 September. How these were measured.