# Tuning

> Tuning Dista: tasks in flight per worker, backpressure limits on the head's queue and a slow driver, the node store and spilling, the memory a driver's results hold, large results in parts, and logging a task's inputs.

[magmalake](https://magmalake.dev/) [Dista](https://magmalake.dev/s/dista)

The defaults suit most jobs. These are the settings that change how much work is in flight, where results are kept, and how much memory the head and workers use. Measured runs against Ray and Rust are on the [performance](https://magmalake.dev/s/dista/performance) page.

## Tasks in flight

The head keeps `pipeline` tasks in flight per worker, default 2: `Runtime[...](pipeline=n)` for a head built into the program, or `dista head --pipeline n`. Deeper hides the round trip to the head for small tasks. On an M4 MacBook, 10,000 tasks of `square` on 4 local workers took 11.0 µs each with no pipelining, 6.2 µs at depth 2 and 5.3 µs at depth 4\. The head's single loop handles about 160,000 empty tasks a second, flat from 4 to 192 workers. `pixi run bench` runs both, from `bench/fanout.mojo`.

## Backpressure

While the tasks queued or waiting for inputs reach `DISTA_MAX_QUEUED` (default 100,000), or their arguments `DISTA_MAX_QUEUED_BYTES` (default 1 GiB), the head stops reading drivers, and `remote` blocks until workers catch up. Past `DISTA_MAX_DRIVER_BUFFER` bytes (default 256 MiB) of results waiting for a driver that isn't reading, that driver's queued tasks wait. 0 turns a limit off. Workers, observers and clients inside tasks are never paused. The head's `/metrics` show `dista_queued_bytes`, `dista_admission_paused` and `dista_drivers_behind`.

## The node store and spilling

A worker writes each large result into a file in the node store, which readers on the same host map without a socket. The store is `/dev/shm` on Linux, `$TMPDIR` elsewhere, or `DISTA_STORE_DIR`. `DISTA_SHM=0` turns it off, keeping large results in the worker's memory. Past `DISTA_STORE_BYTES` of results (no limit when unset), or when a store write fails, results go to `DISTA_SPILL_DIR` on disk (default `$TMPDIR`, else `/tmp`), where readers map or fetch them as before.

On Kubernetes each pod has its own `/dev/shm`, so pods use sockets unless they share a volume for `DISTA_STORE_DIR`. A memory-backed `/dev/shm` evicts a pod that passes its size limit, so set `DISTA_STORE_BYTES` below it; the AWS manifests use 1 GiB per worker.

## Memory held for drivers

A result stays in the driver, at the head (if it's small) and on its worker until the driver releases it. Read results with `rt.take` unless they're read again, and `rt.release` the refs you only passed to other tasks; see [Freeing results](https://magmalake.dev/s/dista/docs/tasks#freeing). The head logs a driver holding `DISTA_WARN_HELD` objects (default 100,000; 0, never), and again at each doubling, and `dista_driver_held_objects_max` shows the largest.

## Large results

A large result is copied once per hop. A worker serving a fetch sends from its file's mapping, or from memory, 1 MiB at a time, so many fetchers of one result don't cost a copy each. A result over 64 MiB comes in parts, which the fetcher assembles into one buffer, so results past 1 GiB cross hosts. A frame has to fit 1 GiB, so a task's argument can't be larger: keep large data in results. In [bench/fetch.mojo](https://github.com/magmalake/dista.mojo/blob/main/bench/fetch.mojo) (`pixi run bench-fetch`), 12 readers of a 256 MB result over sockets took about half as long once results were sent this way.

## Debugging inputs

`DISTA_DEBUG=1` has a worker log each task's inputs as it assembles them. For where time goes, write the [timeline](https://magmalake.dev/s/dista/docs/cli#timeline).

[A magmalake project](https://magmalake.dev/)

Apache 2.0 · © 2026 Marius Seritan