# Failure and recovery

> What Dista does when things fail: a task that raises, a worker that dies or goes silent, a peer that can't be reached, a task that runs too long, a driver that leaves, and a head that restarts; and how it's tested.

[magmalake](https://magmalake.dev/) [Dista](https://magmalake.dev/s/dista)

Dista runs a dead worker's tasks again and rebuilds a restarted head from its peers. It doesn't rebuild results lost with a worker: a job that needs one fails, naming it, and is run again, by [Orca](https://magmalake.dev/s/orca) or by hand. Tasks may run more than once, so they should be safe to repeat, as a pure function is.

## A task that raises

A task that raises fails its `ObjectRef` with the error, which `get` and `take` raise in the driver, and every task that depends on it fails with the same error. The worker carries on.

## A worker that dies

The tasks a dead worker was running go back in the queue for another worker, up to `DISTA_TASK_RETRIES` times (default 3). Past that they fail with `worker lost while running this task`, as they do at once if an input they read from that worker went with it, or if a `TaskMany` had answered some of its results.

The results it held are lost. Dista doesn't rebuild them by running again the tasks that made them, so a task that needs one fails, saying the result was lost with its worker and that the job should be run again. A task on another worker that was fetching one fails with `input lost: the worker holding it (<ip:port>) went away`, and its worker drops the connection and carries on. In local mode, where nothing replaces a dead worker, tasks fail with `no workers left` once every worker is gone, rather than waiting.

A worker that's stopped, rather than killed, [drains](https://magmalake.dev/s/dista/docs/cluster#draining) first: it loses nothing if it finishes within `DISTA_DRAIN_SECONDS`.

## A silent peer

A host that freezes, loses its network or is hard-stopped never closes its connections. Every Dista socket has TCP keepalive: after `DISTA_KEEPALIVE_SECONDS` of silence (default 10; 0 turns it off) the kernel probes, and after 3 unanswered probes, about 25 s in all, the connection fails. The head then drops the worker and its tasks run again; a worker whose head went away rejoins `DISTA_HEAD`, a replacement head included; and a driver resumes with the new head. The kernel answers probes for a process that's only busy, so a long task doesn't trip it. With data unacknowledged on the connection, the kernel's retransmission limit applies instead, about 15 minutes on Linux.

## An unreachable peer

A worker connects to the workers it fetches inputs from inside its event loop, so it keeps running tasks and serving fetches meanwhile. After `DISTA_PEER_CONNECT_TIMEOUT` seconds (default 5, fractions allowed) it fails the tasks waiting on that peer, with `input lost: cannot reach the worker holding it`. A driver's fetch waits the same time. A closed port fails at once; the timeout bounds an address that never answers, which the kernel would otherwise retry for about two minutes.

## A task that runs too long

`DISTA_TASK_TIMEOUT` (seconds; unset, no limit) covers what keepalive can't see, such as a hung task or a stopped process. The head fails a task still running that long after it sent it, with `timed out: ran longer than DISTA_TASK_TIMEOUT`. It isn't retried, its worker isn't stopped, and a late result is dropped. `dista_tasks_timed_out_total` counts them. To stop waiting without failing the task, use [get with a timeout](https://magmalake.dev/s/dista/docs/tasks#giving-up).

## A driver that leaves

When a driver's connection drops, the head deletes its tasks still queued or waiting for inputs, rather than run them for nobody. Tasks already running finish, and their results are dropped.

## A head that restarts

Nothing the head holds is written to disk; a new head is rebuilt from its peers. Workers rejoin and report what they hold, and drivers reconnect under their old number and resubmit what they still need. A task running at that moment may run twice. Actors come back too, from their [checkpoints](https://magmalake.dev/s/dista/docs/actors#checkpoints).

A head numbers new drivers from a random start and gives an epoch in its welcome, so only a driver resuming under its old number claims the results workers report, and a new driver can't be handed another job's. Results no driver comes back for are freed after `DISTA_ORPHAN_SECONDS` (default 120). That covers a head restart, not a head lost for good together with its driver.

## How it's tested

A chaos test kills the head at random under a random task graph and checks every answer. Fuzzers cover the head's frames, the codec and the scaling policy. Other tests kill a worker while two others fetch from it, fetch from a worker that stopped answering, drain a worker mid-job, and restart the head once an actor has a checkpoint. On kind, a worker pod is killed during the taxi queries and the actor's pod is deleted mid-run, and every answer and the actor's count must come out exact.

[A magmalake project](https://magmalake.dev/)

Apache 2.0 · © 2026 Marius Seritan