dista.worker

A small result (up to INLINE_LIMIT bytes) goes back to the head, which forwards it. A larger one stays here: in a shared mapping in the node's store (see store.mojo), or in memory if the store is off. The worker tells the head where it is; a reader on this host maps the file, and one elsewhere fetches it from this worker's object server.

So a worker is an event loop over the head, its listener, the drivers and workers fetching from it, and the workers it fetches from. A task whose inputs live on other workers waits, parked, until they've all arrived; the worker keeps serving fetches meanwhile. No worker ever waits on another one synchronously, so two workers that need each other's results can't deadlock. Running a task does block the loop, so fetches from this worker wait until the current task finishes.

A result here lives until the head says free, once nothing references it. Inputs fetched from peers are copies, kept only until the tasks waiting on them have their arguments.

A worker the head gives an actor builds its state and then runs only that actor's calls, in the order they come. A kill destroys the state, and the worker goes back to running tasks. Between calls the head may ask for a checkpoint: the worker writes the state through the codec to DISTA_CHECKPOINT_URI (see checkpoint.mojo), and deletes the checkpoint before it. An actor restarted elsewhere is restored from the last one.

A worker told to stop (SIGTERM, as Kubernetes sends) drains: it tells the head, which sends it no more work and has other workers adopt the results it holds, each fetching its copy from here. It finishes what it was given, and once the head says stop, it serves fetches a moment longer, for tasks given its old locations, and exits. DISTA_DRAIN_SECONDS (default 25, inside Kubernetes' default grace period of 30) bounds the whole drain.

If the head goes away, run returns and the results here stay. rejoin introduces the worker to a new head and reports what it holds; the tasks it had queued are dropped, since their drivers resubmit them. An actor it hosted is destroyed: actors don't survive a head restart yet, though their checkpoints stay.