Docs

The dista command

dista is a head for any Dista program, and a look at a running cluster. It depends on Dista's core alone, so your program's own binary stays the driver and the worker. Beside it, the head and every worker serve Prometheus metrics, and a driver can write its tasks' timeline.

Commands

Build it from the repository with pixi run cli, to build/dista; the container image has it at /opt/dista/bin/dista. Every command but head connects to the head as an observer, which owns no objects and isn't counted as a driver. Its head is --head host:port, else DISTA_HEAD, else localhost:6390.

--pipeline sets how many tasks the head keeps in flight per worker; see Tuning. stop's workers exit instead of rejoining.

status

dista status prints the head's view: workers, with the program each was built from and what it's running or hosting; drivers, with the objects each holds; and actors, with their worker, calls and last checkpoint.

dista status --json is the head's demand, on one line: tasks ready, waiting for inputs and running, actors waiting for a worker, the pipeline depth, whether there's a checkpoint store, the objects each driver holds (driver_objects), and per worker its address, node, tasks in flight, seconds idle, bytes of results held, actor and whether it's draining.

scale

dista scale --once reads the demand and prints dista.scale.decide's decision, a line each: want N, start N, drain ADDRESS for each worker to drain, and short_since T to pass back as --short-since next time. Its flags are the policy's, and what the loop running it knows:

dista scale --once --min 0 --max 6 --step 4 --up-after 1 --idle 2 \
  --pending 0 --short-since 0

Autoscaling says how the policy decides.

Metrics

With DISTA_METRICS_PORT set, the head and every worker serve Prometheus metrics at /metrics. The kind manifests turn it on at port 9090, for Kubernetes monitoring to scrape. The head's include:

• tasks: dista_tasks_submitted_total, dista_tasks_finished_total by outcome, dista_tasks_retried_total, dista_tasks_timed_out_total, and the dista_task_duration_seconds histogram
• demand: dista_queue_depth, dista_tasks_ready, dista_tasks_waiting, dista_tasks_running, dista_workers, dista_workers_idle, dista_workers_draining
• memory: dista_objects, dista_head_result_bytes, dista_stored_results, dista_queued_bytes, dista_driver_held_objects_max
• backpressure: dista_admission_paused, dista_admission_pauses_total, dista_drivers_behind
• actors: dista_actors, dista_actors_waiting, dista_actor_restarts_total, dista_actors_moved_total, dista_checkpoints_total, and dista_results_moved_total for drains

A worker's metrics count what it runs, stores, spills, copies and adopts: dista_worker_tasks_finished_total, dista_worker_store_bytes, dista_worker_results_spilled_total, dista_worker_bytes_copied_total, dista_worker_fetch_buffer_bytes, dista_worker_results_adopted_total and dista_worker_draining, among others.

The timeline

The head stamps each task's submission, dispatch and completion, and the worker its start and end. rt.timeline(path) writes every finished task, in Chrome's trace format, which Perfetto and the browser's trace viewer open, and returns how many it wrote. Each worker is a track, with a second track for time spent waiting for inputs. The driver can add its own work, so a gap between tasks says what it was:

var t = rt.clock()
var tasks = scan.plan_files()
rt.span("plan", t)          # from t to now, on the driver's track
...
_ = rt.timeline("trace.json")

dista timeline -o trace.json writes every driver's finished tasks, without taking them from the drivers. It names tasks by their wire ID, since a generic head doesn't know their names. A head restart loses the spans it held.