Docs
Roadmap
The goal for resources: a run that needs more memory than expected fails alone and is retried bigger, instead of taking other runs down with it, and a pipeline on a new machine uses its cores without hand-set concurrency settings.
"Never out of memory" isn't promised. Some memory use depends on the data, such as a very large repository's clone or an unusually busy hour, and is only known while the step runs. Packing runs onto a machine optimally is a bin-packing problem; the aim is to keep the machine busy without anything being killed.
Four layers
| Layer | What it does | Today | Next |
|---|---|---|---|
| Measure | Know what each run actually used | Nothing is recorded | orca #200 Record each run's peak memory and CPU |
| Admit | Start a run only if its request fits what's free | max_runs counts runs, not their size | orca #171 Resource requests and grants, "auto" orca #202 Admission by memory pressure |
| Contain | An overrun hits only its own run | first_to_go makes a run's processes the kernel's first choice; a memory cap on control-placed runs | orca #201 A cgroup memory limit per run, retry bigger after an OOM |
| Back pressure inside a run | A job adapts how much work it keeps in flight | Hand-set: DISTA_WORKER_SLOTS, DISTA_MAX_QUEUED, DISTA_STORE_BYTES, and each pipeline's own settings | dista #123 Size in-flight work from the run's grant and live memory |
Order
orca #200, then #171, then #201, then #202 and dista #123. Each step needs the one before it: grants need measurements, containment needs grants to use as limits, and adaptive in-flight work needs both grants and the pressure signal.
Prior art
What each of these gives the plan:
- Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune and John Wilkes, "Large-scale cluster management at Google with Borg", EuroSys 2015. requests against usage, and limits. Link
- Krzysztof Rzadca et al., "Autopilot: workload autoscaling at Google", EuroSys 2020. limits set from usage history, which is what "auto" means here. Link
- Christina Delimitrou and Christos Kozyrakis, "Quasar: resource-efficient and QoS-aware cluster management", ASPLOS 2014. learning what a job needs instead of trusting its declaration. Link
- Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker and Ion Stoica, "Dominant Resource Fairness: fair allocation of multiple resource types", NSDI 2011. admitting with CPU and memory together. Link
- Robert Grandl, Ganesh Ananthanarayanan, Srikanth Kandula, Sriram Rao and Aditya Akella, "Multi-resource packing for cluster schedulers" (Tetris), SIGCOMM 2014. the same, as packing. Link
- Matt Welsh, David Culler and Eric Brewer, "SEDA: an architecture for well-conditioned, scalable internet services", SOSP 2001. admission control and bounded queues. Link
- Kathleen Nichols and Van Jacobson, "Controlling queue delay", ACM Queue, 2012. sizing in-flight work from measured delay. Link
- Johannes Weiner et al., "TMO: transparent memory offloading in datacenters", ASPLOS 2022. memory pressure (PSI) as the signal. Link
- Viktor Leis, Peter Boncz, Alfons Kemper and Thomas Neumann, "Morsel-driven parallelism: a NUMA-aware query evaluation framework for the many-core age", SIGMOD 2014. work in small units sized to the cores and memory available. Link