Cluster Mode¶
Cluster mode turns the orchestrator into a long-lived background service. Clients can connect at any time to submit tasks and receive results without restarting the orchestrator between runs.
How It Works¶
- The orchestrator starts in the background and writes a comm checkpoint to
checkpoint_dirrecording its ZMQ address. - Any number of
ClusterClientinstances read that checkpoint to discover the address and connect. - Clients submit tasks and receive results via
concurrent.futures.Future. - The orchestrator shuts down gracefully on
SIGTERM(sent byel stoporEnsembleLauncher.stop()).
Via the CLI¶
launcher_cluster.json:
{
"task_executor_name": "async_processpool",
"comm_name": "async_zmq",
"nlevels": 1,
"cluster": true,
"checkpoint_dir": "/scratch/my_job/ckpt"
}
# Start the orchestrator in the background
el start my_ensemble.json --launcher-config-file launcher_cluster.json
# Submit tasks from Python (see below)
# Graceful shutdown
el stop
Via the Python API¶
Start / Stop¶
from ensemble_launcher import EnsembleLauncher
from ensemble_launcher.config import LauncherConfig, SystemConfig
el = EnsembleLauncher(
ensemble_file={},
system_config=SystemConfig(name="local", ncpus=8),
launcher_config=LauncherConfig(
cluster=True,
checkpoint_dir="/scratch/my_job/ckpt",
),
Nodes=["node-001", "node-002"],
)
el.start() # non-blocking; spawns orchestrator in a separate process
# ...
el.stop() # sends SIGTERM, waits for graceful exit, force-kills if needed
Context Manager¶
with EnsembleLauncher(...) as el:
# orchestrator is running
...
# stop() called automatically on exit
Submitting Tasks with ClusterClient¶
from ensemble_launcher.orchestrator import ClusterClient
from ensemble_launcher.ensemble import Task
with ClusterClient(checkpoint_dir="/scratch/my_job/ckpt") as client:
futures = {}
for i in range(10):
task = Task(task_id=f"task-{i}", nnodes=1, ppn=1,
executable=my_fn, args=(i,))
futures[task.task_id] = client.submit(task)
results = {tid: fut.result(timeout=60) for tid, fut in futures.items()}
Connecting to a Specific Node¶
Node ID Naming Convention¶
| Node ID | Role |
|---|---|
main |
Global master (root) |
main.w0, main.w1 |
Workers of the global master |
main.m0, main.m1 |
Sub-masters (nlevels=2) |
main.m0.w0 |
Worker under sub-master 0 |
node_id="global" always resolves to the root master (shortest name in the checkpoint directory).
Tuning Scheduling Policies with PolicyClient¶
Alongside the task-submission endpoint, each node serves its scheduling policy state on
a separate endpoint. PolicyClient reads and updates it while the run is in flight, which
is how a controller or autotuner steers scheduling based on workflow state it knows about
and the scheduler does not.
from ensemble_launcher.orchestrator import ClusterClient, PolicyClient
ckpt = "/scratch/my_job/ckpt"
with ClusterClient(checkpoint_dir=ckpt) as cc, \
PolicyClient(ckpt, node_id="main.w0") as pc:
print(pc.get_state())
pc.set_state({"gpu_weight": 8.0}) # affects tasks submitted from here on
futures = [cc.submit(t) for t in tasks]
Prerequisites. Discovery is file-based, so checkpoint_dir must be set, and the
endpoint must be enabled with either cluster=True (which implies it) or
enable_policy_client=True on its own -- the latter lets you tune a plain blocking run().
Node ids are the same as for ClusterClient, including "global". To find out which nodes
are serving an endpoint, or to tune several at once:
from ensemble_launcher.orchestrator import PolicyGroupClient
nodes = PolicyClient.discover_nodes(ckpt) # ['main', 'main.w0', 'main.w1']
with PolicyGroupClient(ckpt, node_ids=nodes) as group:
group.set_state({"gpu_weight": 8.0}) # -> {node_id: resulting state}
A group update is not atomic across nodes, though each node's own update is
all-or-nothing. Already-queued tasks keep their priorities unless you pass rescore=True.
See Custom Scheduling for the full
semantics.