Skip to content

Configuration

Auto-Configuration

The launcher automatically configures itself based on your workload and system:

from ensemble_launcher import EnsembleLauncher

el = EnsembleLauncher(
    ensemble_file="config.json",
    Nodes=["node-001", "node-002"],  # Optional: auto-detects from PBS_NODEFILE
)

SystemConfig

Describes the resources available on each node:

from ensemble_launcher.config import SystemConfig

system_config = SystemConfig(
    name="my_cluster",
    ncpus=104,
    ngpus=12,
    cpus=list(range(104)),
    gpus=list(range(12))
)
Parameter Type Default Description
name str Required Cluster name
ncpus int cpu_count() CPUs per node
ngpus int 0 GPUs per node
cpus List[int] [] Specific CPU IDs
gpus List[Union[str, int]] [] Specific GPU IDs

Sharing a GPU across tasks

Breaking change: SystemConfig used to let you "overload" a GPU by repeating its ID in the gpus list (e.g. gpus=['0', '0', '1', '1']), and the scheduler treated each repeated entry as a separate slot. Duplicate IDs in cpus/gpus now raise a ValidationError instead. That mechanism was cluster-wide and node-scoped -- every task on the node could oversubscribe the GPU, with no way to say how many tasks should actually share it.

Sharing is now expressed per task via Task.ngpus_per_process, which accepts a fraction:

system_config = SystemConfig(
    name="my_cluster",
    cpus=list(range(104)),
    gpus=[0, 1, 2, 3],  # each ID must be unique
)

# Four tasks at ngpus_per_process=0.25 share GPU 0 concurrently; a fifth
# will not be scheduled until one of them completes.
task = Task(
    task_id="t0",
    nnodes=1,
    ppn=1,
    ngpus_per_process=0.25,
    executable="my_program",
)

Internally each GPU ID is tracked as an available fraction starting at 1.0; a request subtracts ngpus_per_process from the ID(s) it is granted and the fraction is restored on completion. This makes the sharing decision explicit at the task that wants it, instead of an invisible cluster-wide setting.

LauncherConfig

Controls orchestration behavior:

from ensemble_launcher.config import LauncherConfig

launcher_config = LauncherConfig(
    child_executor_name="async_mpi",
    task_executor_name="async_processpool",
    comm_name="async_zmq",
    policy_config=PolicyConfig(nlevels=2),
    report_interval=10.0,
    return_stdout=True,
    worker_logs=True,
    master_logs=True,
)
Parameter Type Default Description
child_executor_name str "async_processpool" Executor for launching sub-master/worker processes
task_executor_name str or list "async_processpool" Executor for running tasks
comm_name Literal["async_zmq"] "async_zmq" Communication backend
report_interval float 10.0 Status update frequency (seconds)
return_stdout bool False Capture task stdout
worker_logs bool False Enable worker logging
master_logs bool False Enable master logging
gpu_selector str "ZE_AFFINITY_MASK" Environment variable for GPU affinity
cluster bool False Enable cluster mode
checkpoint_dir str or None None Checkpoint directory for cluster mode
enable_workstealing bool False Enable work-stealing scheduler
enable_policy_client bool False Serve policy state for PolicyClient. Implied by cluster=True; needs checkpoint_dir either way.
children_scheduler_policy str "simple_split_children_policy" Policy for partitioning resources across children
task_scheduler_policy str "fifo_policy" Policy for task scoring/priority. fifo_policy uses an unscored deque; any other policy scores tasks into a priority heap
policy_config PolicyConfig PolicyConfig() Configuration passed to scheduling policies
req_res bool True Enable ACK-based guaranteed delivery for task messages
send_retries int 10 Retry count on ACK timeout (req_res=True only)
send_timeout float 1.0 Per-attempt ACK timeout in seconds (req_res=True only)
comm_io_process bool False Run each node's data connections (parent, children, clients) in a process of their own: socket, sender checks, ACKs and sends happen there, so a busy event loop can't delay ACKs. req_res=True only; heartbeats stay in process
assign_chunk_size int 4096 A master places tasks this many at a time, yielding to the event loop between chunks, so a large update never blocks heartbeats or results
max_inflight_flushes int 1 Result sends in flight per destination; a failed send is resent in its slot. Masters forward results as they arrive; result_flush_interval and result_buffer_size only affect workers
max_inflight_task_sends int 1 TaskUpdates in flight per child; a failed send is resent in its slot before anything newer. Masters send tasks as soon as they are placed, so task_flush_interval and task_buffer_size no longer affect them
task_store str "segment" TaskStore implementation each node keeps its tasks in
task_store_hot_bytes int 1073741824 Task bytes a node keeps in memory (1 GB); the rest is offloaded to disk
task_store_dir str or None None Where offloaded tasks go; None = node-local $TMPDIR. On Aurora, point it at local scratch

PolicyConfig

Hierarchy and scheduling parameters passed to children policies:

from ensemble_launcher.config import PolicyConfig

policy_config = PolicyConfig(
    nlevels=2,
    nchildren=4,
    leaf_nodes=64,
)
Parameter Type Default Description
nlevels int 1 Hierarchy depth
nchildren int 1 Number of children per master
leaf_nodes int 1 Target number of leaf (worker) nodes
strict_priority bool False If True, tasks are scheduled in strict priority order
initial_state Dict[str, Any] {} Seeds the policy's mutable self.state, readable and updatable at runtime via PolicyClient

PolicyConfig accepts extra fields (extra="allow") so custom policies can define their own parameters.

See Custom Scheduling for how initial_state and PolicyClient combine to let a policy be retuned mid-run.

Resource Pinning

Pin tasks to specific CPUs and GPUs for optimal performance:

{
    "ensembles": {
        "pinned_ensemble": {
            "nnodes": 1,
            "ppn": 4,
            "cmd_template": "./gpu_code",
            "cpu_affinity": "0,1,2,3",
            "gpu_affinity": "0,1,2,3",
            "ngpus_per_process": 1
        }
    }
}

The gpu_selector option in LauncherConfig controls which environment variable is set for GPU affinity. It defaults to "ZE_AFFINITY_MASK" (Intel GPUs). For NVIDIA GPUs, set it to "CUDA_VISIBLE_DEVICES".

JSON Configuration Files

Both system and launcher configs can be provided as JSON files via the CLI:

system.json:

{
    "name": "my_cluster",
    "ncpus": 104,
    "ngpus": 12,
    "cpus": [0, 1, 2, 3, 4],
    "gpus": [0, 1, 2, 3]
}

launcher.json:

{
    "child_executor_name": "async_mpi",
    "task_executor_name": "async_mpi",
    "comm_name": "async_zmq",
    "nlevels": 2,
    "report_interval": 10.0,
    "return_stdout": true,
    "worker_logs": true,
    "master_logs": true
}

Performance Tuning

Communication

The communication layer uses a transport-agnostic design with pluggable backends. ZMQ is the default transport for both data and heartbeat channels, supporting 1 to 2048+ nodes.

Monitoring and Debugging

launcher_config = LauncherConfig(
    worker_logs=True,
    master_logs=True,
    report_interval=5.0,
    profile="basic",       # Communication latencies and task runtime
    # profile="timeline",  # Mean, std, sum, counts of orchestrator events
)

Logs are written to logs/master-*.log and logs/worker-*.log. Profiles go to profiles/.