ezpz¶
ezpz is a thin layer over PyTorch distributed that removes the per-machine boilerplate: rank and topology detection, backend selection, hostfile parsing, and launcher invocation. The same script runs on Aurora, Polaris, and Perlmutter without an if machine == ... branch.
Where to read more
- Full docs (canonical): ezpz.cool — install, CLI reference, examples, and guides.
- Source & issues: github.com/saforem2/ezpz
- New in Nov 2026: remote submission via the ALCF IRI API — blog post with walkthrough and API details.
The problem it solves¶
Moving a working PyTorch script between ALCF systems, or from ALCF to NERSC, normally means rewriting the parts that have nothing to do with the model:
| Concern | Aurora / Sunspot | Polaris / Sophia | Perlmutter |
|---|---|---|---|
| Scheduler | PBS Pro | PBS Pro | Slurm |
| Launcher | mpiexec | mpiexec | srun |
| Accelerator | Intel XPU | NVIDIA | NVIDIA |
| Backend | xccl | nccl | nccl |
| Hostfile | $PBS_NODEFILE | $PBS_NODEFILE | $SLURM_NODELIST |
| Local rank var | PALS_LOCAL_RANKID | PMI_LOCAL_RANK | SLURM_LOCALID |
ezpz detects all of it at runtime. The Supported Systems page lists the full matrix, including the hostname prefixes used for detection (x4* for Aurora, x1* for Sunspot, and so on).
It also sets the Aurora-specific environment that is easy to get wrong:
ZE_FLAT_DEVICE_HIERARCHY=FLAT, so all 12 tiles are visible. Without it only 6 devices appear and every rank with local rank ≥ 6 fails withdevice index out of range.CCL_OP_SYNC=1, which works around an oneCCL async-completion hang that affects FSDP2 with tensor parallel degree > 1 (AuroraBugTracking#174).
Setup¶
module load frameworks
uv pip install --no-cache --link-mode=copy "git+https://github.com/saforem2/ezpz" # (1)!
- Install from the repository. PyPI's
ezpzis an unrelated package — always use thegit+https://github.com/saforem2/ezpzURL. Ifuvis not available,pip installworks butuv pip install --no-cache --link-mode=copyis preferred for reproducibility.
Then, inside a job script or an interactive allocation:
ezpz_setup_env loads the right modules for the current machine, activates the Python environment, and exports the machine-specific environment variables. Verify with:
Launching¶
ezpz launch wraps the scheduler's launcher. It reads the hostfile, computes ranks-per-node from the detected accelerator count, and applies CPU binding:
That command is unchanged across every supported machine; on Aurora it becomes a 12-rank-per-node mpiexec --pmi=pmix, on Perlmutter an srun. See ezpz launch for the flags, including --auto-retry for resubmitting around node failures. From your laptop you can also submit remotely — see Remote jobs below.
Minimal PBS job on Aurora
Remote jobs¶
ezpz can submit and manage jobs from outside ALCF via the ALCF IRI API — no login-node SSH session required. The same submit / status / cancel surface also works over SSH for machines the IRI does not cover.
Get a token once from an ALCF UAN (uvx alcf-tokens login iri), then:
# submit from your laptop — returns a job ID like 8936286.aurora-pbs-0001
ezpz submit --remote aurora -N 2 -q debug -A <project> -- python3 train.py
# 8936286.aurora-pbs-0001 (via iri)
# check status (Rich table on a TTY, plain text when piped)
ezpz status --remote aurora --limit 5
ezpz status --remote aurora --states queued,active
# live poll with desktop notifications on Q->R / R->F
ezpz status --remote aurora --watch
# fetch stdout without opening an interactive SSH session
ezpz fetch 8936286 --remote aurora --lines 80 --no-dedupe
# per-example benchmark logs (not the MPI-duplicated spool)
ezpz logs 8936286 --remote aurora --example fsdp --lines 100
# extend walltime or move queues without resubmitting
ezpz extend 8936286 --remote aurora --time 06:00:00
ezpz extend 8936286 --remote aurora --queue prod --account <project> --time 12:00:00
# cancel
ezpz cancel 8936286 --remote aurora
# what machines exist and whether they're up
ezpz resources
ezpz incidents --resource aurora --limit 5
# shell completions (bash / zsh / fish)
ezpz completion bash
# `ezpz status --remote aurora --limit 5` (TTY) renders as:
# JOBID S QUEUE NODES WALL ELAP JOBNAME NOTE
# 8936286 R prod 2 06:00 01:23 train —
# 8936102 Q debug 1 01:00 -- debug-run —
All of status / fetch / logs / extend / cancel / resources / incidents support --json for scripting (| jq), and --dry-run where it matters. See Remote submission for the full auto fallback contract (IRI-first on Aurora / Polaris / Crux, SSH elsewhere), the shared-FS workdir rule, and the --env / --env-setup-script environment story. CLI reference: ezpz submit · ezpz status · ezpz fetch · ezpz logs · ezpz extend · ezpz cancel · ezpz resources · ezpz incidents · ezpz completion.
Using it in your own script¶
Two calls replace the usual distributed preamble:
- Initializes the process group with the correct backend, binds the device, and returns the global rank. No
MASTER_ADDR/MASTER_PORTplumbing.
See Distributed Training for the full API.
Features¶
| Area | What it does | Details |
|---|---|---|
| Launcher | Scheduler-aware mpiexec / srun wrapper with CPU binding and --auto-retry | ezpz launch |
| Setup | Per-machine module loading and environment export | Quickstart |
| Distributed | Backend selection, rank/topology detection, DDP and FSDP2 wrapping | Distributed Training |
| Parallelism | FSDP2, tensor parallel, sequence parallel, 2D meshes | Parallelism |
| Examples | Runnable training examples with a shared model-size ladder (debug → xxxl) | Examples |
| Profiling | torch.profiler wrapper with automatic XPU/CUDA activity selection | Profiling |
| Metrics | History for per-iteration metrics, plots, and W&B / MLflow logging | History |
| Checkpointing | Distributed checkpoint save/restore, resumable after node failure | Checkpoint & Restart |
| Fault handling | Node-kill injection and automatic retry | Fault Injection |
| Remote | Submit / status / fetch / logs / extend / cancel via IRI or SSH | Remote submission |
| Diagnostics | ezpz doctor, ezpz test | ezpz test |
Examples¶
Each example takes a --model size from debug (laptop-runnable) through xxxl (~10B), so the same command scales from a smoke test to a real job:
ezpz launch python3 -m ezpz.examples.test --model small
ezpz launch python3 -m ezpz.examples.fsdp --model large
ezpz launch python3 -m ezpz.examples.fsdp_tp --model large --tp 2 # (1)!
- FSDP2 + tensor parallel. On Aurora this requires
CCL_OP_SYNC=1, whichezpz_setup_envsets; see AuroraBugTracking#174.
The full set is listed under Examples, and ezpz benchmark runs them in sequence with a summary report.
Other systems¶
The same workflow applies on Polaris and Sophia (PBS, nccl) and on Perlmutter (Slurm, srun); the NERSC specifics are in the Perlmutter guide.
Getting help¶
- Docs: ezpz.cool — CLI index, examples, troubleshooting
- Source & issues: github.com/saforem2/ezpz
- Remote walkthrough: samf.sh/posts/2026/10/16/