LAMMPS
Overview
LAMMPS is a general-purpose molecular dynamics software package for massively parallel computers. It is written in an exceptionally clean style that makes it one of the more popular codes for users to extend and it currently has dozens of user-developed extensions.
For details about the code and its usage, see the LAMMPS home page. This page provides information specific to running on Polaris at the ALCF.
Using LAMMPS at ALCF
ALCF provides assistance with build instructions, compiling executables, submitting jobs, and providing prebuilt binaries (upon request). A collection of Makefiles and submission scripts are available in the ALCF GettingStarted repo here. For questions, contact us at support@alcf.anl.gov.
How to Obtain the Code
LAMMPS is an open-source code, which can be downloaded from the LAMMPS website.
Building on Polaris
After LAMMPS has been downloaded and unpacked on an ALCF filesystem, users should see a directory whose name is of the form lammps-<version>. One should then see the Makefile lammps-<version>/src/MAKE/MACHINES/Makefile.polaris in recent versions that can be used as a starting point for compilation on Polaris. Copies of Makefiles for building with the GPU/KOKKOS package using CUDA for GPU support with the GNU/NVHPC compiler are available in the ALCF GettingStarted repo here. For older versions of LAMMPS, you may need to take an existing Makefile (e.g. Makefile.mpi) for your specific version of LAMMPS and edit the top portion appropratiately to create a new Makefile.polaris.
KOKKOS package and GNU compilers
The following modules are useful for this particular build. Note, the cmake module is not required if using the GNU Makefile procedure to build LAMMPS. The initial module restore is just setting the default environment as the starting point. Users may find it useful to copy these module commands into a small helper script to assist with compiling and running LAMMPS (e.g. setup_lammps_gnu.sh).
module restore
module load craype-accel-nvidia80
module swap PrgEnv-nvhpc PrgEnv-gnu
module use /soft/modulefiles
module load cudatoolkit-standalone
module load spack-pe-base cmake
The top portion of Makefile.polaris_gnu_kokkos used to build LAMMPS with the KOKKOS package using GNU as the host-side compiler is shown as an example.
# polaris_nvhpc_kokkos = Flags for NVIDIA A100, GNU Compiler, MPICH, CUDA
# module load craype-accel-nvidia80
# module swap PrgEnv-nvhpc PrgEnv-gnu
# module use /soft/modulefiles
# module load cudatoolkit-standalone
# make polaris_gnu_kokkos -j 32
SHELL = /bin/sh
# ---------------------------------------------------------------------
# compiler/linker settings
# specify flags and libraries needed for your compiler
KOKKOS_DEVICES = Cuda,OpenMP
KOKKOS_ARCH = Ampere80
KOKKOS_ABSOLUTE_PATH = $(shell cd $(KOKKOS_PATH); pwd)
KOKKOS_CUDA_OPTIONS = "enable_lambda,disable_malloc_async"
export NVCC_WRAPPER_DEFAULT_COMPILER = nvc++
CC = $(KOKKOS_ABSOLUTE_PATH)/bin/nvcc_wrapper
CCFLAGS = -g -O3 -mp -DLAMMPS_MEMALIGN=64
#CCFLAGS += -DLAMMPS_BIGBIG
SHFLAGS = -fPIC
DEPFLAGS = -M
LINK = $(CC)
LINKFLAGS = $(CCFLAGS)
LIB =
SIZE = size
With the appropriate LAMMPS Makefile in place an executable can be compiled as in the following example. Note, per-user limits on the login nodes will reduce the maximum parallelism for compilation. Users are encouraged to compile on a compute node in an interactive session if necessary.
KOKKOS package and NVHPC compilers
The following modules are useful for this particular build. Note, the cmake module is not required if using the GNU Makefile procedure to build LAMMPS. The initial module restore is just setting the default environment as the starting point. Users may find it useful to copy these module commands into a small helper script to assist with compiling and running LAMMPS (e.g. setup_lammps_nvhpc.sh).
module restore
module load craype-accel-nvidia80
module use /soft/modulefiles
module load spack-pe-base cmake
The top portion of Makefile.polaris_nvhpc_kokkos used to build LAMMPS with the KOKKOS package using the NVHPC compilers is shown as an example.
# polaris_nvhpc_kokkos = Flags for NVIDIA A100, NVIDIA Compiler, MPICH, CUDA
# make polaris_nvhpc_kokkos -j 16
SHELL = /bin/sh
# ---------------------------------------------------------------------
# compiler/linker settings
# specify flags and libraries needed for your compiler
KOKKOS_DEVICES = Cuda,OpenMP
KOKKOS_ARCH = Ampere80
KOKKOS_ABSOLUTE_PATH = $(shell cd $(KOKKOS_PATH); pwd)
KOKKOS_CUDA_OPTIONS = "enable_lambda,disable_malloc_async"
export NVCC_WRAPPER_DEFAULT_COMPILER = nvc++
CRAY_INC = $(shell CC --cray-print-opts=cflags)
CRAY_LIB = $(shell CC --cray-print-opts=libs)
#$(info CRAY_INC = ${CRAY_INC})
#$(info CRAY_LIB = ${CRAY_LIB})
CC = $(KOKKOS_ABSOLUTE_PATH)/bin/nvcc_wrapper
CCFLAGS = -g -O3 -mp -DLAMMPS_MEMALIGN=64
CCFLAGS += $(CRAY_INC)
SHFLAGS = -fPIC
DEPFLAGS = -M
LINK = $(CC)
LIB =
LIB += $(CRAY_LIB)
SIZE = size
With the appropriate LAMMPS Makefile in place an executable can be compiled as in the following example. Note, per-user limits on the login nodes will reduce the maximum parallelism for compilation. Users are encouraged to compile on a compute node in an interactive session if necessary.
Running Jobs on Polaris
An example submission script for running a 64-node KOKKOS-enabled LAMMPS executable is below as an example. Additional information on LAMMPS application flags and options is described on the LAMMPS website.
#!/bin/sh
#PBS -l select=64:system=polaris
#PBS -l place=scatter
#PBS -l walltime=0:15:00
#PBS -l filesystems=home:grand:eagle
#PBS -q prod
#PBS -A Catalyst
export MPICH_GPU_SUPPORT_ENABLED=1
NNODES=`wc -l < $PBS_NODEFILE`
# per-node settings
NRANKS=4
NDEPTH=8
NTHREADS=1
NGPUS=4
NTOTRANKS=$(( NNODES * NRANKS ))
. ./setup_lammps_gnu.sh
EXE=/home/knight/bin/lmp_polaris_gnu_kokkos
EXE_ARG="-in in.reaxc.hns -k on g ${NGPUS} -sf kk -pk kokkos neigh half neigh/qeq full newton on "
# OMP settings mostly to quiet Kokkos messages
MPI_ARG=" -n ${NTOTRANKS} --ppn ${NRANKS} --depth=${NDEPTH} --cpu-bind depth "
OMP_ARG=" --env OMP_NUM_THREADS=${NTHREADS} --env OMP_PROC_BIND=spread --env OMP_PLACES=cores "
COMMAND="mpiexec ${MPI_ARG} ${OMP_ARG} ${EXE} ${EXE_ARG}"
echo "COMMAND= ${COMMAND}"
${COMMAND}
GPU package
The module environments above can be used to build LAMMPS with the GPU package as well. Copies of Makefiles for building with the GPU package using CUDA for GPU support with the GNU and NVHPC compilers are available in the ALCF GettingStarted repo here.
Performance Notes
Some useful information on accelerator packages and expectations can be found on the LAMMPS website here.