Julia¶

Introduction¶

Julia is a high-level, high-performance programming language designed for technical and scientific computing. It combines the ease of use of dynamic languages with the performance of compiled languages, making it well-suited for large-scale simulations and data analysis.

This guide details how to configure and run Julia on the Aurora supercomputer, focusing on leveraging the system's key architectural features for large-scale parallel and GPU-accelerated computing.

All the source files used in this documentation are located at https://github.com/anlsys/julia_alcf. Feel free to open PRs!

Julia Setup¶

Julia is available on Aurora as a module.

We recommend setting your environment variable $JULIA_DEPOT_PATH to a project directory PROJECT on Flare or Gila for faster file access and to avoid filling up your home directory.

Add the following to your shell configuration file (e.g., ~/.bashrc or ~/.bash_profile):

export JULIA_DEPOT_PATH="/flare/PROJECT/$USER/julia_depot"

If $JULIA_DEPOT_PATH is not set, it defaults to ~/.julia with a warning when you load the module.

Loading Julia¶

Load the Julia module:

module use /soft/modulefiles
module load julia

By default, this loads the latest stable version of Julia. To load a specific version:

module load julia/1.12  # Latest version
module load julia/1.11  # Previous version
module load julia/1.10  # LTS (Long Term Support)

Version Policy¶

Aurora maintains three Julia versions:

• Latest stable release (currently 1.12): The most recent stable version with the newest features and performance improvements
• Previous version (currently 1.11): The previous stable release for compatibility with recent projects
• LTS (Long Term Support) (currently 1.10): Provides long-term stability with bug fixes but no new features, ideal for production workloads requiring consistency

When new versions are released, the oldest non-LTS version is retired (removed from the system and no longer available), and the LTS version is updated according to the Julia LTS release schedule.

Configuring the Programming Environment¶

To leverage Aurora's architecture, you must configure Julia to use the system's optimized libraries for MPI.jl, oneAPI.jl, and HDF5.jl. For a modern, interactive development experience, we recommend using Visual Studio Code with the official Julia and Remote - SSH extensions.

The Julia module on Aurora is pre-configured with system-specific preferences (via LocalPreferences.toml in the system load path) to ensure these packages use the correct system libraries (MPICH, Intel Level Zero, HDF5).

Install the required packages in your Julia environment with the following commands:

using Pkg
Pkg.add(["MPI", "oneAPI", "HDF5", "KernelAbstractions"])

Note: MPIPreferences does not need to be explicitly added as it's a dependency of MPI.jl.

oneAPI-Aware MPI¶

oneAPI-aware MPI is enabled by default on Aurora. You can pass oneArray objects directly to MPI.jl functions without explicit host-device transfers, enabling efficient GPU-to-GPU communication across nodes:

using oneAPI, MPI
MPI.Init()

# Create a oneArray and pass it directly to MPI operations
data = oneAPI.rand(Float64, 100)
MPI.Allreduce!(data, +, MPI.COMM_WORLD)  # GPU-to-GPU communication

Verify Configuration on a Compute Node¶

The Aurora login nodes do not have GPU access. You must request an interactive job to test your GPU configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Request an interactive node
qsub -I -l select=1,walltime=1:00:00,filesystems=home:flare -A [PROJECT] -q debug

# Once on the node, load Julia and run the verification
module use /soft/modulefiles
module load julia
julia -e "using oneAPI; oneAPI.versioninfo()"

# Expected Output Snippet
# oneAPI.jl version: ...
# Intel Level Zero version: ...
# ...
# 12 devices:
#   0: Intel(R) Data Center GPU Max 1550 ...

Example Julia Code for Approximating Pi¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

using oneAPI
using HDF5
using MPI
using Printf
using Random
using KernelAbstractions

# GPU kernel to check if points fall within a circle
@kernel function pi_kernel!(x, y, d)
    idx = @index(Global)
    @inbounds d[idx] = (x[idx] - 0.5)^2 + (y[idx] - 0.5)^2 <= 0.25 ? 1 : 0
end

# Function to run the computation on a single GPU
function approximate_pi_gpu(n::Integer)
    x = oneAPI.rand(Float64, n)
    y = oneAPI.rand(Float64, n)
    d = oneArray{Float64}(undef, n)

    backend = get_backend(d)
    kernel! = pi_kernel!(backend)
    kernel!(x, y, d, ndrange=n)
    KernelAbstractions.synchronize(backend)

    return sum(d)
end

function main()
    n = 100_000  # Number of points per MPI rank

    # Use a fixed random seed for reproducibility
    Random.seed!(1234 + MPI.Comm_rank(MPI.COMM_WORLD))

    # Each rank computes its sum on the GPU, then we reduce across all ranks
    local_sum = approximate_pi_gpu(n)
    total_sum = MPI.Allreduce(local_sum, MPI.SUM, MPI.COMM_WORLD)

    # Calculate final approximation
    comm_size = MPI.Comm_size(MPI.COMM_WORLD)
    pi_approx = (4 * total_sum) / (n * comm_size)

    if MPI.Comm_rank(MPI.COMM_WORLD) == 0
        @printf "Approximation of π: %.10f\n" pi_approx
        @printf "Error:              %.10f\n" abs(pi_approx - π)
    end
    return pi_approx
end

# --- Main Execution ---
MPI.Init()

if !isinteractive()
    pi_approx = main()

    # Rank 0 writes the result to an HDF5 file
    if MPI.Comm_rank(MPI.COMM_WORLD) == 0
        h5open("pi_approximation.h5", "w") do file
            write(file, "pi", pi_approx)
        end
    end
    MPI.Finalize()
end

Job Submission Script¶

This PBS script requests resources and launches the Julia application using mpiexec:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#!/bin/bash -l
#PBS -l select=1:system=aurora
#PBS -l place=scatter
#PBS -l walltime=0:10:00
#PBS -l filesystems=home:flare
#PBS -q debug
#PBS -A YOUR_PROJECT_ID

cd ${PBS_O_WORKDIR}
module use /soft/modulefiles
module load julia

# --- Job Settings ---
NNODES=`wc -l < $PBS_NODEFILE`
NRANKS_PER_NODE=12
NDEPTH=8 # For CPU binding
NTOTRANKS=$(( NNODES * NRANKS_PER_NODE ))

echo "Nodes: ${NNODES}, Total Ranks: ${NTOTRANKS}, Ranks/Node: ${NRANKS_PER_NODE}"

# --- Execution ---
# Path to the Julia executable
JULIA_EXE_PATH=$(which julia)

# mpiexec arguments
MPI_ARGS="-n ${NTOTRANKS} --ppn ${NRANKS_PER_NODE} --depth=${NDEPTH} --cpu-bind depth"

echo "Running Julia from: ${JULIA_EXE_PATH}"
mpiexec ${MPI_ARGS} ${JULIA_EXE_PATH} --project pi.jl