Installation

Even if runko is a python module it is not (at least not yet) installable through pip or equivalent. This page explains the steps required for compiling runko with gpu and gpu aware mpi support.

Requirements

Version numbers are the ones tested to work. Newer ones should also work.

  • LLVM 17 based C++ compiler that supports HIP (tested with GCC 13 libstdc++)

  • Python 3.11.7

  • Rocm 6.0.3

  • HDF-5

  • MPI implementation that supports GPU aware MPI

  • CMake 3.23

Runko depends on following Python packages:

  • h5py

  • matplotlib

  • numpy

  • scipy

  • palettable

Additionally mpi4py package is required. However, the default installation with \(python -m pip install mpi4py\) will most likely link to different mpi implementation that what runko is using. This can lead to problems so we weed to install mpi4py manually:

MPI4PY_BUILD_MPICC=<mpi-aware C compiler> python -m pip install --no-binary=mpi4py mpi4py

Obtaining Runko

Runko depends on additional C++ libraries but they are vendored in as submodules.

git clone https://github.com/hel-astro-lab/runko
cd runko
git checkout dev-v5
git submodule update --init --recursive

Tip

To speed up \(git submodule update --init --recursive\) one can add e.g. \(-j 4\) flag to it, for \(4\) times parallelized git cloning.

Building

Runko is build with CMake. It is also used to locate dependencies. Depending on the system configuration CMake may or may not find correct libraries automatically. See CMake documentation on how they should be given.

Assuming dependencies are found automatically and \($CXX\) is the compiler we want to use, then runko can be build with:

cmake -B<build-dir> -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=$CXX .
make -j<num-of-threads> -C<build-dir>

Note

Runko sets \(CMAKE_CXX_FLAGS_{RELEASE,DEBUG}\) in \(CMakeLists.txt\). These can be adjusted for different optimization options. This might change in the future.

Building runko will install shared library representing \(pyrunko\) to repository root. Similar pattern is used in corgi, so following paths have to be added to \($PYTHONPATH\) in order to use runko:

RUNKO_PATH="<path-to-runko>"
P1="$RUNKO_PATH/"
P2="$RUNKO_PATH/external/corgi/lib"

# For example:
export PYTHONPATH="$PYTHONPATH:$P1:$P2"

LUMI

The easiest method to install runko on LUMI is by using the handy script archs/lumi-install.sh as follows:

git clone git@github.com:hel-astro-lab/runko.git
cd runko
git checkout dev-v5
git submodule update --init --recursive
./archs/lumi-install.sh

This will build runko in one go and append all necessary LUMI module loads to the virtual environment loader which can be used as follows:

source runko-venv/bin/activate

to enable runko and all its required modules on a LUMI node.

Warning: overloading login nodes

Remember not to compile on login nodes. Consider using a specific slurm job instead for building or interactive jobs:

srun --account=<account> --partition=dev-g --time=00:30:00 --nodes=1 -c32 --pty bash

Running on LUMI

For example run script on LUMI see: \(projects/pic-turbulence/jobs/multi-node.lumi?\).

Tip

If runko crashes due to invalid GPU memory access, then adding \(export MPICH_GPU_IPC_ENABLED=0\) to the job script might fix the problem.

Warning: LUMI segfaults

If you run into a problem where python segfaults or crashes due to illegal instruction it can be worked around by adding \(import matplotlib.pyplot\) before \(import runko\).

I have no idea why this happens nor why importing it first fixes the problem.