CP2K¶
This document describes building CP2K with CMake and with dependencies built by XCONFIGURE, with a focus on GPU support by the means of OpenCL (LIBXSTREAM). All packages are placed into a common directory ($HOME is assumed), and the dependencies are installed next to CP2K, where its configure script finds them.
Dependencies¶
| Package | Needed | Source | Installed into |
|---|---|---|---|
| BLAS, LAPACK, ScaLAPACK | required | MKL, NVPL, or OpenBLAS+ScaLAPACK | system (or MKLROOT) |
| FFTW3 | required | MKL, NVPL, or FFTW | fftw/<arch>/gnu |
| DBCSR | required | GitHub | dbcsr/<arch>/gnu-gpu, etc. |
| LIBWIGNERNJ | required | cp2k.org | libwignernj/<arch>/gnu |
| LIBINT | strongly recommended | cp2k.org (preconfigured) | libint/<arch>/gnu |
| LIBXC | strongly recommended | cp2k.org | libxc/<arch>/gnu |
| LIBVORI | optional | cp2k.org | libvori/<arch>/gnu |
| LIBXSTREAM | GPU support (OpenCL) | GitHub | libxstream |
| LIBXS | for LIBXSTREAM, recommended | GitHub | libxs |
| LIBXSMM | optional (with LIBXS) | GitHub | libxsmm |
The sources of the dependencies are best taken from CP2K's download area, which carries the versions CP2K is tested with. LIBINT in particular must be one of CP2K's preconfigured packages (libint-v<version>-cp2k-lmax-<lmax>.tar.xz), since LIBINT's own source code is a generator rather than the library. LIBINT and LIBXC are optional for CP2K but needed to run common workloads: inputs using Hartree-Fock exchange (e.g., hybrid functionals) require LIBINT, and many exchange-correlation functionals require LIBXC. LIBXS is required for GPU support by LIBXSTREAM, and is useful for CPU-only builds as well (small matrix multiplications, timers, memory pool).
Further prerequisites are the GNU Compiler Collection, MPI (compiler wrappers mpicc, mpicxx, and mpif90), CMake, and Python. Intel MKL replaces OpenBLAS, ScaLAPACK, and FFTW at once.
Step-by-step Guide (GPU/OpenCL)¶
The prerequisites can be installed using the package manager, e.g., on Debian or Ubuntu:
sudo apt install build-essential gfortran cmake python3 git wget pkg-config \
libopenmpi-dev openmpi-bin libopenblas-openmp-dev libscalapack-openmpi-dev \
opencl-headers ocl-icd-opencl-dev clinfo
OpenCL consists of headers, the dispatch library (ICD loader libOpenCL.so), and at least one driver registered in /etc/OpenCL/vendors. The above opencl-headers and ocl-icd-opencl-dev packages provide the headers and the dispatch library. The driver is part of the NVIDIA driver (e.g., libnvidia-compute-<version>), or intel-opencl-icd for Intel GPUs (Intel Compute Runtime). CUDA (CUDA_PATH or nvcc in PATH) and Intel oneAPI (CMPLR_ROOT) also provide headers and a dispatch library, and are preferred in this order when found (oneAPI's dispatch library depends on Intel runtime libraries, and sourcing oneAPI is not needed for building with the GNU Compiler Collection). The available devices are listed by clinfo -l.
An older OpenBLAS (e.g., 0.3.20) may not know a newer CPU and compute wrong results with the kernels it selects for it, which OPENBLAS_CORETYPE=Haswell avoids at runtime (or a newer OpenBLAS). GPU-aware MPI is not needed; for Intel GPUs, it is not packaged and requires building MPICH with Level Zero support (--with-ze).
All packages are placed into $HOME, where each step finds the previous ones:
cd $HOME
wget --content-disposition https://github.com/hfp/xconfigure/raw/main/configure-get.sh
chmod +x configure-get.sh
git clone https://github.com/hfp/libxs.git && make -C libxs -j $(nproc)
git clone https://github.com/hfp/libxstream.git && make -C libxstream -j $(nproc)
git clone --recursive https://github.com/cp2k/dbcsr.git && cd dbcsr
../configure-get.sh dbcsr && ./configure-dbcsr-gpu.sh
cmake --build build-*-gnu-gpu --target install -j $(nproc)
cd $HOME
wget https://www.cp2k.org/static/downloads/libwignernj-0.8.0.tar.gz
tar xf libwignernj-0.8.0.tar.gz && cd libwignernj-0.8.0
../configure-get.sh libwignernj && ./configure-libwignernj.sh
cmake --build build-*-gnu --target install -j $(nproc)
cd $HOME
wget https://www.cp2k.org/static/downloads/libxc-7.1.2.tar.bz2
tar xf libxc-7.1.2.tar.bz2 && cd libxc-7.1.2
../configure-get.sh libxc && ./configure-libxc.sh
cmake --build build-*-gnu --target install -j $(nproc)
cd $HOME
wget https://www.cp2k.org/static/downloads/libint-v2.13.1-cp2k-lmax-6.tar.xz
tar xf libint-v2.13.1-cp2k-lmax-6.tar.xz && cd libint-v2.13.1-cp2k-lmax-6
../configure-get.sh libint && ./configure-libint.sh
cmake --build build-*-gnu --target install -j $(nproc)
cd $HOME
wget https://www.fftw.org/fftw-3.3.11.tar.gz
tar xf fftw-3.3.11.tar.gz && cd fftw-3.3.11
../configure-get.sh fftw && ./configure-fftw.sh
make -C build-*-gnu install -j $(nproc)
cd $HOME
git clone https://github.com/cp2k/cp2k.git && cd cp2k
../configure-get.sh cp2k && ./configure-cp2k-gpu.sh
cmake --build build-*-gnu-gpu -j $(nproc)
./run.sh benchmarks/QS/H2O-64.inp
The build-* folders assume a single architecture per source folder (see below); the configure scripts print the exact build command. Building LIBINT takes the longest (several minutes). FFTW is not needed with Intel MKL (MKLROOT), which also replaces OpenBLAS and ScaLAPACK, or with NVPL (e.g., NVIDIA Grace), which provides FFTW's interface. To rely on NVPL's BLAS, LAPACK, and ScaLAPACK instead of OpenBLAS, NVPL=1 must be used for DBCSR and CP2K alike (NVPL=1 ./configure-dbcsr-gpu.sh and NVPL=1 ./configure-cp2k-gpu.sh).
Build Instructions¶
cd $HOME && git clone https://github.com/cp2k/cp2k.git
cd cp2k
wget --content-disposition https://github.com/hfp/xconfigure/raw/main/configure-get.sh
chmod +x configure-get.sh
./configure-get.sh cp2k
To configure and build CP2K (build-<arch>-gnu-cpu), or CP2K with GPU support by the means of OpenCL (build-<arch>-gnu-gpu):
./configure-cp2k-cpu.sh
cmake --build build-<arch>-gnu-cpu -j $(nproc)
./configure-cp2k-gpu.sh
cmake --build build-<arch>-gnu-gpu -j $(nproc)
CP2K can be run from where it was built, e.g., build-<arch>-gnu-cpu/bin/cp2k.psmp (see "Running CP2K" below). Optionally, --target install installs into ../cp2k/<arch>/gnu-cpu (../cp2k/<arch>/gnu-gpu).
Here, <arch> names the target as determined by the compiler (e.g., sapphirerapids or neoverse-v2), which allows to share the folder structure among different systems (XCFG_ARCH overrides the name). For each dependency, the first of ../<name>/<arch>/gnu-cpu (gnu-gpu), ../<name>/<arch>/gnu-cpu-shared (gnu-gpu-shared), ../<name>/<arch>/gnu, ../<name>/<arch>, the same folders without <arch>, $<NAME>ROOT (e.g., LIBXSROOT), and the same folders in $HOME containing a library or package is used, and the corresponding feature of CP2K is enabled. For example, dbcsr/<arch>/gnu can be a symbolic link to the desired flavor of DBCSR. The found locations are printed when configuring CP2K, and a DBCSR that was built but not installed is pointed out. If built with GNU Make, the matching lib/<target> of LIBXS, LIBXSMM, or LIBXSTREAM is used (e.g., libxs/lib/x86_64). Further dependencies can be supplied using the CMAKE_PREFIX_PATH environment variable, and arguments after an optional "tagname" (default: gnu) are passed to CMake:
CMAKE_PREFIX_PATH=$HOME/elpa/gnu ./configure-cp2k-cpu.sh gnu -DCP2K_USE_ELPA=ON
If NVPL is installed (e.g., on NVIDIA Grace, NVPL_ROOT if not in a system location), its FFTW interface is used unless FFTW was built by XCONFIGURE. With NVPL=1, CP2K also relies on NVPL's BLAS, LAPACK, and ScaLAPACK; DBCSR must be configured with NVPL=1 as well to link the same BLAS library:
NVPL=1 ./configure-cp2k-gpu.sh
Code is generated for the system running the script (-march=native, or -mcpu=native on Arm), i.e., a build system matching the target system is assumed.
CP2K on GPUs¶
The OpenCL support enables DBCSR's OpenCL backend as well as CP2K's GPU-enabled DBM/DBT component. The OpenCL backend has comprehensive runtime-control by the means of environment variables. This can be used to assign OpenCL devices, to aggregate sub-devices (devices are split into sub-devices by default), to extract kernel shapes used by a specific workload, and to subsequently tune specific kernels. DBCSR as well as CP2K's DBM component (src/dbm) can be exercised in a standalone fashion. The LIBXSTREAM documentation describes the runtime settings of the OpenCL backend (e.g., device selection and profiling), and its OpenCL-MPS lets several ranks per node share one context per device.
Running CP2K¶
Running CP2K may go beyond a single node, and pinning processes and threads become even more important. There are several schemes available. As a rule of thumb, a high rank-count for lower node-counts may yield best results unless the workload is very memory intensive. In the latter case, lowering the number of MPI-ranks per node is effective especially if a larger amount of memory is replicated rather than partitioned by the rank-count. In contrast (communication bound), a lower rank count for multi-node computations may be desired. To ease running CP2K, there are a number of supportive scripts provided by XCONFIGURE: plan.sh (see here), run.sh (see here), and info.sh.
The run-script can be symlinked into CP2K's root directory (ln -s /path/to/xconfigure/config/cp2k/run.sh). Unless EXE names the executable, the run-script looks for cp2k.psmp (CP2K_VERSION=ssmp for cp2k.ssmp) in the following order:
exe/Linux-x86-64-intelx(legacy build,CP2K_BUILDselects another ARCH)build/binrelative to the current directory (legacy location)<CP2K_BUILD>/binorbuild-<arch>-<CP2K_BUILD>*/binifCP2K_BUILDis given (e.g.,CP2K_BUILD=build-alderlake-gnu-cpuorCP2K_BUILD=gnu-gpu), where<arch>is determined like for building- otherwise
build/binin CP2K's root directory (e.g., a symbolic link), the most recentbuild-<arch>-*/bin, orbuild-*/binif there is only one ../cp2k/<arch>/gnu/binand../cp2k/gnu/bin(installation,CP2K_BUILDselects another tagname)
The run-script's options and environment variables are shown by ./run.sh --help, which also prints the selected executable.
As soon as several experiments are finished, it becomes handy to summarize the log-output. For this case, an info-script (info.sh) is available attempting to present a table (summary of all results), which is generated from log files (.txt and .out extension by default). Log files can be captured with "tee", or the output is captured by the job scheduler.
./run.sh benchmarks/QS/H2O-64.inp | tee cp2k-h2o64-20240725b.txt
ls -1 *.txt
cp2k-h2o64-20240725a.txt
cp2k-h2o64-20240725b.txt
./info.sh [-best] [/path/to/logs]
H2O-64 Nodes R/N T/R Cases/d Seconds
cp2k-h2o64-20240725a 2 32 4 807 107.237
cp2k-h2o64-20240725b 4 16 8 872 99.962
Note: the "Cases/d" metric is calculated with integer arithmetic and hence represents fully completed cases per day (based on 86400 seconds per day). The number of seconds (as shown) is end-to-end (wall time), i.e., total time to solution including any (sequential) phase (initialization, etc.). Performance is higher if the workload requires more iterations (some publications present a metric based on iteration time).
Sanity Check¶
There is nothing that can replace the full regression test suite. However, to quickly check whether a build is sane or not, one can run for instance benchmarks/QS/H2O-64.inp and check if the SCF iteration prints like the following:
Step Update method Time Convergence Total energy Change
------------------------------------------------------------------------------
1 OT DIIS 0.15E+00 0.5 0.01337191 -1059.6804814927 -1.06E+03
2 OT DIIS 0.15E+00 0.3 0.00866338 -1073.3635678409 -1.37E+01
3 OT DIIS 0.15E+00 0.3 0.00615351 -1082.2282197787 -8.86E+00
4 OT DIIS 0.15E+00 0.3 0.00431587 -1088.6720379505 -6.44E+00
5 OT DIIS 0.15E+00 0.3 0.00329037 -1092.3459788564 -3.67E+00
6 OT DIIS 0.15E+00 0.3 0.00250764 -1095.1407783214 -2.79E+00
7 OT DIIS 0.15E+00 0.3 0.00187043 -1097.2047924571 -2.06E+00
8 OT DIIS 0.15E+00 0.3 0.00144439 -1098.4309205383 -1.23E+00
9 OT DIIS 0.15E+00 0.3 0.00112474 -1099.2105625375 -7.80E-01
10 OT DIIS 0.15E+00 0.3 0.00101434 -1099.5709299131 -3.60E-01
[...]
The column called "Convergence" must monotonically converge towards zero.
References¶
https://nholmber.github.io/2017/04/cp2k-build-cray-xc40/
https://xconfigure.readthedocs.io/cp2k/plan/
https://www.cp2k.org/static/downloads
https://www.cp2k.org/howto:compile