Dependency management

Arbor relies on a (small) number of dependencies. We can distinguish three kinds of deps:

  1. Source management dependencies: Git.

  2. Build dependencies. E.g. CMake, compilers like GCC or CUDA.

  3. Linking dependencies. E.g. MPI.

  4. Source dependencies. These are present as git submodules or as copy in ext/. Their use is optional: users who need integration with their package manager (e.g. apt, spack, yum) can link to those versions instead.

Note that the actual dependencies of your build configuration may vary.

In addition, spack/package.py contains a copy of the Spack package definition upstream. Here instructions for both in-repo and configure-time dependencies are defined.

This document contains rules for when and how to update dependencies, and what to be mindful of when you do.

List of dependencies

A full list of dependencies is maintained at doc/dependencies.csv:

List of configure time dependencies

Build option/target

Tool name

Minimum version

Notes

Update criteria

–

CMake

3.19

  • Is available as a module on HPC systems

  • When arbitrary changes in more recently released versions make supported

–

C++ compiler

C++17 support. See below

–

GCC

9.0.0

  • it is supported by the minimum version of CUDA.

  • it is available either by default or can be installed using standard package manager on the supported Linux versions.

–

Clang

10.0

Needs GCC 9 or later for standard library.

–

Apple Clang

11

Apple LLVM version 11.0.0 (clang-1100.0.33.8)

ARB_GPU

Hip Clang

ROCm 3.9

HIP support is currently experimental.

ARB_GPU

CUDA

11.0

  • It is available on all of the target HPC systems (Piz Daint, Juwels Booster)

ARB_WITH_MPI

MPI

Many MPI implementations are supported.

ARB_WITH_MPI, ARB_WITH_PYTHON

mpi4py

3.1.0

Minimum version depends on compatiblity with your Python and MPI versions. 3.1.0 is the first release to support Python > 3.8.

html

 
sphinx>=2.4.0
svgwrite
docutils >= 0.17

See doc/requirements.txt

unit

googletest

1.12.1

submodule ext/gtest.*

bench

Google-benchmark

submodule ext/google-benchmark

–

json

3.11.2

submodule ext/json

–

pugi

1.13

submodule ext/pugixml

–

random123

1.14.0

submodule ext/random123

–

fmt

10.0.0

submodule ext/fmt

–

tinyopt

source copy ext/tinyopt

ARB_WITH_PYTHON

pybind11

2.10.1

submodule ext/pybind11

ARB_WITH_PYTHON

Python

3.8

Python compatiblity is the range between the latest officially released point version and the minimum here specified.

  • it is not more advanced than the version specified by NEP 29.

  • it is available as a cray-python version on Piz Daint

  • it is available on labs.ebrains.eu

    • python3 = Python 3.8.10

  • Our (eventual) tutorial Jupyter Notebook env

    • Google Colab has Python 3.7

  • it is available in the supported Linuxen

Note

CMake can generate an overview of the dependency tree. Run the following in the build dir:

cmake --graphviz=graphviz/arbor-dep-graph.dot . && dot graphviz/arbor-dep-graph.dot -T png -o graphviz/arbor-dep-graph.png

This plot can be tweaked with the CMakeGraphVizOptions.cmake file in the root of the project, which currently excludes tests, the lmorpho morphology generator and all C++ examples.

User platforms

Although Arbor should in principle run everywhere modern compilers are found, a couple of user platforms are essential. These environments must be able to build Arbor without issue, if not we consider this a bug. Also, build instructions for each of them must be given in the documentation.

  • Ubuntu LTS-latest

  • Ubuntu LTS-latest-1

  • MacOS-latest

  • MacOS-latest-1

  • Cray programming environment on Piz Daint

  • Programming environment on Juwels Booster (todo CI at JSC)

  • Github Action venvs, see list.

  • Manylinux containers. For compatibility of manylinux tags, see here.

Dependency update rules

  1. doc/dependencies.csv, git submodules and spack/package.py shall be in sync.

  2. Dependencies shall be set to a (commit hash corresponding to a) specific version tag. (All current dependencies use semver.)

  3. The version shall be compatible with the user platforms (see above).

  4. The version shall be compatible with the requirements in doc/dependencies.csv.

  5. The version shall be the lowest possible, to facilitate building of complex environments.

  6. The submodule shall be set to the highest version provided by the latest Spack release (“Spack stable”). Spack CI tests both Spack stable and develop.

  7. Moreover, dependencies shall not be updated past the most recent version of the dependency in Spack.

    • This prevents Spack builds from pulling in master, when a more recent version than available is required. See here.

    • This is a manual check, e.g. for pybind: see pybind package.py

  8. Actually updating shall remain a manual process. Update may require nontrivial updates to Arbor, and updates to Spack upstream (e.g. make PR for pybind update).

  9. A dependency update shall have a separate PR, and such a PR updates a single dependency at a time, unless the dependency update requires other dependencies to be updated.

  10. This PR requires review by at least two reviewers.

    • There appears to be no way to enforce that, unless we enforce it for all PRs.

    • Optionally we could have a PR template auto-assigning to some or all of us, which means we’ll at least be notified.

  11. We will try to keep compatible to a wide range in dependency versions.

    • This includes making other components in ext git submodules, such that updates are more easily tracked.