Build and install from source¶

This guide covers building and installing Arbor using CMake, which is the recommended method for configuring Arbor for HPC applications and developers.

We start with an overview of the building process, and the various options available to customize the build. Then we cover installing and running on HPC clusters, followed by a troubleshooting guide for common build problems.

Note

To get help in case of problems installing Arbor, please make an issue on the Arbor Github issues page.

Requirements¶

Minimum requirements¶

The non distributed (i.e. no MPI) version of Arbor can be compiled on Linux or OS X systems with very few tools.

Required Tools¶
Tool	Notes
Git	To check out the code, minimum version 2.0.
CMake	To set up the build, minimum version 3.18.
compiler	A C++17 compiler. See compilers.

Compilers¶

Arbor requires a C++ compiler that fully supports C++17. We recommend using GCC or Clang, for which Arbor has been tested and optimised.

Supported Compilers¶
Compiler	Min version	Notes
GCC	8.4.0
Clang	8.0	Needs GCC 8 or later for standard library.
Apple Clang	9	Apple LLVM version 9.0.0 (clang-900.0.39.2)
Hip Clang	ROCm 3.9	HIP support is currently experimental.

Note

The CC and CXX environment variables specify which compiler executable CMake should use. If these are not set, CMake will attempt to automatically choose a compiler, which may be too old to compile Arbor. For example, the default compiler chosen below by CMake was GCC 4.8.5 at /usr/bin/c++, so the CC and CXX variables were used to specify GCC 10.2.0 before calling cmake.

# on this system CMake chooses the following compiler by default
$ c++ --version
c++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)

# check which version of GCC is available
$ g++ --version
g++ (GCC) 10.2.0
Copyright (C) 2020 Free Software Foundation, Inc.

# set environment variables for compilers
$ export CC=`which gcc`; export CXX=`which g++`;

# launch CMake
# the compiler version and path is given in the CMake output
$ cmake ..
-- The C compiler identification is GNU 10.2.0
-- The CXX compiler identification is GNU 10.2.0
-- Check for working C compiler: /cm/local/apps/gcc/10.2.0/bin/gcc
-- Check for working C compiler: /cm/local/apps/gcc/10.2.0/bin/gcc -- works
...

Note

It is commonly assumed that to get the best performance one should use a vendor-specific compiler (e.g. the Intel, Cray or IBM compilers). These compilers are often better at auto-vectorizing loops, however for everything else GCC and Clang nearly always generate more efficient code.

The main computational loops in Arbor are generated from NMODL. The generated code is explicitly vectorised, obviating the need for vendor compilers, and we can take advantage of the benefits of GCC and Clang: faster compilation times; fewer compiler bugs; and better support for C++ standards.

Note

The IBM XL C++ compiler and Intel C++ compiler are not supported, owing to unresolved compiler issues. We strongly recommend building with GCC or Clang instead on PowerPC and Intel platforms.

Optional requirements¶

GPU support¶

Arbor has full support for NVIDIA GPUs, for which the NVIDIA CUDA toolkit version 10 is required. And experimental support for AMD GPUs when compiled with hip-clang (non-release compiler).

Distributed¶

Arbor uses MPI to run on HPC cluster systems. Arbor has been tested on MVAPICH2, OpenMPI, Cray MPI, and IBM MPI. More information on building with MPI is in the HPC cluster section.

Python¶

Arbor has a Python frontend, for which a minimum of Python 3.6 is required. In addition, numpy is a runtime requirement for the Python package. In order to use MPI in combination with the python frontend the mpi4py Python package is recommended. See Python frontend for more information.

NeuroML¶

Arbor supports reading cell morphologies defined in NeuroML version 2 through an additional support library arborio. This library requires libxml2 for the parsing of NeuroML2 XML, if it is built with NeuroML support enabled. See NeuroML support for more information.

Documentation¶

To build a local copy of the html documentation that you are reading now, you will need to install Sphinx.

External dependencies¶

For the (optional) python bindings Arbor uses pybind11, and JSON parsing is faciliated through nlohmann json.

There are two ways to obtain these libraries. The default way is to use them from the system, e.g., installed via apt install python3-pybind11 and apt install nlohmann-json3-dev for a Debian based distribution.

The other possiblity is to use versions of these dependencies that are bundled with Arbor via the CMAKE option ARB_USE_BUNDLED_LIBS. If set, pybind11 is retrieved from a Git submodule (see below) and nlohmann json from a copy in the checked out sources.

It is also possible to select only one of the two libraries to be taken from the system or from Arbor.

Getting the code¶

The easiest way to acquire the latest version of Arbor is to check the code out from the Github repository:

git clone https://github.com/arbor-sim/arbor.git --recurse-submodules

We recommend using a recursive checkout, because Arbor uses Git submodules for some of its library dependencies. The CMake configuration attempts to detect if a required submodule is available, and will print a helpful warning or error message if not, but it is up to the user to ensure that all required submodules are downloaded.

The Git submodules can be updated, or initialized in a project that didn’t use a recursive checkout:

git submodule update --init --recursive

You can also point your browser to Arbor’s Github page and download a zip file. If you use the zip file, then don’t forget to run Git submodule update manually.

Building and installing Arbor¶

Once the Arbor code has been checked out, first run CMake to configure the build, then run make.

Below is a simple workflow for: 1) getting the source; 2) configuring the build; 3) building; 4) running tests; 5) installing.

For more detailed build configuration options, see the quick start guide.

# 1) Clone.
git clone https://github.com/arbor-sim/arbor.git --recurse-submodules
cd arbor

# Make a path for building
mkdir build
cd build

# 2) Use CMake to configure the build.
# By default Arbor builds in release mode, i.e. with optimizations on.
# Release mode should be used for installing and benchmarking Arbor.
cmake .. # add -DARB_USE_BUNDLED_LIBS=ON to use bundled/git-submoduled libs

# 3.1) Build Arbor library.
make -j 4
# 3.2) Build Arbor unit tests.
make -j 4 tests
# 3.3) Build Arbor C++ examples.
make -j 4 examples

# 4) Run tests.
./bin/unit

# 5) Install (by default, to /usr/local).
make install

This will build Arbor in release mode with the default C++ compiler.

Quick start: examples¶

Below are some example of CMake configurations for Arbor. For more detail on individual CMake parameters and flags, follow links to the more detailed descriptions below.

Debug mode with assertions enabled.

If you encounter problems building or running Arbor, compile with these options for testing and debugging.

cmake -DARB_WITH_ASSERTIONS=ON -DCMAKE_BUILD_TYPE=debug

Release mode with Clang.

export CC=`which clang`
export CXX=`which clang++`
cmake

Release mode for the Haswell architecture and explicit vectorization of kernels.

cmake -DARB_VECTORIZE=ON -DARB_ARCH=haswell

Release mode with explicit vectorization, targeting the Broadwell architecture, with support for Nvidia GPUs, and building with GCC 9.

export CC=gcc-9
export CXX=g++-9
cmake -DARB_VECTORIZE=ON -DARB_ARCH=broadwell -DARB_GPU=cuda

Release mode with explicit vectorization, targeting the Broadwell architecture, with support for AMD GPUs, and building with hipcc.

export CC=clang
export CXX=hipcc
cmake -DARB_VECTORIZE=ON -DARB_ARCH=broadwell -DARB_GPU=hip

Release mode with explicit vectorization, optimized for the local system architecture and install in /opt/arbor

cmake -DARB_VECTORIZE=ON -DCMAKE_INSTALL_PREFIX=/opt/arbor

Release mode with profiling enabled

cmake -DARB_WITH_PROFILING=ON

Build target¶

By default, Arbor is built in release mode, which should be used when installing or benchmarking Arbor. To compile in debug mode (which in practical terms means with -g -O0 flags), use the CMAKE_BUILD_TYPE CMake parameter.

cmake -DCMAKE_BUILD_TYPE={debug,release}

Architecture¶

By default, Arbor is built to target whichever architecture is the compiler default, which often involves a sacrifice of performance for binary portability. The target architecture can be explicitly set with the ARB_ARCH configuration option. This will be used to direct the compiler to use the corresponding instruction sets and to optimize for that architecture.

When building and installing on the same machine, a good choice for many environments is to set ARB_ARCH to native:

cmake -DARB_ARCH=native

When deploying on a different machine (cross-compiling) specify the specific architecture of the target machine. The valid values correspond to those given to the -mcpu or -march options for GCC and Clang; the build system will translate these names to corresponding values for other supported compilers.

Specific recent x86-family Intel CPU architectures include broadwell, skylake and knl. Complete lists of architecture names can be found in the compiler documentation: for example GCC x86 options, PowerPC options, and ARM options.

# Intel architectures
cmake -DARB_ARCH=broadwell        # broadwell with avx2
cmake -DARB_ARCH=skylake-avx512   # skylake with avx512 (Xeon server)
cmake -DARB_ARCH=knl              # Xeon Phi KNL

# ARM Arm8a
cmake -DARB_ARCH=armv8-a

# IBM Power8
cmake -DARB_ARCH=power8

Vectorization¶

Explicit vectorization of computational kernels can be enabled in Arbor by setting the ARB_VECTORIZE CMake flag. This option is typically used in conjunction with the ARB_ARCH option to specify the target architecture: without SIMD support in Arbor for the architecture, enabling ARB_VECTORIZE will lead to a compilation error.

cmake -DARB_VECTORIZE=ON -DARB_ARCH=native

With this flag set, the library will use architecture-specific vectorization intrinsics to implement these kernels. Arbor currently has vectorization support for x86 architectures with AVX, AVX2 or AVX512 ISA extensions; and for AArch64 ARM architectures with NEON and SVE (first available on ARMv8-A).

Note

Note that on x86-64 platforms compilation will fail if you enable vectorization, but the CPU or -DARB_ARCH does not support any form of AVX.

GPU backend¶

Compiling for the GPU backend is controlled by the ARB_GPU CMake option which is used to select between NVIDIA and AMD GPUs as well as specify the chosen GPU compiler.

none: The default option. Disables the GPU backend.
cuda: Enables the GPU backend for NVIDIA GPUs and compiles Arbor with nvcc (CUDA files), and the default C++ compiler (C++ files).
cuda-clang: Enables the GPU backend for NVIDIA GPUs and compiles Arbor with clang.
hip: Enables the experimental GPU backend for AMD GPUs and compiles Arbor with hipcc.

NVIDIA GPUs:

Arbor supports NVIDIA GPUs using CUDA. Compiling Arbor for NVIDIA GPUs requires the CUDA Toolkit.

cmake -DARB_GPU=cuda

cmake -DARB_GPU=cuda-clang

Arbor is built for all supported NVIDIA GPUs and the available GPU will be used at runtime.

Depending on the configuration of the system where Arbor is being built, the C++ compiler may not be able to find the cuda.h header when building for NIDIA GPUs. The easiest workaround is to add the path to the include directory containing the header to the CPATH environment variable before configuring and building Arbor, for example:

export CPATH="/opt/cuda/include:$CPATH"
cmake -DARB_GPU=cuda

HIP GPUs:

Arbor has experimental support for AMD GPUs using HIP. The only compiler currently supported is the non-release hip-clang (hipcc) compiler. (For instructions on how to build hipcc, refer to the HIP documentation).

CMake configuration for compiling Arbor with hipcc (CUDA and C++ files):

export CC=clang
export CXX=hipcc
cmake -DARB_GPU=hip

Arbor is built for all supported AMD GPUs and the available GPU will be used at runtime.

Note

Arbor supports and has been tested on Pascal (P100) and Volta (V100) NVIDIA GPUs, as well as Mi50 and Mi60 AMD GPUs.

Python frontend¶

Arbor can be used with a python frontend which is enabled by toggling the CMake ARB_WITH_PYTHON option:

cmake -DARB_WITH_PYTHON=ON

By default ARB_WITH_PYTHON=OFF. When this option is turned on, a Python module called arbor is built.

A specific version of Python can be set when configuring with CMake using the PYTHON_EXECUTABLE variable. For example, to use Python 3.8 installed on a Linux system with the executable in /usr/bin/python3.8:

cmake .. -DARB_WITH_PYTHON=ON -DPYTHON_EXECUTABLE=/usr/bin/python3.8

By default the Python package will be installed in the appropriate sub-directory inside CMAKE_INSTALL_PREFIX, determined by querying Python’s sysconfig library. For example ${CMAKE_INSTALL_PREFIX}/lib/python3.9/site-packages/.

To install the module in a different location, independent of CMAKE_INSTALL_PREFIX, use ARB_PYTHON_LIB_PATH to specify the location where the Python module is to be installed.

cmake .. -DARB_WITH_PYTHON=on -DARB_PYTHON_PATH_LIB=/custom/path

Note

The location of libraries under a prefix in only guaranteed to be standard for Python’s global library location. Therefore, correct installation of the Python package to any other location using CMAKE_INSTALL_PREFIX, such as user directory (e.g. ~/.local), a Python or Conda virtual environment, may result in installation to a wrong path.

python3 -m site --user-site (for user installations) or a path from python3 -c 'import site; print(site.getsitepackages())' (for virtual environment installation) can be used in combination with ARB_PYTHON_LIB_PATH.

In addition, installation via pip or python setup.py is guaranteed to find the right path. Please refer to the Python installation instruction.

# A demonstration using ARB_PYTHON_LIB_PATH

# Set up your venv.
mkdir myenv
cd myenv/
python3 -m venv env
source env/bin/activate

# Install dependencies
pip3 install numpy

# Obtain arbor
git clone --recursive git@github.com:arbor-sim/arbor.git

# Manually set the prefix under which the python package will be installed.
# In this case, the first directory found by querying Python's list of site-package directories.
pyprefix=`python3 -c 'import site; print(site.getsitepackages()[0])'`

# Setup CMake
mkdir build
cd build
cmake ../arbor -DARB_WITH_PYTHON=on       \       # enable python support.
               -DARB_USE_BUNDLED_LIBS=on  \       # use bundled versions of deps.
               -DARB_PYTHON_LIB_PATH="$pyprefix"  # set Python installation path.

# Build and install
make -j4
make install

# Test it out!
python -c "import arbor; print(arbor.__config__)"

The Arbor Python wrapper has optional support for mpi4py, though it is not required to use Arbor with Python and MPI. CMake will attempt to automatically detect mpi4py if configured with both -DARB_WITH_PYTHON=ON and MPI -DARB_WITH_MPI=ON. If CMake fails to find mpi4py when it should, the easiest workaround is to add the path to the include directory for mpi4py to the CPATH environment variable before configuring and building Arbor:

# search for path tp python's site-package mpi4py
for p in `python3 -c 'import sys; print("\n".join(sys.path))'`; do echo ===== $p; ls $p | grep mpi4py; done

===== /path/to/python3/site-packages
mpi4py

# set CPATH and run cmake
export CPATH="/path/to/python3/site-packages/mpi4py/include/:$CPATH"

cmake -DARB_WITH_PYTHON=ON -DARB_WITH_MPI=ON

NeuroML support¶

Arbor has limited support for NeuroML version 2 through an additional library arborio. This library will be built with NeuroML support if the option -DARB_WITH_NEUROML=ON is passed to CMake at configuration time. arborio depends upon the the libxml2 library for XML parsing.

Arbor will additionally install the static library libarborio.a. Applications using this functionality will need to link against this library in addition to the main Arbor library and libxml2. For example:

g++ -std=c++17 -pthread mycode.cpp -larborio -larbor -lxml2

For projects using CMake, Arbor NeuroML support can be required with the component neuroml. The corresponding CMake library target is arbor::arborio.

find_package(arbor COMPONENTS neuroml)
# ...
target_link_libraries(myapp arbor::arborio)

Profiling¶

Arbor has built in profiling that can report the time spent in each step during the simulation that can be toggled with the -DARB_WITH_PROFILING CMake option:

cmake .. -DARB_WITH_PROFILING=ON

By default ARB_WITH_PROFILING=OFF.

Installation¶

Arbor can be installed with make install after configuration. The installation comprises:

The static libraries libarbor.a and libarborenv.a.
Public header files.
The modcc NMODL compiler if built.
The python module if built.
The HTML documentation if built.

The default install path (/usr/local) can be overridden with the CMAKE_INSTALL_PREFIX configuration option.

Provided that Sphinx is available, HTML documentation for Arbor can be built with make html. Note that documentation is not built by default — if built, it too will be included in the installation.

Note that the modcc compiler will not be built by default if the ARB_MODCC configuration setting is used to specify a different executable for modcc. While modcc can be used to translate user-supplied NMODL mechanism descriptions into C++ and CUDA code for use with Arbor, this generated code currently relies upon private headers that are not installed.

HPC clusters¶

HPC clusters offer their own unique challenges when compiling and running software, so we cover some common issues in this section. If you have problems on your target system that are not covered here, please make an issue on the Arbor Github issues page. We will do our best to help you directly, and update this guide to help other users.

MPI¶

Arbor uses MPI for distributed systems. By default it is built without MPI support, which can enabled by setting the ARB_WITH_MPI configuration flag. An example of building a ‘release’ (optimized) version of Arbor with MPI is:

# set the compiler wrappers
export CC=`which mpicc`
export CXX=`which mpicxx`

# configure with mpi
cmake -DARB_WITH_MPI=ON

# run MPI-specific unit tests on 2 MPI ranks
mpirun -n 2 ./bin/unit-mpi

The example above sets the CC and CXX environment variables to use compiler wrappers provided by the MPI implementation. While the configuration process will attempt to find MPI libraries and build options automatically, we recommend using the supplied MPI compiler wrappers in preference.

Note

MPI distributions provide compiler wrappers for compiling MPI applications.

In the example above the compiler wrappers for C and C++ called mpicc and mpicxx respectively. The name of the compiler wrapper is dependent on the MPI distribution.

The wrapper forwards the compilation to a compiler, like GCC, and you have to ensure that this compiler is able to compile Arbor. For wrappers that call GCC or Clang compilers, pass the --version flag to the wrapper. For example, on a Cray system, where the C++ wrapper is called CC:

$ CC --version
g++ (GCC) 6.2.0 20160822 (Cray Inc.)

Cray systems¶

The compiler used by the MPI wrappers is set using a “programming environment” module. The first thing to do is change this module, which by default is set to the Cray programming environment, to a compiler that can compile Arbor. For example, to use the GCC compilers, select the GNU programming environment:

module swap PrgEnv-cray PrgEnv-gnu

The version of GCC can then be set by choosing an appropriate gcc module. In the example below we use module avail to see which versions of GCC are available, then choose GCC 7.1.0

$ module avail gcc      # see all available gcc versions

------------------------- /opt/modulefiles ---------------------------
gcc/4.9.3    gcc/6.1.0    gcc/7.1.0    gcc/5.3.0(default)    gcc/6.2.0

$ module swap gcc/7.1.0 # swap gcc 5.3.0 for 7.1.0

$ CC --version          # test that the wrapper uses gcc 7.1.0
g++ (GCC) 7.1.0 20170502 (Cray Inc.)

# set compiler wrappers
$ export CC=`which cc`
$ export CXX=`which CC`

Note that the C and C++ compiler wrappers are called cc and CC respectively on Cray systems.

CMake detects that it is being run in the Cray programming environment, which makes our lives a little bit more difficult (CMake sometimes tries a bit too hard to help). To get CMake to correctly link our code, we need to set the CRAYPE_LINK_TYPE environment variable to dynamic.

export CRAYPE_LINK_TYPE=dynamic

Putting it all together, a typical workflow to build Arbor on a Cray system is:

export CRAYPE_LINK_TYPE=dynamic
module swap PrgEnv-cray PrgEnv-gnu
module swap gcc/7.1.0
export CC=`which cc`; export CXX=`which CC`;
cmake -DARB_WITH_MPI=ON    # MPI support

Note

If CRAYPE_LINK_TYPE isn’t set, there will be warnings like the following when linking:

warning: Using 'dlopen' in statically linked applications requires at runtime
         the shared libraries from the glibc version used for linking

Often the library or executable will work, however if a different glibc is loaded, Arbor will crash at runtime with obscure errors that are very difficult to debug.

Troubleshooting¶

Cross compiling NMODL¶

Care must be taken when Arbor is compiled on a system with a different architecture to the target system where Arbor will run. This occurs quite frequently on HPC systems, for example when building on a login/service node that has a different architecture to the compute nodes.

Note

If building Arbor on a laptop or desktop system, i.e. on the same computer that you will run Arbor on, cross compilation is not an issue.

Note

The ARB_ARCH setting is not applied to the building of modcc. On systems where the build node and compute node have different architectures within the same family, this may mean that separate compilation of modcc is not necessary.

Warning

Illegal instruction errors are a sure sign that Arbor is running on a system that does not support the architecture it was compiled for.

When cross compiling, we have to take care that the modcc compiler, which is used to convert NMODL to C++/CUDA code, is able to run on the compilation node.

By default, building Arbor will build the modcc executable from source, and then use that to build the built-in mechanisms specified in NMODL. This behaviour can be overridden with the ARB_MODCC configuration option, for example:

cmake -DARB_MODCC=path-to-local-modcc

Here we will use the example of compiling for Intel KNL on a Cray system, which has Intel Sandy Bridge CPUs on login nodes that don’t support the AVX512 instructions used by KNL.

#
#   Step 1: Build modcc.
#

module swap PrgEnv-cray PrgEnv-gnu
# Important: use GNU compilers directly, not the compiler wrappers,
# which generate code for KNL, not the login nodes.
export CC=`which gcc`; export CXX=`which g++`;
export CRAYPE_LINK_TYPE=dynamic

# make a path for the modcc build
mkdir build_modcc
cd build_modcc

# configure and make modcc
cmake ..
make -j modcc

#
#   Step 2: Build Arbor.
#

cd ..
mkdir build; cd build;
# use the compiler wrappers to build Arbor
export CC=`which cc`; export CXX=`which CC`;
cmake .. -DCMAKE_BUILD_TYPE=release           \
         -DARB_WITH_MPI=ON                    \
         -DARB_ARCH=knl                       \
         -DARB_VECTORIZE=ON                   \
         -DARB_MODCC=../build_modcc/bin/modcc

Note

Cross compilation issues can occur when there are minor differences between login and compute nodes, e.g. when the login node has Intel Haswell, and the compute nodes have Intel Broadwell.

Other systems, such as IBM BGQ, have very different architectures for login and compute nodes.

If the modcc compiler was not compiled for the login node, illegal instruction errors will occur when building, e.g.

$ make
...
[ 40%] modcc generating: /users/bcumming/arbor_knl/mechanisms/multicore/pas_cpu.hpp
/bin/sh: line 1: 12735 Illegal instruction     (core dumped) /users/bcumming/arbor_knl/build_modcc/modcc/modcc -t cpu -s\ avx512 -o /users/bcumming/arbor_knl/mechanisms/multicore/pas /users/bcumming/arbor_knl/mechanisms/mod/pas.mod
mechanisms/CMakeFiles/build_all_mods.dir/build.make:69: recipe for target '../mechanisms/multicore/pas_cpu.hpp' failed

If you have errors when running the tests or a miniapp, then either the wrong ARB_ARCH target architecture was selected; or you might have forgot to launch on the compute node. e.g.:

$ ./bin/unit
Illegal instruction (core dumped)

On the Cray KNL system, srun is used to launch (it might be mpirun or similar on your system):

$ srun -n1 -c1 ./bin/unit
[==========] Running 609 tests from 108 test cases.
[----------] Global test environment set-up.
[----------] 15 tests from algorithms
[ RUN      ] algorithms.parallel_sort
[       OK ] algorithms.parallel_sort (15 ms)
[ RUN      ] algorithms.sum
[       OK ] algorithms.sum (0 ms)
...

Debugging¶

Sometimes things go wrong: tests fail, simulations give strange results, segmentation faults occur and exceptions are thrown.

A good first step when things to wrong is to turn on additional assertions that can catch errors. These are turned off by default (because they slow things down a lot), and have to be turned on by setting the ARB_WITH_ASSERTIONS CMake option:

cmake -DARB_WITH_ASSERTIONS=ON

Note

These assertions are in the form of arb_assert macros inside the code, for example:

void decrement_min_remaining() {
    arb_assert(min_remaining_steps_>0);
    if (!--min_remaining_steps_) {
        compute_min_remaining();
    }
}

A failing arb_assert indicates that an error inside the Arbor library, caused either by a logic error in Arbor, or incorrectly checked user input.

If this occurs, it is highly recommended that you attach the output to the bug report you send to the Arbor developers!

CMake Git submodule warnings¶

When running CMake, warnings like the following indicate that the Git submodules need to be updated.

The Git submodule for rtdtheme is not available.
To check out all submodules use the following commands:
    git submodule init
    git submodule update
Or download submodules recursively when checking out:
    git clone --recurse-submodules https://github.com/arbor-sim/arbor.git