.. _in_build_install: Build and install from source ############################# This guide covers building and installing Arbor using CMake, which is the recommended method for configuring Arbor for HPC applications and developers. We start with an overview of the building process, and the various options available to customize the build. Then we cover installing and running on `HPC clusters `_, followed by a `troubleshooting guide `_ for common build problems. .. note:: To get help in case of problems installing Arbor, please make an issue on the Arbor `Github issues `_ page. Getting the code ================ The easiest way to acquire the latest version of Arbor is to check the code out from the `Github repository `_: .. code-block:: bash git clone https://github.com/arbor-sim/arbor.git --recurse-submodules We recommend using a recursive checkout, because Arbor uses Git submodules for some of its library dependencies. The CMake configuration attempts to detect if a required submodule is available, and will print a helpful warning or error message if not, but it is up to the user to ensure that all required submodules are downloaded. The Git submodules can be updated, or initialized in a project that didn't use a recursive checkout: .. code-block:: bash git submodule update --init --recursive You can also point your browser to Arbor's `Github page `_ and download a zip file. If you use the zip file, then don't forget to run Git submodule update manually. .. _install_requirements: Requirements ============ Dependencies ------------ A full list of dependencies and minimum versions supported thereof is maintained in the file ``doc/dependencies.csv``, for completeness shown also in the following table. .. csv-table:: List of requirements :file: ../dependencies.csv :widths: 10, 20, 10, 70, 1 :header-rows: 1 Note, that while we consider these to be our minimally supported versions, Arbor might still compile and work fine using older versions. We will, however, not offer much support for these versions, nor will we fix issues specific to them. Minimum requirements -------------------- Arbor can be configured with various features, which may introduce additional dependencies. The non distributed (i.e. no MPI) version of Arbor can be compiled on Linux or OS X systems with very few tools. In the table above the blank lines in the "Build option/target" column reflect the minimum you need, provided you have obtained a copy of Arbor which includes the git submodules. .. _install-compilers: Compilers ~~~~~~~~~ Arbor requires a C++ compiler that fully supports C++17. We recommend using GCC or Clang, for which Arbor has been tested and optimised, see notes in the above table. .. _note_CC: .. Note:: The ``CC`` and ``CXX`` environment variables specify which compiler executable CMake should use. If these are not set, CMake will attempt to automatically choose a compiler, which may be too old to compile Arbor. For example, the default compiler chosen below by CMake was GCC 4.8.5 at ``/usr/bin/c++``, so the ``CC`` and ``CXX`` variables were used to specify GCC 10.2.0 before calling ``cmake``. .. code-block:: bash # on this system CMake chooses the following compiler by default $ c++ --version c++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) # check which version of GCC is available $ g++ --version g++ (GCC) 10.2.0 Copyright (C) 2020 Free Software Foundation, Inc. # set environment variables for compilers $ export CC=`which gcc`; export CXX=`which g++`; # launch CMake # the compiler version and path is given in the CMake output $ cmake .. -- The C compiler identification is GNU 10.2.0 -- The CXX compiler identification is GNU 10.2.0 -- Check for working C compiler: /cm/local/apps/gcc/10.2.0/bin/gcc -- Check for working C compiler: /cm/local/apps/gcc/10.2.0/bin/gcc -- works ... .. Note:: It is commonly assumed that to get the best performance one should use a vendor-specific compiler (e.g. the Intel, Cray or IBM compilers). These compilers are often better at auto-vectorizing loops, however for everything else GCC and Clang nearly always generate more efficient code. The main computational loops in Arbor are generated from `NMODL `_. The generated code is explicitly vectorised, obviating the need for vendor compilers, and we can take advantage of the benefits of GCC and Clang: faster compilation times; fewer compiler bugs; and better support for C++ standards. .. Note:: The IBM XL C++ compiler and Intel C++ compiler are not supported, owing to unresolved compiler issues. We strongly recommend building with GCC or Clang instead on PowerPC and Intel platforms. Optional requirements --------------------- GPU support ~~~~~~~~~~~ Arbor has full support for NVIDIA GPUs, for which the NVIDIA CUDA toolkit version 11 is required. And experimental support for AMD GPUs when compiled with hip-clang (non-release compiler). Distributed ~~~~~~~~~~~ Arbor uses MPI to run on HPC cluster systems. Arbor has been tested on MVAPICH2, OpenMPI, Cray MPI, and IBM MPI. More information on building with MPI is in the `HPC cluster section `_. Python ~~~~~~ Arbor has a Python frontend, for which a minimum of Python 3.9 is required. In addition, `numpy` is a runtime requirement for the Python package. In order to use MPI in combination with the python frontend the `mpi4py `_ Python package is recommended. See :ref:`install-python` for more information. NeuroML ~~~~~~~ Arbor supports reading cell morphologies defined in NeuroML version 2 through an additional support library ``arborio``. See :ref:`install-neuroml` for more information. Boost ~~~~~ When ``ARB_BACKTRACE`` is set to ``ON`` during configure we use ``Boost::stacktrace`` to print stacktraces upon failed assertions and attach them to the base exception types ``arbor_exception`` and ``arbor_internal_error`` as ``where``. Documentation ~~~~~~~~~~~~~~ To build a local copy of the html documentation that you are reading now, you will need to install `Sphinx `_. .. _install-downloading: .. _building: Building and installing Arbor ============================= Once the Arbor code has been checked out, first run CMake to configure the build, then run make. Below is a simple workflow for: **1)** getting the source; **2)** configuring the build; **3)** building; **4)** running tests; **5)** installing. For more detailed build configuration options, see the `quick start `_ guide. .. code-block:: bash # 1) Clone. git clone https://github.com/arbor-sim/arbor.git --recurse-submodules cd arbor # Make a path for building mkdir build cd build # 2) Use CMake to configure the build. # By default Arbor builds in release mode, i.e. with optimizations on. # Release mode should be used for installing and benchmarking Arbor. cmake .. # 3.1) Build Arbor library. make -j 4 # 3.2) Build Arbor unit tests. make -j 4 tests # 3.3) Build Arbor C++ examples. make -j 4 examples # 4) Run tests. ./bin/unit # 5) Install (by default, to /usr/local). make install This will build Arbor in release mode with the `default C++ compiler `_. .. _quickstart: Quick start: examples --------------------- Below are some example of CMake configurations for Arbor. For more detail on individual CMake parameters and flags, follow links to the more detailed descriptions below. .. topic:: `Debug `_ mode with `assertions `_ enabled. If you encounter problems building or running Arbor, compile with these options for testing and debugging. .. code-block:: bash cmake -DARB_WITH_ASSERTIONS=ON -DCMAKE_BUILD_TYPE=debug .. topic:: `Release `_ mode with `Clang `_. .. code-block:: bash export CC=`which clang` export CXX=`which clang++` cmake .. topic:: `Release `_ mode for the `Haswell architecture `_ and `explicit vectorization `_ of kernels. .. code-block:: bash cmake -DARB_VECTORIZE=ON -DARB_ARCH=haswell .. topic:: `Release `_ mode with `explicit vectorization `_, targeting the `Broadwell architecture `_, with support for `Nvidia GPUs `_, and building with `GCC 9 `_. .. code-block:: bash export CC=gcc-9 export CXX=g++-9 cmake -DARB_VECTORIZE=ON -DARB_ARCH=broadwell -DARB_GPU=cuda .. topic:: `Release `_ mode with `explicit vectorization `_, targeting the `Broadwell architecture `_, with support for `AMD GPUs `_, and building with `hipcc `_. .. code-block:: bash export CC=clang export CXX=hipcc cmake -DARB_VECTORIZE=ON -DARB_ARCH=broadwell -DARB_GPU=hip .. topic:: `Release `_ mode with `explicit vectorization `_, optimized for the local system architecture and `install `_ in ``/opt/arbor`` .. code-block:: bash cmake -DARB_VECTORIZE=ON -DCMAKE_INSTALL_PREFIX=/opt/arbor .. topic:: `Release `_ mode with profiling enabled .. code-block:: bash cmake -DARB_WITH_PROFILING=ON .. _buildtarget: Build target ------------ By default, Arbor is built in release mode, which should be used when installing or benchmarking Arbor. To compile in debug mode (which in practical terms means with ``-g -O0`` flags), use the ``CMAKE_BUILD_TYPE`` CMake parameter. .. code-block:: bash cmake -DCMAKE_BUILD_TYPE={debug,release} .. _install-architecture: Architecture ------------ By default, Arbor is built to target whichever architecture is the compiler default, which often involves a sacrifice of performance for binary portability. The target architecture can be explicitly set with the ``ARB_ARCH`` configuration option. This will be used to direct the compiler to use the corresponding instruction sets and to optimize for that architecture. When building and installing on the same machine, a good choice for many environments is to set ``ARB_ARCH`` to ``native``: .. code-block:: bash cmake -DARB_ARCH=native When deploying on a different machine (cross-compiling) specify the specific architecture of the target machine. The valid values correspond to those given to the ``-mcpu`` or ``-march`` options for GCC and Clang; the build system will translate these names to corresponding values for other supported compilers. Specific recent x86-family Intel CPU architectures include ``broadwell``, ``skylake`` and ``knl``. Complete lists of architecture names can be found in the compiler documentation: for example GCC `x86 options `_, `PowerPC options `_, and `ARM options `_. .. code-block:: bash # Intel architectures cmake -DARB_ARCH=broadwell # broadwell with avx2 cmake -DARB_ARCH=skylake-avx512 # skylake with avx512 (Xeon server) cmake -DARB_ARCH=knl # Xeon Phi KNL # ARM Arm8a cmake -DARB_ARCH=armv8-a # IBM Power8 cmake -DARB_ARCH=power8 .. _install-vectorize: Vectorization ------------- Explicit vectorization of computational kernels can be enabled in Arbor by setting the ``ARB_VECTORIZE`` CMake flag. This option is typically used in conjunction with the ``ARB_ARCH`` option to specify the target architecture: without SIMD support in Arbor for the architecture, enabling ``ARB_VECTORIZE`` will lead to a compilation error. .. code-block:: bash cmake -DARB_VECTORIZE=ON -DARB_ARCH=native With this flag set, the library will use architecture-specific vectorization intrinsics to implement these kernels. Arbor currently has vectorization support for x86 architectures with AVX, AVX2 or AVX512 ISA extensions; and for AArch64 ARM architectures with NEON and SVE (first available on ARMv8-A). .. note:: Note that on x86-64 platforms compilation will fail if you enable vectorization, but the CPU or ``-DARB_ARCH`` does not support any form of AVX. .. _install-gpu: GPU backend ----------- Compiling for the GPU backend is controlled by the ``ARB_GPU`` CMake option which is used to select between NVIDIA and AMD GPUs as well as specify the chosen GPU compiler. * ``none``: The default option. Disables the GPU backend. * ``cuda``: Enables the GPU backend for NVIDIA GPUs and compiles Arbor with nvcc (CUDA files), and the default C++ compiler (C++ files). * ``cuda-clang``: Enables the GPU backend for NVIDIA GPUs and compiles Arbor with clang. * ``hip``: Enables the experimental GPU backend for AMD GPUs and compiles Arbor with hipcc. **NVIDIA GPUs**: Arbor supports NVIDIA GPUs using CUDA. Compiling Arbor for NVIDIA GPUs requires the CUDA Toolkit. .. code-block:: bash cmake -DARB_GPU=cuda .. code-block:: bash cmake -DARB_GPU=cuda-clang Arbor is built for all supported NVIDIA GPUs and the available GPU will be used at runtime. Depending on the configuration of the system where Arbor is being built, the C++ compiler may not be able to find the ``cuda.h`` header when building for NIDIA GPUs. The easiest workaround is to add the path to the include directory containing the header to the ``CPATH`` environment variable before configuring and building Arbor, for example: .. code-block:: bash export CPATH="/opt/cuda/include:$CPATH" cmake -DARB_GPU=cuda **HIP GPUs**: Arbor has experimental support for AMD GPUs using HIP. The only compiler currently supported is the non-release hip-clang (``hipcc``) compiler. (For instructions on how to build hipcc, refer to the `HIP documentation `_). *CMake configuration for compiling Arbor with hipcc (CUDA and C++ files):* .. code-block:: bash export CC=clang export CXX=hipcc cmake -DARB_GPU=hip Arbor is built for all supported AMD GPUs and the available GPU will be used at runtime. .. Note:: Arbor supports and has been tested on Pascal (P100) and Volta (V100) NVIDIA GPUs, as well as Mi50 and Mi60 AMD GPUs. .. _install-python: Python frontend ---------------- Arbor can be used with a python frontend which is enabled by toggling the CMake ``ARB_WITH_PYTHON`` option: .. code-block:: bash cmake -DARB_WITH_PYTHON=ON By default ``ARB_WITH_PYTHON=OFF``. When this option is turned on, a Python module called :py:mod:`arbor` is built. A specific version of Python can be set when configuring with CMake using the ``PYTHON_EXECUTABLE`` variable. For example, to use Python 3.11 installed on a Linux system with the executable in ``/usr/bin/python3.11``: .. code-block:: bash cmake .. -DARB_WITH_PYTHON=ON -DPYTHON_EXECUTABLE=/usr/bin/python3.11 By default the Python package will be installed in the appropriate sub-directory inside ``CMAKE_INSTALL_PREFIX``, determined by querying Python's sysconfig library. For example ``${CMAKE_INSTALL_PREFIX}/lib/python3.10/site-packages/``. To install the module in a different location, independent of ``CMAKE_INSTALL_PREFIX``, use ``ARB_PYTHON_LIB_PATH`` to specify the location where the Python module is to be installed. .. code-block:: bash cmake .. -DARB_WITH_PYTHON=on -DARB_PYTHON_PATH_LIB=/custom/path .. note:: The location of libraries under a prefix in only guaranteed to be standard for Python's global library location. Therefore, correct installation of the Python package to any other location using ``CMAKE_INSTALL_PREFIX``, such as user directory (e.g. `~/.local`), a Python or Conda virtual environment, may result in installation to a wrong path. ``python -m site --user-site`` (for user installations) or a path from ``python -c 'import site; print(site.getsitepackages())'`` (for virtual environment installation) can be used in combination with ``ARB_PYTHON_LIB_PATH``. In addition, installation via ``pip`` is guaranteed to find the right path. Please refer to the :ref:`Python installation instruction `. .. code-block:: bash # A demonstration using ARB_PYTHON_LIB_PATH # Set up your venv. python -m venv env source env/bin/activate # Install dependencies pip install numpy # Obtain arbor git clone --recursive git@github.com:arbor-sim/arbor.git # Manually set the prefix under which the python package will be installed. # In this case, the first directory found by querying Python's list of site-package directories. pyprefix=`python -c 'import site; print(site.getsitepackages()[0])'` # Setup CMake mkdir build cd build cmake ../arbor -DARB_WITH_PYTHON=on \ # enable python support. -DARB_PYTHON_LIB_PATH="$pyprefix" # set Python installation path. # Build and install make -j4 make install # Test it out! python -c "import arbor; arbor.print_config()" The Arbor Python wrapper has optional support for mpi4py, though it is not required to use Arbor with Python and MPI. CMake will attempt to automatically detect ``mpi4py`` if configured with both ``-DARB_WITH_PYTHON=ON`` and MPI ``-DARB_WITH_MPI=ON``. If CMake fails to find ``mpi4py`` when it should, the easiest workaround is to add the path to the include directory for ``mpi4py`` to the ``CPATH`` environment variable before configuring and building Arbor: .. code-block:: bash # search for path tp python's site-package mpi4py $ for p in `python -c 'import sys; print("\n".join(sys.path))'`; do echo ===== $p; ls $p | grep mpi4py; done ===== /path/to/python/site-packages mpi4py # set CPATH and run cmake $ export CPATH="/path/to/python/site-packages/mpi4py/include/:$CPATH" $ cmake -DARB_WITH_PYTHON=ON -DARB_WITH_MPI=ON .. _install-neuroml: NeuroML support --------------- Arbor has limited support for NeuroML version 2 through an additional library ``arborio``. Arbor will additionally install the static library ``libarborio.a``. Applications using this functionality will need to link against this library in addition to the main Arbor library. For example: .. code-block:: bash g++ -std=c++17 -pthread mycode.cpp -larborio -larbor -lxml2 For projects using CMake, Arbor NeuroML support can be required with the component ``neuroml``. The corresponding CMake library target is ``arbor::arborio``. .. code-block:: cmake find_package(arbor COMPONENTS neuroml) # ... target_link_libraries(myapp arbor::arborio) .. _install-profiling: Profiling --------- Arbor has built in profiling that can report the time spent in each step during the simulation that can be toggled with the ``-DARB_WITH_PROFILING`` CMake option: .. code-block:: bash cmake .. -DARB_WITH_PROFILING=ON By default ``ARB_WITH_PROFILING=OFF``. .. _install: Installation ------------ Arbor can be installed with ``make install`` after configuration. The installation comprises: - The static libraries ``libarbor.a`` and ``libarborenv.a``. - Public header files. - The ``modcc`` NMODL compiler if built. - The python module if built. - The HTML documentation if built. The default install path (``/usr/local``) can be overridden with the ``CMAKE_INSTALL_PREFIX`` configuration option. Provided that Sphinx is available, HTML documentation for Arbor can be built with ``make html``. Note that documentation is not built by default — if built, it too will be included in the installation. Note that the ``modcc`` compiler will not be built by default if the ``ARB_MODCC`` configuration setting is used to specify a different executable for ``modcc``. While ``modcc`` can be used to translate user-supplied NMODL mechanism descriptions into C++ and CUDA code for use with Arbor, this generated code currently relies upon private headers that are not installed. .. _cluster: HPC clusters ============ HPC clusters offer their own unique challenges when compiling and running software, so we cover some common issues in this section. If you have problems on your target system that are not covered here, please make an issue on the Arbor `Github issues `_ page. We will do our best to help you directly, and update this guide to help other users. .. warning:: On many HPC systems a tool called ``module`` or ``ml`` is installed, which use the ``CPATH`` environment variable to set up include paths for building. The contents of this variable are forced on all compilations and by extension to dependency generation. This can lead to the wrong headers being picked up despite ``CMake`` reporting the correct versions, which can produce spurious errors. If you are using one of these tools (and possibly ```easybuid``) and encounter such problems, try to ``module unload`` as many modules as possibe. One example has been found with JSC clusters and an outdated pybind11 which was brought in by ``ml SciPy-Stack``. .. _install-mpi: MPI --- Arbor uses MPI for distributed systems. By default it is built without MPI support, which can enabled by setting the ``ARB_WITH_MPI`` configuration flag. An example of building a 'release' (optimized) version of Arbor with MPI is: .. code-block:: bash # set the compiler wrappers export CC=`which mpicc` export CXX=`which mpicxx` # configure with mpi cmake -DARB_WITH_MPI=ON # run MPI-specific unit tests on 2 MPI ranks mpirun -n 2 ./bin/unit-mpi The example above sets the ``CC`` and ``CXX`` environment variables to use compiler wrappers provided by the MPI implementation. While the configuration process will attempt to find MPI libraries and build options automatically, we recommend using the supplied MPI compiler wrappers in preference. .. Note:: MPI distributions provide **compiler wrappers** for compiling MPI applications. In the example above the compiler wrappers for C and C++ called ``mpicc`` and ``mpicxx`` respectively. The name of the compiler wrapper is dependent on the MPI distribution. The wrapper forwards the compilation to a compiler, like GCC, and you have to ensure that this compiler is able to compile Arbor. For wrappers that call GCC or Clang compilers, pass the ``--version`` flag to the wrapper. For example, on a Cray system, where the C++ wrapper is called ``CC``: .. code-block:: bash $ CC --version g++ (GCC) 6.2.0 20160822 (Cray Inc.) Heterogeneous systems --------------------- Some HPC clusters offer different types of nodes, with different hardware and where some may have GPUs. In order for the compilers to correctly target the intended hardware and link to the appropriate libraries it may be necessary to load a top-level module for cross-compiling. For example, on the hybrid Juwels Booster system, one would execute: .. code-block:: bash module load Stages/2026 CUDA This loads the required dependencies for the GPU node architecture. Bull / Atos Systems ------------------- We need to add implementations for MPI and some build tools .. code-block:: bash module load OpenMPI GCC CMake and if the Python interface is desired .. code-block:: bash module load Python SciPy-bundle mpi4py then, in a proper build directory, we can configure the build. A common seltection of options might be .. code-block:: bash cmake .. -DARB_VECTORIZE=ON -DCMAKE_BUILD_TYPE=release # use SIMD and select production build \ -DARB_WITH_MPI=ON # use MPI \ -DARB_GPU=cuda -DCMAKE_CUDA_ARCHITECTURES="80;90" # enable GPU support for A100 and H100 \ -DARB_WITH_PYTHON=ON -DARB_BUILD_PYTHON_STUBS=OFF # enable the Python interface if GPU support is not required, the third line is omitted and MPI or Python support can similarly disabled by setting the relevant options to ``OFF``. These options should be correct on JSC systems like JUWELS and JUPITER. Alternatively, the ParaStation MPI can be used, likewise available as a module. Cray systems ------------ The compiler used by the MPI wrappers is set using a "programming environment" module. The first thing to do is change this module, which by default is set to the Cray programming environment, to a compiler that can compile Arbor. For example, to use the GCC compilers, select the GNU programming environment: .. note:: While the specific versions mentioned here may be outdated, the general workflow is still correct. .. code-block:: bash module swap PrgEnv-cray PrgEnv-gnu The version of GCC can then be set by choosing an appropriate gcc module. In the example below we use ``module avail`` to see which versions of GCC are available, then choose GCC 7.1.0 .. code-block:: bash $ module avail gcc # see all available gcc versions ------------------------- /opt/modulefiles --------------------------- gcc/8.1.0 gcc/8.3.0 gcc/9.3.0 gcc/10.3.0 gcc/11.2.0(default) $ CC --version # test that the wrapper uses gcc 7.1.0 g++ (GCC) 11.2.0 20210728 (Cray Inc.) # set compiler wrappers $ export CC=`which cc` $ export CXX=`which CC` Note that the C and C++ compiler wrappers are called ``cc`` and ``CC`` respectively on Cray systems. .. note:: When targeting the cuda backend, the compiler has to be compatible with the respective cuda toolkit version. Thus, it may be necessary to switch the compiler. On Piz Daint, for example, one would do the following at the time of this writing: .. code-block:: bash $ module load daint-gpu/21.09 $ module load craype-accel-nvidia60 $ module swap cudatoolkit/21.5_11.3 # switch to newer cuda toolkit $ module swap gcc/9.3.0 # switch to 9.3.0 from 11.2.0 CMake detects that it is being run in the Cray programming environment, which makes our lives a little bit more difficult (CMake sometimes tries a bit too hard to help). To get CMake to correctly link our code, we need to set the ``CRAYPE_LINK_TYPE`` environment variable to ``dynamic``. .. code-block:: bash $ export CRAYPE_LINK_TYPE=dynamic .. note:: This step is no longer required beginning with the Cray PE 19.06 release, where the default linking mode is dynamic. On older systems, if ``CRAYPE_LINK_TYPE`` isn't set, there will be warnings like the following when linking: .. code-block:: none warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking Often the library or executable will work, however if a different glibc is loaded, Arbor will crash at runtime with obscure errors that are very difficult to debug. For building Arbor's Python interface, it may be necessary to load a Python that knows about the target system; in the case of the CSCS Cray system: python version, which knows about the Cray system: .. code-block:: bash $ module load cray-python/3.9.4.1 Putting it all together, a typical workflow to build Arbor on a Cray system is: .. code-block:: bash export CRAYPE_LINK_TYPE=dynamic # only required if Cray PE version < 19.06 # For GPU setup module load daint-gpu/21.09 # system specific module load craype-accel-nvidia60 # system specific module swap cudatoolkit/21.5_11.3 # system specific module swap PrgEnv-cray PrgEnv-gnu # Load newest or cuda-toolkit compatible compiler module swap gcc/9.3.0 export CC=`which cc`; export CXX=`which CC`; module load cray-python/3.9.4.1 # For CPU builds cmake -DARB_WITH_MPI=ON # MPI support # For GPU builds cmake -DARB_WITH_MPI=ON \ # MPI support -DARB_GPU=cuda \ # target cuda backend -DCMAKE_CUDA_RUNTIME_LIBRARY=Shared \ # use shared cuda libraries -DCUDA_USE_STATIC_CUDA_RUNTIME=OFF .. _troubleshooting: Troubleshooting =============== .. _crosscompiling: Cross compiling NMODL --------------------- Care must be taken when Arbor is compiled on a system with a different architecture to the target system where Arbor will run. This occurs quite frequently on HPC systems, for example when building on a login/service node that has a different architecture to the compute nodes. .. Note:: If building Arbor on a laptop or desktop system, i.e. on the same computer that you will run Arbor on, cross compilation is not an issue. .. Note:: The ``ARB_ARCH`` setting is not applied to the building of ``modcc``. On systems where the build node and compute node have different architectures within the same family, this may mean that separate compilation of ``modcc`` is not necessary. .. Warning:: ``Illegal instruction`` errors are a sure sign that Arbor is running on a system that does not support the architecture it was compiled for. When cross compiling, we have to take care that the *modcc* compiler, which is used to convert NMODL to C++/CUDA code, is able to run on the compilation node. By default, building Arbor will build the ``modcc`` executable from source, and then use that to build the built-in mechanisms specified in NMODL. This behaviour can be overridden with the ``ARB_MODCC`` configuration option, for example: .. code-block:: bash cmake -DARB_MODCC=path-to-local-modcc Here we will use the example of compiling for Intel KNL on a Cray system, which has Intel Sandy Bridge CPUs on login nodes that don't support the AVX512 instructions used by KNL. .. code-block:: bash # # Step 1: Build modcc. # module swap PrgEnv-cray PrgEnv-gnu # Important: use GNU compilers directly, not the compiler wrappers, # which generate code for KNL, not the login nodes. export CC=`which gcc`; export CXX=`which g++`; export CRAYPE_LINK_TYPE=dynamic # make a path for the modcc build mkdir build_modcc cd build_modcc # configure and make modcc cmake .. make -j modcc # # Step 2: Build Arbor. # cd .. mkdir build; cd build; # use the compiler wrappers to build Arbor export CC=`which cc`; export CXX=`which CC`; cmake .. -DCMAKE_BUILD_TYPE=release \ -DARB_WITH_MPI=ON \ -DARB_ARCH=knl \ -DARB_VECTORIZE=ON \ -DARB_MODCC=../build_modcc/bin/modcc .. Note:: Cross compilation issues can occur when there are minor differences between login and compute nodes, e.g. when the login node has Intel Haswell, and the compute nodes have Intel Broadwell. Other systems, such as IBM BGQ, have very different architectures for login and compute nodes. If the *modcc* compiler was not compiled for the login node, illegal instruction errors will occur when building, e.g. .. code-block:: none $ make ... [ 40%] modcc generating: /users/bcumming/arbor_knl/mechanisms/multicore/pas_cpu.hpp /bin/sh: line 1: 12735 Illegal instruction (core dumped) /users/bcumming/arbor_knl/build_modcc/modcc/modcc -t cpu -s\ avx512 -o /users/bcumming/arbor_knl/mechanisms/multicore/pas /users/bcumming/arbor_knl/mechanisms/mod/pas.mod mechanisms/CMakeFiles/build_all_mods.dir/build.make:69: recipe for target '../mechanisms/multicore/pas_cpu.hpp' failed If you have errors when running the tests or a miniapp, then either the wrong ``ARB_ARCH`` target architecture was selected; or you might have forgot to launch on the compute node. e.g.: .. code-block:: none $ ./bin/unit Illegal instruction (core dumped) On the Cray KNL system, ``srun`` is used to launch (it might be ``mpirun`` or similar on your system): .. code-block:: none $ srun -n1 -c1 ./bin/unit [==========] Running 609 tests from 108 test cases. [----------] Global test environment set-up. [----------] 15 tests from algorithms [ RUN ] algorithms.parallel_sort [ OK ] algorithms.parallel_sort (15 ms) [ RUN ] algorithms.sum [ OK ] algorithms.sum (0 ms) ... .. _debugging: Debugging --------- Sometimes things go wrong: tests fail, simulations give strange results, segmentation faults occur and exceptions are thrown. A good first step when things to wrong is to turn on additional assertions that can catch errors. These are turned off by default (because they slow things down a lot), and have to be turned on by setting the ``ARB_WITH_ASSERTIONS`` CMake option: .. code-block:: bash cmake -DARB_WITH_ASSERTIONS=ON .. Note:: These assertions are in the form of ``arb_assert`` macros inside the code, for example: .. code-block:: cpp void decrement_min_remaining() { arb_assert(min_remaining_steps_>0); if (!--min_remaining_steps_) { compute_min_remaining(); } } A failing ``arb_assert`` indicates that an error inside the Arbor library, caused either by a logic error in Arbor, or incorrectly checked user input. If this occurs, it is highly recommended that you attach the output to the `bug report `_ you send to the Arbor developers! CMake Git submodule warnings ---------------------------- When running CMake, warnings like the following indicate that the Git submodules need to be `updated `_. .. code-block:: none The Git submodule for rtdtheme is not available. To check out all submodules use the following commands: git submodule init git submodule update Or download submodules recursively when checking out: git clone --recurse-submodules https://github.com/arbor-sim/arbor.git Build in conda virtual environment using miniconda ================================================== If you hope to install Arbor from source in a virtual environment in order not to interfere with other Arbor versions or software you have installed, or if you want to use features that are not included in the official release, you can use miniconda for this purpose. Otherwise, type pip install arbor in your terminal to install arbor. But you will have to fix two issues, one is feeding the correct Python path, and the other is 'GLIBCXX 3.4.30 not found in conda environment'. (Please note: This is assuming Linux and MacOS is not yet covered.) .. code-block:: bash #create a virtual environment conda create --name arbor_test conda activate arbor_test #go to the folder and clone the Arbor source package from GitHub cd ~/miniconda3/envs/arbor_test/ mkdir src cd src git clone https://github.com/arbor-sim/arbor.git --recurse-submodules #install python and numpy in this environment conda install python=3.12.2 conda install numpy #start the build cd arbor mkdir build cd build cmake .. -GNinja -DCMAKE_CXX_COMPILER=$(which g++) -DCMAKE_C_COMPILER=$(which gcc) -DARB_WITH_PYTHON=ON -DARB_VECTORIZE=ON -DPython3_EXECUTABLE=$(which python3) -DARB_USE_BUNDLED_LIBS=ON #activate ninja to install ninja sudo ninja install #correct the path to the site packages and the libc files #first request the right Python site package path python -c 'import numpy; print(numpy.__path__)' #load the right path to the one used for installing #replace with the path you get in the previous operation before ‘/numpy’ cp -r ~/miniconda3/envs/arbor_test/src/arbor/build/python/arbor #redirect the libc files such that the miniconda environment can access it ln -sf /lib/x86_64-linux-gnu/libstdc++.so.6 ~/miniconda3/envs/arbor_test/bin/../lib/libstdc++.so.6 #go to any working directory to try if you successfully installed arbor, by starting python and importing arbor. #one thing to add here could be testing for the version, i.e., python -c 'import arbor; print(arbor.__version__)' #should work without errors and print something like 0.91-dev. #then deactivate the environment if no more actions are planned. In the future, always first activate the virtual environment with and then use arbor in this environment with: conda activate arbor_test python >>>import arbor