Installing the module¶
The module is distributed in pre-compiled Python wheels. Wheels are available for any (sufficiently recent) Linux distribution, as long as for Windows 10.
The following table lists the Python versions for which pre-compiled wheels are available.
| OS | Currently Supported Python Versions |
|---|---|
| Linux | 3.5, 3.6, 3.7, 3.8, 3.9 |
| Windows 10 | 3.6, 3.7, 3.8 |
The installation on both systems can be easily accomplished by typing the following command in the bash prompt (Linux) or the Anaconda Prompt / Powershell (Windows):
$ python3 -m pip install scatman --extra-index-url https://gitlab+deploy-token-257392:xxKAcLMWssj_sTpjCEkQ@gitlab.com/api/v4/projects/18794850/packages/pypi/simple
For GPU support, the CUDA Runtime library (>=10.1) is required for both Linux and Windows. For the installation of the scatman with GPU acceleration, the package scatman-cuda should be installed instead:
$ python3 -m pip install scatman-cuda --extra-index-url https://gitlab+deploy-token-257392:xxKAcLMWssj_sTpjCEkQ@gitlab.com/api/v4/projects/18794850/packages/pypi/simple
Note
Installing on Windows
On windows, the module is only tested when installed via pip on Anaconda/Miniconda prompt. There, the Python executable is called with the command python instead of the Linux command python3.
Compiling the module¶
This chapter is dedicated to how to compile and install the module from source code, in case the pre-compiled packages don’t meet your OS/Python configuration.
The module, as told before, is written in C++. The binding between the C++ code and Python is made thanks to the cool PyBind11 library. PyBind11 exposes the C++ functions and classes to Python.
Compilation Prerequisites¶
The following table shows the compilation requirements:
| Component Name | Version | Reason | Link | Install on Debian-like systems |
|---|---|---|---|---|
| Python3 | >=3.4 | Python interpreter and C++ header files | https://www.python.org/ | sudo apt install python3
| sudo apt install python3-dev |
| CMake | >=3.17 | Build system | https://cmake.org/ | sudo apt install cmake |
| GCC | >=7 | C++ compiler | https://gcc.gnu.org/ | sudo apt install gcc |
| FFTW3 | Any | Fast Fourier Transform | http://www.fftw.org/ | sudo apt install libfftw3-dev |
| NVidia CUDA Toolkit (optional) | >=10 | Enable GPU computing | https://docs.nvidia.com/cuda/ | Have a look here |
| Component Name | Version | Reason | Link |
|---|---|---|---|
| Python3 (Anaconda) | >=3.4 | Python interpreter and C++ header files | https://www.anaconda.com/ |
| Visual Studio | >=2017 | C++ compiler | https://visualstudio.microsoft.com/ |
| Intel MKL | Any | Fast Fourier Transform | https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html |
| NVidia CUDA Toolkit (optional) | >=10 | Enable GPU computing | https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html |
Note
GPU acceleration is optional. To enable it, it is necessary to set the envirnoment variable SCATMAN_CUDA to 1.
As you may have noticed, there is no dependence on the PyBind11 library. This is just because it is a header only library, and it is automatically downloaded at compile time. downloads the source from the gitlab repository,
Once that the requirements are satisfied, it is now possible to proceed with the compilation step. The compilation and installation steps should be completely transparent to the user, thanks to the use of pip.
Building on Linux¶
Just open the command shell, and type the following command:
$ python3 -m pip install git+https://gitlab.ethz.ch/nux/numerical-physics/pyscatman.git --user
If everything goes smoothly, this line downloads the source from the gitlab repository, compiles it, and installs the proper files in the proper directory
Building on Windows¶
These installation steps are for the Anaconda prompt. First, open the Anaconda prompt, and type:
$ conda install -c anaconda git
$ python -m pip install cmake
This commands will install git, that will be used to clone the scatman repository, and CMake, needed to build the scatman. Afterthat, we clone the repository and we move inside:
$ git clone https://gitlab.ethz.ch/nux/numerical-physics/pyscatman.git
$ cd pyscatman
The last step is the actual compilation and installation step:
$ python -m pip install . --user
If everything goes smoothly, this line compiles the source and installs the proper files in the proper directory.
Building without pip (Linux Only)¶
Preparing for compilation¶
As a first step, we need to download the repository. This can be accompished with git by executing the following command in the shell:
$ git clone git@gitlab.ethz.ch:nux/numerical-physics/pyscatman.git
Note
as long as the repository is not public, you are required to insert your credentials.
Then, move to the pyscatman folder and create a directory named build, where the compilation process will be executed (you can decide the name that you prefer for the folder, actually).
$ cd pyscatman
$ mkdir build
$ cd build
Now we are ready for the compilation.
As a first attempt, it is possible to simply type the following command:
$ cmake ..
This command will run the cmake software, that will try to automatically detect all the mandatory libraries that are necessary for the building process, and it will create a Makefile for us. If, from one side, it is likely that it will succeed, we have to take a particular care about the detected python version.
Being a compiled Python module, it is necessary that the Python version detected by cmake is the same version that will be used to import the scatman module and make the simulations. It is possible that there are multiple Python versions installed on the PC. For this reason, is better to provide to CMake the exact Python executable.
To accompish this, we have to first retrieve the full path of the Python executable that we want to use. It can be done in this way:
$ which python3
/usr/bin/python3
Once we have the path, it is possible to add it to CMake by adding the -DPYTHON_EXECUTABLE option, in the following way:
cmake -DPYTHON_EXECUTABLE=/usr/bin/python3 ..
Note
It is possible that CMake fails to find the location of the FFTW library. In this case, it is possible to provide library path to CMake, by adding the option -DFFTW_LIBRARIES=/usr/lib/libfftw.so ..
Enabling GPUs¶
As told before, it is possible to enable the support for the GPU computing. At the moment, only CUDA-capable Nvidia GPUs are supported.
To do this, an additional option -DWITH_CUDA=On has to be provided to CMake.
After that, CMake will try to search for the NVidia CUDA compiler nvcc. If it fails to find it, but the CUDA compiler is installed, it is still possible to explicitly provide the full path for the nvcc compiler, via the option -DCMAKE_CUDA_COMPILER.
In this case, the full command should look like:
$ cmake -DPYTHON_EXECUTABLE=/path/to/python3 -DWITH_CUDA=On -DCMAKE_CUDA_COMPILER=/path/to/nvcc ..
Compilation¶
Once that CMake executed correctly, it automatically created a Makefile in the build directory. At this point, it is possible to compile the code by just running the makefile, i.e.:
$ make
If everything goes well, you should see a .so file in the build directory:
$ ls
CMakeCache.txt CMakeFiles cmake_install.cmake _deps libacquire.a libcontext.a libmsft.a libshapes.a libtuning.a Makefile scatman.cpython-38-x86_64-linux-gnu.so
The file name will have the form scatman.cpython-XX-YYYY-ZZZZZZ.so, where XX is the major and minor Python version (in this example we used Python 3.8, so it will be 38), YYYY is the architecture (in the example x86_64) and ZZZZZZ is the Operative System.
Installation¶
Once you have the .so (i.e. the shared library object), it is sufficient to place it in the same folder where you open the Python interpreter, or where your Python script is placed, as like as a normal module written in Python.
However, it is more convenient to put it in a place where Python search for modules.
To install the module such that it is accessible by all users (root privileges required) you can place it in the folder /usr/lib/python3.8 (or whatever version you are using):
$ sudo cp scatman.cpython-38-x86_64-linux-gnu.so /usr/lib/python3.8/
It may be more convenient (or even mandatory, if you don’t have a root access to the machine) to place it in a local folder, and add the local folder to the PYTHONPATH environment variable. You can choose the local folder you prefer. I suggest to do the following (replace 3.8 with your python version):
$ mkdir ~/.local/lib/python3.8
$ cp scatman.cpython-38-x86_64-linux-gnu.so ~/.local/lib/python3.8
$ export PYTHONPATH=$PYTHONPATH:~/.local/lib/python3.8
The PYTHONPATH variables allows to add custom folders where Python will search for modules. To make this PYTHONPATH setting permanent, just add the following line to the file ~/.bashrc:
export PYTHONPATH=$PYTHONPATH:$HOME/.local/lib/python3.8
In this way, you can import the scatman module from anywhere.