Installation¶

Although Melissa was initially designed to be used on supercomputers, it can also be installed and executed on a local machine. Users are even recommended to start with a local installation in order to become familiar with the framework before using it at scale.

Due to the code structure of Melissa, the installation is organized in two parts. The API (C code wrapped in other languages) is compiled and installed with CMake while the launcher and server components as well as their dependencies are built with pip. Depending on the user's working environment and needs, a specific installation method may be more appropriate than another. This tutorial will guide users through the steps of each procedure and help them find the best suited.

Before building Melissa¶

The following dependencies must be installed before building Melissa:

• CMake 3.7.2 or newer
• GNU Make
• A C99 compiler
• An MPI implementation
• Python 3.8 or newer
• ZeroMQ 4.1.5 or newer

On debian based systems with root access, these dependencies can be installed via:

sudo apt-get install cmake build-essential libopenmpi-dev python3 python3-dev python3-venv libzmq3-dev

In addition, Melissa has three levels of Python dependencies that will be installed with pip at build:

1. basic dependencies sufficient to run the sensitivity analysis server which are listed in requirements.txt.

2. deep learning dependencies to run a deep-surrogate training which are listed in requirements_deep_learning.txt.

3. extra-dependencies for development purposes which are listed in requirements_dev.txt.

Quick installation¶

Manual installation¶

The following guidelines outline how to manually install Melissa.

First the melissa folder can be created by cloning it from its INRIA GitLab repository:

git clone https://gitlab.inria.fr/melissa/melissa.git
cd melissa

The API can then be built with the following commands:

# from melissa/
mkdir build install && cd build
cmake -DCMAKE_INSTALL_PREFIX=../install ..
make
make install

As evidenced by the following table, users installing on a local machine may benefit from including the -DINSTALL_ZMQ=ON flag.

Next, in order not to pollute the user's environment, a virtual environment can be created as follows:

# from melissa/build
cd ..
python3 -m venv .venv                                                                      # create a pip venv
source .venv/bin/activate                                                                  # activate the venv
python3 -m pip install --upgrade pip                                                       # upgrade pip
pip3 install -r requirements.txt -r requirements_deep_learning.txt -r requirements_dev.txt # install all requirements in the venv
pip3 install -e .                                                                          # install Melissa py-modules and make them directly available to the venv

This will have for effect to install all dependencies including both torch and tensorflow. In addition, the Melissa sources will be installed in editable mode which is particularly convenient for developers. Users are nevertheless free to adapt the pip commands and requirement files to their needs. For deep surrogate training for instance, the presence of only one of the two frameworks is strictly required.

After successful installation, users can open a new terminal and either source their virtual environment or the environment file melissa_set_env.sh before executing their first Melissa command:

# from melissa/
source .venv/bin/activate # activate venv if created earlier
source melissa_set_env.sh # add ~/.local/bin to the PYTHONPATH if no virtual environment was used
melissa-launcher -h       # print melissa-launcher help

Installation with Spack¶

1. Install and set up Spack: Spack documentation: Getting Started

2. Build Melissa in its latest stable version:

   spack install melissa-api            # install the API with its dependencies
   spack install py-melissa-core +torch # install the Melissa Python modules with the deep learning requirements and torch only
   

3. Load Melissa:

    spack load melissa-api
    spack load py-melissa-core
   

At this point, the user should have a working Melissa installation.

spack install melissa-api@target
spack install py-melissa-core@target

Spack and modules¶

If you have installed Melissa on a supercomputer which works with modules, you may want to have Melissa build depending on one of such module. Typically the MPI module optimized for your supercomputer, here is a basic example on how to proceed:

• Load the module:

  $ module load openmpi/4.0.5
  
  Loading openmpi/4.0.5
  Loading requirement: intel-compilers/19.1.3
  

• Tell Spack to search for installed (so-called "external") packages:

  $ spack external find
  
   ==> The following specs have been detected on this system and added to /home/user/.spack/packages.yaml
   openmpi@4.0.5
  

• Afterwards the modules can be unloaded:

  module purge
  

• The module may have been compiled with a specific compiler. Add the compiler to Spack:

   $ spack compiler find
  
   ==> Added 1 new compiler to /home/user/.spack/linux/compilers.yaml
    intel@19.1.3.304
   ==> Compilers are defined in the following files:
    /home/user/.spack/linux/compilers.yaml
  

• Spack can only recognize a small number of packages automatically and in many cases it is advisable or even necessary to edit the list of available packages manually. The list of packages can be found at ~/.spack/packages.yaml. If you perform these modifications, you must find out:

  • the path to the package or the relevant module(s) to load.
  • the Spack package name (e.g., the Spack package name of CMake is cmake)
  • the package version,
  • and the compiler that was used.

The compiler dependency is very important because in the author's experience different compiler modules may depend on different versions of essential libraries such as libc or binutils. Using a package that was built with one compiler while a different compiler module is loaded may lead to crashes in the best case and silent data corruption or a waste of supercomputer CPU hours in the worst case.

An example with Intel MPI:

packages:
  gmake:
    externals:
    - spec: gmake@4.2.1
      prefix: /usr
  intel-mpi:
    externals:
    - spec: intel-mpi@2019.5.281%intel@19.0.5.281
      modules: [intel-all/2019.5]

• In some environments existing packages must be used; they should never be built by Spack. For example on supercomputers, MPI must often be linked against libraries of the computers' batch scheduler for otherwise, MPI applications cannot be launched. You can prohibit building packages from scratch. Example:

  packages:
    mpi:
      buildable: false
  

• Build Melissa with specific compiler and package dependencies:

   spack install melissa-api %intel@19.1.3.304 ^openmpi@4.0.5
  

Checking the installation¶

A full confirmation of the installation can be performed by following one of the available example tutorials:

• Sensitivity-Analysis with the Heat-PDE use-case here.
• Deep Surrogate training with the Heat-PDE use-case here.
• Deep Surrogate training with the Lorenz use-case here.

If Melissa was installed with Spack, the user is advised to clone the repository and work off of any of the proposed examples:

git clone https://gitlab.inria.fr/melissa/melissa.git
cd melissa/examples

Advanced installation¶

On a supercomputer, a proper management of the dependencies and environment can quickly become a burden.

Again, the melissa folder can be created by cloning it from its INRIA GitLab repository:

git clone https://gitlab.inria.fr/melissa/melissa.git
cd melissa

Regarding the API, the user must make sure that it is compiled with the same dependencies as those of the instrumented solver:

1. load all modules required by the solver,

2. build/install the Melissa API:

# from melissa/
mkdir build install && cd build
cmake -DCMAKE_INSTALL_PREFIX=../install ..
make
make install

# from melissa/build/
cd ..
source melissa_set_env.sh

On the other hand, the Python dependencies may be constrained by the available packages. On Jean-Zay for instance, torch and tensorflow versions optimized for GPU execution are provided in their own conda environments and with Python versions different from the default one. From a clean environment, the deep learning framework module must be loaded before pip installing Melissa:

# from melissa/
module purge                                   # purge all pre-loaded modules
module load pytorch-gpu/py3/1.13.0             # switch to torch-gpu Python environment (this can change the Python version)
pip3 install -r requirements.txt               # add missing main requirements to torch-gpu Python environment
pip3 install -r requirements_deep_learning.txt # add missing deep learning Python packages to torch-gpu Python environment 
pip3 install -e .                              # install Melissa launcher and server in the torch-gpu Python environment

Still on Jean-Zay supercomputer, pytorch-gpu/py3/1.13.0 comes with Python 3.10.8 which is different from the frontend node version. All pip install commands will hence add packages to ~/.local/lib/python3.10/site-packages and the executables (namely melissa-launcher, melissa-server and melissa-monitor) to ~/.local/bin/. Sourcing melissa_set_env.sh has for effect to add ~/.local/ and ~/.local/bin to the PYTHONPATH.

In addition, for a Melissa command to successfully be executed (from the frontend node or embedded in an script), the Python environment will need to be the same as the one in which the pip install commands were executed. For our example on Jean-Zay, this means that any call to melissa-launcher or melissa-server must be preceded by module load pytorch-gpu/py3/1.13.0:

# from melissa/
module load pytorch-gpu/py3/1.13.0
source melissa_set_env.sh
melissa-launcher -h

If the installation is successful, the user should be prompted with the available command line arguments.

Multiple Melissa installations¶

As explained at the beginning of this tutorial, if the installation is not performed in a virtual environment, the missing packages as well as the Melissa sources and executables will end up in ~/.local/. In order to have multiple Melissa versions coexist for instance, the executables must be placed at a specific location made accessible at runtime. To do so, users should do the following.

Again, let us start by adding the melissa folder from its INRIA GitLab repository:

git clone https://gitlab.inria.fr/melissa/melissa.git
cd melissa

For any Python version (here denoted X.X), the pip install command is then run with the --target option so that the executables (melissa-launcher, melissa-server and melissa-monitor) are placed in a specific installation folder:

pip3 install -r requirements.txt -r requirements_deep_learning.txt -r requirements_dev.txt # install all missing requirements in ~/.local/lib/pythonX.X/site-packages
pip3 install --target=<install_folder> -e .                                                # install Melissa py-modules and place executables in <install_folder>/bin

Finally, the API can be built with the following commands:

mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=../<install_folder> ..
make
make install

Thus, the melissa_set_env.sh script created by CMake will make sure melissa/<install_folder>/bin is added to PATH as it is sourced.

The melissa_set_env.sh script¶

Due to the structure of Melissa, environment variables must be set consistently with the way the software is built. To ensure this, Melissa relies on an automatic environment setup script melissa_set_env.sh that is created at build by CMake from melissa_set_env.sh.in:

#! /bin/sh

export PATH=@CMAKE_INSTALL_PREFIX@/bin:$PATH
export LD_LIBRARY_PATH=@CMAKE_INSTALL_PREFIX@/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=@ZEROMQ_LIB_DIR@/lib:$LD_LIBRARY_PATH
export MELISSA_INSTALL_PREFIX=@CMAKE_INSTALL_PREFIX@
export PYTHONPATH=$PYTHONPATH:@CMAKE_INSTALL_PREFIX@/lib:@CMAKE_INSTALL_PREFIX@/melissa:@CMAKE_SOURCE_DIR@
export PATH=$HOME/.local/bin:$PATH
export PYTHONPATH=$HOME/.local/:${PYTHONPATH:-}

melissa_set_env.sh is copied both in CMAKE_INSTALL_PREFIX/bin and in melissa.

This piece of code serves two main purposes. The first one is to ensure that the API sources and executables are added to LD_LIBRARY_PATH which is required in order to compile the Melissa instrumented solver. The second objective is to update PATH and PYTHONPATH with the Melissa Python executables and required packages if they were installed in ~/.local.

Melissa ensures that this file is sourced every time a new server or client job is executed.

. /path/to/melissa/melissa_set_env.sh

"client_config": {
    "melissa_client_env": "/path/to/melissa/<installation_folder>",
},
"server_config": {
    "melissa_server_env": "/path/to/melissa/<installation_folder>",
}

Dependency propagation during execution¶

As explained in the Building a new use-case tutorial, specific dependencies and environment variables must be set using the preprocessing_commands lists in the user configuration file. These bash commands are executed before the data generator executable, on each individual job node. Therefore, they need to prepare the node for the data generator and the melissa-server. For example, loading modules to prepare for a data generator would set client/server_config to look as follows:

    "client_config": {
        "executable_command": "/path/to/melissa/examples/heat-pde/executables/build/heatc",
        "preprocessing_commands":[
                    "module -v load zeromq/4.2.5",
                    "module -v load openmpi/4.0.2-cuda"]
    },
    "server_config": {
        "preprocessing_commands":[
                    "module load -v pytorch-gpu/py3/1.10.0"]
    }

See examples/heat-pde and examples/lorenz for more configuration examples.