General overview¶

Terminology¶

Here are some key terms to help users understand the Melissa configuration:

• client refers to an instance of the user code (e.g., the data-generator, typically a physics-based application) that is instrumented to send data to the server. The number of client instances corresponds to the size of the user-defined parameter space.

• group represents a job that comprises multiple clients. This is applicable when Sobol indices are computed or when the group_size configuration option is set to a value greater than 1. If group_size=1, then a client is equivalent to a group. When group_size is greater than 1, non-unit groups are submitted using OpenMPI MPMD launch mode or as slurm heterogeneous jobs.

• server is a single instance that collects and processes data generated by the clients. It also directs the launcher to submit new client instances with updated parameter combinations.

• launcher is a single instance responsible for submitting the server job and, subsequently, following instructions from the server to submit client jobs. The launcher interacts with the cluster scheduler and currently supports OAR, slurm, and OpenMPI.

• study refers to the entire process, which includes launching parameter combinations, generating data, collecting data, and processing it.

File Structure Overview¶

Melissa relies on a set of configuration files that define various aspects of the user simulation. These include:

• <project-dir>/<config-file>.json
  A JSON file that configures the parametric sweep of the data-generator, the data processing logic, and the launcher.

• <project-dir>/<use-case>_server.py
  A user-defined server class that typically includes all customizations for data treatment (e.g., deep learning training, environment configurations, etc.).

Here, <project-dir> refers to a user-created folder, which should follow a structure similar to the examples/* directories included with Melissa.

Configuration Structure¶

The <config-file>.json provides a flexible way to define configurations for the current study. However, it requires specific key: value pairs tailored to the specific parent server class being used. Each server class expects different configurations depending on the type of study being conducted.

Configuration options¶

These options are required regardless of the server type. To understand each option in detail, execute:

melissa-launcher --print-options

• server_filename: The name of the file containing the user defined server. Assumed to be in the same folder as the config. Type string.
• server_class: The class name of the user defined server inside the server_filename file. Type string.
• output_dir: The output directory to write results and logs. If relative path, then it is assumed relative to the CWD of the melissa-launcher command. Type string.
• study_options: A custom dictionary which is accessible inside the server_class for users to parameterize their studies. Type object.
• server_config: Special configuration for the server only. Type object. Useful for specifying preprocessing commands before executing the melissa-server command in server.sh script.
• client_config: Special configuration for the client only. Type object. Useful for specifying executable_command path to the data-generator as well as specifying preprocessing commands before running it in client.*.sh scripts.
• launcher_config: Special configuration for the launcher only. Type object. Useful for specifying batch scheduler configuration supported by Melissa. This option requires scheduler, scheduler_arg_client, scheduler_arg_server, job_limit, and server_timeout.