Skip to content

Fault Tolerance

Melissa provides a robust fault tolerance and checkpointing system for managing various components of an experiment. Users can enable fault tolerance by configuring the settings as follows: 

{
    "launcher_config": {
        "fault_tolerance": true
    }
}

When fault_tolerance is enabled, the launcher automatically restarts any client that is terminated for any reason, including:

  • Consecutive messages from a client exceed study_options.simulation_timeout seconds (possibly due to a failure).
  • The client encounters scheduling issues on the supercomputer.
  • The client reaches its allocated wall time and is terminated by the scheduler.

Additionally, the launcher continuously monitors the server's status and will automatically relaunch it if a failure is detected. The server may be restarted if:

  • No messages are received from the server for launcher_config.server_timeout seconds.
  • The server encounters scheduling issues on the supercomputer.
  • The server reaches its allocated wall time and is terminated by the scheduler.

Checkpointing the Server

The server maintains critical state information that must be checkpointed to ensure a seamless restart after failure. This state information includes:

  • Completed and unfinished clients.
  • Buffer (reservoir) object (for Deep Learning studies).
  • Neural Network weights, biases, optimizer state, and batch progress (for Deep Learning studies).

Checkpointing is triggered by calling self.checkpoint() within the server.

Deep Learning

For DL, the checkpointing approach is similar to SA, where checkpoint_interval specifies the number of processed batches before a checkpoint is saved. Users can configure this setting as follows: 

{
    "dl_config": {
        "checkpoint_interval": 100
    }
}

By default, checkpoint_interval is set to the value of nb_batches_update.

Tip

For greater flexibility, users can manually trigger a checkpoint at any batch index by calling self.checkpoint(batch_idx) on their custom servers.

Error Handling and the Restart Cycle

All server-side exceptions are categorized under the MelissaError hierarchy. When an exception occurs during the data reception or training loops, the server captures the error, logs it, and terminates. The Melissa Launcher, acting as the orchestrator, monitors the exit status of these server jobs. By design, the Launcher interprets any non-zero exit code or standard exception (such as ReceptionError or TrainingError) as a transient failure likely caused by hardware instability, network jitter, or a node crash. Consequently, the Launcher automatically triggers a restart of the server to resume from the last checkpoint.

The significant flaw in this "restart-by-default" policy lies in its agnosticism toward error causes. The Launcher only ceases its restart attempts if it encounters a FatalError (like errors in the configuration or the initial setup before the server calls start()). If the server encounters deterministic errors such as AssertionError, FileNotFoundError (caused by a typo), AttributeError, etc. These are identically to a transient one. Because these errors are rooted in the code/configuration, they may persist identically across every subsequent attempt. Without a mechanism to distinguish these "permanent" Python exceptions from "possibly recoverable" system failures, the Launcher will continue to resubmit the server job indefinitely, or until the total walltime of the allocation is exhausted. This may lead to a significant waste of computational resources, as the system enters an infinite loop of failure and re-initialization that no amount of restarting can resolve.

Launcher Failure

If the launcher fails, the simulation must be restarted manually. However, users can resume the simulation from the last checkpoint by enabling load_from_checkpoint in the launcher_config: 

{
    "launcher_config": {
        "load_from_checkpoint": true
    }
}

With load_from_checkpoint enabled, the launcher ensures that the server is restored from the latest checkpoint, just as it would be if the server had failed and restarted automatically. This guarantees continuity, allowing the experiment to resume without (too much) loss of progress.