Buffer Types

melissa.server.deep_learning.reservoir.BufferType

Bases: Enum

Enum representing the different types of buffer strategies available for sampling.

  • FIFO = 0
  • FIRO = 1
  • Reservoir = 2
  • BatchReservoir = 3
  • SimPairedReservoir = 4

Creating a buffer

melissa.server.deep_learning.reservoir.make_buffer(buffer_size, buffer_t=BufferType.FIFO, per_server_watermark=None, pseudo_epochs=None, batch_size=None, sweep_size=None, nb_time_steps=None, comm_size=1)

Factory function to create different types of buffers based on the specified buffer type.

This function initializes and returns a buffer (queue) object based on the given buffer size and type. The available buffer types are defined by the BufferType enum, and each type has specific requirements. The function raises a ValueError if the necessary parameters are not provided for the selected buffer type.

Parameters
  • buffer_size (int): The maximum size of the buffer.
  • buffer_t (BufferType, optional): The type of buffer to create. Default is BufferType.FIFO.
  • per_server_watermark (int, optional): A threshold used in some buffer types (e.g., Reservoir, FIRO, and BatchReservoir).
  • pseudo_epochs (int, optional): A parameter for FIRO buffer type to define the number of epochs for the buffer.
  • batch_size (int, optional):A parameter for BatchReservoir buffer type to defien the size of a batch.
  • sweep_size (int, optional): The number of distinct simulation trajectories.
  • nb_time_steps (int, optional): The number of time steps per simulation trajectory.
  • comm_size (int, optional): The number of parallel processes or threads.
Returns

BaseQueue: A buffer object of the specified type.

Buffer Classes

melissa.server.deep_learning.reservoir.FIFO

Bases: CounterMixin, ReceptionDependant, BaseQueue

First In First Out (FIFO) Queue.

A queue implementation that follows the FIFO principle, where the first item added is the first one to be retrieved. This queue also supports counting the number of times each item has been seen (via CounterMixin) and can signal when data reception is over (via ReceptionDependant).

melissa.server.deep_learning.reservoir.FIRO

Bases: CounterMixin, ReceptionDependant, BaseQueue

First In First Out (FIFO) Queue.

A queue implementation that follows the FIFO principle, where the first item added is the first one to be retrieved. This queue also supports counting the number of times each item has been seen (via CounterMixin) and can signal when data reception is over (via ReceptionDependant).

melissa.server.deep_learning.reservoir.Reservoir

Bases: CounterMixin, ThresholdMixin, RandomEvictOnWriteQueue

First In Random Out (FIRO) with eviction on write.

This queue implementation combines the First In strategy (FIFO) with random eviction upon writing. It behaves like a traditional FIFO queue for insertion, but when the queue reaches its capacity, it randomly evicts one of the previously added elements to make room for the new item. The class also tracks the number of times each item has been seen (via CounterMixin) and enforces a threshold for sampling (via ThresholdMixin).

Attributes
  • maxsize (int): The maximum size of the queue. Once the size exceeds this limit, eviction occurs.
  • threshold (int): The minimum number of items required in the queue to allow sampling.

melissa.server.deep_learning.SimPairedReservoir

Bases: ThresholdMixin, SimPairedQueue

A reservoir sampling queue that retrieves paired simulation data.

This class extends the SimPairedQueue to incorporate reservoir sampling techniques with a threshold-based sampling readiness condition. It's designed for scenarios where paired samples from the same simulation are needed for training, while also maintaining a diverse and representative set of samples through reservoir sampling.

It maintains an efficient data structure for storing simulation trajectories and implements adaptive sampling strategies that prioritize undersampled time differences.

Attributes:
  • maxsize (int): Maximum number of items to store in the reservoir
  • threshold (int): Minimum number of items required before sampling
  • sweep_size (int): Number of simulation trajectories to manage
  • nb_time_steps (int): Number of time steps in each simulation trajectory

melissa.server.deep_learning.reservoir.BatchReservoir

Bases: BatchGetMixin, Reservoir

A queue implementing a batch version of the Reservoir sampling algorithm.

This class extends the Reservoir queue to support batch sampling. It combines the functionality of the Reservoir sampling algorithm (with random eviction on write) with the ability to retrieve data in batches rather than individual samples. It ensures that the queue never exceeds a specified maximum size and maintains a threshold for sampling readiness.

Attributes
  • threshold (int): The minimum number of items required in the queue to allow sampling.
  • batch_size (int): The number of items to retrieve in each batch.