Conduit Integration in Melissa¶

Why Conduit?¶

Before this migration, every message between a Melissa client and the server was a hand-packed byte buffer. A fixed-size header (timestep, sim ID, client rank, payload size, metadata size, field name) was written field by field with memcpy, and the receiving end on the server had to unpack that exact same layout using struct.unpack with a format string that mirrored the C struct byte for byte. Any change to that layout, even adding a single field, meant updating both sides in lockstep which is not scalable.

Conduit removes this fragility by making every message self-describing. A conduit::Node carries its own schema (shape, type, and layout of its data) alongside the data itself. The receiving end doesn't need to know in advance what's inside; it reads the schema first and then interprets the data accordingly. This is the foundation the entire client-server protocol now rests on.

Every Exchange Is a Conduit Node¶

This is not limited to the data payload. The handshake, the parameter fetch protocol, and the termination signal are all Conduit nodes now, serialised the same way and carried over the same ZMQ transport. There is exactly one wire format in the entire system, defined once in conduit_bridge, used everywhere:

the connection request the client sends on startup (comm size, simulation ID)
the connection response the server sends back (server topology, memmap path, push endpoints)
the data envelope wrapping every simulation send
the parameter fetch request and response in persistent client mode
the termination message the client sends at finalize

Nothing in the protocol is a raw int or a manually packed struct anymore. If the schema needs to grow. it's matter of adding a line to a node in question.

The Wire Format¶

Every node, whatever it contains, is serialised the same way before it touches a ZMQ socket:

a int64_t giving the length of the node's schema as JSON
followed by that JSON schema char8 itself
followed by the uint8_t compacted binary data.

The receiving side reads the length, decodes the schema, and reconstructs the node from the bytes that follow. This framing is deliberately simple and deliberately uniform. There's no negotiation, no protocol versioning byte, no special-casing per message type. A connection response and a multi-gigabyte simulation field go over the wire through the identical code path.

The Envelope (what-you-send-is-what-you-get)¶

User data is never sent naked. Whatever the user hands to melissa::send, it gets wrapped in an envelope that carries the routing information the server needs to know where this data belongs: which simulation, which client rank (only rank 0 at this moment) sent it, which named channel (node_name) it was registered under, and which timestep this is. The user's actual data lives under a single payload key inside that envelope, untouched and unmodified.

This separation matters because it means the payload schema is entirely the user's to define. Melissa doesn't know or care what's inside payload. It could be a single float array, a deeply nested mesh structure with coordinates and connectivity and field values, or a handful of scalar diagnostics. The envelope's job is purely routing; the payload's job is purely data.

The Three Overloads of `melissa::send`¶

The C++ API exposes melissa::send for three different ways a user might already have their data sitting in memory, and all three converge on exactly the same underlying path.

If you hand it a conduit::Node directly, that node becomes the payload as-is no copying, no transformation, whatever schema you built is exactly what the server receives.
If you hand it a raw double* and a size, the function builds a conduit::Node around that pointer using set_external_float64_ptr. Conduit wraps the existing memory rather than copying it, describes it as a float64 array of the given length, and that becomes the payload.
If you hand it a raw float* and a size, the same thing happens with set_external_float32_ptr.

All three paths terminate in the same call to the conduit::Node& overload of send, which builds the envelope, serialises it, and pushes it over the socket. The convenience overloads exist purely so that a user with a plain array doesn't have to construct a Conduit node by hand for the common case.

Python Native Upcasting

An additional benefit of unifying these workflows is data type consistency at the endpoint. Raw memory arrays transmitted via the C++ or C pointer-overload signatures are automatically decoded directly into fully contiguous NumPy arrays on the Python server side.

Gathering Before Sending¶

Melissa operates in a 1xM communication pattern: many simulation ranks, one server-facing send. This means that before anything goes over the wire, the contributions from every MPI rank running the simulation have to come together on rank 0, which is the only rank that actually talks to the server.

This is where Conduit's relay::mpi::gather_using_schema comes in. Each rank builds its own local node and rank 0 gathers all of them. What happens to the data during that gather depends entirely on what shape the user's node was in to begin with, and this is a distinction worth understanding clearly because it changes what the server sees on the other end.

If every rank handed in a plain flat array under the same key, Melissa recognizes this as homogeneous numeric data and the gather concatenates the arrays end to end, in rank order, into a single flat array on rank 0. Four ranks each contributing 2500 doubles become one rank with 10000 doubles. This is the natural behaviour for the common case where a simulation's domain is decomposed across ranks and each rank just holds its local slice of one global field.
If instead the user's node is a more elaborate structure that simply can't be concatenated. The gather instead consolidates the contributions into a structure indexed by rank. Rank 0 ends up with a node that has each rank's full structure preserved underneath a rank-indexed key, rather than any attempt to merge or concatenate dissimilar schemas together. There is no silent flattening of structure Conduit can't reconcile; if the data doesn't have an unambiguous way to combine, it's kept apart and labeled by origin.

This means the user has real control over what the server eventually sees, just by choosing what shape to build their per-rank node in. A simulation sending one large decomposed field gets a single concatenated array on the server with no extra wrapping. A simulation sending richer per-rank diagnostics such as mesh-local statistics that don't make sense merged, then it gets a clean per-rank breakdown instead, with nothing lost and nothing forced together that shouldn't be.

It's worth being deliberate about which of these you want, because the schema you build per rank is what determines it there's no separate flag or setting controlling gather behaviour, it falls directly out of whether your local node's schema is uniform and flat across ranks or not.

Decoding using Python¶

On the Python server side, deserializing incoming ZMQ messages follows the inverse of the wire format:

Schema Unpacking: The header int64_t is unpacked to determine the exact byte length of the trailing JSON schema string.
Schema Instantiation: The JSON string is extracted and parsed into a conduit.Schema object.
Payload Binding: The remaining byte slice is mapped to a temporary envelope node using set_external(schema, payload_bytes).
Deep Compaction: Because set_external creates non-owning, transient references into the network socket buffer, the temporary node is immediately copied using envelope_node.compact_to(compacted_node).

The resulting compacted_node owns its memory allocations on the C++ heap, allowing payload fields (such as sub-nodes or NumPy arrays) to be safely extracted and retained by the server without risk of memory corruption or dangling references.

Memory Alignment & Buffer Realignment

Conduit schema headers have variable byte lengths, meaning the payload binary data often begins at an unaligned memory offset (offset % 8 != 0).

On 64-bit architectures, binding unaligned byte buffers directly via set_external causes PyConduit's C++ pointer casts (double*, int64_t*) to misalign, leading to silent data corruption or segmentation faults when parsing large arrays.

To guarantee 8-byte boundary alignment, the payload byte slice must be re-aligned into a fresh buffer (e.g., via bytes(msg[offset:])) prior to calling set_external(), followed immediately by compact_to() to solidify ownership.

What This Buys Going Forward?¶

Because the payload schema is entirely open, Melissa's wire protocol no longer constrains what a simulation can send. Now, a simulation training a surrogate model can send a payload with named features, coordinates, and auxiliary metadata sitting alongside the field data, all in one node, all self-describing, and the server reads it back exactly as built without either side needing prior knowledge of that particular simulation's schema.

The cost of this flexibility is honesty about gather semantics. A user who wants concatenation needs a uniform flat schema per rank, and a user who wants per-rank structure preserved needs to embrace that their gathered node will be rank-indexed on the server side. There's no API call that hides this distinction or tries to guess what you meant.