v2506: New and improved parallel operation

This release includes a number of low-level changes in the bridge code between OpenFOAM and MPI. Although the changes are largely hidden from the general user, they are useful for scalability and creating future communication patterns.

Direct mapping to MPI data types and operations

Most communication is now able to map directly to fundamental MPI data types and predefined aggregate MPI data types. This change means that fields now send the number of elements instead of the number of bytes. Although the amount of data communicated remains the same, the number of data items that can be communicated increases accordingly. For vector, this means that the size of the communicated field increases by a factor 24 before hitting the MPI limit. Mapping of data types occurs at compile-time. In addition to the mapping to MPI data types, the infrastructure now also provides compile-time mapping to fundamental MPI operators. With these changes, the following type of code:

reduce(someVectors, maxOp());

now maps to a single MPI Allreduce call instead of a series of point-to-point communications followed by a broadcast.

Support for node topology communication

On startup the following node communicators are always created:

• UPstream::commInterNode() - communicator between nodes
• UPstream::commLocalNode() - communicator within a node

These help when writing topology-aware algorithms that leverage faster local node communication.

Often in large parallel runs using cyclicA(C)MI patches, only a small subset of processors host AMI patch faces. In this release, transfers are limited to processors that participate in A(C)MI specific communication by allocating a local 'communicator'. On large cases with multiple AMI sets this can make a dramatic difference in the amount (and overlap) of communication. The new behaviour is controlled by the localAMIComm setting in the OptimisationSwitches section of the controlDict:

• localAMIComm = 0 : old behaviour - all processors included in the communication
• localAMIComm = 1 : default - only processors with patch faces
• localAMIComm = 2 : like 1 but includes always processor 0

The difference between option 1 and 2 is that in option 1 any statistics, e.g. min/max weight of AMI matching, will come from the lowest numbered processor that has patch faces instead of the usual processor 0. Depending on the parallel running method the output might be at a different location in the log file (all the other messages are from processor0). Always adding processor 0 (option 2) guarantees consistent output but introduces an additional bottleneck. A typical example considering a large case with 4 AMI pairs on 4096 cores yielded the following timings:

Note that these results are taken from a single run; on a lower number of cores / less AMI pairs the benefit will decrease.

Source code

• $FOAM_SRC/meshTools/AMIInterpolation/AMIInterpolation

Code

• Commit 34c933