Parallel Computing

Parallel Computing

The availability of parallel computing provides an opportunity to simulate problems of greater complexity, with greater accuracy, more quickly. Users can benefit greatly from the parallel computing capabilities available in OpenFOAM, e.g.

  • Almost everything, from meshing, pre-processing, through simulation, to post-processing, runs in parallel as standard; no stage is limited to run on a single CPU on the memory allocated to that CPU.
  • The cost of parallel simulation is affordable to all because OpenFOAM is free.
  • Parallelisation is robust and integrated at a low level, so in general, new applications require no ‘parallel-specific’ coding; they will run in parallel by default.
  • A convenient interface, Pstream, to plug any Message Passing Interface (MPI) library into OpenFOAM

Domain decomposition and MPI

OpenFOAM employs domain decomposition, with its decomposePar utility, to split the mesh and fields into a number of sub-domains and allocate them to separate processors. Applications can then run in parallel on separate sub-domains, with communication between processors with software that uses the MPI communications protocol. While OpenFOAM is shipped with the OpenMPI library, any MPI library, such as those optimised for particular hardware platforms, can be used with OpenFOAM by “plugging” it in through the Pstream interface.

Performance

There have been many studies into the parallel performance of OpenFOAM. Performance results vary depending on:

  • the nature of the tests, e.g. fixed number of total cells or fixed number of cells per node;
  • class of parallel computer, e.g. multicore, cluster, massively-parallel;
  • choice of Message Passing Interface (MPI) software for distributed memory machines, e.g. OpenMPI, native system MPI, etc.;

While it is difficult to make conclusive statements based on the study data available, it appears that with a suitable system set up, OpenFOAM scales well up to at least 1000 CPUs. A study, for example, by the CSC - IT Center for Science, Finland tested a 3 dimensional cavity flow in OpenFOAM on the Cray XT4 on cases of 13.8 and 22 million cells. It concludes that “On Cray XT OpenFOAM exhibits super-linear scalability.” The scalability graph does indeed show slightly above linear scaling on simulations from 64 to 1024 CPUs.

A study by the EPCC on HECToR, another Cray XT supercomputer, shows good scaling for cases that are sufficiently large that the problem is not dominated by inter-processor communication. In the scaling results of that study, the jet simulation on the “fine” mesh shows good scaling up to 512 cores and the report concludes that “OpenFOAM scales well on HECToR for both simple tutorial cases and for complex industrial cases”. In addition, the fine jet simulation used only 19 million cells, so that at 1024 cores, there are only 18,500 cells per core. With a larger number of cells per core, e.g. 100,000+, better scaling would be expected.