When running a simulation in parallel, the geometry must first be decomposed (segmented) into individual geometries for each MPI process. These separate geometries are connected together with special processor boundary patches. The processor-specific constant/polyMesh/boundary
files will contain this type of entry:
procBoundary0to14 { type processor; inGroups 1(processor); nFaces 131; startFace 34983; matchTolerance 0.0001; transform unknown; myProcNo 0; neighbProcNo 14; }
The decomposePar utility is a commonly used method to decompose domains and subsequently distribute the fields. The reconstructPar and reconstructParMesh utilities can be used to reconstruct a single domain from the processor sub-domains. In addition to the above non-parallel decomposition and reconstruction tools there is a parallel all-in-one tool redistributePar
:
mpirun -np XXX redistributePar -decompose -parallel
mpirun -np XXX redistributePar -reconstruct -parallel
The decomposeParDict
is required by decompose utilities and for any solvers or utilities running in parallel. It is normally located in the simulation system
directory. The -decomposeParDict name
command-line option can be used to specify an alternate file.
The numberOfSubdomains
entry is mandatory:
numberOfSubdomains <int>;
The method
entry is required for the decomposePar utility and specifies the decomposition method type:
method <word>;
The method
entry is generally not required when running a simulation.
OpenFOAM offers a variety of decomposition methods and interfaces to external, third-party decomposition routines. The types of decomposition methods available will thus depend on your particular installation.
Name | Class |
---|---|
none | Foam::noDecomp |
manual | Foam::manualDecomp |
simple | Foam::simpleGeomDecomp |
hierarchical | Foam::hierarchGeomDecomp |
kahip | Foam::kahipDecomp |
metis | Foam::metisDecomp |
scotch | Foam::scotchDecomp |
structured | Foam::structuredDecomp |
multiLevel | Foam::multiLevelDecomp |
If a decomposition method requires any additional configuration controls, these are specified either within in a generic coeffs
dictionary that or a method-specific version. For example,
method hierarchical; coeffs { n (4 2 3); } // ----- method metis; metisCoeffs { method k-way; }
For simplicity, the generic coeffs
dictionary is generally preferable. However, for some specific decomposition methods, e.g., multiLevel) only the method-specific coefficients dictionary is permitted.
When running multi-region simulations, it may be desirable to use different decomposition methods for one or more regions, or even to have fewer processors allocated to a particular region. If, for example, the multi-region simulation contains a large fluid region and a very small solid region, it can be advantageous to decompose the solid onto fewer processors.
The region-wise specification is contained in a regions sub-dictionary with decomposeParDict
. For example,
numberOfSubdomains 2048; method metis; regions { heater { numberOfSubdomains 2; method hierarchical; coeffs { n (2 1 1); } } "*.solid" { numberOfSubdomains 16; method scotch; } }
numberOfSubdomains
remains mandatory, since this specifies the number of domains for the entire simulation. The individual regions may use the same number or fewer domains. The numberOfSubdomains
entry within a region specification is only needed if the value differs.The Foam::multiLevelDecomp decomposition provides a general means of successively decomposing with different methods. Each application of the decomposition is termed a level. For example,
numberOfSubdomains 2048; method multiLevel; multiLevelCoeffs { nodes { numberOfSubdomains 128; method hierarchical; coeffs { n (16 4 2); } } cpus { numberOfSubdomains 2; method scotch; } cores { numberOfSubdomains 8; method metis; } }
For cases where the same method is applied at each level, this can also be conveniently written in a much shorter form:
numberOfSubdomains 2048; method multiLevel; multiLevelCoeffs { method scotch domains (128 2 8); }
When the specified domains
is smaller than numberOfSubdomains
but can be resolved as an integral multiple, this integral multiple is used as the first level. This can make it easier to manage when changing the number of domains for the simulation. For example,
numberOfSubdomains 1024; method multiLevel; multiLevelCoeffs { method scotch domains (2 8); //< inferred as domains (64 2 8); }
These are constraints applied to the decomposition. Typical uses might be e.g. to keep a cyclicAMI patch on a single processor (might speed up simulation) or have maximum local donors in an overset case.
constraints { // Keep owner and neighbour of baffles on same processor // (ie, keep it detectable as a baffle). // Baffles are two boundary face sharing the same points baffles { type preserveBaffles; enabled true; } // Keep owner and neighbour on same processor for faces in zones faces { type preserveFaceZones; zones (".*"); enabled true; } // Keep owner and neighbour on same processor for faces in patches // (only makes sense for cyclic patches. Not suitable for e.g. // cyclicAMI since these are not coupled on the patch level. // Use singleProcessorFaceSets for those. patches { type preservePatches; patches (".*"); enabled true; } // Keep all of faceSet on a single processor. This puts all cells // connected with a point, edge or face on the same processor. // (just having face connected cells might not guarantee a balanced // decomposition) // The processor can be -1 (the decompositionMethod chooses the // processor for a good load balance) or explicitly provided (upsets // balance) processors { type singleProcessorFaceSets; sets ((f1 -1)); enabled true; } // Decompose cells such that all cell originating from single cell // end up on same processor refinement { type refinementHistory; enabled true; } // Prevent decomposition splitting of the geometric regions // Uses any topoSetFaceSource for selecting the constrained faces geometric { type geometric; grow false; selection { box1 { source box; min (-10 -10 -10); max (1 1 1); } ball1 { source sphere; origin (-2 -2 1); radius 1; } arbitrary { source surface; surfaceType triSurfaceMesh; surfaceName blob.obj; } } } }
Would you like to suggest an improvement to this page? | Create an issue |
Copyright © 2018 OpenCFD Ltd.