hostCollatedFileOperation Class Reference

Version of collatedFileOperation with multiple read/write ranks. More...

Inheritance diagram for hostCollatedFileOperation:
[legend]
Collaboration diagram for hostCollatedFileOperation:
[legend]

Public Member Functions

 TypeName ("hostCollated")
 Runtime type information. More...
 
 hostCollatedFileOperation (const bool verbose)
 Default construct. More...
 
virtual ~hostCollatedFileOperation ()
 Destructor. More...
 
- Public Member Functions inherited from collatedFileOperation
 TypeName ("collated")
 Runtime type information. More...
 
 collatedFileOperation (bool verbose)
 Default construct. More...
 
 collatedFileOperation (const label comm, const labelList &ioRanks, const word &typeName, bool verbose)
 Construct from user communicator. More...
 
virtual ~collatedFileOperation ()
 Destructor. More...
 
virtual fileName objectPath (const IOobject &io, const word &typeName) const
 Generate disk file name for object. Opposite of filePath. More...
 
virtual bool writeObject (const regIOobject &, IOstreamOption streamOpt=IOstreamOption(), const bool valid=true) const
 Writes a regIOobject (so header, contents and divider). More...
 
virtual void flush () const
 Forcibly wait until all output done. Flush any cached data. More...
 
virtual word processorsDir (const IOobject &) const
 Actual name of processors dir. More...
 
virtual word processorsDir (const fileName &) const
 Actual name of processors dir. More...
 
virtual void setNProcs (const label nProcs)
 
- Public Member Functions inherited from masterUncollatedFileOperation
 TypeName ("masterUncollated")
 Runtime type information. More...
 
 masterUncollatedFileOperation (bool verbose)
 Default construct. More...
 
 masterUncollatedFileOperation (const label comm, bool verbose)
 Construct from communicator. More...
 
virtual ~masterUncollatedFileOperation ()
 Destructor. More...
 
virtual bool mkDir (const fileName &, mode_t=0777) const
 Make directory. More...
 
virtual bool chMod (const fileName &, const mode_t) const
 Set the file mode. More...
 
virtual mode_t mode (const fileName &, const bool followLink=true) const
 Return the file mode. More...
 
virtual fileName::Type type (const fileName &, const bool followLink=true) const
 Return the file type: DIRECTORY, FILE or LINK. More...
 
virtual bool exists (const fileName &, const bool checkGzip=true, const bool followLink=true) const
 Does the name exist (as DIRECTORY or FILE) in the file system? More...
 
virtual bool isDir (const fileName &, const bool followLink=true) const
 Does the name exist as a DIRECTORY in the file system? More...
 
virtual bool isFile (const fileName &, const bool checkGzip=true, const bool followLink=true) const
 Does the name exist as a FILE in the file system? More...
 
virtual off_t fileSize (const fileName &, const bool followLink=true) const
 Return size of file. More...
 
virtual time_t lastModified (const fileName &, const bool followLink=true) const
 Return time of last file modification. More...
 
virtual double highResLastModified (const fileName &, const bool followLink=true) const
 Return time of last file modification. More...
 
virtual fileNameList readDir (const fileName &, const fileName::Type=fileName::FILE, const bool filtergz=true, const bool followLink=true) const
 Read a directory and return the entries as a string list. More...
 
virtual bool cp (const fileName &src, const fileName &dst, const bool followLink=true) const
 Copy, recursively if necessary, the source to the destination. More...
 
virtual bool ln (const fileName &src, const fileName &dst) const
 Create a softlink. dst should not exist. Returns true if. More...
 
virtual bool mv (const fileName &src, const fileName &dst, const bool followLink=false) const
 Rename src to dst. More...
 
virtual bool mvBak (const fileName &, const std::string &ext="bak") const
 Rename to a corresponding backup file. More...
 
virtual bool rm (const fileName &) const
 Remove a file, returning true if successful otherwise false. More...
 
virtual bool rmDir (const fileName &dir, const bool silent=false) const
 Remove a directory and its contents. More...
 
virtual fileName filePath (const bool checkGlobal, const IOobject &io, const word &typeName, const bool search) const
 Search for an object. checkGlobal : also check undecomposed case. More...
 
virtual fileName dirPath (const bool checkGlobal, const IOobject &io, const bool search) const
 Search for a directory. checkGlobal : also check undecomposed. More...
 
virtual fileNameList readObjects (const objectRegistry &db, const fileName &instance, const fileName &local, word &newInstance) const
 Search directory for objects. Used in IOobjectList. More...
 
virtual bool readHeader (IOobject &, const fileName &, const word &typeName) const
 Read object header from supplied file. More...
 
virtual autoPtr< ISstreamreadStream (regIOobject &, const fileName &, const word &typeName, const bool valid=true) const
 Reads header for regIOobject and returns an ISstream. More...
 
virtual bool read (regIOobject &, const bool masterOnly, const IOstreamOption::streamFormat format, const word &typeName) const
 Top-level read. More...
 
virtual autoPtr< ISstreamNewIFstream (const fileName &) const
 Generate an ISstream that reads a file. More...
 
virtual autoPtr< OSstreamNewOFstream (const fileName &pathname, IOstreamOption streamOpt=IOstreamOption(), const bool valid=true) const
 Generate an OSstream that writes a file. More...
 
virtual label addWatch (const fileName &) const
 Add watching of a file. Returns handle. More...
 
virtual bool removeWatch (const label) const
 Remove watch on a file (using handle) More...
 
virtual label findWatch (const labelList &watchIndices, const fileName &) const
 Find index (or -1) of file in list of handles. More...
 
virtual void addWatches (regIOobject &, const fileNameList &) const
 Helper: add watches for list of regIOobjects. More...
 
virtual fileName getFile (const label) const
 Get name of file being watched (using handle) More...
 
virtual void updateStates (const bool masterOnly, const bool syncPar) const
 Update state of all files. More...
 
virtual fileMonitor::fileState getState (const label) const
 Get current state of file (using handle) More...
 
virtual void setUnmodified (const label) const
 Set current state of file (using handle) to unmodified. More...
 
virtual instantList findTimes (const fileName &, const word &) const
 Get sorted list of times. More...
 
virtual IOobject findInstance (const IOobject &io, const scalar startValue, const word &stopInstance) const
 Find instance where IOobject is. Fails if cannot be found. More...
 
virtual void setTime (const Time &) const
 Callback for time change. More...
 
const HashPtrTable< instantList > & times () const
 Return cached times. More...
 
- Public Member Functions inherited from fileOperation
 TypeName ("fileOperation")
 Runtime type information. More...
 
 fileOperation (const label comm, const bool distributedRoots=false)
 Construct from communicator, optionally with distributed roots. More...
 
 declareRunTimeSelectionTable (autoPtr, fileOperation, word,(bool verbose),(verbose))
 
virtual ~fileOperation ()=default
 Destructor. More...
 
bool distributed () const noexcept
 Distributed roots (parallel run) More...
 
bool distributed (bool on) const noexcept
 Set distributed roots on/off (mutable) More...
 
virtual fileName filePath (const fileName &) const
 Search for a file or directory. Use IOobject version in. More...
 
virtual label nProcs (const fileName &dir, const fileName &local="") const
 Get number of processor directories/results. Used for e.g. More...
 
fileName processorsCasePath (const IOobject &, const word &procDir) const
 Generate path (like io.path) from root+casename with any. More...
 
fileName processorsPath (const IOobject &, const word &instance, const word &procDir) const
 Generate path (like io.path) with provided instance and any. More...
 
fileName processorsPath (const fileName &, const word &procDir) const
 Operating on fileName: replace processorXXX with procDir. More...
 

Additional Inherited Members

- Public Types inherited from fileOperation
enum  pathType : int {
  NOTFOUND = 0, ABSOLUTE, OBJECT, WRITEOBJECT,
  PROCUNCOLLATED, PROCBASEOBJECT = PROCUNCOLLATED + 1, PROCOBJECT = PROCBASEOBJECT + 1, PARENTOBJECT,
  FINDINSTANCE, PROCUNCOLLATEDINSTANCE, PROCBASEINSTANCE, PROCINSTANCE
}
 Enumeration for the location of an IOobject. More...
 
typedef Tuple2< fileName, Tuple2< pathType, int > > dirIndex
 Augment fileName with pathType and local offset. More...
 
typedef List< dirIndexdirIndexList
 
typedef IntRange< int > procRangeType
 
- Static Public Member Functions inherited from masterUncollatedFileOperation
static bool uniformFile (const fileNameList &)
 Same file? More...
 
- Static Public Member Functions inherited from fileOperation
static autoPtr< fileOperationNewUncollated ()
 Static construct the commonly used uncollatedFileOperation. More...
 
static autoPtr< fileOperationNew (const word &handlerType, bool verbose=false)
 Select fileHandler-type. More...
 
static instantList sortTimes (const fileNameList &dirEntries, const word &constantName="constant")
 Sort directory entries according to time value,. More...
 
static label splitProcessorPath (const fileName &objectPath, fileName &path, fileName &procDir, fileName &local, procRangeType &group, label &nProcs)
 Split objectPath into part before 'processor' and part after. More...
 
static label detectProcessorPath (const fileName &objPath)
 Detect processor number from '/aa/bb/processorDDD/cc'. More...
 
- Static Public Attributes inherited from collatedFileOperation
static float maxThreadFileBufferSize
 Max size of thread buffer size. This is the overall size of. More...
 
- Static Public Attributes inherited from masterUncollatedFileOperation
static float maxMasterFileBufferSize
 Max size of parallel communications. Switches from non-blocking. More...
 
- Static Public Attributes inherited from fileOperation
static const Enum< pathTypepathTypeNames_
 
static word processorsBaseDir = "processors"
 Return the processors directory name (usually "processors") More...
 
static word defaultFileHandler
 Name of the default fileHandler. More...
 
static autoPtr< fileOperationfileHandlerPtr_
 Static fileOperation. More...
 
- Protected Member Functions inherited from collatedFileOperation
void printBanner (const bool printRanks=false) const
 Print banner information, optionally with io ranks. More...
 
bool isMasterRank (const label proci) const
 
bool appendObject (const regIOobject &io, const fileName &pathName, IOstreamOption streamOpt) const
 Append to processorsNN/ file. More...
 
- Protected Member Functions inherited from masterUncollatedFileOperation
template<class Type >
Type scatterList (const UList< Type > &, const int, const label comm) const
 
template<class Type , class fileOp >
Type masterOp (const fileName &, const fileOp &fop, const int tag, const label comm) const
 
template<class Type , class fileOp >
Type masterOp (const fileName &, const fileName &, const fileOp &fop, const int tag, const label comm) const
 
virtual fileName filePathInfo (const bool checkGlobal, const bool isFile, const IOobject &, const bool search, pathType &searchType, word &processorsDir, word &instance) const
 Search (locally!) for object; return info on how it was found. More...
 
fileName localObjectPath (const IOobject &, const pathType &searchType, const word &processorsDir, const word &instancePath) const
 Construct filePath. More...
 
bool exists (const dirIndexList &, IOobject &io) const
 Helper: check IO for local existence. Like filePathInfo but. More...
 
- Protected Member Functions inherited from fileOperation
fileMonitormonitor () const
 Get or create fileMonitor singleton. More...
 
refPtr< dirIndexListlookupAndCacheProcessorsPath (const fileName &objectPath, const bool syncPar) const
 Lookup name of processorsDDD using cache. More...
 
virtual refPtr< dirIndexListlookupProcessorsPath (const fileName &objectPath) const
 Lookup name of processorsDDD using cache. More...
 
bool exists (IOobject &io) const
 
- Static Protected Member Functions inherited from masterUncollatedFileOperation
static labelList subRanks (const label n)
 Get the list of processors that are part of this communicator. More...
 
static word findInstancePath (const instantList &timeDirs, const instant &t)
 Equivalent of Time::findInstance. More...
 
static void readAndSend (const fileName &filePath, const labelUList &procs, PstreamBuffers &pBufs)
 Read file contents and send to processors. More...
 
static autoPtr< ISstreamread (IOobject &io, const label comm, const bool uniform, const fileNameList &filePaths, const boolList &procValid)
 Read files on comms master. More...
 
- Static Protected Member Functions inherited from fileOperation
static labelList ioRanks ()
 Retrieve list of IO ranks from FOAM_IORANKS env variable. More...
 
static void mergeTimes (const instantList &extraTimes, const word &constantName, instantList &times)
 Merge two times. More...
 
static bool isFileOrDir (const bool isFile, const fileName &)
 Helper: check for file (isFile) or directory (!isFile) More...
 
- Protected Attributes inherited from collatedFileOperation
const label myComm_
 Any communicator allocated by me. More...
 
OFstreamCollator writer_
 Threaded writer. More...
 
label nProcs_
 Number of processors (overall) More...
 
const labelList ioRanks_
 Ranks of IO handlers. More...
 
- Protected Attributes inherited from masterUncollatedFileOperation
const label myComm_
 Any communicator allocated by me. More...
 
HashPtrTable< instantListtimes_
 Cached times for a given directory. More...
 
- Protected Attributes inherited from fileOperation
const label comm_
 Communicator to use. More...
 
bool distributed_
 Distributed roots (parallel run) More...
 
HashTable< dirIndexListprocsDirs_
 Detected processors directories. More...
 
autoPtr< fileMonitormonitorPtr_
 File-change monitor for all registered files. More...
 

Detailed Description

Version of collatedFileOperation with multiple read/write ranks.

In parallel it will assume ranks are sorted according to hostname and the lowest rank per hostname will be the IO rank. The output directories will get a unique name processors<N>_<low>-<high> where N is the overall number of processors and low and high is the range of ranks contained in the files. Each of these subsets uses its own communicator.

Instead of using the hostnames the IO ranks can be assigned using the FOAM_IORANKS environment variable (also when running non-parallel), e.g. when decomposing into 4:

FOAM_IORANKS='(0 2)' decomposePar -fileHandler hostCollated

will generate

processors4_0-1/ containing data for processors 0 to 1 processors4_2-3/ containing data for processors 2 to 3

See also
collatedFileOperation
Source files

Definition at line 75 of file hostCollatedFileOperation.H.

Constructor & Destructor Documentation

◆ hostCollatedFileOperation()

hostCollatedFileOperation ( const bool  verbose)
explicit

Default construct.

Definition at line 128 of file hostCollatedFileOperation.C.

References Foam::infoDetailLevel.

◆ ~hostCollatedFileOperation()

Destructor.

Definition at line 153 of file hostCollatedFileOperation.C.

References UPstream::freeCommunicator(), and UPstream::worldComm.

Here is the call graph for this function:

Member Function Documentation

◆ TypeName()

TypeName ( "hostCollated"  )

Runtime type information.


The documentation for this class was generated from the following files: