There are a number of runtime parameters that are used to control the output and frequency of IO files. A list of all the runtime parameters and their descriptions for the IO unit can be found online all of them. Additional description is located in Table 9.2 for checkpoint parameters, Table 9.3 for plotfile parameters, Table 9.4 for particle file parameters, Table 9.5 for flash.dat parameters, and Table 9.6 for genereal IO parameters.
The API routine for writing a checkpoint file is IO_writeCheckpoint. Users usually will not need to call this routine directly because the FLASH IO unit calls IO_writeCheckpoint from the routine IO_output which checks the runtime parameters to see if it is appropriate to write a checkpoint file at this time. There are a number of ways to get FLASH to produce a checkpoint file for restarting. Within the flash.par, runtime parameters can be set to dump output. A checkpoint file can be dumped based on elapsed simulation time, elapsed wall clock time or the number of timesteps advanced. A checkpoint file is also produced when the simulation ends, when the max simulation time tmax, the minimum cosmological redshift, or the total number of steps nend has been reached. A user can force a dump to a checkpoint file at another time by creating a file named .dump_checkpoint in the output directory of the master processor. This manual action causes FLASH to write a checkpoint in the next timestep. Checkpoint files will continue to be dumped after every timestep as long as the code finds a .dump_checkpoint file in the output directory, so the user must remember to remove the file once all the desired checkpoint files have been dumped. Creating a file named .dump_restart in the output directory will cause FLASH to output a checkpoint file and then stop the simulation. This technique is useful for producing one last checkpoint file to save time evolution since the last checkpoint, if the machine is going down or a queue window is about to end. These different methods can be combined without problems. Each counter (number of timesteps between last checkpoint, amount of simulation time single last checkpoint, the change in cosmological redshift, and the amount of wall clock time elapsed since the last checkpoint) is independent of the others, and are not influenced by the use of a .dump_checkpoint or .dump_restart.
Runtime Parameters used to control checkpoint file output include:
|checkpointFileNumber||INTEGER||0||The number of the initial checkpoint file. This number is appended to the end of the filename and incremented at each subsequent output. When restarting a simulation, this indicates which checkpoint file to use.|
|checkpointFileIntervalStep||INTEGER||0||The number of timesteps desired between subsequent checkpoint files.|
|checkpointFileIntervalTime||REAL||1.||The amount of simulation time desired between subsequent checkpoint files.|
|checkpointFileIntervalZ||REAL||HUGE(1.)||The amount of cosmological redshift change that is desired between subsequent checkpoint files.|
|rolling_checkpoint||INTEGER||10000||The number of checkpoint files to keep available at any point in the simulation. If a checkpoint number is greater than rolling_checkpoint, then the checkpoint number is reset to 0. There will be at most rolling_checkpoint checkpoint files kept. This parameter is intended to be used when disk space is at a premium.|
|wall_clock_checkpoint||REAL||43200.||The maximum amount of wall clock time (seconds) to elapse between checkpoints. When the simulation is started, the current time is stored. If wall_clock_checkpoint seconds elapse over the course of the simulation, a checkpoint file is stored. This is useful for ensuring that a checkpoint file is produced before a queue closes.|
|restart||BOOLEAN||.false.||A logical variable indicating whether the simulation is restarting from a checkpoint file (.true.) or starting from scratch (.false.).|
FLASH is capable of restarting from any of the checkpoint files it produces. The user should make sure that the checkpoint file is valid (e.g., the code did not stop while outputting). To tell FLASH to restart, set the restart runtime parameter to .true. in the flash.par. Also, set checkpointFileNumber to the number of the file from which you wish to restart. If plotfiles or particle files are being produced set plotfileNumber and particleFileNumber to the number of the next plotfile and particle file you want FLASH to output. In FLASH4 plotfiles and particle file outputs are forced whenever a checkpoint file is written. Sometimes several plotfiles may be produced after the last valid checkpoint file. Resetting plotfileNumber to the first plotfile produced after the checkpoint from which you are restarting will ensure that there are no gaps in the output. See Sec:Plotfiles for more details on plotfiles.
A plotfile contains all the information needed to
interpret the grid data maintained by FLASH. The data in plotfiles,
including the grid metadata such as coordinates and block sizes,
are stored at single precision to preserve space. This can, however,
be overridden by setting the runtime parameters plotfileMetadataDP
and/or plotfileGridQuantityDP to true to set the grid metadata and the
quantities stored on the grid (dens, pres, temp, etc.) to use double precision,
respectively. Users must choose
which variables to output with the runtime parameters
plot_var_2, etc., by
setting them in the flash.par file. For example:
plot_var_1 = "dens" plot_var_2 = "pres"
Currently, we support a number of plotvars named plot_var_n up to the number of UNKVARS in a given simulation. Similarly, scratch variables may be output to plot files lbl:OutputScratchVariables. At this time, the plotting of face centered quantities is not supported.
The interface for writing a plotfile is the routine
IO_writePlotfile. As with
checkpoint files, the user will not need to call this routine directly
because it is invoked indirectly through calling
IO_output when, based on runtime parameters,
FLASH4 needs to write a plotfile. FLASH can produce plotfiles in much the same
manner as it does with checkpoint files. They can be dumped based on
elapsed simulation time, on steps since the last plotfile dump or by
forcing a plotfile to be written by hand by creating a.dump_plotfile
in the output
directory. A plotfile will also be written at the termination of a simulation
If plotfiles are being kept at particular intervals (such as time intervals) for purposes such as visualization or analysis, it is also possible to have FLASH denote a plotfile as ``forced". This designation places the word forced between the basename and the file format type identifier (or the split number if splitting is used). These files are numbered separately from normal plotfiles. By default, plotfiles are considered forced if output for any reason other than the change in simulation time, change in cosmological redshift, change in step number, or the termination of a simulation from reaching nend , zFinal, or tmax. This option can be disabled by setting ignoreForcedPlot to true in a simulations flash.par file. The following runtime parameters pertain to controlling plotfiles:
|plotFileNumber||INTEGER||0||The number of the starting (or restarting) plotfile. This number is appended to the filename.|
|plotFileIntervalTime||REAL||1.||The amount of simulation time desired between subsequent plotfiles.|
|plotFileIntervalStep||INTEGER||0||The number of timesteps desired between subsequent plotfiles.|
|plotFileIntervalZ||INTEGER||HUGE(1.)||The change in cosmological redshift desired between subsequent plotfiles.|
|plot_var_1, ..., plot_var_n||STRING||"none"||Name of the variables to store in a plotfile. Up to 12 variables can be selected for storage, and the standard 4-character variable name can be used to select them.|
|ignoreForcedPlot||BOOLEAN||.false.||A logical variable indicating whether or not to denote certain plotfiles as forced.|
|forcedPlotfileNumber||INTEGER||0||An integer that sets the starting number for a forced plotfile.|
|plotfileMetadataDP||BOOLEAN||.false.||A logical variable indicating whether or or not to output the normally single-precision grid metadata fields as double precision in plotfiles. This specifically affects coordinates, block size, and bounding box.|
|plotfileGridQuantityDP||BOOLEAN||.false.||A logical variable that sets whether or not quantities stored on the grid, such as those stored in unk, are output in single precision or double precision in plotfiles.|
|particleFileNumber||INTEGER||0||The number of the starting (or restarting) particle file. This number is appended to the end of the filename.|
|particleFileIntervalTime||REAL||1.||The amount of simulation time desired between subsequent particle file dumps.|
|particleFileIntervalStep||INTEGER||0||The number of timesteps desired between subsequent particle file dumps.|
|particleFileIntervalZ||REAL||HUGE(1.)||The change in cosmological redshift desired between subsequent particle file dumps.|
All the code necessary to output particle data is contained in the IO
subunit called IOParticles. Whenever the Particles unit is included in
a simulation the correct IOParticles subunit will also be included.
For example as setup:
./setup IsentropicVortex -2d -auto -unit=Particles +ug
will include the IO unit IO/IOMain/hdf5/serial/UG and the correct
IO/IOParticles/hdf5/serial/UG. The shortcuts +parallelio, +pnetcdf, +ug will also cause the setup script to pick up the correct IOParticles subunit as long as a Particles unit is included in the simulation.
|stats_file||STRING||"flash.dat"||Name of the file to which the integral quantities are written.|
|wr_integrals_freq||INTEGER||1||The number of timesteps to elapse between outputs to the scalar/integral data file (flash.dat)|
There are several runtime parameters that pertain to the general IO unit or multiple output files rather than one particular output file. They are listed in the table below.
|basenm||STRING||"flash_"||The main part of the output filenames. The full filename consists of the base name, a series of three-character abbreviations indicating whether it is a plotfile, particle file or checkpoint file, the file format, and a 4-digit file number. See Sec:Output file names for a description of how FLASH output files are named.|
|output_directory||STRING||""||Output directory for plotfiles, particle files and checkpoint files. The default is the directory in which the executable sits. output_directory can be an absolute or relative path.|
|memory_stat_freq||INTEGER||100000||The number of timesteps to elapse between memory statistic dumps to the log file (flash.log).|
|useCollectiveHDF5||BOOLEAN||.true.||When using the parallel HDF5 implementation of IO, will enable collective mode for HDF5.|
|summaryOutputOnly||BOOLEAN||.false.||When set to .true. write an integrated grid quantities file only. Checkpoint, plot and particle files are not written unless the user creates a .dump_plotfile, .dump_checkpoint, .dump_restart or .dump_particle file.|
Generic Flash Docs User 2019-10-18