[FLASH-USERS] Details of speedup after restart

[Dan Sheeler]

Does this run have just 64 blocks (4 x 16)?  If this is a standard flash 
setup with local physics, having less than one block per process probably 
will produce weird performance numbers.  In a standard run, 2d 8x8 blocks 
require very little ram or work per process.  Single processes are happy 
working on thousands of blocks.  Furthermore, work is distributed to the 
processors in nothing smaller than block-sized chunks.  If your run is a 
typical setup, then half of the processes have more-or-less nothing to do 
but add communication, and I can imagine that would cause runtime to 
fluctuate non-deterministically.

Dan

  --
Dan Sheeler
ASC Flash Center
sheeler at flash.uchicago.edu
(773) 834-3236

On Tue, 5 Jun 2007, sanjib gupta wrote:

> Hi,
>
> I am attaching 2 log files - the initial run on 128 processors, then 
> immediately killing the job and restarting from the first checkpoint file 
> "hc-rt-hdf5_chk_0000"
> notice about 4 timesteps per second initially, then ~30 timesteps/sec after 
> restart.
>
> On 64 processors I noticed the gain was higher , but my resolution was lower 
> (half the number of nblocky, same nblockx, this is a 2D run)- sorry did not 
> keep the logfiles.
>
> However this "gain" cannot be predicted......sometimes I don't get it on the 
> first restart, so I restart a couple of times!
> As you'all can guess, this plays havoc with any benchmarking efforts 
> .......and we do intend to showcase our results from FLASH soon ...   :-)
>
> We compile with intel fortran 9.1.033 and openmpi 1.1 on a linux cluster 
> ....and hdf5 version 1.6.5 ......Makefile.h is attached.
> Architecture - 64 bit AMD Opteron
> running FC3 linux  + BProcV4 (cluster OS) with kernel = 2.6.14
>
> Thanks much for your help/insight/suggestions,
> Sanjib.
>