From evan.scannapieco at asu.edu Tue Jul 1 16:31:15 2008 From: evan.scannapieco at asu.edu (Evan Scannapieco) Date: Tue, 1 Jul 2008 14:31:15 -0700 Subject: [FLASH-USERS] bn_pzExtr Message-ID: <8361436A-05D2-409F-9DC3-0BE1804C91CD@asu.edu> We've run in some trouble trying to use the Bader solver It appears that in bn_pzExtr delta is computed from x(iest-k1) but x is only given initialized at iest. Are we missing something, or does this routine get into trouble if iest is not 1? Evan ************************** Evan Scannapieco Assistant Professor School of Earth and Space Exploration Arizona State University PO. Box 871404 Tempe, AZ 85287-1404 480-727-6788 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://flash.uchicago.edu/pipermail/flash-users/attachments/20080701/40814214/attachment.html From townsley at uchicago.edu Tue Jul 1 19:11:59 2008 From: townsley at uchicago.edu (Dean Townsley) Date: Tue, 01 Jul 2008 19:11:59 -0500 Subject: [FLASH-USERS] bn_pzExtr In-Reply-To: Message from Evan Scannapieco of "Tue, 01 Jul 2008 14:31:15 PDT." <8361436A-05D2-409F-9DC3-0BE1804C91CD@asu.edu> References: <8361436A-05D2-409F-9DC3-0BE1804C91CD@asu.edu> Message-ID: Hi Evan, I do believe that you have found a rather scary bug. I'm comparing this implementation to Ed Brown's network code, which implements the same solver in c++. I think an equivalent comparison can be made to any of Frank Timmes' downloadable networks. (in which this code is closely related to the flash code for this.) Not only x but also qcol should be static objects across calls to this function. (i.e. this function updates them) This means they should either be declared save or be promoted to module-scope variables instead of local variables. Probably this was a mistake in porting this code to flash3. I would suggest just adding save to them. Though I would have to look more carefully if any other similar mistakes have been made. -Dean On Tue, 1 Jul 2008 14:31:15 -0700 Evan Scannapieco wrote: > > We've run in some trouble trying to use the Bader solver > > It appears that in bn_pzExtr delta is computed from x(iest-k1) > but x is only given initialized at iest. Are we missing something, > or does this routine get into trouble if iest is not 1? > > Evan > > From townsley at uchicago.edu Tue Jul 1 23:32:20 2008 From: townsley at uchicago.edu (Dean Townsley) Date: Tue, 01 Jul 2008 23:32:20 -0500 Subject: [FLASH-USERS] bn_pzExtr In-Reply-To: Message from Dean Townsley of "Tue, 01 Jul 2008 19:11:59 CDT." Message-ID: For anyone following this, Ivo Seitenzahl has pointed out that there is a save stament at the beginning of that subroutine, so actually this was a false alarm and things actually seem ok. -Dean On Tue, 01 Jul 2008 19:11:59 -0500 Dean Townsley wrote: > Hi Evan, > > I do believe that you have found a rather scary bug. > I'm comparing this implementation to Ed Brown's network code, which > implements the same solver in c++. I think an equivalent comparison > can be made to any of Frank Timmes' downloadable networks. (in which > this code is closely related to the flash code for this.) > > Not only x but also qcol should be static objects across calls to this > function. (i.e. this function updates them) This means they should > either be declared save or be promoted to module-scope variables > instead of local variables. > > Probably this was a mistake in porting this code to flash3. > > I would suggest just adding save to them. Though I would have to look > more carefully if any other similar mistakes have been made. > > -Dean > > > > On Tue, 1 Jul 2008 14:31:15 -0700 Evan Scannapieco wrote: > > > > We've run in some trouble trying to use the Bader solver > > > > It appears that in bn_pzExtr delta is computed from x(iest-k1) > > but x is only given initialized at iest. Are we missing something, > > or does this routine get into trouble if iest is not 1? > > > > Evan > > > > From apoland at gmu.edu Wed Jul 2 10:48:39 2008 From: apoland at gmu.edu (Arthur Poland) Date: Wed, 02 Jul 2008 11:48:39 -0400 Subject: [FLASH-USERS] Problem viewing output results in IDL Message-ID: <486BA357.7000502@gmu.edu> We are using IDL to look at results from flash 2.5 and want to look at the energy balance and importance of each term in equation 9.32 in the flash users guide. Since the diffusive quantities are calculated together for efficiency we cannot get each one put out in the plot file, they are not calculated individually within flash. Note that my problem is only in looking at the data after flash is done with the calculations. We are looking at the results in IDL where quantities are interpolated onto a constant grid. In principle we can calculate the quantities within IDL. However, within the blocks quantities seem to be just set to the value in the center of the block rather than interpolated. We are having particular problems with Del dot sigma grad T in equation 9.32. Has anyone developed a better interpolation scheme for filling the detailed grid so we can calculate and look at these terms? I think our problem seems to be in rebin, called from merge_amr_nnint.pro, called from loaddata. It works with each block without knowing what is next to it, so it thinks it is extrapolating. Rebin won't extrapolate, so it sets quantities to a constant value. This wrecks havoc with derivatives and second derivatives, so we can't get energy balance. Any ideas around it? Thanks Art Poland -- Dr. Art Poland George Mason University Mail Stop 6A2 4400 University Dr. Fairfax VA 22030 Research Building I Rm. 346 Dept. of Computational and Data Sciences Dept. of Physics and Astronomy Office 703-993-8404 Home 540-347-5003 Cell 540-222-1940 Fax 540-347-5526 From kdere at gmu.edu Sat Jul 5 16:00:49 2008 From: kdere at gmu.edu (Ken Dere) Date: Sat, 05 Jul 2008 17:00:49 -0400 Subject: [FLASH-USERS] undefined references in Simulation_init during linking Message-ID: <486FE101.4040903@gmu.edu> I am trying to set up a new problem that bombs when I get to the link stage. I get the following: Simulation_init.o: In function `simulation_init_': Simulation_init.F90:(.text+0xa): undefined reference to `__simulation_data_MOD_simgamma' Simulation_init.F90:(.text+0x1e): undefined reference to `__simulation_data_MOD_simgdirec' Simulation_init.F90:(.text+0x32): undefined reference to `__simulation_data_MOD_simgvector' plus a few more like this this comes from calling, in Simulation_init call RuntimeParameters_get('gamma', simGamma) call RuntimeParameters_get('grv_direc', simGdirec) call RuntimeParameters_get('grv_vector' ,simGvector) simGamma is defined in Simulation_data. I am able to compile and link the sample programs that I have tried and I just don't see what I am doing wrong. suggestions are welcomed. best regards, Ken Dere From ewommer at physics.arizona.edu Sat Jul 5 17:38:37 2008 From: ewommer at physics.arizona.edu (ewommer at physics.arizona.edu) Date: Sat, 5 Jul 2008 15:38:37 -0700 Subject: [FLASH-USERS] problem with Grid_dump.F90 Message-ID: <20080705153837.xjhaplpa1kco8w4c@bohr.physics.arizona.edu> Hi, all. I've recently updated to FLASH3.0 and am getting the following error message when using "gmake" to compile my simulation: fortcom: Error: Grid_dump.F90, line 181: This name does not have a type, and must have an explicit type. [MPI_MODE_CREATE] MPI_MODE_CREATE+MPI_MODE_RDWR,MPI_INFO_NULL,fh,ierr) ----------^ fortcom: Error: Grid_dump.F90, line 181: This name does not have a type, and must have an explicit type. [MPI_MODE_RDWR] MPI_MODE_CREATE+MPI_MODE_RDWR,MPI_INFO_NULL,fh,ierr) --------------------------^ compilation aborted for Grid_dump.F90 (code 1) gmake: *** [Grid_dump.o] Error 1 I haven't modified Grid_dump.F90 in any way. Also, I didn't encounter this error when I was using FLASH3.0_beta. Any suggestions would be appreciated! Elizabeth :) ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. From tangsk at astro.umass.edu Sat Jul 5 21:06:13 2008 From: tangsk at astro.umass.edu (Shikui Tang) Date: Sat, 05 Jul 2008 22:06:13 -0400 Subject: [FLASH-USERS] problem with Grid_dump.F90 In-Reply-To: <20080705153837.xjhaplpa1kco8w4c@bohr.physics.arizona.edu> References: <20080705153837.xjhaplpa1kco8w4c@bohr.physics.arizona.edu> Message-ID: <48702895.5060904@astro.umass.edu> Hi, I met the same problem before. If I compile FLASH3 with lammpi, I have to add the following line into Flash_mpi.h include "mpiof.h" But there is not need for mpich2. Hope this is useful. Shikui ewommer at physics.arizona.edu wrote: > Hi, all. > I've recently updated to FLASH3.0 and am getting the following error > message when using "gmake" to compile my simulation: > > > fortcom: Error: Grid_dump.F90, line 181: This name does not have a > type, and must have an explicit type. [MPI_MODE_CREATE] > MPI_MODE_CREATE+MPI_MODE_RDWR,MPI_INFO_NULL,fh,ierr) > ----------^ > fortcom: Error: Grid_dump.F90, line 181: This name does not have a > type, and must have an explicit type. [MPI_MODE_RDWR] > MPI_MODE_CREATE+MPI_MODE_RDWR,MPI_INFO_NULL,fh,ierr) > --------------------------^ > compilation aborted for Grid_dump.F90 (code 1) > gmake: *** [Grid_dump.o] Error 1 > > > I haven't modified Grid_dump.F90 in any way. Also, I didn't encounter > this error when I was using FLASH3.0_beta. > > Any suggestions would be appreciated! > Elizabeth :) > > ---------------------------------------------------------------- > This message was sent using IMP, the Internet Messaging Program. > > From lynnreid at flash.uchicago.edu Sat Jul 5 21:33:11 2008 From: lynnreid at flash.uchicago.edu (Lynn Reid) Date: Sat, 05 Jul 2008 21:33:11 -0500 Subject: [FLASH-USERS] undefined references in Simulation_init during linking In-Reply-To: <486FE101.4040903@gmu.edu> References: <486FE101.4040903@gmu.edu> Message-ID: <48702EE7.2040406@flash.uchicago.edu> I assume you've remembered to say use Simulation_data, ONLY: simGamma, simGdirec, simGvector in the declarations of the Simulation_init routine. Also, the routine RuntimeParameters_get is an overloaded function. You MUST include a statement similar to use RuntimeParameters_interface, ONLY : RuntimeParameters_get at the top of your Simulation_init routine. If you don't, all sorts of mysterious things happen which depend upon the particular compiler. The parameter 'gamma' should appear in your flash.par and simGamma in your Fortran file. And finally, don't forget to create a Makefile in your SimulationWhatever directory like this: # Makefile for Ken Dere's problem # [/source/Simulation/SimulationMain/Dere] Simulation += Simulation_data.o If you don't have a Makefile, the compiler won't realize that there are dependencies. Lynn Ken Dere wrote: > I am trying to set up a new problem that bombs when I get to the link > stage. I get the following: > > Simulation_init.o: In function `simulation_init_': > Simulation_init.F90:(.text+0xa): undefined reference to > `__simulation_data_MOD_simgamma' > Simulation_init.F90:(.text+0x1e): undefined reference to > `__simulation_data_MOD_simgdirec' > Simulation_init.F90:(.text+0x32): undefined reference to > `__simulation_data_MOD_simgvector' > > plus a few more like this > > this comes from calling, in Simulation_init > > call RuntimeParameters_get('gamma', simGamma) > > call RuntimeParameters_get('grv_direc', simGdirec) > call RuntimeParameters_get('grv_vector' ,simGvector) > > simGamma is defined in Simulation_data. > > I am able to compile and link the sample programs that I have tried > and I just don't see what I am doing wrong. > > suggestions are welcomed. > > best regards, > > Ken Dere -- Dr. Lynn B. Reid lynnreid at uchicago.edu Scientific Programmer, ASC Flash Center, RI473 University of Chicago, IL USA 60637 +1-773-702-0624 Don't anthropomorphize computers. They don't like it. From dubey at flash.uchicago.edu Sat Jul 5 22:00:29 2008 From: dubey at flash.uchicago.edu (Anshu Dubey) Date: Sat, 5 Jul 2008 22:00:29 -0500 Subject: [FLASH-USERS] problem with Grid_dump.F90 In-Reply-To: <48702895.5060904@astro.umass.edu> References: <20080705153837.xjhaplpa1kco8w4c@bohr.physics.arizona.edu> <48702895.5060904@astro.umass.edu> Message-ID: <5b942a610807052000qb21c8c8x63b2ea156216464@mail.gmail.com> Also, Grid_dump is a purely diagnostic routine. You can delete the file "source/Grid/GridMain/Grid_dump.F90" from your copy of FLASH3, the code will work fine with the null implementation in "source/Grid/Grid_dump.F90" Anshu On Sat, Jul 5, 2008 at 9:06 PM, Shikui Tang wrote: > Hi, > > I met the same problem before. If I compile FLASH3 with lammpi, I have to > add the following line into Flash_mpi.h > include "mpiof.h" > But there is not need for mpich2. Hope this is useful. > > Shikui > > ewommer at physics.arizona.edu wrote: >> >> Hi, all. >> I've recently updated to FLASH3.0 and am getting the following error >> message when using "gmake" to compile my simulation: >> >> >> fortcom: Error: Grid_dump.F90, line 181: This name does not have a type, >> and must have an explicit type. [MPI_MODE_CREATE] >> MPI_MODE_CREATE+MPI_MODE_RDWR,MPI_INFO_NULL,fh,ierr) >> ----------^ >> fortcom: Error: Grid_dump.F90, line 181: This name does not have a type, >> and must have an explicit type. [MPI_MODE_RDWR] >> MPI_MODE_CREATE+MPI_MODE_RDWR,MPI_INFO_NULL,fh,ierr) >> --------------------------^ >> compilation aborted for Grid_dump.F90 (code 1) >> gmake: *** [Grid_dump.o] Error 1 >> >> >> I haven't modified Grid_dump.F90 in any way. Also, I didn't encounter >> this error when I was using FLASH3.0_beta. >> >> Any suggestions would be appreciated! >> Elizabeth :) >> >> ---------------------------------------------------------------- >> This message was sent using IMP, the Internet Messaging Program. >> >> > > From josef.stoeckl at uibk.ac.at Sun Jul 6 04:44:35 2008 From: josef.stoeckl at uibk.ac.at (=?iso-8859-1?Q?Josef_St=F6ckl?=) Date: Sun, 6 Jul 2008 11:44:35 +0200 Subject: [FLASH-USERS] undefined references in Simulation_init during linking In-Reply-To: <486FE101.4040903@gmu.edu> References: <486FE101.4040903@gmu.edu> Message-ID: <000601c8df4c$e721f510$b565df30$@stoeckl@uibk.ac.at> Hi Ken, Did you check your dependencies in the Makefile? This looks like you haven't specified the module dependencies, so the FLASH setup script doesn't know that you need the Simulation_data module for linking. Please check, if you have something like the following two lines in the Makefile within your simulation's directory (source/Simulation/SimulationMain//Makefile): Simulation += Simulation_data.o Simulation_init.o : Simulation_data.o I hope this helps. After modifying the Makefile you need to call the FLASH setup script again, so that the proper build environment gets reworked. Best regards, Josef > -----Urspr?ngliche Nachricht----- > Von: flash-users-bounces at flash.uchicago.edu [mailto:flash-users- > bounces at flash.uchicago.edu] Im Auftrag von Ken Dere > Gesendet: Samstag, 05. Juli 2008 23:01 > An: flash-users at flash.uchicago.edu > Betreff: [FLASH-USERS] undefined references in Simulation_init during > linking > > I am trying to set up a new problem that bombs when I get to the link > stage. I get the following: > > Simulation_init.o: In function `simulation_init_': > Simulation_init.F90:(.text+0xa): undefined reference to > `__simulation_data_MOD_simgamma' > Simulation_init.F90:(.text+0x1e): undefined reference to > `__simulation_data_MOD_simgdirec' > Simulation_init.F90:(.text+0x32): undefined reference to > `__simulation_data_MOD_simgvector' > > plus a few more like this > > this comes from calling, in Simulation_init > > call RuntimeParameters_get('gamma', simGamma) > > call RuntimeParameters_get('grv_direc', simGdirec) > call RuntimeParameters_get('grv_vector' ,simGvector) > > simGamma is defined in Simulation_data. > > I am able to compile and link the sample programs that I have tried and > I just don't see what I am doing wrong. > > suggestions are welcomed. > > best regards, > > Ken Dere From kdere at gmu.edu Sun Jul 6 09:53:52 2008 From: kdere at gmu.edu (Ken Dere) Date: Sun, 06 Jul 2008 10:53:52 -0400 Subject: [FLASH-USERS] undefined references in Simulation_init during linking In-Reply-To: <486FE101.4040903@gmu.edu> References: <486FE101.4040903@gmu.edu> Message-ID: <4870DC80.3040901@gmu.edu> Ken Dere wrote: > I am trying to set up a new problem that bombs when I get to the link > stage. I get the following: > > Simulation_init.o: In function `simulation_init_': > Simulation_init.F90:(.text+0xa): undefined reference to > `__simulation_data_MOD_simgamma' > Simulation_init.F90:(.text+0x1e): undefined reference to > `__simulation_data_MOD_simgdirec' > Simulation_init.F90:(.text+0x32): undefined reference to > `__simulation_data_MOD_simgvector' > > plus a few more like this > > this comes from calling, in Simulation_init > > call RuntimeParameters_get('gamma', simGamma) > > call RuntimeParameters_get('grv_direc', simGdirec) > call RuntimeParameters_get('grv_vector' ,simGvector) > > simGamma is defined in Simulation_data. > > I am able to compile and link the sample programs that I have tried > and I just don't see what I am doing wrong. > > suggestions are welcomed. > > best regards, > > Ken Dere > the problem was the lack of a makefile in my directory. thanks for the quick help, best regards, Ken From latife at astro.rug.nl Mon Jul 7 09:12:35 2008 From: latife at astro.rug.nl (M.A. Latife) Date: Mon, 07 Jul 2008 16:12:35 +0200 Subject: [FLASH-USERS] Dark Matter Density Message-ID: <48722453.1060505@astro.rug.nl> Hi Dear All, I am including particles in my simulations to include the effect of dark matter. In user guide it is mentioned that pden is mesh mapped particle density but it seems me that it is sum of particle density + gas density. Is it true? My second question is ,I want to create density profile like rho=rho0*Radius^(-alpha) for dark matter . To create a dark matter density profile should i initialize more particles where i want to create profile (i mean changing Particle positions file )or FLASH will do it by putting profile of dark matter in simulation _init Block.f90 as it works for gas. Thanks in advance Best regards Latif From klaus at flash.uchicago.edu Mon Jul 7 11:31:20 2008 From: klaus at flash.uchicago.edu (Klaus Weide) Date: Mon, 7 Jul 2008 11:31:20 -0500 (CDT) Subject: [FLASH-USERS] undefined references in Simulation_init during linking In-Reply-To: <000601c8df4c$e721f510$b565df30$@stoeckl@uibk.ac.at> References: <486FE101.4040903@gmu.edu> <000601c8df4c$e721f510$b565df30$@stoeckl@uibk.ac.at> Message-ID: On Sun, 6 Jul 2008, Josef St?ckl wrote: > Hi Ken, > > Did you check your dependencies in the Makefile? This looks like you haven't > specified the module dependencies, so the FLASH setup script doesn't know > that you need the Simulation_data module for linking. Please check, if you > have something like the following two lines in the Makefile within your > simulation's directory (source/Simulation/SimulationMain/ name>/Makefile): > > Simulation += Simulation_data.o > Simulation_init.o : Simulation_data.o A minor additional remark: The second of those lines should not be necessary any more (but it doesn't hurt either). The setup procedure figures out these kinds of dependencies automatically (by looking at Fortran USE statements) and puts them into Makefile.Depend, and that gets included from the Makefile. Klaus From dubey at flash.uchicago.edu Mon Jul 7 12:33:40 2008 From: dubey at flash.uchicago.edu (Anshu Dubey) Date: Mon, 7 Jul 2008 12:33:40 -0500 Subject: [FLASH-USERS] Dark Matter Density In-Reply-To: <48722453.1060505@astro.rug.nl> References: <48722453.1060505@astro.rug.nl> Message-ID: <5b942a610807071033o44d501dalaf9ab65928f425bd@mail.gmail.com> On Mon, Jul 7, 2008 at 9:12 AM, M.A. Latife wrote: > Hi > Dear All, > I am including particles in my simulations to include the effect of dark > matter. In user guide it is mentioned that pden is mesh mapped particle > density but it seems me that it is sum of particle density + gas density. Is > it true? pden is as stated in the users guide outside of the gravity unit. In the routine Grid_potentialListOfBlocks, pden briefly becomes a combination of partlcle and gas density to calculate the potential. Once potential is calculated, the value of gas density is deleted from pden before returning from the routine. > My second question is ,I want to create density profile like > rho=rho0*Radius^(-alpha) for dark matter . To create a dark matter density > profile should i initialize more particles where i want to create profile > (i mean changing Particle positions file )or FLASH will do it by putting > profile of dark matter in simulation _init Block.f90 as it works for gas. > Thanks in advance > Best regards > Latif >cd You will have to write a customized "Particles_initPositions" (see PoisParticles setup for an example) or pt_initPositions (see Orbit setup for example) in your setup directory. > From turcotte at apollo.ubishops.ca Mon Jul 7 15:18:01 2008 From: turcotte at apollo.ubishops.ca (turcotte) Date: Mon, 7 Jul 2008 16:18:01 -0400 (EDT) Subject: [FLASH-USERS] multigrid in 2-D Message-ID: <200807072018.m67KI1a26576@apollo.ubishops.ca> Hello all, I have been trying to use the multigrid solver for gravity in 2-D and have have had difficulty making it work well. I have a cartsian grid set up with a small "star" with the rest of the domain filled with low density "circumstellar" matter. I eventually have inflowing matter from one edge of the domain. If I define grav_boundary_type as "isolated" the code crashes in gr_isoMpoleInit because it is trying to use the multipole solver that does not work in 2-D cartesian. If I define grav_boundary_type as "periodic" the code crashes and I get the following error message: gr_hgMapBcType: An unexpected gr_hgBndConditions was requested for the WORK arr ay: 1 -32 Now I have managed to make it work if I use grav_boundary_type = "periodic" and if I short-circuit gr_hgMapBcType by adding bcTypeToApply = GRIDBC_MG_EXTRAPOLATE return at the top of the routine before anything else is executed. No crashes and the code seems to be doing the right thing eventually (the potential looks right) but the potential is not ok during the first several time steps. There are strong features in the gravitational potential at the boundaries that vary strongly with time. Apart from the problems at the start of the run it seems that there should be a better way of doing things! Thank you for any ideas! And belated thanks for all who have helped in the past! Sylvain Sylvain Turcotte Physics Department turcotte at apollo.ubishops.ca Bishop's University tel: (819) 822-9600 x 2287 Lennoxville (Quebec) fax: (819) 822-9661 Canada J1M 1Z7 office: J 11B URL: http://apollo.ubishops.ca/~turcotte From hudson at mcs.anl.gov Wed Jul 9 15:36:56 2008 From: hudson at mcs.anl.gov (Randy Hudson) Date: Wed, 9 Jul 2008 15:36:56 -0500 Subject: [FLASH-USERS] Visit 1.9.1 & Curve plots of 1d data Message-ID: <41F326AD-FD08-4C4F-995E-C98F8BD44963@mcs.anl.gov> FYI, the latest visit (1.9.1) will now plot curves of single-variable expressions of 1d flash data. Randy. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://flash.uchicago.edu/pipermail/flash-users/attachments/20080709/628bbf06/attachment.html From bernalcg at astroscu.unam.mx Wed Jul 9 19:32:01 2008 From: bernalcg at astroscu.unam.mx (Cristian Giovanny Bernal) Date: Wed, 9 Jul 2008 18:32:01 -0600 Subject: [FLASH-USERS] conserve_var? Message-ID: <20080710001927.M73607@astroscu.unam.mx> Hi people, I have a question. The module MHD of Flash2.5 conserve the variables (like the mass)? I am running a simulation with a rate of injection of mass constant by one of the boundaries (Yr). The other boundaries are periodic (Xl, Xr) and reflecting (Yl). I have two setups (HYDRO & MHD) with exactly the same physic and boundary condition. Nevertheless, when I plotting the flash.dat (using 1:2 "mass vs time"), I see that HYDRO_setup conserve the mass in the dominion (see the attached plot), but MHD not. I do not why! The magnetic field affect the conservation of the variables? I'm setting the conserved_var = .true. in the flash par, but the result is the same. I had tried with several magnetic field configuration in the dominion, and with reflecting boundary lower (Yl) or user_boundary with conduction properties, but this no change the result... THE MHD_SETUP loss mass (no conserve the mass). I had saw the dens_profile both Setups and the inflow mass is constant, but in the HYDRO_SETUP the mass is accumulated onto lower boundary while in the MHD_SETUP the mass si not accumulated. The physics used are: gravity (planepar), EOS (Helmholtz), neutrinoloss (a modified version of radloss), Hydro and MHD. Cartesian coordinates are used in 2D. any idea??? The FLASH3.0 also suffers from this? very thanks for your help... -- Instituto de Astronomia Universidad Nacional Autonoma de Mexico (UNAM) http://www.astroscu.unam.mx -------------- next part -------------- A non-text attachment was scrubbed... Name: hydro.gif Type: image/gif Size: 7948 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080709/f19ab033/attachment.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: mhd.gif Type: image/gif Size: 10598 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080709/f19ab033/attachment-0001.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: dens_cheva_hydro_mhd_bucle.gif Type: image/gif Size: 357427 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080709/f19ab033/attachment-0002.gif From klaus at flash.uchicago.edu Thu Jul 10 17:35:25 2008 From: klaus at flash.uchicago.edu (Klaus Weide) Date: Thu, 10 Jul 2008 17:35:25 -0500 (CDT) Subject: [FLASH-USERS] multigrid in 2-D In-Reply-To: <200807072018.m67KI1a26576@apollo.ubishops.ca> References: <200807072018.m67KI1a26576@apollo.ubishops.ca> Message-ID: On Mon, 7 Jul 2008, turcotte wrote: > Hello all, > > I have been trying to use the multigrid solver for gravity in 2-D > and have have had difficulty making it work well. > > I have a cartsian grid set up with a small "star" with the rest of the domain > filled with low density "circumstellar" matter. I eventually have inflowing > matter from one edge of the domain. > > If I define grav_boundary_type as "isolated" the code crashes in > gr_isoMpoleInit because it is trying to use the multipole solver that does > not work in 2-D cartesian. > > If I define grav_boundary_type as "periodic" the code crashes and I get the > following error message: > gr_hgMapBcType: An unexpected gr_hgBndConditions was requested for the WORK arr > ay: 1 -32 > > Now I have managed to make it work if I use grav_boundary_type = "periodic" > and if I short-circuit gr_hgMapBcType by adding > bcTypeToApply = GRIDBC_MG_EXTRAPOLATE > return > at the top of the routine before anything else is executed. > No crashes and the code seems to be doing the right thing eventually (the > potential looks right) but the potential is not ok during the first > several time steps. There are strong features in the gravitational potential at > the boundaries that vary strongly with time. > > Apart from the problems at the start of the run it seems that there should be a > better way of doing things! > > Thank you for any ideas! > And belated thanks for all who have helped in the past! > Sylvain Hi Sylvain, It seems you have run into a limitation I will describe below. Short summary: parameter combinations like (I.) geometry = "cartesian" xl_boundary_type = "periodic" ... grav_boundary_type = "isolated" or (II.) geometry = "cartesian" xl_boundary_type = "outflow" (or similar) ... grav_boundary_type = "periodic" are not expected to work. I will try to give some explanation, then come back to your concrete problem at the end. With the Paramesh 3 (or 4) Grid implementation: FLASH provides some implementation of boundary conditions (BCs) for three different purposes: (1) For Hydro, and variants like MHD. (2) For certain additional physics solvers - here, the multigrid Poisson solver used for Gravity. (3) For Particles. I will not discuss (3) here since that is a somewhat separate topic, as it deals with the updating of particle properties rather than of mesh-represented solution variables. As for (2), this uses really the same mechanisms as (1). In some sense, there isn't a mechanism for gravitational BCs that is separate from that for Hydro BCs. There only is one Grid mechanism for BCs. To go into more detail: FLASH implements boundary conditions by filling guard cells of boundary blocks in certain special ways, whenever it's time to fill guard cells. (FLASH currently only supports BCs that can be expressed this way.) Guard cells are filled at certain synchronization points, when Grid_fillGuardcells (or its underlying PARAMESH implementation) is called, based on the needs of the algorithm of a solver. "Boundary blocks" are blocks that have some guard cells that lie outside the physical domain. FLASH relies entirely on PARAMESH to keep track of things like - which are the neighbors of a block in any given direction, - which blocks are boundary blocks, - which cell regions in boundary blocks require calls to BC routines in order to have guard cells filled, - which BC type to apply in any direction in any given boundary block. PARAMESH weaves separate blocks into a mesh with the help of a few arrays of per-block metainformation. (these are local in the sense that each processor stores only metainformation on blocks that reside on it, although metainformation on blocks of neighboring processors may be cached.) Primarily, this is the "surr_blks" array. For each possible direction, including diagonals, it indicates what's there in that direction from the block's center: (a) another block at the same refinement level, indicated by a (procNo, localID) pair; (b) inside the domain but no block at the same refinement level, indicated by -1 (there must then be block at a lower refinement level in that direction, which can be identified by following "parent" and other meta- information); (c) no block whatsoever, since there is an external boundary in that direction, indicated by a negative value <= PARAMESH_PHYSICAL_BOUNDARY. Now here is the important part: PERIODIC boundaries are not represented by a special indicator in surr_blks as in case (c), but by directly identifying the wrapped-around neighbor. So in this case the presence of a domain boundary is not apparent in the neighbor metainformation, it looks like case (a) or (b). The "surr_blks" representation is at the core of the PARAMESH Grid representation. This is a natural and efficient way of representing the topological relations between blocks in a periodic domain. Unfortunately it does not allow for treating a given domain boundary as periodic in one context and non-periodic in another. At least not if in each of these contexts we want guard cells being filled by PARAMESH, but according to different ideas about the mesh's topology. So the bottom line: - The [xyz][lr]_boundary_type runtime parameters determine the topology of the mesh that PARAMESH constructs (and updates). - The grav_boundary_type runtime parameter must not contradict that topology. (Certainly at least when the Multigrid gravity solver is used, since that needs guard cell filling done by PARAMESH internally.) Could this limitation be overcome? Not without a lot of messing around with the code, I think. Some possibilities come to my mind, such as - enhance PARAMESH so it can hold (and manage) more than one view of the mesh's topology; - somehow create two alternative views of the mesh topology, and swap them around under PARAMESH's feet as needed without PARAMESH needing to be aware of it; - duplicate a significant part of PARAMESH functionality in FLASH, in order to effect exchange of guard cells among blocks that PARAMESH does not consider neighbors, and/or too fill guard cells according to some BC method where PARAMESH wants them filled from neighboring blocks (depending on which of the cases (I.) or (II.) way above are of interest). None of this sounds too attractive, certainly not easy. Finally, back to the original problem. It seems you tried your hand at making case (II.) work. Calling gr_hgMapBcType resulted in an error because in a mesh consistent with grav_boundary_type=="periodic" (i.e., a 3D torus) there would not actually be any boundary blocks as far as PARAMESH is concerned, so gr_hgMapBcType should never be called at all from within the multigrid solver. By short-circuiting gr_hgMapBcType, you trick PARAMESH into doing something to the boundary block guard cells, but that is not the right thing: the right thing consistent with grav_boundary_type== "periodic" would be filling of those guard cells with data from neighboring (wrapped) blocks. I don't know what would be the meaning of any eventual "solution" gotten this way; even if your potential eventually looks right, that may be deceiving. Anyway, it seems you didn't really want to mix periodic with non-periodic BCs originally, and I don't know if that would actually be physically meaningful. You were just looking for a way to work around a limitation of the multigrid solver (doesn't work in 2D Cartesian). So perhaps the best way forward is to make the auxiliary multipole solver used for isolated gravity boundary conditions work in 2D Cartesian. Please excuse the length of this reply, in part this is because I was trying to work these things out for myself; maybe the details can help some others, too. Klaus From seyit at astro.rug.nl Tue Jul 15 07:50:11 2008 From: seyit at astro.rug.nl (Seyit Hocuk) Date: Tue, 15 Jul 2008 14:50:11 +0200 Subject: [FLASH-USERS] Problem with Compiling Flash3.0 In-Reply-To: <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> References: <48441DC4.9050701@astro.rug.nl> <5b942a610806020923m20556b61pa7c99be31cc46562@mail.gmail.com> <48455BAA.8050405@astro.rug.nl> <2467fdc0806031010p22ed2a2bwb7bf9d4497af5d95@mail.gmail.com> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> Message-ID: <487C9D03.90800@astro.rug.nl> Hi, I reinstalled everything and it seems to work! The annoying remarks of loops being vectorized is gone if I put -vec-report0 and remove the -fast option. I tested and found that the -ipo within the -fast option is the cause of this report output. I hope that by removing this -ipo (which in my compiler is shown as -ipo[n]), it doesn't create a different problem. At the moment it looks like it works fine now. Is it dangerous to do it like this? Thanks to all the people who replied, Tomek, Rusty, Nathan, Chris,Tomek again, Brock and Anshu. Kind regards, Seyit Tomasz Plewa wrote: > Seyit - > > I would suggest a clean installation starting with reviewing your > environment. Once your environment is clean of references to old > compilers, etc., get the new compiler in, then mpich, little guys like > zlib and/or szip, and finally hdf5. And since you are at it, you may > want to install a serial hdf5 version as well. > > The best if you do it in one sequence. > > Tomek From tomek at scs.fsu.edu Tue Jul 15 09:01:52 2008 From: tomek at scs.fsu.edu (Tomasz Plewa) Date: Tue, 15 Jul 2008 10:01:52 -0400 Subject: [FLASH-USERS] Problem with Compiling Flash3.0 In-Reply-To: <487C9D03.90800@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <5b942a610806020923m20556b61pa7c99be31cc46562@mail.gmail.com> <48455BAA.8050405@astro.rug.nl> <2467fdc0806031010p22ed2a2bwb7bf9d4497af5d95@mail.gmail.com> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> Message-ID: <20080715100152.ma647g3gkgos80ks@webmail.scs.fsu.edu> Inlining is tricky, frequently broken with oversimplified compiler heuristics. Test before using! ;) Tomek -- Department of Scientific Computing, DSL 443 ph: 850.644.1959 Florida State University fax: 850.644.0098 Tallahassee, FL 32306 web: people.scs.fsu.edu/~tomek/ -- Seyit Hocuk wrote: > Hi, > > I reinstalled everything and it seems to work! The annoying remarks of > loops being vectorized is gone if I put -vec-report0 and remove the > -fast option. I tested and found that the -ipo within the -fast option > is the cause of this report output. I hope that by removing this -ipo > (which in my compiler is shown as -ipo[n]), it doesn't create a > different problem. At the moment it looks like it works fine now. Is it > dangerous to do it like this? > > Thanks to all the people who replied, Tomek, Rusty, Nathan, Chris,Tomek > again, Brock and Anshu. > > Kind regards, > Seyit > > > > Tomasz Plewa wrote: >> Seyit - >> >> I would suggest a clean installation starting with reviewing your >> environment. Once your environment is clean of references to old >> compilers, etc., get the new compiler in, then mpich, little guys >> like zlib and/or szip, and finally hdf5. And since you are at it, >> you may want to install a serial hdf5 version as well. >> >> The best if you do it in one sequence. >> >> Tomek > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From nhearn at uchicago.edu Tue Jul 15 09:24:58 2008 From: nhearn at uchicago.edu (Nathan Hearn) Date: Tue, 15 Jul 2008 09:24:58 -0500 Subject: [FLASH-USERS] Problem with Compiling Flash3.0 In-Reply-To: <487C9D03.90800@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <5b942a610806020923m20556b61pa7c99be31cc46562@mail.gmail.com> <48455BAA.8050405@astro.rug.nl> <2467fdc0806031010p22ed2a2bwb7bf9d4497af5d95@mail.gmail.com> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> Message-ID: <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> Hi Seyit, It is certainly not dangerous to do this -- in fact "fast" options sometimes cause the compiler to produce machine code that is not fully equivalent to the code being compiled (from a mathematical standpoint). However, you may be losing out on some of the performance capabilities of your hardware. The messages about loop vectorization -- although, admittedly, they are annoying -- are the compiler announcing that it has found ways to use the vector hardware on your CPU (e.g., SSE, AltiVec), which can result in multiple floating point operations occurring simultaneously. In other words, a single operation produces two or more results in parallel, instead of just one. (In your case, did the number of vectorization messages decrease by using the "-vec-report0" option?) The inter-procedural optimizations option (ipo) can produce more tightly-coupled code when many function calls are made from a given location in the code. Typically, this option will perform a lot of operations in the final stage of building the complete program, and trigger many of the other compiler operations to be called again. (Thus, you can end up with twice the number of vectorization messages when you are done.) Again, these are just performance optimizations, and removing them should not produce incorrect code. But, it may be worth comparing the performance (time-to-completion) of code that is compiled with and without these optimizations. My recommendation would be to look at the documentation for your compiler, and select some specific optimizations instead of simply using "fast". (It sounds like you may have already done this.) I have a feeling that you will get a bigger performance boost from the vectorization than the interprocedural optimizations, so I would start there first. - Nathan On Tue, Jul 15, 2008 at 7:50 AM, Seyit Hocuk wrote: > Hi, > > I reinstalled everything and it seems to work! The annoying remarks of loops > being vectorized is gone if I put -vec-report0 and remove the -fast option. > I tested and found that the -ipo within the -fast option is the cause of > this report output. I hope that by removing this -ipo (which in my compiler > is shown as -ipo[n]), it doesn't create a different problem. At the moment > it looks like it works fine now. Is it dangerous to do it like this? > > Thanks to all the people who replied, Tomek, Rusty, Nathan, Chris,Tomek > again, Brock and Anshu. > > Kind regards, > Seyit From seyit at astro.rug.nl Tue Jul 15 11:55:56 2008 From: seyit at astro.rug.nl (Seyit Hocuk) Date: Tue, 15 Jul 2008 18:55:56 +0200 Subject: [FLASH-USERS] Problem with Compiling Flash3.0 In-Reply-To: <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> References: <48441DC4.9050701@astro.rug.nl> <5b942a610806020923m20556b61pa7c99be31cc46562@mail.gmail.com> <48455BAA.8050405@astro.rug.nl> <2467fdc0806031010p22ed2a2bwb7bf9d4497af5d95@mail.gmail.com> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> Message-ID: <487CD69C.1060403@astro.rug.nl> Hi Nathan, > In your case, did the number of vectorization messages decrease by using the "-vec-report0" > option? > Only if I don't use -ipo or -fast (contains -ipo). Because -ipo puts these remarks at the end wheter you used -vec-report or not. > But, it may be worth comparing the > performance (time-to-completion) of code that is compiled with and > without these optimizations. My recommendation would be to look at > the documentation for your compiler, and select some specific > optimizations instead of simply using "fast". (It sounds like you may > have already done this.) I did indeed. I also compared with different optimizations (with/without -ipo and/or -fast). My initial quick checks showed that computation speed is always unaffected. > I have a feeling that you will get a bigger > performance boost from the vectorization than the interprocedural > optimizations, so I would start there first. > I didn't have any performance boost. It was exactly the same. At-least in the initial parts of my simulation. There is one huge problem that I noticed though. Normally I am always doing simulations on my own computer or at the cluster available to me. However, now that I am doing on this new computer with this new compiler, my initial results are different and the difference seems to grow bigger during the simulation. I use exactly! the same initial conditions. But I immediately notice that refinement is different (refines less quicker) and dtHydro is slightly different and growing more different compared to my computer. Could this be due to the fact that the new machine is 64 bit and mine is 32 bit and thus more precise? Although of-course the compiler is also different as are the libraries (Hdf5 and Mpich) as is the operating system. Seyit > > - Nathan > > > On Tue, Jul 15, 2008 at 7:50 AM, Seyit Hocuk wrote: > >> Hi, >> >> I reinstalled everything and it seems to work! The annoying remarks of loops >> being vectorized is gone if I put -vec-report0 and remove the -fast option. >> I tested and found that the -ipo within the -fast option is the cause of >> this report output. I hope that by removing this -ipo (which in my compiler >> is shown as -ipo[n]), it doesn't create a different problem. At the moment >> it looks like it works fine now. Is it dangerous to do it like this? >> >> Thanks to all the people who replied, Tomek, Rusty, Nathan, Chris,Tomek >> again, Brock and Anshu. >> >> Kind regards, >> Seyit >> From richp at flash.uchicago.edu Tue Jul 15 12:43:56 2008 From: richp at flash.uchicago.edu (Paul M. Rich) Date: Tue, 15 Jul 2008 12:43:56 -0500 Subject: [FLASH-USERS] Problem with Compiling Flash3.0 In-Reply-To: <487CD69C.1060403@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <5b942a610806020923m20556b61pa7c99be31cc46562@mail.gmail.com> <48455BAA.8050405@astro.rug.nl> <2467fdc0806031010p22ed2a2bwb7bf9d4497af5d95@mail.gmail.com> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> Message-ID: <487CE1DC.8080505@flash.uchicago.edu> Seyit Hocuk wrote: > There is one huge problem that I noticed though. Normally I am always > doing simulations on my own computer or at the cluster available to > me. However, now that I am doing on this new computer with this new > compiler, my initial results are different and the difference seems to > grow bigger during the simulation. I use exactly! the same initial > conditions. But I immediately notice that refinement is different > (refines less quicker) and dtHydro is slightly different and growing > more different compared to my computer. > > Could this be due to the fact that the new machine is 64 bit and mine > is 32 bit and thus more precise? Although of-course the compiler is > also different as are the libraries (Hdf5 and Mpich) as is the > operating system. Yes, this most certainly can cause appreciable differences between runs. Different computer architectures and compilers can and will result in different optimizations being used, and will pretty much ensure that the order of floating point operations will be changed. Depending on the problem, this can give very different-appearing results when output files are compared with sfocu, particularly if the problem has a lot of very sharp discontinuities or other small, rapidly changing, features. If the initial differences are small, at about machine precision for double values, it is probably due to this. It also may be the case that one compiler was automatically initializing data differently, or that uninitialized data is being used in the problem at initialization time unintentionally. If you're getting large errors at initialization, I'd check this. Hope this clarifies things for you, Paul Rich From klaus at flash.uchicago.edu Wed Jul 16 14:58:50 2008 From: klaus at flash.uchicago.edu (Klaus Weide) Date: Wed, 16 Jul 2008 14:58:50 -0500 (CDT) Subject: [FLASH-USERS] Hydrostatic BC, etc: How to make it work in FLASH 3.0 Message-ID: Summarizing from recent exchanges with Mateusz : 1) To make the hydrostatic boundary conditions included with FLASH3 (or any user-supplied alternative implementation) work correctly, the following should be added to a simulations's flash.par file: enableMaskedGCFill = .FALSE. This was an oversight; this flag probably should default to FALSE anyway. Background: The runtime parameter enableMaskedGCFill turns "masked guard cell filling" on, for improved performance (not all variables need to have guard cells filled all the time when Grid_fillGuardCells is called). We have tested the masked guard cell filling before the FLASH3 release quite thoroughly, and don't know of any problems with it except in connection with hydrostatic BCs. The Grid unit includes logic that determines the set of variables for which guard cells need to be communicated (based on requirements specified by the caller of Grid_fillGuardCells). This logic does not take into account that some variables are always required as input for hydrostatic BC implementations (in particular, pressure). This will be fixed in a future FLASH release, by making enableMaskedGCFill default to .FALSE. and/or by considering such requirements in the guard cell masking logic. 2) To make hydrostatic boundary conditions work with the staggered mesh MHD solver - or, more generally, with any setup that uses face variables [FACEVAR keyword in a Config file]: Handling for some overlooked cases (filling of guard cells for face variables) needs to be added to the code. Probably the simples way is the following: In source/Grid/GridBoundaryConditions/OneRow/Grid_applyBCEdge.F90, in order to have guard cells for a face variable at domain boundaries of type "hydrostatic+nvrefl" filled as if this was a "reflecting" boundary, change case(REFLECTING) to case(REFLECTING,HYDROSTATIC_F2_NVREFL) Similarly, to have guard cells for a face variable at domain boundaries of type "hydrostatic+nvout" filled as if this was an "outflow" boundary, change case(OUTFLOW) to case(OUTFLOW,HYDROSTATIC_F2_NVOUT) and to have guard cells for a face variable at domain boundaries of type "hydrostatic+nvdiode" filled as if this was a "diode" boundary, change case(DIODE) to case(DIODE,HYDROSTATIC_F2_NVDIODE) Background: For simple boundary conditions like "outflow", "reflecting", and "diode", guard cells for face variables are generally filled at domain boundaries by using the same rules as for cell-centered variables: by copying values from the last interior cell for "outflow", or copying values from the last few cell layers for "reflecting" (with some special handling for velocities), etc. The above changes let the code handle face variables for hydrostatic-variant BCs in similar ways. Whether this handling is appropriate depends on the problem, of course. The staggered mesh MHD implementation uses face variables for divergence-free magnetic fields. 3) Finally, Mateusz found some errors in the code that actually implements the FLASH2-style hydrostatic boundary conditions, in ../source/Grid/GridBoundaryConditions/Flash2HSE/gr_applyFlash2HSEBC.F90. The errors will eventually be corrected, but they are in code that can only be enabled by editing the source file. Basically, be warned if you are editing gr_applyFlash2HSEBC.F90, don't assume that the code that is disabled is correct. Klaus Weide From seyit at astro.rug.nl Fri Jul 18 11:32:37 2008 From: seyit at astro.rug.nl (Seyit Hocuk) Date: Fri, 18 Jul 2008 18:32:37 +0200 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <487CE1DC.8080505@flash.uchicago.edu> References: <48441DC4.9050701@astro.rug.nl> <5b942a610806020923m20556b61pa7c99be31cc46562@mail.gmail.com> <48455BAA.8050405@astro.rug.nl> <2467fdc0806031010p22ed2a2bwb7bf9d4497af5d95@mail.gmail.com> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> Message-ID: <4880C5A5.50708@astro.rug.nl> Oh man, I am using 3 different computers (with different architectures, compilers, optimizations, environments). All three start differently immediately from the beginning (difference in dt_hydro and refinement). Should I worry very very very much? I'm now doing a complete simulation and am going to check if my end results also change. I'm trembling by the thought..... Paul M. Rich wrote: > Seyit Hocuk wrote: >> There is one huge problem that I noticed though. Normally I am always >> doing simulations on my own computer or at the cluster available to >> me. However, now that I am doing on this new computer with this new >> compiler, my initial results are different and the difference seems >> to grow bigger during the simulation. I use exactly! the same initial >> conditions. But I immediately notice that refinement is different >> (refines less quicker) and dtHydro is slightly different and growing >> more different compared to my computer. >> >> Could this be due to the fact that the new machine is 64 bit and mine >> is 32 bit and thus more precise? Although of-course the compiler is >> also different as are the libraries (Hdf5 and Mpich) as is the >> operating system. > > Yes, this most certainly can cause appreciable differences between > runs. Different computer architectures and compilers can and will > result in different optimizations being used, and will pretty much > ensure that the order of floating point operations will be changed. > Depending on the problem, this can give very different-appearing > results when output files are compared with sfocu, particularly if the > problem has a lot of very sharp discontinuities or other small, > rapidly changing, features. If the initial differences are small, at > about machine precision for double values, it is probably due to this. > > It also may be the case that one compiler was automatically > initializing data differently, or that uninitialized data is being > used in the problem at initialization time unintentionally. If you're > getting large errors at initialization, I'd check this. > > Hope this clarifies things for you, > > Paul Rich From tomek at scs.fsu.edu Fri Jul 18 11:57:53 2008 From: tomek at scs.fsu.edu (Tomasz Plewa) Date: Fri, 18 Jul 2008 12:57:53 -0400 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <4880C5A5.50708@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <5b942a610806020923m20556b61pa7c99be31cc46562@mail.gmail.com> <48455BAA.8050405@astro.rug.nl> <2467fdc0806031010p22ed2a2bwb7bf9d4497af5d95@mail.gmail.com> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> Message-ID: <4880CB91.2050803@scs.fsu.edu> Seyit - Try to simplify your problem. If you run in parallel, can you confirm that the results differ when you run on a single processor? If you run on many levels, how about a single block/level solution? As a rule of thumb: try to simplify your problem. This may help you identifying the source of discrepancies. As you mentioned, there are many components involved, not only the application code. Tomek -- Seyit Hocuk wrote: > Oh man, > > I am using 3 different computers (with different architectures, > compilers, optimizations, environments). All three start differently > immediately from the beginning (difference in dt_hydro and refinement). > Should I worry very very very much? > I'm now doing a complete simulation and am going to check if my end > results also change. > > I'm trembling by the thought..... > > > > > > > > > > Paul M. Rich wrote: >> Seyit Hocuk wrote: >>> There is one huge problem that I noticed though. Normally I am always >>> doing simulations on my own computer or at the cluster available to >>> me. However, now that I am doing on this new computer with this new >>> compiler, my initial results are different and the difference seems >>> to grow bigger during the simulation. I use exactly! the same initial >>> conditions. But I immediately notice that refinement is different >>> (refines less quicker) and dtHydro is slightly different and growing >>> more different compared to my computer. >>> >>> Could this be due to the fact that the new machine is 64 bit and mine >>> is 32 bit and thus more precise? Although of-course the compiler is >>> also different as are the libraries (Hdf5 and Mpich) as is the >>> operating system. >> >> Yes, this most certainly can cause appreciable differences between >> runs. Different computer architectures and compilers can and will >> result in different optimizations being used, and will pretty much >> ensure that the order of floating point operations will be changed. >> Depending on the problem, this can give very different-appearing >> results when output files are compared with sfocu, particularly if the >> problem has a lot of very sharp discontinuities or other small, >> rapidly changing, features. If the initial differences are small, at >> about machine precision for double values, it is probably due to this. >> >> It also may be the case that one compiler was automatically >> initializing data differently, or that uninitialized data is being >> used in the problem at initialization time unintentionally. If you're >> getting large errors at initialization, I'd check this. >> >> Hope this clarifies things for you, >> >> Paul Rich > > -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- A non-text attachment was scrubbed... Name: tomek.vcf Type: text/x-vcard Size: 296 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080718/fabf9a56/attachment.vcf From nhearn at uchicago.edu Fri Jul 18 12:10:13 2008 From: nhearn at uchicago.edu (Nathan Hearn) Date: Fri, 18 Jul 2008 12:10:13 -0500 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <4880C5A5.50708@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> Message-ID: <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> Hi Seyit, Are you seeing these differences in the standard test problems included in Flash (e.g., Noh, Sod, Sedov), or are they specific to the problem that you are running. (Really, this follows from Tomek's advice to look at the simplest configurations where the differences are seen.) Also, it would be good to include some more information in your messages, for instance: the problem that you are running, the specific architectures and compilers, the optimization flags, and the library versions for HDF5 and MPI that you are using. Details like these can be really helpful in evaluating and fixing the issues you are seeing. - Nathan On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk wrote: > Oh man, > > I am using 3 different computers (with different architectures, compilers, > optimizations, environments). All three start differently immediately from > the beginning (difference in dt_hydro and refinement). > Should I worry very very very much? > I'm now doing a complete simulation and am going to check if my end results > also change. > > I'm trembling by the thought..... From seyit at astro.rug.nl Fri Jul 18 12:16:15 2008 From: seyit at astro.rug.nl (Seyit Hocuk) Date: Fri, 18 Jul 2008 19:16:15 +0200 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <4880CB91.2050803@scs.fsu.edu> References: <48441DC4.9050701@astro.rug.nl> <5b942a610806020923m20556b61pa7c99be31cc46562@mail.gmail.com> <48455BAA.8050405@astro.rug.nl> <2467fdc0806031010p22ed2a2bwb7bf9d4497af5d95@mail.gmail.com> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <4880CB91.2050803@scs.fsu.edu> Message-ID: <4880CFDF.7080100@astro.rug.nl> Hi Tomek, > If you run in parallel, can you confirm that the results differ when > you run on a single processor? Quick answer, yes. If I run with multiple processors instead of 1, the results do differ! > > If you run on many levels, how about a single block/level solution? I should try that. But I assume they might differ also, since 1 of the three computers started directly with a different dt_hydro. I'm not sure if that would matter. I know for a fact that choosing gamma 1.001 works on one computer and not on the other (Vcycle problem). Guess it's a too small value. Seyit.. *shivering* ;) From tomek at scs.fsu.edu Fri Jul 18 12:30:06 2008 From: tomek at scs.fsu.edu (Tomasz Plewa) Date: Fri, 18 Jul 2008 13:30:06 -0400 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <4880CFDF.7080100@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <5b942a610806020923m20556b61pa7c99be31cc46562@mail.gmail.com> <48455BAA.8050405@astro.rug.nl> <2467fdc0806031010p22ed2a2bwb7bf9d4497af5d95@mail.gmail.com> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <4880CB91.2050803@scs.fsu.edu> <4880CFDF.7080100@astro.rug.nl> Message-ID: <4880D31E.4090809@scs.fsu.edu> Seyit - >> If you run in parallel, can you confirm that the results differ when >> you run on a single processor? > > Quick answer, yes. If I run with multiple processors instead of 1, the > results do differ! The idea is to narrow the number of options you use. If serial execution produces consistent results, you have a good basis to start. For one, such consistency may indicate that your compilers produce fair results. >> If you run on many levels, how about a single block/level solution? > > I should try that. But I assume they might differ also, since 1 of the > three computers started directly with a different dt_hydro. I'm not sure > if that would matter. Information is generally helpful, but since we are in murky waters then not just any information is equally helpful. If you try a single block, you eliminate perhaps 90% of the code complexity. > I know for a fact that choosing gamma 1.001 works on one computer and > not on the other (Vcycle problem). Guess it's a too small value. Try a single block first, if possible. If results do differ before evolution starts (initial time step differs), there is a problem with initilizing your application. This will narrow your focus to perhaps 1% of the code. Tomek -- -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- A non-text attachment was scrubbed... Name: tomek.vcf Type: text/x-vcard Size: 296 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080718/a07a1e2e/attachment.vcf From seyit at astro.rug.nl Fri Jul 18 12:49:00 2008 From: seyit at astro.rug.nl (Seyit Hocuk) Date: Fri, 18 Jul 2008 19:49:00 +0200 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> Message-ID: <4880D78C.4020708@astro.rug.nl> Hi Nathan, Thats a good Idea, I'll start simple and test those first thing monday. It's too late for me now. To answer your questions: The problem I am running is my own home made, actually it's a modified version of Jeans. Basically including heat, cool, eos gamma, compositions, adjusted refinement criteria similar to one of the other problems (with delta_ref and delta_deref) and a scala of other small things. 1) Centos5 Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 i686 i686 i386 GNU/Linux ifort intel compiler version 8.0 hdf5 1.6.6 (serial) mpich2 Version: 1.0.3 gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include 2) Debian (??) Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC 2005 x86_64 GNU/Linux pgf90 6.0-5 32-bit target on x86-64 Linux hdf5 1.4.5 (serial) mpich2 version 1.0.5 gcc version 3.3.5 (Debian 1:3.3.5-13) FFLAGS_OPT = -c -fast -r8 -i4 3) Ubuntu Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux ifort intel compiler version 10.1.012 hdf5 1.6.7 (serial) mpich2 Version: 1.0.6 gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I $(MPI_PATH)/include or (only -ipo difference, result seemingly doesn't change) FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include Nathan Hearn wrote: > Hi Seyit, > > Are you seeing these differences in the standard test problems > included in Flash (e.g., Noh, Sod, Sedov), or are they specific to the > problem that you are running. (Really, this follows from Tomek's > advice to look at the simplest configurations where the differences > are seen.) > > Also, it would be good to include some more information in your > messages, for instance: the problem that you are running, the specific > architectures and compilers, the optimization flags, and the library > versions for HDF5 and MPI that you are using. Details like these can > be really helpful in evaluating and fixing the issues you are seeing. > > > - Nathan > > > On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk wrote: > >> Oh man, >> >> I am using 3 different computers (with different architectures, compilers, >> optimizations, environments). All three start differently immediately from >> the beginning (difference in dt_hydro and refinement). >> Should I worry very very very much? >> I'm now doing a complete simulation and am going to check if my end results >> also change. >> >> I'm trembling by the thought..... >> From tomek at scs.fsu.edu Fri Jul 18 13:08:25 2008 From: tomek at scs.fsu.edu (Tomasz Plewa) Date: Fri, 18 Jul 2008 14:08:25 -0400 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <4880D78C.4020708@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> Message-ID: <4880DC19.7070205@scs.fsu.edu> Although I prefer a low cost approach to debugging which avoids dealing with setup/system/compiler information at first and attempts to eliminate 99% of the source code, here are quick comments on your systems. (1) pgi 6.0 is a quite antique compiler. It's so old that now I do not even remember what problems we may have had with it. I suspect quite a few and none of them might be relevant... (2) As you might noticed, intel has a newer compiler version. Your CentOS5 system needs upgrades on both mpich2 and compiler sides. (3) Your Ubuntu system seems relatively up-to-date although HDF5 1.8.x is out there and seems working fine. (4) Compilers are known to do strange things to perfect codes! So if you run into a trouble on a new system, one of the first things to try is to lower compiler optimization level. Avoiding -fast or -ip or -inline is usually a good idea. Simple -O0 may suffice for a couple of steps. As a bonus, the code compiles fast. Then the trick is to get it crashing quickly... Again, I suggest to simplify your application, isolate the problem, fix it, then find the best compiler, mpich, options, etc., to run your production code. Enjoy your weekend! Tomek -- Seyit Hocuk wrote: > Hi Nathan, > > Thats a good Idea, I'll start simple and test those first thing monday. > It's too late for me now. > > To answer your questions: > > The problem I am running is my own home made, actually it's a modified > version of Jeans. Basically including heat, cool, eos gamma, > compositions, adjusted refinement criteria similar to one of the other > problems (with delta_ref and delta_deref) and a scala of other small > things. > > 1) > Centos5 > Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 i686 > i686 i386 GNU/Linux > ifort intel compiler version 8.0 > hdf5 1.6.6 (serial) > mpich2 Version: 1.0.3 > gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) > FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include > > 2) > Debian (??) > Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC 2005 > x86_64 GNU/Linux > pgf90 6.0-5 32-bit target on x86-64 Linux > hdf5 1.4.5 (serial) > mpich2 version 1.0.5 > gcc version 3.3.5 (Debian 1:3.3.5-13) > FFLAGS_OPT = -c -fast -r8 -i4 > > 3) > Ubuntu > Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux > ifort intel compiler version 10.1.012 > hdf5 1.6.7 (serial) > mpich2 Version: 1.0.6 > gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) > FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I > $(MPI_PATH)/include > or (only -ipo difference, result seemingly doesn't change) > FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include > > > > > Nathan Hearn wrote: >> Hi Seyit, >> >> Are you seeing these differences in the standard test problems >> included in Flash (e.g., Noh, Sod, Sedov), or are they specific to the >> problem that you are running. (Really, this follows from Tomek's >> advice to look at the simplest configurations where the differences >> are seen.) >> >> Also, it would be good to include some more information in your >> messages, for instance: the problem that you are running, the specific >> architectures and compilers, the optimization flags, and the library >> versions for HDF5 and MPI that you are using. Details like these can >> be really helpful in evaluating and fixing the issues you are seeing. >> >> >> - Nathan >> >> >> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk wrote: >> >>> Oh man, >>> >>> I am using 3 different computers (with different architectures, >>> compilers, >>> optimizations, environments). All three start differently immediately >>> from >>> the beginning (difference in dt_hydro and refinement). >>> Should I worry very very very much? >>> I'm now doing a complete simulation and am going to check if my end >>> results >>> also change. >>> >>> I'm trembling by the thought..... >>> > > -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- A non-text attachment was scrubbed... Name: tomek.vcf Type: text/x-vcard Size: 296 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080718/e8145df7/attachment.vcf From bperret at asu.edu Mon Jul 21 04:28:27 2008 From: bperret at asu.edu (Beatrice Perret) Date: Mon, 21 Jul 2008 02:28:27 -0700 Subject: [FLASH-USERS] Visit 1.9.1/Flash2.5 Message-ID: <69f3c8d10807210228s5258bfc2wc54b41a90c90c7dd@mail.gmail.com> I am using Flash 2.5 and can not switch at this point to Flash3.0. I visualize my data remotely on my laptop which is a Windows machine using visit 1.9.1 and, 1.10.1 as soon as it will be available. The image I am getting is wrong. A friend of mine who has visit 1.6 has looked at the picture, it is what is expected (I mean: it is not a bug in my program), but the same file seen under visit 1.9.1 has a kind of grid mixed with the data. Do you have a special plugin available to visualize data generated by Flash2.5 using any latest visit version 1.9.1 or newer? If you need more information, description or the files I am generating, please let me know. Thanks. Beatrice From hudson at mcs.anl.gov Mon Jul 21 05:22:23 2008 From: hudson at mcs.anl.gov (Randy Hudson) Date: Mon, 21 Jul 2008 05:22:23 -0500 Subject: [FLASH-USERS] Visit 1.9.1/Flash2.5 In-Reply-To: <69f3c8d10807210228s5258bfc2wc54b41a90c90c7dd@mail.gmail.com> References: <69f3c8d10807210228s5258bfc2wc54b41a90c90c7dd@mail.gmail.com> Message-ID: Beatrice, Can you send me images from the two versions of visit? Thanks. On Jul 21, 2008, at 4:28 AM, Beatrice Perret wrote: > I am using Flash 2.5 and can not switch at this point to Flash3.0. I > visualize my data remotely on my laptop which is a Windows machine > using visit 1.9.1 and, 1.10.1 as soon as it will be available. The > image I am getting is wrong. A friend of mine who has visit 1.6 has > looked at the picture, it is what is expected (I mean: it is not a bug > in my program), but the same file seen under visit 1.9.1 has a kind of > grid mixed with the data. Do you have a special plugin available to > visualize data generated by Flash2.5 using any latest visit version > 1.9.1 or newer? > If you need more information, description or the files I am > generating, please let me know. > > Thanks. > > Beatrice Randy. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://flash.uchicago.edu/pipermail/flash-users/attachments/20080721/655eb68b/attachment.html From lia at astro.columbia.edu Tue Jul 22 17:11:09 2008 From: lia at astro.columbia.edu (Lia Corrales) Date: Tue, 22 Jul 2008 18:11:09 -0400 Subject: [FLASH-USERS] getting timestep from dBasePropertyReal Message-ID: <48865AFD.9070509@astro.columbia.edu> Hi all, I am using Flash 2.5 and need to use the timestep (dt) for a calculation. When I include dBasePropertyReal in my eos3d.F90 code, using: dt = dBasePropertyReal("TimeStep") my simulation freezes up during initialization, with the output stating: dBase: property name - Timestep - not found. >From the user's guide I cannot find a problem with the way I have used it. Can anyone help? Thanks - Lia - From latife at astro.rug.nl Thu Jul 24 05:18:08 2008 From: latife at astro.rug.nl (M.A. Latife) Date: Thu, 24 Jul 2008 12:18:08 +0200 Subject: [FLASH-USERS] problem in initializing more than 2GB pr CPU Message-ID: <488856E0.7020801@astro.rug.nl> Hi Dear all, I am using super computer with following specifications. it has 128 Gb internal memory and 16 processors. it is the same pc about which Seyit mentioned compiling problems in previous emails. Ubuntu Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux ifort intel compiler version 10.1.012 hdf5 1.6.7 (serial) mpich2 Version: 1.0.6 gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I $(MPI_PATH)/include or (only -ipo difference, result seemingly doesn't change) FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include Now we have an other problem that when we try to initiate 2000 blocks which in my case has more than 2 GB per processor it starts giving this error and make is not complete. we can't understand why it is not making when we have more than 2 GB per processor. it works fine for less than less than 2 GBper processor . Is it compiler problem or some thing else? dBaseTree.o: In function `dbasetree_mp_dbasetreesetreal3d_': dBaseTree.F90:(.text+0x22): relocation truncated to fit: R_X86_64_32S against symbol `tree_mp_bnd_box_' defined in COMMON section in tree.o dBaseTree.o: In function `dbasetree_mp_dbasetreesetint3d_': dBaseTree.F90:(.text+0xb4): relocation truncated to fit: R_X86_64_32S against symbol `tree_mp_child_' defined in COMMON section in tree.o dBaseTree.F90:(.text+0xea): relocation truncated to fit: R_X86_64_32S against symbol `tree_mp_neigh_' defined in COMMON section in tree.o dBaseTree.o: In function `dbasetree_mp_dbasetreesetreal2d_': dBaseTree.F90:(.text+0x176): relocation truncated to fit: R_X86_64_32S against symbol `tree_mp_bsize_' defined in COMMON section in tree.o dBaseTree.F90:(.text+0x19c): relocation truncated to fit: R_X86_64_32S against symbol `tree_mp_coord_' defined in COMMON section in tree.o dBaseTree.o: In function `dbasetree_mp_dbasetreesetint2d_': dBaseTree.F90:(.text+0x244): relocation truncated to fit: R_X86_64_32S against symbol `tree_mp_child_type_' defined in COMMON section in tree.o dBaseTree.F90:(.text+0x269): relocation truncated to fit: R_X86_64_32S against symbol `tree_mp_neigh_type_' defined in COMMON section in tree.o dBaseTree.F90:(.text+0x289): relocation truncated to fit: R_X86_64_32S against symbol `tree_mp_parent_' defined in COMMON section in tree.o dBaseTree.o: In function `dbasetree_mp_dbasetreesetreal1d_': dBaseTree.F90:(.text+0x2a0): relocation truncated to fit: R_X86_64_32S against symbol `tree_mp_work_block_' defined in COMMON section in tree.o dBaseTree.o: In function `dbasetree_mp_dbasetreesetint1d_': dBaseTree.F90:(.text+0x330): additional relocation overflows omitted from the output make[1]: *** [flash2] Error 1 make[1]: Leaving directory `/home/users/latife/FLASH2.5/object' make: *** [default] Error 2 Thanks in advance Best Regards Latif From cdaley at flash.uchicago.edu Thu Jul 24 06:59:22 2008 From: cdaley at flash.uchicago.edu (Chris Daley) Date: Thu, 24 Jul 2008 06:59:22 -0500 Subject: [FLASH-USERS] problem in initializing more than 2GB pr CPU In-Reply-To: <488856E0.7020801@astro.rug.nl> References: <488856E0.7020801@astro.rug.nl> Message-ID: <48886E9A.8020902@flash.uchicago.edu> M.A. Latife wrote: > > Hi > Dear all, > I am using super computer with following specifications. it has 128 Gb > internal memory and 16 processors. > it is the same pc about which Seyit mentioned compiling problems in > previous emails. > Ubuntu > Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux > ifort intel compiler version 10.1.012 > hdf5 1.6.7 (serial) > mpich2 Version: 1.0.6 > gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) > FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I > $(MPI_PATH)/include > or (only -ipo difference, result seemingly doesn't change) > FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include > > > Now we have an other problem that when we try to initiate 2000 blocks > which in my case has more than 2 GB per processor it starts giving > this error and make is not complete. we can't understand why it is not > making when we have more than 2 GB per processor. it works fine for > less than less than 2 GBper processor . Is it compiler problem or some > thing else? > > > > dBaseTree.o: In function `dbasetree_mp_dbasetreesetreal3d_': > dBaseTree.F90:(.text+0x22): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_bnd_box_' defined in COMMON section in tree.o > dBaseTree.o: In function `dbasetree_mp_dbasetreesetint3d_': > dBaseTree.F90:(.text+0xb4): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_child_' defined in COMMON section in tree.o > dBaseTree.F90:(.text+0xea): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_neigh_' defined in COMMON section in tree.o > dBaseTree.o: In function `dbasetree_mp_dbasetreesetreal2d_': > dBaseTree.F90:(.text+0x176): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_bsize_' defined in COMMON section in tree.o > dBaseTree.F90:(.text+0x19c): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_coord_' defined in COMMON section in tree.o > dBaseTree.o: In function `dbasetree_mp_dbasetreesetint2d_': > dBaseTree.F90:(.text+0x244): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_child_type_' defined in COMMON section in tree.o > dBaseTree.F90:(.text+0x269): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_neigh_type_' defined in COMMON section in tree.o > dBaseTree.F90:(.text+0x289): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_parent_' defined in COMMON section in tree.o > dBaseTree.o: In function `dbasetree_mp_dbasetreesetreal1d_': > dBaseTree.F90:(.text+0x2a0): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_work_block_' defined in COMMON section in tree.o > dBaseTree.o: In function `dbasetree_mp_dbasetreesetint1d_': > dBaseTree.F90:(.text+0x330): additional relocation overflows omitted > from the output > make[1]: *** [flash2] Error 1 > make[1]: Leaving directory `/home/users/latife/FLASH2.5/object' > make: *** [default] Error 2 > Thanks in advance > Best Regards > Latif > > > Hi Latif, This is not a compiler problem, it is simply because you have crossed the 2GB code + data limit. If you want to use more than 2GB, you can try passing the -mcmodel option to ifort during code compilation, i.e. -mcmodel=medium or -mcmodel=large. Please refer to the ifort man page for details. Regards, Chris From tomek at scs.fsu.edu Thu Jul 24 07:15:42 2008 From: tomek at scs.fsu.edu (Tomasz Plewa) Date: Thu, 24 Jul 2008 08:15:42 -0400 Subject: [FLASH-USERS] [Fwd: Re: problem in initializing more than 2GB pr CPU] Message-ID: <4888726E.9050000@scs.fsu.edu> Latif - I am inclined to follow http://flash.uchicago.edu/pipermail/flash-users/2005-November/002057.html In case of the Intel compiler, check -mcmodel=medium or -mcmodel=large. This opion needs to be used consistently among all application components, I believe. By the way, I am not sure there exist systems capable of evolving more than 2 GB of application memory for real problems. At that memory use level, execution per time step will be prohibitively slow. That is, unless you allocate additional memory only for occasional use, which is not the case here, your application will crawl to completion. Tomek -- M.A. Latife wrote: > > Hi > Dear all, > I am using super computer with following specifications. it has 128 Gb > internal memory and 16 processors. > it is the same pc about which Seyit mentioned compiling problems in > previous emails. > Ubuntu > Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux > ifort intel compiler version 10.1.012 > hdf5 1.6.7 (serial) > mpich2 Version: 1.0.6 > gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) > FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I > $(MPI_PATH)/include > or (only -ipo difference, result seemingly doesn't change) > FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include > > > Now we have an other problem that when we try to initiate 2000 blocks > which in my case has more than 2 GB per processor it starts giving this > error and make is not complete. we can't understand why it is not making > when we have more than 2 GB per processor. it works fine for less than > less than 2 GBper processor . Is it compiler problem or some thing else? > > > > dBaseTree.o: In function `dbasetree_mp_dbasetreesetreal3d_': > dBaseTree.F90:(.text+0x22): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_bnd_box_' defined in COMMON section in tree.o > dBaseTree.o: In function `dbasetree_mp_dbasetreesetint3d_': > dBaseTree.F90:(.text+0xb4): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_child_' defined in COMMON section in tree.o > dBaseTree.F90:(.text+0xea): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_neigh_' defined in COMMON section in tree.o > dBaseTree.o: In function `dbasetree_mp_dbasetreesetreal2d_': > dBaseTree.F90:(.text+0x176): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_bsize_' defined in COMMON section in tree.o > dBaseTree.F90:(.text+0x19c): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_coord_' defined in COMMON section in tree.o > dBaseTree.o: In function `dbasetree_mp_dbasetreesetint2d_': > dBaseTree.F90:(.text+0x244): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_child_type_' defined in COMMON section in tree.o > dBaseTree.F90:(.text+0x269): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_neigh_type_' defined in COMMON section in tree.o > dBaseTree.F90:(.text+0x289): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_parent_' defined in COMMON section in tree.o > dBaseTree.o: In function `dbasetree_mp_dbasetreesetreal1d_': > dBaseTree.F90:(.text+0x2a0): relocation truncated to fit: R_X86_64_32S > against symbol `tree_mp_work_block_' defined in COMMON section in tree.o > dBaseTree.o: In function `dbasetree_mp_dbasetreesetint1d_': > dBaseTree.F90:(.text+0x330): additional relocation overflows omitted > from the output > make[1]: *** [flash2] Error 1 > make[1]: Leaving directory `/home/users/latife/FLASH2.5/object' > make: *** [default] Error 2 > Thanks in advance > Best Regards > Latif > > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: tomek.vcf Type: text/x-vcard Size: 296 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080724/0bf565a8/attachment.vcf From tomek at scs.fsu.edu Thu Jul 24 08:32:27 2008 From: tomek at scs.fsu.edu (Tomasz Plewa) Date: Thu, 24 Jul 2008 09:32:27 -0400 Subject: [FLASH-USERS] problem in initializing more than 2GB pr CPU In-Reply-To: <48887F5F.6090209@astro.rug.nl> References: <488856E0.7020801@astro.rug.nl> <48886E9A.8020902@flash.uchicago.edu> <48887617.7070209@astro.rug.nl> <48887970.1050908@flash.uchicago.edu> <48887F5F.6090209@astro.rug.nl> Message-ID: <4888846B.70604@scs.fsu.edu> Latif - It may not solve your problem, but you may still want to estimate whether you are able to complete your simulation in a sensible time frame. To put things into perspective, let's assume that you eventually produced a 2+ GB per processor code. Let's say, 800 blocks 16^3 zones each. These days typical execution times for FLASH are slightly less than 1 16^3 block per processor per second per cycle (double time step). If you use all blocks in your model, and assuming perfect scaling, you need 800 seconds per time step. You can advance your model at a rate of about 100 cycles per day. A typical 3-D model costs perhaps several 1,000 cycles. So in your case this translates into several weeks of uninterrupted computing on your whole system per a single model. Tomek -- -------------- next part -------------- A non-text attachment was scrubbed... Name: tomek.vcf Type: text/x-vcard Size: 296 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080724/fa787448/attachment.vcf From tomek at scs.fsu.edu Thu Jul 24 09:00:08 2008 From: tomek at scs.fsu.edu (Tomasz Plewa) Date: Thu, 24 Jul 2008 10:00:08 -0400 Subject: [FLASH-USERS] problem in initializing more than 2GB pr CPU In-Reply-To: <48887F5F.6090209@astro.rug.nl> References: <488856E0.7020801@astro.rug.nl> <48886E9A.8020902@flash.uchicago.edu> <48887617.7070209@astro.rug.nl> <48887970.1050908@flash.uchicago.edu> <48887F5F.6090209@astro.rug.nl> Message-ID: <48888AE8.1070204@scs.fsu.edu> Latif - I have managed to build an executable for sedov 3d with 16^3 blocks with intel 10.1, -mcmodel=medium and -shared-intel, following Chris' suggestion. I did not include an io module as I do not have hdf5 build for a non-default mcmodel. For 2,000 blocks the executable is about 4.8 GB in size. Even for 1,000 blocks it is still 2.4 GB. The memory demand would increase still more if hdf5 is added. I doubt you can efficiently use any of those execs on your system. I expect lots of swapping action. Tomek -- -------------- next part -------------- A non-text attachment was scrubbed... Name: tomek.vcf Type: text/x-vcard Size: 296 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080724/dcad5ba4/attachment.vcf From seyit at astro.rug.nl Tue Jul 29 12:52:33 2008 From: seyit at astro.rug.nl (Seyit Hocuk) Date: Tue, 29 Jul 2008 19:52:33 +0200 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <4880DC19.7070205@scs.fsu.edu> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> Message-ID: <488F58E1.9050306@astro.rug.nl> Hi, It has been some time now, but to continue the discussion, I have tested Jeans standard problem and Sedov standard problem including changing the refinement levels. The result is exactly same for all three different computers. Unfortunately my own setup shows differences and it seems the end result changes by the coice of computer/environment/compiler whatever. My own setup has a lot more physics than the simpe test problems. One thing I am thinking about is that I adjusted the refinement criteria to use a reference density instead of looking at the second order derivative. I don't know if this could be the problem, but dt_hydro should not depend on the refinement criteria so I guess there is more to it. Unfortunately it is not easy to change compilers and some other stuff (like cpu/kernel etc.) on the different computers because I have limited rights on those machines. I don't want to find out I have to choose one machine and stick with it. How do I know which machine is the best. Anyway, I am going to try at least to get the libraries the same (hdf5 / mpi) for all machines and see if that helps. Btw the latest hdf5 (1.8.1) doesn't seem to work with Flash2.5. I will also play with compiler options. Try to make them exactly the same and avoid aggressive optimizations. I hope this is where the solution lies. Thanks, Seyit Tomasz Plewa wrote: > Although I prefer a low cost approach to debugging which avoids > dealing with setup/system/compiler information at first and attempts > to eliminate 99% of the source code, here are quick comments on your > systems. > > (1) pgi 6.0 is a quite antique compiler. It's so old that now I do not > even remember what problems we may have had with it. I suspect quite a > few and none of them might be relevant... > > (2) As you might noticed, intel has a newer compiler version. Your > CentOS5 system needs upgrades on both mpich2 and compiler sides. > > (3) Your Ubuntu system seems relatively up-to-date although HDF5 1.8.x > is out there and seems working fine. > > (4) Compilers are known to do strange things to perfect codes! So if > you run into a trouble on a new system, one of the first things to try > is to lower compiler optimization level. Avoiding -fast or -ip or > -inline is usually a good idea. Simple -O0 may suffice for a couple of > steps. As a bonus, the code compiles fast. Then the trick is to get it > crashing quickly... > > Again, I suggest to simplify your application, isolate the problem, > fix it, then find the best compiler, mpich, options, etc., to run your > production code. > > Enjoy your weekend! > > Tomek > -- > Seyit Hocuk wrote: >> Hi Nathan, >> >> Thats a good Idea, I'll start simple and test those first thing >> monday. It's too late for me now. >> >> To answer your questions: >> >> The problem I am running is my own home made, actually it's a >> modified version of Jeans. Basically including heat, cool, eos gamma, >> compositions, adjusted refinement criteria similar to one of the >> other problems (with delta_ref and delta_deref) and a scala of other >> small things. >> >> 1) >> Centos5 >> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 >> i686 i686 i386 GNU/Linux >> ifort intel compiler version 8.0 >> hdf5 1.6.6 (serial) >> mpich2 Version: 1.0.3 >> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include >> >> 2) >> Debian (??) >> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC >> 2005 x86_64 GNU/Linux >> pgf90 6.0-5 32-bit target on x86-64 Linux >> hdf5 1.4.5 (serial) >> mpich2 version 1.0.5 >> gcc version 3.3.5 (Debian 1:3.3.5-13) >> FFLAGS_OPT = -c -fast -r8 -i4 >> >> 3) >> Ubuntu >> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux >> ifort intel compiler version 10.1.012 >> hdf5 1.6.7 (serial) >> mpich2 Version: 1.0.6 >> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >> $(MPI_PATH)/include >> or (only -ipo difference, result seemingly doesn't change) >> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >> >> >> >> >> Nathan Hearn wrote: >>> Hi Seyit, >>> >>> Are you seeing these differences in the standard test problems >>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific to the >>> problem that you are running. (Really, this follows from Tomek's >>> advice to look at the simplest configurations where the differences >>> are seen.) >>> >>> Also, it would be good to include some more information in your >>> messages, for instance: the problem that you are running, the specific >>> architectures and compilers, the optimization flags, and the library >>> versions for HDF5 and MPI that you are using. Details like these can >>> be really helpful in evaluating and fixing the issues you are seeing. >>> >>> >>> - Nathan >>> >>> >>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk >>> wrote: >>> >>>> Oh man, >>>> >>>> I am using 3 different computers (with different architectures, >>>> compilers, >>>> optimizations, environments). All three start differently >>>> immediately from >>>> the beginning (difference in dt_hydro and refinement). >>>> Should I worry very very very much? >>>> I'm now doing a complete simulation and am going to check if my end >>>> results >>>> also change. >>>> >>>> I'm trembling by the thought..... >>>> >> >> > From tplewa at fsu.edu Tue Jul 29 13:12:02 2008 From: tplewa at fsu.edu (Tomasz Plewa) Date: Tue, 29 Jul 2008 14:12:02 -0400 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <488F58E1.9050306@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> Message-ID: <488F5D72.1040305@fsu.edu> Seyit - Avoiding aggressive optimization is certainly good idea when it comes to debugging implementation. Alas, given you experience problems with 3 different compilers it is rather unlilkely that compilers are at fault. If your grid does not resolve structures correctly (persumably due to lax refinement criteria), the code may quite easily produce an unphysical state. But that should not happen during the initial grid generation. Any luck with a single block/single level runs? Tomek -- Seyit Hocuk wrote: > Hi, > > It has been some time now, but to continue the discussion, I have > tested Jeans standard problem and Sedov standard problem including > changing the refinement levels. The result is exactly same for all > three different computers. Unfortunately my own setup shows > differences and it seems the end result changes by the coice of > computer/environment/compiler whatever. My own setup has a lot more > physics than the simpe test problems. > > One thing I am thinking about is that I adjusted the refinement > criteria to use a reference density instead of looking at the second > order derivative. I don't know if this could be the problem, but > dt_hydro should not depend on the refinement criteria so I guess there > is more to it. > > Unfortunately it is not easy to change compilers and some other stuff > (like cpu/kernel etc.) on the different computers because I have > limited rights on those machines. I don't want to find out I have to > choose one machine and stick with it. How do I know which machine is > the best. Anyway, I am going to try at least to get the libraries the > same (hdf5 / mpi) for all machines and see if that helps. Btw the > latest hdf5 (1.8.1) doesn't seem to work with Flash2.5. > > I will also play with compiler options. Try to make them exactly the > same and avoid aggressive optimizations. I hope this is where the > solution lies. > > Thanks, > Seyit > > > > > Tomasz Plewa wrote: >> Although I prefer a low cost approach to debugging which avoids >> dealing with setup/system/compiler information at first and attempts >> to eliminate 99% of the source code, here are quick comments on your >> systems. >> >> (1) pgi 6.0 is a quite antique compiler. It's so old that now I do >> not even remember what problems we may have had with it. I suspect >> quite a few and none of them might be relevant... >> >> (2) As you might noticed, intel has a newer compiler version. Your >> CentOS5 system needs upgrades on both mpich2 and compiler sides. >> >> (3) Your Ubuntu system seems relatively up-to-date although HDF5 >> 1.8.x is out there and seems working fine. >> >> (4) Compilers are known to do strange things to perfect codes! So if >> you run into a trouble on a new system, one of the first things to >> try is to lower compiler optimization level. Avoiding -fast or -ip or >> -inline is usually a good idea. Simple -O0 may suffice for a couple >> of steps. As a bonus, the code compiles fast. Then the trick is to >> get it crashing quickly... >> >> Again, I suggest to simplify your application, isolate the problem, >> fix it, then find the best compiler, mpich, options, etc., to run >> your production code. >> >> Enjoy your weekend! >> >> Tomek >> -- >> Seyit Hocuk wrote: >>> Hi Nathan, >>> >>> Thats a good Idea, I'll start simple and test those first thing >>> monday. It's too late for me now. >>> >>> To answer your questions: >>> >>> The problem I am running is my own home made, actually it's a >>> modified version of Jeans. Basically including heat, cool, eos >>> gamma, compositions, adjusted refinement criteria similar to one of >>> the other problems (with delta_ref and delta_deref) and a scala of >>> other small things. >>> >>> 1) >>> Centos5 >>> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 >>> i686 i686 i386 GNU/Linux >>> ifort intel compiler version 8.0 >>> hdf5 1.6.6 (serial) >>> mpich2 Version: 1.0.3 >>> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include >>> >>> 2) >>> Debian (??) >>> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC >>> 2005 x86_64 GNU/Linux >>> pgf90 6.0-5 32-bit target on x86-64 Linux >>> hdf5 1.4.5 (serial) >>> mpich2 version 1.0.5 >>> gcc version 3.3.5 (Debian 1:3.3.5-13) >>> FFLAGS_OPT = -c -fast -r8 -i4 >>> >>> 3) >>> Ubuntu >>> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 >>> GNU/Linux >>> ifort intel compiler version 10.1.012 >>> hdf5 1.6.7 (serial) >>> mpich2 Version: 1.0.6 >>> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >>> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >>> $(MPI_PATH)/include >>> or (only -ipo difference, result seemingly doesn't change) >>> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >>> >>> >>> >>> >>> Nathan Hearn wrote: >>>> Hi Seyit, >>>> >>>> Are you seeing these differences in the standard test problems >>>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific to the >>>> problem that you are running. (Really, this follows from Tomek's >>>> advice to look at the simplest configurations where the differences >>>> are seen.) >>>> >>>> Also, it would be good to include some more information in your >>>> messages, for instance: the problem that you are running, the specific >>>> architectures and compilers, the optimization flags, and the library >>>> versions for HDF5 and MPI that you are using. Details like these can >>>> be really helpful in evaluating and fixing the issues you are seeing. >>>> >>>> >>>> - Nathan >>>> >>>> >>>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk >>>> wrote: >>>> >>>>> Oh man, >>>>> >>>>> I am using 3 different computers (with different architectures, >>>>> compilers, >>>>> optimizations, environments). All three start differently >>>>> immediately from >>>>> the beginning (difference in dt_hydro and refinement). >>>>> Should I worry very very very much? >>>>> I'm now doing a complete simulation and am going to check if my >>>>> end results >>>>> also change. >>>>> >>>>> I'm trembling by the thought..... >>>>> >>> >>> >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: tplewa.vcf Type: text/x-vcard Size: 297 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080729/31389082/attachment.vcf From richp at flash.uchicago.edu Tue Jul 29 15:10:59 2008 From: richp at flash.uchicago.edu (Paul M. Rich) Date: Tue, 29 Jul 2008 15:10:59 -0500 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <488F58E1.9050306@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> Message-ID: <488F7953.4010802@flash.uchicago.edu> Seyit, I can clear up the HDF5 1.8 issue, I think. I've experimented with HDF5 1.8.1 in FLASH3 and that will only work if you have built the v1.8.x implementation with support for the 1.6 api. I believe that FLASH2.5's will work in a similar fashion. Is the version of HDF5 1.8.1 you are using built with the 1.6 api bindings? Paul Rich Seyit Hocuk wrote: > Hi, > > It has been some time now, but to continue the discussion, I have > tested Jeans standard problem and Sedov standard problem including > changing the refinement levels. The result is exactly same for all > three different computers. Unfortunately my own setup shows > differences and it seems the end result changes by the coice of > computer/environment/compiler whatever. My own setup has a lot more > physics than the simpe test problems. > > One thing I am thinking about is that I adjusted the refinement > criteria to use a reference density instead of looking at the second > order derivative. I don't know if this could be the problem, but > dt_hydro should not depend on the refinement criteria so I guess there > is more to it. > > Unfortunately it is not easy to change compilers and some other stuff > (like cpu/kernel etc.) on the different computers because I have > limited rights on those machines. I don't want to find out I have to > choose one machine and stick with it. How do I know which machine is > the best. Anyway, I am going to try at least to get the libraries the > same (hdf5 / mpi) for all machines and see if that helps. Btw the > latest hdf5 (1.8.1) doesn't seem to work with Flash2.5. > > I will also play with compiler options. Try to make them exactly the > same and avoid aggressive optimizations. I hope this is where the > solution lies. > > Thanks, > Seyit > > > > > Tomasz Plewa wrote: >> Although I prefer a low cost approach to debugging which avoids >> dealing with setup/system/compiler information at first and attempts >> to eliminate 99% of the source code, here are quick comments on your >> systems. >> >> (1) pgi 6.0 is a quite antique compiler. It's so old that now I do >> not even remember what problems we may have had with it. I suspect >> quite a few and none of them might be relevant... >> >> (2) As you might noticed, intel has a newer compiler version. Your >> CentOS5 system needs upgrades on both mpich2 and compiler sides. >> >> (3) Your Ubuntu system seems relatively up-to-date although HDF5 >> 1.8.x is out there and seems working fine. >> >> (4) Compilers are known to do strange things to perfect codes! So if >> you run into a trouble on a new system, one of the first things to >> try is to lower compiler optimization level. Avoiding -fast or -ip or >> -inline is usually a good idea. Simple -O0 may suffice for a couple >> of steps. As a bonus, the code compiles fast. Then the trick is to >> get it crashing quickly... >> >> Again, I suggest to simplify your application, isolate the problem, >> fix it, then find the best compiler, mpich, options, etc., to run >> your production code. >> >> Enjoy your weekend! >> >> Tomek >> -- >> Seyit Hocuk wrote: >>> Hi Nathan, >>> >>> Thats a good Idea, I'll start simple and test those first thing >>> monday. It's too late for me now. >>> >>> To answer your questions: >>> >>> The problem I am running is my own home made, actually it's a >>> modified version of Jeans. Basically including heat, cool, eos >>> gamma, compositions, adjusted refinement criteria similar to one of >>> the other problems (with delta_ref and delta_deref) and a scala of >>> other small things. >>> >>> 1) >>> Centos5 >>> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 >>> i686 i686 i386 GNU/Linux >>> ifort intel compiler version 8.0 >>> hdf5 1.6.6 (serial) >>> mpich2 Version: 1.0.3 >>> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include >>> >>> 2) >>> Debian (??) >>> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC >>> 2005 x86_64 GNU/Linux >>> pgf90 6.0-5 32-bit target on x86-64 Linux >>> hdf5 1.4.5 (serial) >>> mpich2 version 1.0.5 >>> gcc version 3.3.5 (Debian 1:3.3.5-13) >>> FFLAGS_OPT = -c -fast -r8 -i4 >>> >>> 3) >>> Ubuntu >>> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 >>> GNU/Linux >>> ifort intel compiler version 10.1.012 >>> hdf5 1.6.7 (serial) >>> mpich2 Version: 1.0.6 >>> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >>> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >>> $(MPI_PATH)/include >>> or (only -ipo difference, result seemingly doesn't change) >>> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >>> >>> >>> >>> >>> Nathan Hearn wrote: >>>> Hi Seyit, >>>> >>>> Are you seeing these differences in the standard test problems >>>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific to the >>>> problem that you are running. (Really, this follows from Tomek's >>>> advice to look at the simplest configurations where the differences >>>> are seen.) >>>> >>>> Also, it would be good to include some more information in your >>>> messages, for instance: the problem that you are running, the specific >>>> architectures and compilers, the optimization flags, and the library >>>> versions for HDF5 and MPI that you are using. Details like these can >>>> be really helpful in evaluating and fixing the issues you are seeing. >>>> >>>> >>>> - Nathan >>>> >>>> >>>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk >>>> wrote: >>>> >>>>> Oh man, >>>>> >>>>> I am using 3 different computers (with different architectures, >>>>> compilers, >>>>> optimizations, environments). All three start differently >>>>> immediately from >>>>> the beginning (difference in dt_hydro and refinement). >>>>> Should I worry very very very much? >>>>> I'm now doing a complete simulation and am going to check if my >>>>> end results >>>>> also change. >>>>> >>>>> I'm trembling by the thought..... >>>>> >>> >>> >> From nhearn at uchicago.edu Tue Jul 29 15:24:44 2008 From: nhearn at uchicago.edu (Nathan Hearn) Date: Tue, 29 Jul 2008 15:24:44 -0500 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <488F7953.4010802@flash.uchicago.edu> References: <48441DC4.9050701@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> <488F7953.4010802@flash.uchicago.edu> Message-ID: <2467fdc0807291324t5368140ax904909774b1bf582@mail.gmail.com> For everyone's information, the HDF Group has a nice page on the compatibility macros that are meant to bridge between the 1.6 and 1.8 versions, as well as a discussion of the associated configuration options for HDF5: http://www.hdfgroup.org/HDF5/doc/RM/APICompatMacros.html - Nathan On Tue, Jul 29, 2008 at 3:10 PM, Paul M. Rich wrote: > Seyit, > > I can clear up the HDF5 1.8 issue, I think. I've experimented with HDF5 > 1.8.1 in FLASH3 and that will only work if you have built the v1.8.x > implementation with support for the 1.6 api. I believe that FLASH2.5's will > work in a similar fashion. Is the version of HDF5 1.8.1 you are using built > with the 1.6 api bindings? > > Paul Rich > > Seyit Hocuk wrote: >> >> Hi, >> >> It has been some time now, but to continue the discussion, I have tested >> Jeans standard problem and Sedov standard problem including changing the >> refinement levels. The result is exactly same for all three different >> computers. Unfortunately my own setup shows differences and it seems the end >> result changes by the coice of computer/environment/compiler whatever. My >> own setup has a lot more physics than the simpe test problems. >> >> One thing I am thinking about is that I adjusted the refinement criteria >> to use a reference density instead of looking at the second order >> derivative. I don't know if this could be the problem, but dt_hydro should >> not depend on the refinement criteria so I guess there is more to it. >> >> Unfortunately it is not easy to change compilers and some other stuff >> (like cpu/kernel etc.) on the different computers because I have limited >> rights on those machines. I don't want to find out I have to choose one >> machine and stick with it. How do I know which machine is the best. Anyway, >> I am going to try at least to get the libraries the same (hdf5 / mpi) for >> all machines and see if that helps. Btw the latest hdf5 (1.8.1) doesn't seem >> to work with Flash2.5. >> >> I will also play with compiler options. Try to make them exactly the same >> and avoid aggressive optimizations. I hope this is where the solution lies. >> >> Thanks, >> Seyit >> >> >> >> >> Tomasz Plewa wrote: >>> >>> Although I prefer a low cost approach to debugging which avoids dealing >>> with setup/system/compiler information at first and attempts to eliminate >>> 99% of the source code, here are quick comments on your systems. >>> >>> (1) pgi 6.0 is a quite antique compiler. It's so old that now I do not >>> even remember what problems we may have had with it. I suspect quite a few >>> and none of them might be relevant... >>> >>> (2) As you might noticed, intel has a newer compiler version. Your >>> CentOS5 system needs upgrades on both mpich2 and compiler sides. >>> >>> (3) Your Ubuntu system seems relatively up-to-date although HDF5 1.8.x is >>> out there and seems working fine. >>> >>> (4) Compilers are known to do strange things to perfect codes! So if you >>> run into a trouble on a new system, one of the first things to try is to >>> lower compiler optimization level. Avoiding -fast or -ip or -inline is >>> usually a good idea. Simple -O0 may suffice for a couple of steps. As a >>> bonus, the code compiles fast. Then the trick is to get it crashing >>> quickly... >>> >>> Again, I suggest to simplify your application, isolate the problem, fix >>> it, then find the best compiler, mpich, options, etc., to run your >>> production code. >>> >>> Enjoy your weekend! >>> >>> Tomek >>> -- >>> Seyit Hocuk wrote: >>>> >>>> Hi Nathan, >>>> >>>> Thats a good Idea, I'll start simple and test those first thing monday. >>>> It's too late for me now. >>>> >>>> To answer your questions: >>>> >>>> The problem I am running is my own home made, actually it's a modified >>>> version of Jeans. Basically including heat, cool, eos gamma, compositions, >>>> adjusted refinement criteria similar to one of the other problems (with >>>> delta_ref and delta_deref) and a scala of other small things. >>>> >>>> 1) >>>> Centos5 >>>> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 i686 >>>> i686 i386 GNU/Linux >>>> ifort intel compiler version 8.0 >>>> hdf5 1.6.6 (serial) >>>> mpich2 Version: 1.0.3 >>>> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>>> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include >>>> >>>> 2) >>>> Debian (??) >>>> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC 2005 >>>> x86_64 GNU/Linux >>>> pgf90 6.0-5 32-bit target on x86-64 Linux >>>> hdf5 1.4.5 (serial) >>>> mpich2 version 1.0.5 >>>> gcc version 3.3.5 (Debian 1:3.3.5-13) >>>> FFLAGS_OPT = -c -fast -r8 -i4 >>>> >>>> 3) >>>> Ubuntu >>>> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux >>>> ifort intel compiler version 10.1.012 >>>> hdf5 1.6.7 (serial) >>>> mpich2 Version: 1.0.6 >>>> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >>>> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >>>> $(MPI_PATH)/include >>>> or (only -ipo difference, result seemingly doesn't change) >>>> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >>>> >>>> >>>> >>>> >>>> Nathan Hearn wrote: >>>>> >>>>> Hi Seyit, >>>>> >>>>> Are you seeing these differences in the standard test problems >>>>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific to the >>>>> problem that you are running. (Really, this follows from Tomek's >>>>> advice to look at the simplest configurations where the differences >>>>> are seen.) >>>>> >>>>> Also, it would be good to include some more information in your >>>>> messages, for instance: the problem that you are running, the specific >>>>> architectures and compilers, the optimization flags, and the library >>>>> versions for HDF5 and MPI that you are using. Details like these can >>>>> be really helpful in evaluating and fixing the issues you are seeing. >>>>> >>>>> >>>>> - Nathan >>>>> >>>>> >>>>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk >>>>> wrote: >>>>> >>>>>> >>>>>> Oh man, >>>>>> >>>>>> I am using 3 different computers (with different architectures, >>>>>> compilers, >>>>>> optimizations, environments). All three start differently immediately >>>>>> from >>>>>> the beginning (difference in dt_hydro and refinement). >>>>>> Should I worry very very very much? >>>>>> I'm now doing a complete simulation and am going to check if my end >>>>>> results >>>>>> also change. >>>>>> >>>>>> I'm trembling by the thought..... >>>>>> >>>> >>>> >>> > > -- Nathan C. Hearn nhearn at uchicago.edu ASC Flash Center Computational Physics Group University of Chicago From seyit at astro.rug.nl Wed Jul 30 04:28:09 2008 From: seyit at astro.rug.nl (Seyit Hocuk) Date: Wed, 30 Jul 2008 11:28:09 +0200 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <488F5D72.1040305@fsu.edu> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> <488F5D72.1040305@fsu.edu> Message-ID: <48903429.5090007@astro.rug.nl> Hi Tomek good morning, Let me answer your first question. * /Any luck with a single block/single level runs? / Nope. Even single level (lref_min = lref_max) runs differ in dt_hydro. That means I don't use refinement at all. I guess it is also affecting the end results. I have to do a complete simulation again to be sure. But the initial files show small differences already. Here the initial results. Note 1: the differences in precision in k, kJ, c0 etc... Note 2: pc 1 and 3 are not that much differen, but grow more different later on. *pc 1* flash: initializing for jeans problem. kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 k = 1.756203682760181E-019 kJ = 6.508776532382562E-019 c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 velamp = 64881.6252564676 perturbation is unstable with growth time 8776137041619.21 min_blocks 5 max_blocks 5 tot_blocks 5 min_blocks 21 max_blocks 21 tot_blocks 21 min_blocks 85 max_blocks 85 tot_blocks 85 min_blocks 341 max_blocks 341 tot_blocks 341 INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will send 520 blocks per message. Warning! The HDF5 header files included by this application do not match the version used by the HDF5 library to which this application is linked. Data corruption or segmentation faults may occur if the application is allowed to continue. You can, at your own risk, disable this check by setting the environment variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. Setting it to 2 will suppress the warning totally. Headers are 1.6.6, library is 1.6.7 Bye... forrtl: info: Fortran error message number is 76. forrtl: warning: Could not open message catalog: ifcore_msg.cat. forrtl: info: Check environment variable NLSPATH and protection of /usr/lib/ifcore_msg.cat. Abort [1] + Done emacs flash.par [seyit at celsius objectT3]$ setenv HDF5_DISABLE_VERSION_CHECK 1 [seyit at celsius objectT3]$ ./flash2 flash: initializing for jeans problem. kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 k = 1.756203682760181E-019 kJ = 6.508776532382562E-019 c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 velamp = 64881.6252564676 perturbation is unstable with growth time 8776137041619.21 min_blocks 5 max_blocks 5 tot_blocks 5 min_blocks 21 max_blocks 21 tot_blocks 21 min_blocks 85 max_blocks 85 tot_blocks 85 min_blocks 341 max_blocks 341 tot_blocks 341 INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will send 520 blocks per message. [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 Progress: | *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) *** number of variables found for plotfile storage = 3 dens pres temp *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** n t dt | dt_hydro 1 2.0000E+05 2.0000E+05 | 4.240E+11 2 6.0000E+05 4.0000E+05 | 4.240E+11 3 1.4000E+06 8.0000E+05 | 4.240E+11 4 3.0000E+06 1.6000E+06 | 4.240E+11 5 6.2000E+06 3.2000E+06 | 4.240E+11 6 1.2600E+07 6.4000E+06 | 4.240E+11 7 2.5400E+07 1.2800E+07 | 4.240E+11 8 5.1000E+07 2.5600E+07 | 4.240E+11 9 1.0220E+08 5.1200E+07 | 4.240E+11 10 2.0460E+08 1.0240E+08 | 4.240E+11 11 4.0940E+08 2.0480E+08 | 4.241E+11 12 8.1900E+08 4.0960E+08 | 4.241E+11 13 1.6382E+09 8.1920E+08 | 4.241E+11 14 3.2766E+09 1.6384E+09 | 4.242E+11 15 6.5534E+09 3.0000E+09 | 4.243E+11 16 1.2553E+10 3.0000E+09 | 4.247E+11 17 1.8553E+10 3.0000E+09 | 4.250E+11 18 2.4553E+10 3.0000E+09 | 4.255E+11 19 3.0553E+10 3.0000E+09 | 4.259E+11 20 3.6553E+10 3.0000E+09 | 4.266E+11 21 4.2553E+10 3.0000E+09 | 4.273E+11 22 4.8553E+10 3.0000E+09 | 4.281E+11 23 5.4553E+10 3.0000E+09 | 4.288E+11 24 6.0553E+10 3.0000E+09 | 4.296E+11 25 6.6553E+10 3.0000E+09 | 4.304E+11 26 7.2553E+10 3.0000E+09 | 4.313E+11 27 7.8553E+10 3.0000E+09 | 4.321E+11 28 8.4553E+10 3.0000E+09 | 4.330E+11 29 9.0553E+10 3.0000E+09 | 4.339E+11 30 9.6553E+10 3.0000E+09 | 4.348E+11 31 1.0255E+11 3.0000E+09 | 4.357E+11 32 1.0855E+11 3.0000E+09 | 4.366E+11 33 1.1455E+11 3.0000E+09 | 4.301E+11 34 1.2055E+11 3.0000E+09 | 3.803E+11 35 1.2655E+11 3.0000E+09 | 3.379E+11 36 1.3255E+11 3.0000E+09 | 3.057E+11 37 1.3855E+11 3.0000E+09 | 2.868E+11 38 1.4455E+11 3.0000E+09 | 2.876E+11 39 1.5055E+11 3.0000E+09 | 2.880E+11 40 1.5655E+11 3.0000E+09 | 2.903E+11 41 1.6255E+11 3.0000E+09 | 2.936E+11 42 1.6855E+11 3.0000E+09 | 2.978E+11 43 1.7455E+11 3.0000E+09 | 4.475E+11 44 1.8055E+11 3.0000E+09 | 4.486E+11 45 1.8655E+11 3.0000E+09 | 4.496E+11 46 1.9255E+11 3.0000E+09 | 4.507E+11 47 1.9855E+11 3.0000E+09 | 4.517E+11 48 2.0455E+11 3.0000E+09 | 4.528E+11 49 2.1055E+11 3.0000E+09 | 4.538E+11 50 2.1655E+11 3.0000E+09 | 4.549E+11 *pc 2* flash: initializing for jeans problem. kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 k = 1.7562036827601810E-019 kJ = 6.5087765323825604E-019 c0 = 181807.2941759182 omega^2 = -1.2983542608611234E-026 velamp = 64881.62525646757 perturbation is unstable with growth time 8776137041619.213 min_blocks 5 max_blocks 5 tot_blocks 5 min_blocks 21 max_blocks 21 tot_blocks 21 min_blocks 85 max_blocks 85 tot_blocks 85 min_blocks 341 max_blocks 341 tot_blocks 341 INITIAL TIMESTEP = 100000.0000000000 [CHECKPOINT_WR] NOTE: will send 520 blocks per message. [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 Progress: | *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) *** number of variables found for plotfile storage = 3 dens pres temp *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** n t dt | dt_hydro 1 2.0000E+05 2.0000E+05 | 4.406E+11 2 6.0000E+05 4.0000E+05 | 4.406E+11 3 1.4000E+06 8.0000E+05 | 4.406E+11 4 3.0000E+06 1.6000E+06 | 4.406E+11 5 6.2000E+06 3.2000E+06 | 4.406E+11 6 1.2600E+07 6.4000E+06 | 4.406E+11 7 2.5400E+07 1.2800E+07 | 4.406E+11 8 5.1000E+07 2.5600E+07 | 4.406E+11 9 1.0220E+08 5.1200E+07 | 4.406E+11 10 2.0460E+08 1.0240E+08 | 4.405E+11 11 4.0940E+08 2.0480E+08 | 4.404E+11 12 8.1900E+08 4.0960E+08 | 4.403E+11 13 1.6382E+09 8.1920E+08 | 4.399E+11 14 3.2766E+09 1.6384E+09 | 4.393E+11 15 6.5534E+09 3.0000E+09 | 4.381E+11 16 1.2553E+10 3.0000E+09 | 4.365E+11 17 1.8553E+10 3.0000E+09 | 4.353E+11 18 2.4553E+10 3.0000E+09 | 4.345E+11 19 3.0553E+10 3.0000E+09 | 4.340E+11 20 3.6553E+10 3.0000E+09 | 4.337E+11 21 4.2553E+10 3.0000E+09 | 4.337E+11 22 4.8553E+10 3.0000E+09 | 4.339E+11 23 5.4553E+10 3.0000E+09 | 4.343E+11 24 6.0553E+10 3.0000E+09 | 4.348E+11 25 6.6553E+10 3.0000E+09 | 4.355E+11 26 7.2553E+10 3.0000E+09 | 4.363E+11 27 7.8553E+10 3.0000E+09 | 4.371E+11 28 8.4553E+10 3.0000E+09 | 4.382E+11 29 9.0553E+10 3.0000E+09 | 4.393E+11 30 9.6553E+10 3.0000E+09 | 4.405E+11 31 1.0255E+11 3.0000E+09 | 4.418E+11 32 1.0855E+11 3.0000E+09 | 4.431E+11 33 1.1455E+11 3.0000E+09 | 4.447E+11 34 1.2055E+11 3.0000E+09 | 4.463E+11 35 1.2655E+11 3.0000E+09 | 4.479E+11 36 1.3255E+11 3.0000E+09 | 4.496E+11 37 1.3855E+11 3.0000E+09 | 4.513E+11 38 1.4455E+11 3.0000E+09 | 4.531E+11 39 1.5055E+11 3.0000E+09 | 4.550E+11 40 1.5655E+11 3.0000E+09 | 4.570E+11 41 1.6255E+11 3.0000E+09 | 4.590E+11 42 1.6855E+11 3.0000E+09 | 4.611E+11 43 1.7455E+11 3.0000E+09 | 4.632E+11 44 1.8055E+11 3.0000E+09 | 4.653E+11 45 1.8655E+11 3.0000E+09 | 4.675E+11 46 1.9255E+11 3.0000E+09 | 4.697E+11 47 1.9855E+11 3.0000E+09 | 4.719E+11 48 2.0455E+11 3.0000E+09 | 4.733E+11 49 2.1055E+11 3.0000E+09 | 4.734E+11 50 2.1655E+11 3.0000E+09 | 4.736E+11 *pc 3* flash: initializing for jeans problem. kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 k = 1.756203682760181E-019 kJ = 6.508776532382562E-019 c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 velamp = 64881.6252564676 perturbation is unstable with growth time 8776137041619.21 min_blocks 5 max_blocks 5 tot_blocks 5 min_blocks 21 max_blocks 21 tot_blocks 21 min_blocks 85 max_blocks 85 tot_blocks 85 min_blocks 341 max_blocks 341 tot_blocks 341 INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will send 520 blocks per message. [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 Progress: | *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) *** number of variables found for plotfile storage = 3 dens pres temp *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** n t dt | dt_hydro 1 2.0000E+05 2.0000E+05 | 4.240E+11 2 6.0000E+05 4.0000E+05 | 4.240E+11 3 1.4000E+06 8.0000E+05 | 4.240E+11 4 3.0000E+06 1.6000E+06 | 4.240E+11 5 6.2000E+06 3.2000E+06 | 4.240E+11 6 1.2600E+07 6.4000E+06 | 4.240E+11 7 2.5400E+07 1.2800E+07 | 4.240E+11 8 5.1000E+07 2.5600E+07 | 4.240E+11 9 1.0220E+08 5.1200E+07 | 4.240E+11 10 2.0460E+08 1.0240E+08 | 4.240E+11 11 4.0940E+08 2.0480E+08 | 4.241E+11 12 8.1900E+08 4.0960E+08 | 4.241E+11 13 1.6382E+09 8.1920E+08 | 4.241E+11 14 3.2766E+09 1.6384E+09 | 4.242E+11 15 6.5534E+09 3.0000E+09 | 4.244E+11 16 1.2553E+10 3.0000E+09 | 4.247E+11 17 1.8553E+10 3.0000E+09 | 4.251E+11 18 2.4553E+10 3.0000E+09 | 4.255E+11 19 3.0553E+10 3.0000E+09 | 4.259E+11 20 3.6553E+10 3.0000E+09 | 4.266E+11 21 4.2553E+10 3.0000E+09 | 4.273E+11 22 4.8553E+10 3.0000E+09 | 4.281E+11 23 5.4553E+10 3.0000E+09 | 4.288E+11 24 6.0553E+10 3.0000E+09 | 4.296E+11 25 6.6553E+10 3.0000E+09 | 4.304E+11 26 7.2553E+10 3.0000E+09 | 4.313E+11 27 7.8553E+10 3.0000E+09 | 4.321E+11 28 8.4553E+10 3.0000E+09 | 4.330E+11 29 9.0553E+10 3.0000E+09 | 4.338E+11 30 9.6553E+10 3.0000E+09 | 4.347E+11 31 1.0255E+11 3.0000E+09 | 4.356E+11 32 1.0855E+11 3.0000E+09 | 4.366E+11 33 1.1455E+11 3.0000E+09 | 4.375E+11 34 1.2055E+11 3.0000E+09 | 4.384E+11 35 1.2655E+11 3.0000E+09 | 4.394E+11 36 1.3255E+11 3.0000E+09 | 4.404E+11 37 1.3855E+11 3.0000E+09 | 4.413E+11 38 1.4455E+11 3.0000E+09 | 4.423E+11 39 1.5055E+11 3.0000E+09 | 4.433E+11 40 1.5655E+11 3.0000E+09 | 4.444E+11 41 1.6255E+11 3.0000E+09 | 4.454E+11 42 1.6855E+11 3.0000E+09 | 4.464E+11 43 1.7455E+11 3.0000E+09 | 4.474E+11 44 1.8055E+11 3.0000E+09 | 4.484E+11 45 1.8655E+11 3.0000E+09 | 4.495E+11 46 1.9255E+11 3.0000E+09 | 4.505E+11 47 1.9855E+11 3.0000E+09 | 4.516E+11 48 2.0455E+11 3.0000E+09 | 4.526E+11 49 2.1055E+11 3.0000E+09 | 4.537E+11 50 2.1655E+11 3.0000E+09 | 4.547E+11 *here again the 3 different pc's* 1) Centos5 Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 i686 i686 i386 GNU/Linux ifort intel compiler version 8.0 hdf5 1.6.6 (serial) *(updated recently to 1.6.7)* mpich2 Version: 1.0.3 gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include 2) Debian (??) Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC 2005 x86_64 GNU/Linux pgf90 6.0-5 32-bit target on x86-64 Linux hdf5 1.4.5 (serial) mpich2 version 1.0.5 gcc version 3.3.5 (Debian 1:3.3.5-13) FFLAGS_OPT = -c -fast -r8 -i4 3) Ubuntu Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux ifort intel compiler version 10.1.012 hdf5 1.6.7 (serial) mpich2 Version: 1.0.6 gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I $(MPI_PATH)/include or (only -ipo difference, result seemingly doesn't change) FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include Tomasz Plewa wrote: > Seyit - > > Avoiding aggressive optimization is certainly good idea when it comes > to debugging implementation. Alas, given you experience problems with > 3 different compilers it is rather unlilkely that compilers are at fault. > > If your grid does not resolve structures correctly (persumably due to > lax refinement criteria), the code may quite easily produce an > unphysical state. But that should not happen during the initial grid > generation. > > Any luck with a single block/single level runs? > > Tomek > -- > Seyit Hocuk wrote: >> Hi, >> >> It has been some time now, but to continue the discussion, I have >> tested Jeans standard problem and Sedov standard problem including >> changing the refinement levels. The result is exactly same for all >> three different computers. Unfortunately my own setup shows >> differences and it seems the end result changes by the coice of >> computer/environment/compiler whatever. My own setup has a lot more >> physics than the simpe test problems. >> >> One thing I am thinking about is that I adjusted the refinement >> criteria to use a reference density instead of looking at the second >> order derivative. I don't know if this could be the problem, but >> dt_hydro should not depend on the refinement criteria so I guess >> there is more to it. >> >> Unfortunately it is not easy to change compilers and some other stuff >> (like cpu/kernel etc.) on the different computers because I have >> limited rights on those machines. I don't want to find out I have to >> choose one machine and stick with it. How do I know which machine is >> the best. Anyway, I am going to try at least to get the libraries the >> same (hdf5 / mpi) for all machines and see if that helps. Btw the >> latest hdf5 (1.8.1) doesn't seem to work with Flash2.5. >> >> I will also play with compiler options. Try to make them exactly the >> same and avoid aggressive optimizations. I hope this is where the >> solution lies. >> >> Thanks, >> Seyit >> >> >> >> >> Tomasz Plewa wrote: >>> Although I prefer a low cost approach to debugging which avoids >>> dealing with setup/system/compiler information at first and attempts >>> to eliminate 99% of the source code, here are quick comments on >>> your systems. >>> >>> (1) pgi 6.0 is a quite antique compiler. It's so old that now I do >>> not even remember what problems we may have had with it. I suspect >>> quite a few and none of them might be relevant... >>> >>> (2) As you might noticed, intel has a newer compiler version. Your >>> CentOS5 system needs upgrades on both mpich2 and compiler sides. >>> >>> (3) Your Ubuntu system seems relatively up-to-date although HDF5 >>> 1.8.x is out there and seems working fine. >>> >>> (4) Compilers are known to do strange things to perfect codes! So if >>> you run into a trouble on a new system, one of the first things to >>> try is to lower compiler optimization level. Avoiding -fast or -ip >>> or -inline is usually a good idea. Simple -O0 may suffice for a >>> couple of steps. As a bonus, the code compiles fast. Then the trick >>> is to get it crashing quickly... >>> >>> Again, I suggest to simplify your application, isolate the problem, >>> fix it, then find the best compiler, mpich, options, etc., to run >>> your production code. >>> >>> Enjoy your weekend! >>> >>> Tomek >>> -- >>> Seyit Hocuk wrote: >>>> Hi Nathan, >>>> >>>> Thats a good Idea, I'll start simple and test those first thing >>>> monday. It's too late for me now. >>>> >>>> To answer your questions: >>>> >>>> The problem I am running is my own home made, actually it's a >>>> modified version of Jeans. Basically including heat, cool, eos >>>> gamma, compositions, adjusted refinement criteria similar to one of >>>> the other problems (with delta_ref and delta_deref) and a scala of >>>> other small things. >>>> >>>> 1) >>>> Centos5 >>>> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 >>>> i686 i686 i386 GNU/Linux >>>> ifort intel compiler version 8.0 >>>> hdf5 1.6.6 (serial) >>>> mpich2 Version: 1.0.3 >>>> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>>> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include >>>> >>>> 2) >>>> Debian (??) >>>> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC >>>> 2005 x86_64 GNU/Linux >>>> pgf90 6.0-5 32-bit target on x86-64 Linux >>>> hdf5 1.4.5 (serial) >>>> mpich2 version 1.0.5 >>>> gcc version 3.3.5 (Debian 1:3.3.5-13) >>>> FFLAGS_OPT = -c -fast -r8 -i4 >>>> >>>> 3) >>>> Ubuntu >>>> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 >>>> GNU/Linux >>>> ifort intel compiler version 10.1.012 >>>> hdf5 1.6.7 (serial) >>>> mpich2 Version: 1.0.6 >>>> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >>>> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >>>> $(MPI_PATH)/include >>>> or (only -ipo difference, result seemingly doesn't change) >>>> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >>>> >>>> >>>> >>>> >>>> Nathan Hearn wrote: >>>>> Hi Seyit, >>>>> >>>>> Are you seeing these differences in the standard test problems >>>>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific to >>>>> the >>>>> problem that you are running. (Really, this follows from Tomek's >>>>> advice to look at the simplest configurations where the differences >>>>> are seen.) >>>>> >>>>> Also, it would be good to include some more information in your >>>>> messages, for instance: the problem that you are running, the >>>>> specific >>>>> architectures and compilers, the optimization flags, and the library >>>>> versions for HDF5 and MPI that you are using. Details like these can >>>>> be really helpful in evaluating and fixing the issues you are seeing. >>>>> >>>>> >>>>> - Nathan >>>>> >>>>> >>>>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk >>>>> wrote: >>>>> >>>>>> Oh man, >>>>>> >>>>>> I am using 3 different computers (with different architectures, >>>>>> compilers, >>>>>> optimizations, environments). All three start differently >>>>>> immediately from >>>>>> the beginning (difference in dt_hydro and refinement). >>>>>> Should I worry very very very much? >>>>>> I'm now doing a complete simulation and am going to check if my >>>>>> end results >>>>>> also change. >>>>>> >>>>>> I'm trembling by the thought..... >>>>>> >>>> >>>> >>> >> From seyit at astro.rug.nl Wed Jul 30 04:35:49 2008 From: seyit at astro.rug.nl (Seyit Hocuk) Date: Wed, 30 Jul 2008 11:35:49 +0200 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <48903429.5090007@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> <488F5D72.1040305@fsu.edu> <48903429.5090007@astro.rug.nl> Message-ID: <489035F5.5020609@astro.rug.nl> BTW... The results also differ when I am using multiple cpu's instead of 1. mpiexec -np 4 ./flash2 (on pc 3 instead of running just ./flash2) Note: k,kJ,c0 etc. are the same flash: initializing for jeans problem. kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 k = 1.756203682760181E-019 kJ = 6.508776532382562E-019 c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 velamp = 64881.6252564676 perturbation is unstable with growth time 8776137041619.21 min_blocks 1 max_blocks 2 tot_blocks 5 min_blocks 5 max_blocks 6 tot_blocks 21 min_blocks 21 max_blocks 22 tot_blocks 85 min_blocks 85 max_blocks 86 tot_blocks 341 INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will send 520 blocks per message. [CHECKPOINT_WR] Writing checkpoint file jeans_r13vB_hdf5_chk_0000 Progress: |... *** Wrote output to jeans_r13vB_hdf5_chk_0000 ( 341 blocks ) *** number of variables found for plotfile storage = 3 dens pres temp *** Wrote output to jeans_r13vB_hdf5_plt_cnt_0000 *** n t dt | dt_hydro 1 2.0000E+05 2.0000E+05 | 4.197E+11 2 6.0000E+05 4.0000E+05 | 4.197E+11 3 1.4000E+06 8.0000E+05 | 4.197E+11 4 3.0000E+06 1.6000E+06 | 4.197E+11 5 6.2000E+06 3.2000E+06 | 4.197E+11 6 1.2600E+07 6.4000E+06 | 4.197E+11 7 2.5400E+07 1.2800E+07 | 4.197E+11 8 5.1000E+07 2.5600E+07 | 4.197E+11 9 1.0220E+08 5.1200E+07 | 4.197E+11 10 2.0460E+08 1.0240E+08 | 4.198E+11 11 4.0940E+08 2.0480E+08 | 4.198E+11 12 8.1900E+08 4.0960E+08 | 4.198E+11 13 1.6382E+09 8.1920E+08 | 4.199E+11 14 3.2766E+09 1.6384E+09 | 4.201E+11 15 6.5534E+09 3.0000E+09 | 4.205E+11 16 1.2553E+10 3.0000E+09 | 4.213E+11 17 1.8553E+10 3.0000E+09 | 4.221E+11 18 2.4553E+10 3.0000E+09 | 4.229E+11 19 3.0553E+10 3.0000E+09 | 4.237E+11 20 3.6553E+10 3.0000E+09 | 4.246E+11 21 4.2553E+10 3.0000E+09 | 4.254E+11 22 4.8553E+10 3.0000E+09 | 4.263E+11 23 5.4553E+10 3.0000E+09 | 4.272E+11 24 6.0553E+10 3.0000E+09 | 4.281E+11 25 6.6553E+10 3.0000E+09 | 4.290E+11 26 7.2553E+10 3.0000E+09 | 4.300E+11 27 7.8553E+10 3.0000E+09 | 4.309E+11 28 8.4553E+10 3.0000E+09 | 4.319E+11 29 9.0553E+10 3.0000E+09 | 4.328E+11 30 9.6553E+10 3.0000E+09 | 4.338E+11 31 1.0255E+11 3.0000E+09 | 4.348E+11 32 1.0855E+11 3.0000E+09 | 4.358E+11 33 1.1455E+11 3.0000E+09 | 4.369E+11 34 1.2055E+11 3.0000E+09 | 4.380E+11 35 1.2655E+11 3.0000E+09 | 4.393E+11 36 1.3255E+11 3.0000E+09 | 4.405E+11 37 1.3855E+11 3.0000E+09 | 4.416E+11 38 1.4455E+11 3.0000E+09 | 4.429E+11 39 1.5055E+11 3.0000E+09 | 4.441E+11 40 1.5655E+11 3.0000E+09 | 4.454E+11 41 1.6255E+11 3.0000E+09 | 4.465E+11 42 1.6855E+11 3.0000E+09 | 4.478E+11 43 1.7455E+11 3.0000E+09 | 4.491E+11 44 1.8055E+11 3.0000E+09 | 4.503E+11 45 1.8655E+11 3.0000E+09 | 4.516E+11 46 1.9255E+11 3.0000E+09 | 4.529E+11 47 1.9855E+11 3.0000E+09 | 4.541E+11 48 2.0455E+11 3.0000E+09 | 4.554E+11 49 2.1055E+11 3.0000E+09 | 4.567E+11 50 2.1655E+11 3.0000E+09 | 4.576E+11 Seyit Hocuk wrote: > Hi Tomek good morning, > > Let me answer your first question. > > * /Any luck with a single block/single level runs? / > > Nope. Even single level (lref_min = lref_max) runs differ in dt_hydro. > That means I don't use refinement at all. I guess it is also affecting > the end results. I have to do a complete simulation again to be sure. > But the initial files show small differences already. > > Here the initial results. > Note 1: the differences in precision in k, kJ, c0 etc... > Note 2: pc 1 and 3 are not that much differen, but grow more different > later on. > > > *pc 1* > > flash: initializing for jeans problem. > > kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 > > k = 1.756203682760181E-019 > kJ = 6.508776532382562E-019 > c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 > velamp = 64881.6252564676 > perturbation is unstable with growth time 8776137041619.21 > min_blocks 5 max_blocks 5 tot_blocks 5 > min_blocks 21 max_blocks 21 tot_blocks 21 > min_blocks 85 max_blocks 85 tot_blocks 85 > min_blocks 341 max_blocks 341 tot_blocks 341 > INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will > send 520 blocks per message. > Warning! The HDF5 header files included by this application do not > match the > version used by the HDF5 library to which this application is linked. > Data > corruption or segmentation faults may occur if the application is > allowed to continue. You can, at your own risk, disable this check by > setting > the environment variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. > Setting it to 2 will suppress the warning totally. > Headers are 1.6.6, library is 1.6.7 > Bye... > forrtl: info: Fortran error message number is 76. > forrtl: warning: Could not open message catalog: ifcore_msg.cat. > forrtl: info: Check environment variable NLSPATH and protection of > /usr/lib/ifcore_msg.cat. > Abort > [1] + Done emacs flash.par > [seyit at celsius objectT3]$ setenv HDF5_DISABLE_VERSION_CHECK 1 > [seyit at celsius objectT3]$ ./flash2 > > flash: initializing for jeans problem. > > kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 > > k = 1.756203682760181E-019 > kJ = 6.508776532382562E-019 > c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 > velamp = 64881.6252564676 > perturbation is unstable with growth time 8776137041619.21 > min_blocks 5 max_blocks 5 tot_blocks 5 > min_blocks 21 max_blocks 21 tot_blocks 21 > min_blocks 85 max_blocks 85 tot_blocks 85 > min_blocks 341 max_blocks 341 tot_blocks 341 > INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will > send 520 blocks per message. > [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 > Progress: | > *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) *** > number of variables found for plotfile storage = 3 > dens pres temp > *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** > n t dt | dt_hydro > 1 2.0000E+05 2.0000E+05 | 4.240E+11 > 2 6.0000E+05 4.0000E+05 | 4.240E+11 > 3 1.4000E+06 8.0000E+05 | 4.240E+11 > 4 3.0000E+06 1.6000E+06 | 4.240E+11 > 5 6.2000E+06 3.2000E+06 | 4.240E+11 > 6 1.2600E+07 6.4000E+06 | 4.240E+11 > 7 2.5400E+07 1.2800E+07 | 4.240E+11 > 8 5.1000E+07 2.5600E+07 | 4.240E+11 > 9 1.0220E+08 5.1200E+07 | 4.240E+11 > 10 2.0460E+08 1.0240E+08 | 4.240E+11 > 11 4.0940E+08 2.0480E+08 | 4.241E+11 > 12 8.1900E+08 4.0960E+08 | 4.241E+11 > 13 1.6382E+09 8.1920E+08 | 4.241E+11 > 14 3.2766E+09 1.6384E+09 | 4.242E+11 > 15 6.5534E+09 3.0000E+09 | 4.243E+11 > 16 1.2553E+10 3.0000E+09 | 4.247E+11 > 17 1.8553E+10 3.0000E+09 | 4.250E+11 > 18 2.4553E+10 3.0000E+09 | 4.255E+11 > 19 3.0553E+10 3.0000E+09 | 4.259E+11 > 20 3.6553E+10 3.0000E+09 | 4.266E+11 > 21 4.2553E+10 3.0000E+09 | 4.273E+11 > 22 4.8553E+10 3.0000E+09 | 4.281E+11 > 23 5.4553E+10 3.0000E+09 | 4.288E+11 > 24 6.0553E+10 3.0000E+09 | 4.296E+11 > 25 6.6553E+10 3.0000E+09 | 4.304E+11 > 26 7.2553E+10 3.0000E+09 | 4.313E+11 > 27 7.8553E+10 3.0000E+09 | 4.321E+11 > 28 8.4553E+10 3.0000E+09 | 4.330E+11 > 29 9.0553E+10 3.0000E+09 | 4.339E+11 > 30 9.6553E+10 3.0000E+09 | 4.348E+11 > 31 1.0255E+11 3.0000E+09 | 4.357E+11 > 32 1.0855E+11 3.0000E+09 | 4.366E+11 > 33 1.1455E+11 3.0000E+09 | 4.301E+11 > 34 1.2055E+11 3.0000E+09 | 3.803E+11 > 35 1.2655E+11 3.0000E+09 | 3.379E+11 > 36 1.3255E+11 3.0000E+09 | 3.057E+11 > 37 1.3855E+11 3.0000E+09 | 2.868E+11 > 38 1.4455E+11 3.0000E+09 | 2.876E+11 > 39 1.5055E+11 3.0000E+09 | 2.880E+11 > 40 1.5655E+11 3.0000E+09 | 2.903E+11 > 41 1.6255E+11 3.0000E+09 | 2.936E+11 > 42 1.6855E+11 3.0000E+09 | 2.978E+11 > 43 1.7455E+11 3.0000E+09 | 4.475E+11 > 44 1.8055E+11 3.0000E+09 | 4.486E+11 > 45 1.8655E+11 3.0000E+09 | 4.496E+11 > 46 1.9255E+11 3.0000E+09 | 4.507E+11 > 47 1.9855E+11 3.0000E+09 | 4.517E+11 > 48 2.0455E+11 3.0000E+09 | 4.528E+11 > 49 2.1055E+11 3.0000E+09 | 4.538E+11 > 50 2.1655E+11 3.0000E+09 | 4.549E+11 > > > > *pc 2* > > flash: initializing for jeans problem. > > kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 > > k = 1.7562036827601810E-019 > kJ = 6.5087765323825604E-019 > c0 = 181807.2941759182 omega^2 = -1.2983542608611234E-026 > velamp = 64881.62525646757 > perturbation is unstable with growth time 8776137041619.213 > min_blocks 5 max_blocks 5 tot_blocks 5 > min_blocks 21 max_blocks 21 tot_blocks 21 > min_blocks 85 max_blocks 85 tot_blocks 85 > min_blocks 341 max_blocks 341 tot_blocks 341 > INITIAL TIMESTEP = 100000.0000000000 [CHECKPOINT_WR] NOTE: > will send 520 blocks per message. > [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 > Progress: | > *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) *** > number of variables found for plotfile storage = 3 > dens pres temp > *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** > n t dt | dt_hydro > 1 2.0000E+05 2.0000E+05 | 4.406E+11 > 2 6.0000E+05 4.0000E+05 | 4.406E+11 > 3 1.4000E+06 8.0000E+05 | 4.406E+11 > 4 3.0000E+06 1.6000E+06 | 4.406E+11 > 5 6.2000E+06 3.2000E+06 | 4.406E+11 > 6 1.2600E+07 6.4000E+06 | 4.406E+11 > 7 2.5400E+07 1.2800E+07 | 4.406E+11 > 8 5.1000E+07 2.5600E+07 | 4.406E+11 > 9 1.0220E+08 5.1200E+07 | 4.406E+11 > 10 2.0460E+08 1.0240E+08 | 4.405E+11 > 11 4.0940E+08 2.0480E+08 | 4.404E+11 > 12 8.1900E+08 4.0960E+08 | 4.403E+11 > 13 1.6382E+09 8.1920E+08 | 4.399E+11 > 14 3.2766E+09 1.6384E+09 | 4.393E+11 > 15 6.5534E+09 3.0000E+09 | 4.381E+11 > 16 1.2553E+10 3.0000E+09 | 4.365E+11 > 17 1.8553E+10 3.0000E+09 | 4.353E+11 > 18 2.4553E+10 3.0000E+09 | 4.345E+11 > 19 3.0553E+10 3.0000E+09 | 4.340E+11 > 20 3.6553E+10 3.0000E+09 | 4.337E+11 > 21 4.2553E+10 3.0000E+09 | 4.337E+11 > 22 4.8553E+10 3.0000E+09 | 4.339E+11 > 23 5.4553E+10 3.0000E+09 | 4.343E+11 > 24 6.0553E+10 3.0000E+09 | 4.348E+11 > 25 6.6553E+10 3.0000E+09 | 4.355E+11 > 26 7.2553E+10 3.0000E+09 | 4.363E+11 > 27 7.8553E+10 3.0000E+09 | 4.371E+11 > 28 8.4553E+10 3.0000E+09 | 4.382E+11 > 29 9.0553E+10 3.0000E+09 | 4.393E+11 > 30 9.6553E+10 3.0000E+09 | 4.405E+11 > 31 1.0255E+11 3.0000E+09 | 4.418E+11 > 32 1.0855E+11 3.0000E+09 | 4.431E+11 > 33 1.1455E+11 3.0000E+09 | 4.447E+11 > 34 1.2055E+11 3.0000E+09 | 4.463E+11 > 35 1.2655E+11 3.0000E+09 | 4.479E+11 > 36 1.3255E+11 3.0000E+09 | 4.496E+11 > 37 1.3855E+11 3.0000E+09 | 4.513E+11 > 38 1.4455E+11 3.0000E+09 | 4.531E+11 > 39 1.5055E+11 3.0000E+09 | 4.550E+11 > 40 1.5655E+11 3.0000E+09 | 4.570E+11 > 41 1.6255E+11 3.0000E+09 | 4.590E+11 > 42 1.6855E+11 3.0000E+09 | 4.611E+11 > 43 1.7455E+11 3.0000E+09 | 4.632E+11 > 44 1.8055E+11 3.0000E+09 | 4.653E+11 > 45 1.8655E+11 3.0000E+09 | 4.675E+11 > 46 1.9255E+11 3.0000E+09 | 4.697E+11 > 47 1.9855E+11 3.0000E+09 | 4.719E+11 > 48 2.0455E+11 3.0000E+09 | 4.733E+11 > 49 2.1055E+11 3.0000E+09 | 4.734E+11 > 50 2.1655E+11 3.0000E+09 | 4.736E+11 > > > > *pc 3* > > > flash: initializing for jeans problem. > > kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 > > k = 1.756203682760181E-019 > kJ = 6.508776532382562E-019 > c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 > velamp = 64881.6252564676 > perturbation is unstable with growth time 8776137041619.21 > min_blocks 5 max_blocks 5 tot_blocks 5 > min_blocks 21 max_blocks 21 tot_blocks 21 > min_blocks 85 max_blocks 85 tot_blocks 85 > min_blocks 341 max_blocks 341 tot_blocks 341 > INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will > send 520 blocks per message. > [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 > Progress: | > *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) *** > number of variables found for plotfile storage = 3 > dens pres temp > *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** > n t dt | dt_hydro > 1 2.0000E+05 2.0000E+05 | 4.240E+11 > 2 6.0000E+05 4.0000E+05 | 4.240E+11 > 3 1.4000E+06 8.0000E+05 | 4.240E+11 > 4 3.0000E+06 1.6000E+06 | 4.240E+11 > 5 6.2000E+06 3.2000E+06 | 4.240E+11 > 6 1.2600E+07 6.4000E+06 | 4.240E+11 > 7 2.5400E+07 1.2800E+07 | 4.240E+11 > 8 5.1000E+07 2.5600E+07 | 4.240E+11 > 9 1.0220E+08 5.1200E+07 | 4.240E+11 > 10 2.0460E+08 1.0240E+08 | 4.240E+11 > 11 4.0940E+08 2.0480E+08 | 4.241E+11 > 12 8.1900E+08 4.0960E+08 | 4.241E+11 > 13 1.6382E+09 8.1920E+08 | 4.241E+11 > 14 3.2766E+09 1.6384E+09 | 4.242E+11 > 15 6.5534E+09 3.0000E+09 | 4.244E+11 > 16 1.2553E+10 3.0000E+09 | 4.247E+11 > 17 1.8553E+10 3.0000E+09 | 4.251E+11 > 18 2.4553E+10 3.0000E+09 | 4.255E+11 > 19 3.0553E+10 3.0000E+09 | 4.259E+11 > 20 3.6553E+10 3.0000E+09 | 4.266E+11 > 21 4.2553E+10 3.0000E+09 | 4.273E+11 > 22 4.8553E+10 3.0000E+09 | 4.281E+11 > 23 5.4553E+10 3.0000E+09 | 4.288E+11 > 24 6.0553E+10 3.0000E+09 | 4.296E+11 > 25 6.6553E+10 3.0000E+09 | 4.304E+11 > 26 7.2553E+10 3.0000E+09 | 4.313E+11 > 27 7.8553E+10 3.0000E+09 | 4.321E+11 > 28 8.4553E+10 3.0000E+09 | 4.330E+11 > 29 9.0553E+10 3.0000E+09 | 4.338E+11 > 30 9.6553E+10 3.0000E+09 | 4.347E+11 > 31 1.0255E+11 3.0000E+09 | 4.356E+11 > 32 1.0855E+11 3.0000E+09 | 4.366E+11 > 33 1.1455E+11 3.0000E+09 | 4.375E+11 > 34 1.2055E+11 3.0000E+09 | 4.384E+11 > 35 1.2655E+11 3.0000E+09 | 4.394E+11 > 36 1.3255E+11 3.0000E+09 | 4.404E+11 > 37 1.3855E+11 3.0000E+09 | 4.413E+11 > 38 1.4455E+11 3.0000E+09 | 4.423E+11 > 39 1.5055E+11 3.0000E+09 | 4.433E+11 > 40 1.5655E+11 3.0000E+09 | 4.444E+11 > 41 1.6255E+11 3.0000E+09 | 4.454E+11 > 42 1.6855E+11 3.0000E+09 | 4.464E+11 > 43 1.7455E+11 3.0000E+09 | 4.474E+11 > 44 1.8055E+11 3.0000E+09 | 4.484E+11 > 45 1.8655E+11 3.0000E+09 | 4.495E+11 > 46 1.9255E+11 3.0000E+09 | 4.505E+11 > 47 1.9855E+11 3.0000E+09 | 4.516E+11 > 48 2.0455E+11 3.0000E+09 | 4.526E+11 > 49 2.1055E+11 3.0000E+09 | 4.537E+11 > 50 2.1655E+11 3.0000E+09 | 4.547E+11 > > > > > *here again the 3 different pc's* > > > 1) > Centos5 > Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 > i686 i686 i386 GNU/Linux > ifort intel compiler version 8.0 > hdf5 1.6.6 (serial) *(updated recently to 1.6.7)* > mpich2 Version: 1.0.3 > gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) > FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include > > 2) > Debian (??) > Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC 2005 > x86_64 GNU/Linux > pgf90 6.0-5 32-bit target on x86-64 Linux > hdf5 1.4.5 (serial) > mpich2 version 1.0.5 > gcc version 3.3.5 (Debian 1:3.3.5-13) > FFLAGS_OPT = -c -fast -r8 -i4 > > 3) > Ubuntu > Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux > ifort intel compiler version 10.1.012 > hdf5 1.6.7 (serial) > mpich2 Version: 1.0.6 > gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) > FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I > $(MPI_PATH)/include > or (only -ipo difference, result seemingly doesn't change) > FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include > > > > > > > Tomasz Plewa wrote: >> Seyit - >> >> Avoiding aggressive optimization is certainly good idea when it comes >> to debugging implementation. Alas, given you experience problems with >> 3 different compilers it is rather unlilkely that compilers are at >> fault. >> >> If your grid does not resolve structures correctly (persumably due to >> lax refinement criteria), the code may quite easily produce an >> unphysical state. But that should not happen during the initial grid >> generation. >> >> Any luck with a single block/single level runs? >> >> Tomek >> -- >> Seyit Hocuk wrote: >>> Hi, >>> >>> It has been some time now, but to continue the discussion, I have >>> tested Jeans standard problem and Sedov standard problem including >>> changing the refinement levels. The result is exactly same for all >>> three different computers. Unfortunately my own setup shows >>> differences and it seems the end result changes by the coice of >>> computer/environment/compiler whatever. My own setup has a lot more >>> physics than the simpe test problems. >>> >>> One thing I am thinking about is that I adjusted the refinement >>> criteria to use a reference density instead of looking at the second >>> order derivative. I don't know if this could be the problem, but >>> dt_hydro should not depend on the refinement criteria so I guess >>> there is more to it. >>> >>> Unfortunately it is not easy to change compilers and some other >>> stuff (like cpu/kernel etc.) on the different computers because I >>> have limited rights on those machines. I don't want to find out I >>> have to choose one machine and stick with it. How do I know which >>> machine is the best. Anyway, I am going to try at least to get the >>> libraries the same (hdf5 / mpi) for all machines and see if that >>> helps. Btw the latest hdf5 (1.8.1) doesn't seem to work with Flash2.5. >>> >>> I will also play with compiler options. Try to make them exactly the >>> same and avoid aggressive optimizations. I hope this is where the >>> solution lies. >>> >>> Thanks, >>> Seyit >>> >>> >>> >>> >>> Tomasz Plewa wrote: >>>> Although I prefer a low cost approach to debugging which avoids >>>> dealing with setup/system/compiler information at first and >>>> attempts to eliminate 99% of the source code, here are quick >>>> comments on your systems. >>>> >>>> (1) pgi 6.0 is a quite antique compiler. It's so old that now I do >>>> not even remember what problems we may have had with it. I suspect >>>> quite a few and none of them might be relevant... >>>> >>>> (2) As you might noticed, intel has a newer compiler version. Your >>>> CentOS5 system needs upgrades on both mpich2 and compiler sides. >>>> >>>> (3) Your Ubuntu system seems relatively up-to-date although HDF5 >>>> 1.8.x is out there and seems working fine. >>>> >>>> (4) Compilers are known to do strange things to perfect codes! So >>>> if you run into a trouble on a new system, one of the first things >>>> to try is to lower compiler optimization level. Avoiding -fast or >>>> -ip or -inline is usually a good idea. Simple -O0 may suffice for a >>>> couple of steps. As a bonus, the code compiles fast. Then the trick >>>> is to get it crashing quickly... >>>> >>>> Again, I suggest to simplify your application, isolate the problem, >>>> fix it, then find the best compiler, mpich, options, etc., to run >>>> your production code. >>>> >>>> Enjoy your weekend! >>>> >>>> Tomek >>>> -- >>>> Seyit Hocuk wrote: >>>>> Hi Nathan, >>>>> >>>>> Thats a good Idea, I'll start simple and test those first thing >>>>> monday. It's too late for me now. >>>>> >>>>> To answer your questions: >>>>> >>>>> The problem I am running is my own home made, actually it's a >>>>> modified version of Jeans. Basically including heat, cool, eos >>>>> gamma, compositions, adjusted refinement criteria similar to one >>>>> of the other problems (with delta_ref and delta_deref) and a scala >>>>> of other small things. >>>>> >>>>> 1) >>>>> Centos5 >>>>> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT >>>>> 2008 i686 i686 i386 GNU/Linux >>>>> ifort intel compiler version 8.0 >>>>> hdf5 1.6.6 (serial) >>>>> mpich2 Version: 1.0.3 >>>>> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>>>> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I >>>>> $(MPI_PATH)/include >>>>> >>>>> 2) >>>>> Debian (??) >>>>> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC >>>>> 2005 x86_64 GNU/Linux >>>>> pgf90 6.0-5 32-bit target on x86-64 Linux >>>>> hdf5 1.4.5 (serial) >>>>> mpich2 version 1.0.5 >>>>> gcc version 3.3.5 (Debian 1:3.3.5-13) >>>>> FFLAGS_OPT = -c -fast -r8 -i4 >>>>> >>>>> 3) >>>>> Ubuntu >>>>> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 >>>>> GNU/Linux >>>>> ifort intel compiler version 10.1.012 >>>>> hdf5 1.6.7 (serial) >>>>> mpich2 Version: 1.0.6 >>>>> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >>>>> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >>>>> $(MPI_PATH)/include >>>>> or (only -ipo difference, result seemingly doesn't change) >>>>> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >>>>> >>>>> >>>>> >>>>> >>>>> Nathan Hearn wrote: >>>>>> Hi Seyit, >>>>>> >>>>>> Are you seeing these differences in the standard test problems >>>>>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific >>>>>> to the >>>>>> problem that you are running. (Really, this follows from Tomek's >>>>>> advice to look at the simplest configurations where the differences >>>>>> are seen.) >>>>>> >>>>>> Also, it would be good to include some more information in your >>>>>> messages, for instance: the problem that you are running, the >>>>>> specific >>>>>> architectures and compilers, the optimization flags, and the library >>>>>> versions for HDF5 and MPI that you are using. Details like these >>>>>> can >>>>>> be really helpful in evaluating and fixing the issues you are >>>>>> seeing. >>>>>> >>>>>> >>>>>> - Nathan >>>>>> >>>>>> >>>>>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk >>>>>> wrote: >>>>>> >>>>>>> Oh man, >>>>>>> >>>>>>> I am using 3 different computers (with different architectures, >>>>>>> compilers, >>>>>>> optimizations, environments). All three start differently >>>>>>> immediately from >>>>>>> the beginning (difference in dt_hydro and refinement). >>>>>>> Should I worry very very very much? >>>>>>> I'm now doing a complete simulation and am going to check if my >>>>>>> end results >>>>>>> also change. >>>>>>> >>>>>>> I'm trembling by the thought..... >>>>>>> >>>>> >>>>> >>>> >>> > From seyit at astro.rug.nl Wed Jul 30 04:44:03 2008 From: seyit at astro.rug.nl (Seyit Hocuk) Date: Wed, 30 Jul 2008 11:44:03 +0200 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <488F7953.4010802@flash.uchicago.edu> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> <488F7953.4010802@flash.uchicago.edu> Message-ID: <489037E3.1050400@astro.rug.nl> Hi Paul and good morning, Thanks for your reply. To answer your question " Is the version of HDF5 1.8.1 you are using built with the 1.6 api bindings? ", I don't know. I didn't check that. I will try that, when I know how to do that. I just downloaded the source file and did configure-make-make install. Where does the bindings come in? I will try to find it out. Greetz, Seyit Paul M. Rich wrote: > Seyit, > > I can clear up the HDF5 1.8 issue, I think. I've experimented with > HDF5 1.8.1 in FLASH3 and that will only work if you have built the > v1.8.x implementation with support for the 1.6 api. I believe that > FLASH2.5's will work in a similar fashion. Is the version of HDF5 > 1.8.1 you are using built with the 1.6 api bindings? > > Paul Rich > > Seyit Hocuk wrote: >> Hi, >> >> It has been some time now, but to continue the discussion, I have >> tested Jeans standard problem and Sedov standard problem including >> changing the refinement levels. The result is exactly same for all >> three different computers. Unfortunately my own setup shows >> differences and it seems the end result changes by the coice of >> computer/environment/compiler whatever. My own setup has a lot more >> physics than the simpe test problems. >> >> One thing I am thinking about is that I adjusted the refinement >> criteria to use a reference density instead of looking at the second >> order derivative. I don't know if this could be the problem, but >> dt_hydro should not depend on the refinement criteria so I guess >> there is more to it. >> >> Unfortunately it is not easy to change compilers and some other stuff >> (like cpu/kernel etc.) on the different computers because I have >> limited rights on those machines. I don't want to find out I have to >> choose one machine and stick with it. How do I know which machine is >> the best. Anyway, I am going to try at least to get the libraries the >> same (hdf5 / mpi) for all machines and see if that helps. Btw the >> latest hdf5 (1.8.1) doesn't seem to work with Flash2.5. >> >> I will also play with compiler options. Try to make them exactly the >> same and avoid aggressive optimizations. I hope this is where the >> solution lies. >> >> Thanks, >> Seyit >> >> >> >> >> Tomasz Plewa wrote: >>> Although I prefer a low cost approach to debugging which avoids >>> dealing with setup/system/compiler information at first and attempts >>> to eliminate 99% of the source code, here are quick comments on >>> your systems. >>> >>> (1) pgi 6.0 is a quite antique compiler. It's so old that now I do >>> not even remember what problems we may have had with it. I suspect >>> quite a few and none of them might be relevant... >>> >>> (2) As you might noticed, intel has a newer compiler version. Your >>> CentOS5 system needs upgrades on both mpich2 and compiler sides. >>> >>> (3) Your Ubuntu system seems relatively up-to-date although HDF5 >>> 1.8.x is out there and seems working fine. >>> >>> (4) Compilers are known to do strange things to perfect codes! So if >>> you run into a trouble on a new system, one of the first things to >>> try is to lower compiler optimization level. Avoiding -fast or -ip >>> or -inline is usually a good idea. Simple -O0 may suffice for a >>> couple of steps. As a bonus, the code compiles fast. Then the trick >>> is to get it crashing quickly... >>> >>> Again, I suggest to simplify your application, isolate the problem, >>> fix it, then find the best compiler, mpich, options, etc., to run >>> your production code. >>> >>> Enjoy your weekend! >>> >>> Tomek >>> -- >>> Seyit Hocuk wrote: >>>> Hi Nathan, >>>> >>>> Thats a good Idea, I'll start simple and test those first thing >>>> monday. It's too late for me now. >>>> >>>> To answer your questions: >>>> >>>> The problem I am running is my own home made, actually it's a >>>> modified version of Jeans. Basically including heat, cool, eos >>>> gamma, compositions, adjusted refinement criteria similar to one of >>>> the other problems (with delta_ref and delta_deref) and a scala of >>>> other small things. >>>> >>>> 1) >>>> Centos5 >>>> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 >>>> i686 i686 i386 GNU/Linux >>>> ifort intel compiler version 8.0 >>>> hdf5 1.6.6 (serial) >>>> mpich2 Version: 1.0.3 >>>> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>>> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include >>>> >>>> 2) >>>> Debian (??) >>>> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC >>>> 2005 x86_64 GNU/Linux >>>> pgf90 6.0-5 32-bit target on x86-64 Linux >>>> hdf5 1.4.5 (serial) >>>> mpich2 version 1.0.5 >>>> gcc version 3.3.5 (Debian 1:3.3.5-13) >>>> FFLAGS_OPT = -c -fast -r8 -i4 >>>> >>>> 3) >>>> Ubuntu >>>> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 >>>> GNU/Linux >>>> ifort intel compiler version 10.1.012 >>>> hdf5 1.6.7 (serial) >>>> mpich2 Version: 1.0.6 >>>> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >>>> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >>>> $(MPI_PATH)/include >>>> or (only -ipo difference, result seemingly doesn't change) >>>> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >>>> >>>> >>>> >>>> >>>> Nathan Hearn wrote: >>>>> Hi Seyit, >>>>> >>>>> Are you seeing these differences in the standard test problems >>>>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific to >>>>> the >>>>> problem that you are running. (Really, this follows from Tomek's >>>>> advice to look at the simplest configurations where the differences >>>>> are seen.) >>>>> >>>>> Also, it would be good to include some more information in your >>>>> messages, for instance: the problem that you are running, the >>>>> specific >>>>> architectures and compilers, the optimization flags, and the library >>>>> versions for HDF5 and MPI that you are using. Details like these can >>>>> be really helpful in evaluating and fixing the issues you are seeing. >>>>> >>>>> >>>>> - Nathan >>>>> >>>>> >>>>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk >>>>> wrote: >>>>> >>>>>> Oh man, >>>>>> >>>>>> I am using 3 different computers (with different architectures, >>>>>> compilers, >>>>>> optimizations, environments). All three start differently >>>>>> immediately from >>>>>> the beginning (difference in dt_hydro and refinement). >>>>>> Should I worry very very very much? >>>>>> I'm now doing a complete simulation and am going to check if my >>>>>> end results >>>>>> also change. >>>>>> >>>>>> I'm trembling by the thought..... >>>>>> >>>> >>>> >>> > From seyit at astro.rug.nl Wed Jul 30 04:48:44 2008 From: seyit at astro.rug.nl (Seyit Hocuk) Date: Wed, 30 Jul 2008 11:48:44 +0200 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <2467fdc0807291324t5368140ax904909774b1bf582@mail.gmail.com> References: <48441DC4.9050701@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> <488F7953.4010802@flash.uchicago.edu> <2467fdc0807291324t5368140ax904909774b1bf582@mail.gmail.com> Message-ID: <489038FC.2030005@astro.rug.nl> Ah thanks Nathan, I see there is a good link to the api bindings. That answers the question I posted a minute ago I guess. I think I just have to configure hdf5 with a certain flag. " |--with-default-api-version=v16 |" Let me try that out. Greetz, Seyit Nathan Hearn wrote: > For everyone's information, the HDF Group has a nice page on the > compatibility macros that are meant to bridge between the 1.6 and 1.8 > versions, as well as a discussion of the associated configuration > options for HDF5: > > http://www.hdfgroup.org/HDF5/doc/RM/APICompatMacros.html > > - Nathan > > > On Tue, Jul 29, 2008 at 3:10 PM, Paul M. Rich wrote: > >> Seyit, >> >> I can clear up the HDF5 1.8 issue, I think. I've experimented with HDF5 >> 1.8.1 in FLASH3 and that will only work if you have built the v1.8.x >> implementation with support for the 1.6 api. I believe that FLASH2.5's will >> work in a similar fashion. Is the version of HDF5 1.8.1 you are using built >> with the 1.6 api bindings? >> >> Paul Rich >> >> Seyit Hocuk wrote: >> >>> Hi, >>> >>> It has been some time now, but to continue the discussion, I have tested >>> Jeans standard problem and Sedov standard problem including changing the >>> refinement levels. The result is exactly same for all three different >>> computers. Unfortunately my own setup shows differences and it seems the end >>> result changes by the coice of computer/environment/compiler whatever. My >>> own setup has a lot more physics than the simpe test problems. >>> >>> One thing I am thinking about is that I adjusted the refinement criteria >>> to use a reference density instead of looking at the second order >>> derivative. I don't know if this could be the problem, but dt_hydro should >>> not depend on the refinement criteria so I guess there is more to it. >>> >>> Unfortunately it is not easy to change compilers and some other stuff >>> (like cpu/kernel etc.) on the different computers because I have limited >>> rights on those machines. I don't want to find out I have to choose one >>> machine and stick with it. How do I know which machine is the best. Anyway, >>> I am going to try at least to get the libraries the same (hdf5 / mpi) for >>> all machines and see if that helps. Btw the latest hdf5 (1.8.1) doesn't seem >>> to work with Flash2.5. >>> >>> I will also play with compiler options. Try to make them exactly the same >>> and avoid aggressive optimizations. I hope this is where the solution lies. >>> >>> Thanks, >>> Seyit >>> >>> >>> >>> >>> Tomasz Plewa wrote: >>> >>>> Although I prefer a low cost approach to debugging which avoids dealing >>>> with setup/system/compiler information at first and attempts to eliminate >>>> 99% of the source code, here are quick comments on your systems. >>>> >>>> (1) pgi 6.0 is a quite antique compiler. It's so old that now I do not >>>> even remember what problems we may have had with it. I suspect quite a few >>>> and none of them might be relevant... >>>> >>>> (2) As you might noticed, intel has a newer compiler version. Your >>>> CentOS5 system needs upgrades on both mpich2 and compiler sides. >>>> >>>> (3) Your Ubuntu system seems relatively up-to-date although HDF5 1.8.x is >>>> out there and seems working fine. >>>> >>>> (4) Compilers are known to do strange things to perfect codes! So if you >>>> run into a trouble on a new system, one of the first things to try is to >>>> lower compiler optimization level. Avoiding -fast or -ip or -inline is >>>> usually a good idea. Simple -O0 may suffice for a couple of steps. As a >>>> bonus, the code compiles fast. Then the trick is to get it crashing >>>> quickly... >>>> >>>> Again, I suggest to simplify your application, isolate the problem, fix >>>> it, then find the best compiler, mpich, options, etc., to run your >>>> production code. >>>> >>>> Enjoy your weekend! >>>> >>>> Tomek >>>> -- >>>> Seyit Hocuk wrote: >>>> >>>>> Hi Nathan, >>>>> >>>>> Thats a good Idea, I'll start simple and test those first thing monday. >>>>> It's too late for me now. >>>>> >>>>> To answer your questions: >>>>> >>>>> The problem I am running is my own home made, actually it's a modified >>>>> version of Jeans. Basically including heat, cool, eos gamma, compositions, >>>>> adjusted refinement criteria similar to one of the other problems (with >>>>> delta_ref and delta_deref) and a scala of other small things. >>>>> >>>>> 1) >>>>> Centos5 >>>>> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 i686 >>>>> i686 i386 GNU/Linux >>>>> ifort intel compiler version 8.0 >>>>> hdf5 1.6.6 (serial) >>>>> mpich2 Version: 1.0.3 >>>>> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>>>> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include >>>>> >>>>> 2) >>>>> Debian (??) >>>>> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC 2005 >>>>> x86_64 GNU/Linux >>>>> pgf90 6.0-5 32-bit target on x86-64 Linux >>>>> hdf5 1.4.5 (serial) >>>>> mpich2 version 1.0.5 >>>>> gcc version 3.3.5 (Debian 1:3.3.5-13) >>>>> FFLAGS_OPT = -c -fast -r8 -i4 >>>>> >>>>> 3) >>>>> Ubuntu >>>>> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux >>>>> ifort intel compiler version 10.1.012 >>>>> hdf5 1.6.7 (serial) >>>>> mpich2 Version: 1.0.6 >>>>> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >>>>> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >>>>> $(MPI_PATH)/include >>>>> or (only -ipo difference, result seemingly doesn't change) >>>>> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >>>>> >>>>> >>>>> >>>>> >>>>> Nathan Hearn wrote: >>>>> >>>>>> Hi Seyit, >>>>>> >>>>>> Are you seeing these differences in the standard test problems >>>>>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific to the >>>>>> problem that you are running. (Really, this follows from Tomek's >>>>>> advice to look at the simplest configurations where the differences >>>>>> are seen.) >>>>>> >>>>>> Also, it would be good to include some more information in your >>>>>> messages, for instance: the problem that you are running, the specific >>>>>> architectures and compilers, the optimization flags, and the library >>>>>> versions for HDF5 and MPI that you are using. Details like these can >>>>>> be really helpful in evaluating and fixing the issues you are seeing. >>>>>> >>>>>> >>>>>> - Nathan >>>>>> >>>>>> >>>>>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk >>>>>> wrote: >>>>>> >>>>>> >>>>>>> Oh man, >>>>>>> >>>>>>> I am using 3 different computers (with different architectures, >>>>>>> compilers, >>>>>>> optimizations, environments). All three start differently immediately >>>>>>> from >>>>>>> the beginning (difference in dt_hydro and refinement). >>>>>>> Should I worry very very very much? >>>>>>> I'm now doing a complete simulation and am going to check if my end >>>>>>> results >>>>>>> also change. >>>>>>> >>>>>>> I'm trembling by the thought..... >>>>>>> >>>>>>> >>>>> >> > > > > From tplewa at fsu.edu Wed Jul 30 05:46:10 2008 From: tplewa at fsu.edu (Tomasz Plewa) Date: Wed, 30 Jul 2008 06:46:10 -0400 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <48903429.5090007@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> <488F5D72.1040305@fsu.edu> <48903429.5090007@astro.rug.nl> Message-ID: <48904672.6060907@fsu.edu> Seyit - To begin with, you are using 2 different processor architectures, 3 different Linux kernels, 3 different Fortran compilers, and, finally, 3 different versions of one mpi implementation (mpich). In this situation it is expected that you will see at least some variation in results of floating point calculations (1) due to different processor even if you use the same compiler (i.e. pc1 vs pc3) or (2) compiler even if you use the same architecture (i.e. pc2 vs pc3). Kernel may play some rather minimal role and mpi version is perhaps even the least concern. So the bottom line is: differences in numerical results are expected even at t=0. And since you are working with a physically unstable problem, physics will pick up that additional perturbative signal and drive solutions over sightly different trajectories. Are those differences important? Well, that depends on what you are after. If sheer numerics and consistency of floating point calculcations is crucial, perhaps yes. That might a computer science question. But if physics is of primary concern then you need to judge from the evolution of the system. Formally, you may want to do a convergence study and check whether all your models converge to the same solution (using some metric and norm). If not, pick one system that does converge. Please keep in mind that convergence in a sense of numerical solution does not imply correctness in terms of physics. This aspect of numerical modeling is once in a while confused... Tomek -- Seyit Hocuk wrote: > Hi Tomek good morning, > > Let me answer your first question. > > * /Any luck with a single block/single level runs? / > > Nope. Even single level (lref_min = lref_max) runs differ in dt_hydro. > That means I don't use refinement at all. I guess it is also affecting > the end results. I have to do a complete simulation again to be sure. > But the initial files show small differences already. > > Here the initial results. > Note 1: the differences in precision in k, kJ, c0 etc... > Note 2: pc 1 and 3 are not that much differen, but grow more different > later on. > > > *pc 1* > > flash: initializing for jeans problem. > > kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 > > k = 1.756203682760181E-019 > kJ = 6.508776532382562E-019 > c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 > velamp = 64881.6252564676 > perturbation is unstable with growth time 8776137041619.21 > min_blocks 5 max_blocks 5 tot_blocks 5 > min_blocks 21 max_blocks 21 tot_blocks 21 > min_blocks 85 max_blocks 85 tot_blocks 85 > min_blocks 341 max_blocks 341 tot_blocks 341 > INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will > send 520 blocks per message. > Warning! The HDF5 header files included by this application do not > match the > version used by the HDF5 library to which this application is linked. > Data > corruption or segmentation faults may occur if the application is > allowed to continue. You can, at your own risk, disable this check by > setting > the environment variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. > Setting it to 2 will suppress the warning totally. > Headers are 1.6.6, library is 1.6.7 > Bye... > forrtl: info: Fortran error message number is 76. > forrtl: warning: Could not open message catalog: ifcore_msg.cat. > forrtl: info: Check environment variable NLSPATH and protection of > /usr/lib/ifcore_msg.cat. > Abort > [1] + Done emacs flash.par > [seyit at celsius objectT3]$ setenv HDF5_DISABLE_VERSION_CHECK 1 > [seyit at celsius objectT3]$ ./flash2 > > flash: initializing for jeans problem. > > kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 > > k = 1.756203682760181E-019 > kJ = 6.508776532382562E-019 > c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 > velamp = 64881.6252564676 > perturbation is unstable with growth time 8776137041619.21 > min_blocks 5 max_blocks 5 tot_blocks 5 > min_blocks 21 max_blocks 21 tot_blocks 21 > min_blocks 85 max_blocks 85 tot_blocks 85 > min_blocks 341 max_blocks 341 tot_blocks 341 > INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will > send 520 blocks per message. > [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 > Progress: | > *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) *** > number of variables found for plotfile storage = 3 > dens pres temp > *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** > n t dt | dt_hydro > 1 2.0000E+05 2.0000E+05 | 4.240E+11 > 2 6.0000E+05 4.0000E+05 | 4.240E+11 > 3 1.4000E+06 8.0000E+05 | 4.240E+11 > 4 3.0000E+06 1.6000E+06 | 4.240E+11 > 5 6.2000E+06 3.2000E+06 | 4.240E+11 > 6 1.2600E+07 6.4000E+06 | 4.240E+11 > 7 2.5400E+07 1.2800E+07 | 4.240E+11 > 8 5.1000E+07 2.5600E+07 | 4.240E+11 > 9 1.0220E+08 5.1200E+07 | 4.240E+11 > 10 2.0460E+08 1.0240E+08 | 4.240E+11 > 11 4.0940E+08 2.0480E+08 | 4.241E+11 > 12 8.1900E+08 4.0960E+08 | 4.241E+11 > 13 1.6382E+09 8.1920E+08 | 4.241E+11 > 14 3.2766E+09 1.6384E+09 | 4.242E+11 > 15 6.5534E+09 3.0000E+09 | 4.243E+11 > 16 1.2553E+10 3.0000E+09 | 4.247E+11 > 17 1.8553E+10 3.0000E+09 | 4.250E+11 > 18 2.4553E+10 3.0000E+09 | 4.255E+11 > 19 3.0553E+10 3.0000E+09 | 4.259E+11 > 20 3.6553E+10 3.0000E+09 | 4.266E+11 > 21 4.2553E+10 3.0000E+09 | 4.273E+11 > 22 4.8553E+10 3.0000E+09 | 4.281E+11 > 23 5.4553E+10 3.0000E+09 | 4.288E+11 > 24 6.0553E+10 3.0000E+09 | 4.296E+11 > 25 6.6553E+10 3.0000E+09 | 4.304E+11 > 26 7.2553E+10 3.0000E+09 | 4.313E+11 > 27 7.8553E+10 3.0000E+09 | 4.321E+11 > 28 8.4553E+10 3.0000E+09 | 4.330E+11 > 29 9.0553E+10 3.0000E+09 | 4.339E+11 > 30 9.6553E+10 3.0000E+09 | 4.348E+11 > 31 1.0255E+11 3.0000E+09 | 4.357E+11 > 32 1.0855E+11 3.0000E+09 | 4.366E+11 > 33 1.1455E+11 3.0000E+09 | 4.301E+11 > 34 1.2055E+11 3.0000E+09 | 3.803E+11 > 35 1.2655E+11 3.0000E+09 | 3.379E+11 > 36 1.3255E+11 3.0000E+09 | 3.057E+11 > 37 1.3855E+11 3.0000E+09 | 2.868E+11 > 38 1.4455E+11 3.0000E+09 | 2.876E+11 > 39 1.5055E+11 3.0000E+09 | 2.880E+11 > 40 1.5655E+11 3.0000E+09 | 2.903E+11 > 41 1.6255E+11 3.0000E+09 | 2.936E+11 > 42 1.6855E+11 3.0000E+09 | 2.978E+11 > 43 1.7455E+11 3.0000E+09 | 4.475E+11 > 44 1.8055E+11 3.0000E+09 | 4.486E+11 > 45 1.8655E+11 3.0000E+09 | 4.496E+11 > 46 1.9255E+11 3.0000E+09 | 4.507E+11 > 47 1.9855E+11 3.0000E+09 | 4.517E+11 > 48 2.0455E+11 3.0000E+09 | 4.528E+11 > 49 2.1055E+11 3.0000E+09 | 4.538E+11 > 50 2.1655E+11 3.0000E+09 | 4.549E+11 > > > > *pc 2* > > flash: initializing for jeans problem. > > kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 > > k = 1.7562036827601810E-019 > kJ = 6.5087765323825604E-019 > c0 = 181807.2941759182 omega^2 = -1.2983542608611234E-026 > velamp = 64881.62525646757 > perturbation is unstable with growth time 8776137041619.213 > min_blocks 5 max_blocks 5 tot_blocks 5 > min_blocks 21 max_blocks 21 tot_blocks 21 > min_blocks 85 max_blocks 85 tot_blocks 85 > min_blocks 341 max_blocks 341 tot_blocks 341 > INITIAL TIMESTEP = 100000.0000000000 [CHECKPOINT_WR] NOTE: > will send 520 blocks per message. > [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 > Progress: | > *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) *** > number of variables found for plotfile storage = 3 > dens pres temp > *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** > n t dt | dt_hydro > 1 2.0000E+05 2.0000E+05 | 4.406E+11 > 2 6.0000E+05 4.0000E+05 | 4.406E+11 > 3 1.4000E+06 8.0000E+05 | 4.406E+11 > 4 3.0000E+06 1.6000E+06 | 4.406E+11 > 5 6.2000E+06 3.2000E+06 | 4.406E+11 > 6 1.2600E+07 6.4000E+06 | 4.406E+11 > 7 2.5400E+07 1.2800E+07 | 4.406E+11 > 8 5.1000E+07 2.5600E+07 | 4.406E+11 > 9 1.0220E+08 5.1200E+07 | 4.406E+11 > 10 2.0460E+08 1.0240E+08 | 4.405E+11 > 11 4.0940E+08 2.0480E+08 | 4.404E+11 > 12 8.1900E+08 4.0960E+08 | 4.403E+11 > 13 1.6382E+09 8.1920E+08 | 4.399E+11 > 14 3.2766E+09 1.6384E+09 | 4.393E+11 > 15 6.5534E+09 3.0000E+09 | 4.381E+11 > 16 1.2553E+10 3.0000E+09 | 4.365E+11 > 17 1.8553E+10 3.0000E+09 | 4.353E+11 > 18 2.4553E+10 3.0000E+09 | 4.345E+11 > 19 3.0553E+10 3.0000E+09 | 4.340E+11 > 20 3.6553E+10 3.0000E+09 | 4.337E+11 > 21 4.2553E+10 3.0000E+09 | 4.337E+11 > 22 4.8553E+10 3.0000E+09 | 4.339E+11 > 23 5.4553E+10 3.0000E+09 | 4.343E+11 > 24 6.0553E+10 3.0000E+09 | 4.348E+11 > 25 6.6553E+10 3.0000E+09 | 4.355E+11 > 26 7.2553E+10 3.0000E+09 | 4.363E+11 > 27 7.8553E+10 3.0000E+09 | 4.371E+11 > 28 8.4553E+10 3.0000E+09 | 4.382E+11 > 29 9.0553E+10 3.0000E+09 | 4.393E+11 > 30 9.6553E+10 3.0000E+09 | 4.405E+11 > 31 1.0255E+11 3.0000E+09 | 4.418E+11 > 32 1.0855E+11 3.0000E+09 | 4.431E+11 > 33 1.1455E+11 3.0000E+09 | 4.447E+11 > 34 1.2055E+11 3.0000E+09 | 4.463E+11 > 35 1.2655E+11 3.0000E+09 | 4.479E+11 > 36 1.3255E+11 3.0000E+09 | 4.496E+11 > 37 1.3855E+11 3.0000E+09 | 4.513E+11 > 38 1.4455E+11 3.0000E+09 | 4.531E+11 > 39 1.5055E+11 3.0000E+09 | 4.550E+11 > 40 1.5655E+11 3.0000E+09 | 4.570E+11 > 41 1.6255E+11 3.0000E+09 | 4.590E+11 > 42 1.6855E+11 3.0000E+09 | 4.611E+11 > 43 1.7455E+11 3.0000E+09 | 4.632E+11 > 44 1.8055E+11 3.0000E+09 | 4.653E+11 > 45 1.8655E+11 3.0000E+09 | 4.675E+11 > 46 1.9255E+11 3.0000E+09 | 4.697E+11 > 47 1.9855E+11 3.0000E+09 | 4.719E+11 > 48 2.0455E+11 3.0000E+09 | 4.733E+11 > 49 2.1055E+11 3.0000E+09 | 4.734E+11 > 50 2.1655E+11 3.0000E+09 | 4.736E+11 > > > > *pc 3* > > > flash: initializing for jeans problem. > > kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 > > k = 1.756203682760181E-019 > kJ = 6.508776532382562E-019 > c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 > velamp = 64881.6252564676 > perturbation is unstable with growth time 8776137041619.21 > min_blocks 5 max_blocks 5 tot_blocks 5 > min_blocks 21 max_blocks 21 tot_blocks 21 > min_blocks 85 max_blocks 85 tot_blocks 85 > min_blocks 341 max_blocks 341 tot_blocks 341 > INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will > send 520 blocks per message. > [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 > Progress: | > *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) *** > number of variables found for plotfile storage = 3 > dens pres temp > *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** > n t dt | dt_hydro > 1 2.0000E+05 2.0000E+05 | 4.240E+11 > 2 6.0000E+05 4.0000E+05 | 4.240E+11 > 3 1.4000E+06 8.0000E+05 | 4.240E+11 > 4 3.0000E+06 1.6000E+06 | 4.240E+11 > 5 6.2000E+06 3.2000E+06 | 4.240E+11 > 6 1.2600E+07 6.4000E+06 | 4.240E+11 > 7 2.5400E+07 1.2800E+07 | 4.240E+11 > 8 5.1000E+07 2.5600E+07 | 4.240E+11 > 9 1.0220E+08 5.1200E+07 | 4.240E+11 > 10 2.0460E+08 1.0240E+08 | 4.240E+11 > 11 4.0940E+08 2.0480E+08 | 4.241E+11 > 12 8.1900E+08 4.0960E+08 | 4.241E+11 > 13 1.6382E+09 8.1920E+08 | 4.241E+11 > 14 3.2766E+09 1.6384E+09 | 4.242E+11 > 15 6.5534E+09 3.0000E+09 | 4.244E+11 > 16 1.2553E+10 3.0000E+09 | 4.247E+11 > 17 1.8553E+10 3.0000E+09 | 4.251E+11 > 18 2.4553E+10 3.0000E+09 | 4.255E+11 > 19 3.0553E+10 3.0000E+09 | 4.259E+11 > 20 3.6553E+10 3.0000E+09 | 4.266E+11 > 21 4.2553E+10 3.0000E+09 | 4.273E+11 > 22 4.8553E+10 3.0000E+09 | 4.281E+11 > 23 5.4553E+10 3.0000E+09 | 4.288E+11 > 24 6.0553E+10 3.0000E+09 | 4.296E+11 > 25 6.6553E+10 3.0000E+09 | 4.304E+11 > 26 7.2553E+10 3.0000E+09 | 4.313E+11 > 27 7.8553E+10 3.0000E+09 | 4.321E+11 > 28 8.4553E+10 3.0000E+09 | 4.330E+11 > 29 9.0553E+10 3.0000E+09 | 4.338E+11 > 30 9.6553E+10 3.0000E+09 | 4.347E+11 > 31 1.0255E+11 3.0000E+09 | 4.356E+11 > 32 1.0855E+11 3.0000E+09 | 4.366E+11 > 33 1.1455E+11 3.0000E+09 | 4.375E+11 > 34 1.2055E+11 3.0000E+09 | 4.384E+11 > 35 1.2655E+11 3.0000E+09 | 4.394E+11 > 36 1.3255E+11 3.0000E+09 | 4.404E+11 > 37 1.3855E+11 3.0000E+09 | 4.413E+11 > 38 1.4455E+11 3.0000E+09 | 4.423E+11 > 39 1.5055E+11 3.0000E+09 | 4.433E+11 > 40 1.5655E+11 3.0000E+09 | 4.444E+11 > 41 1.6255E+11 3.0000E+09 | 4.454E+11 > 42 1.6855E+11 3.0000E+09 | 4.464E+11 > 43 1.7455E+11 3.0000E+09 | 4.474E+11 > 44 1.8055E+11 3.0000E+09 | 4.484E+11 > 45 1.8655E+11 3.0000E+09 | 4.495E+11 > 46 1.9255E+11 3.0000E+09 | 4.505E+11 > 47 1.9855E+11 3.0000E+09 | 4.516E+11 > 48 2.0455E+11 3.0000E+09 | 4.526E+11 > 49 2.1055E+11 3.0000E+09 | 4.537E+11 > 50 2.1655E+11 3.0000E+09 | 4.547E+11 > > > > > *here again the 3 different pc's* > > > 1) > Centos5 > Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 > i686 i686 i386 GNU/Linux > ifort intel compiler version 8.0 > hdf5 1.6.6 (serial) *(updated recently to 1.6.7)* > mpich2 Version: 1.0.3 > gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) > FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include > > 2) > Debian (??) > Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC 2005 > x86_64 GNU/Linux > pgf90 6.0-5 32-bit target on x86-64 Linux > hdf5 1.4.5 (serial) > mpich2 version 1.0.5 > gcc version 3.3.5 (Debian 1:3.3.5-13) > FFLAGS_OPT = -c -fast -r8 -i4 > > 3) > Ubuntu > Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux > ifort intel compiler version 10.1.012 > hdf5 1.6.7 (serial) > mpich2 Version: 1.0.6 > gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) > FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I > $(MPI_PATH)/include > or (only -ipo difference, result seemingly doesn't change) > FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include > > > > > > > Tomasz Plewa wrote: >> Seyit - >> >> Avoiding aggressive optimization is certainly good idea when it comes >> to debugging implementation. Alas, given you experience problems with >> 3 different compilers it is rather unlilkely that compilers are at >> fault. >> >> If your grid does not resolve structures correctly (persumably due to >> lax refinement criteria), the code may quite easily produce an >> unphysical state. But that should not happen during the initial grid >> generation. >> >> Any luck with a single block/single level runs? >> >> Tomek >> -- >> Seyit Hocuk wrote: >>> Hi, >>> >>> It has been some time now, but to continue the discussion, I have >>> tested Jeans standard problem and Sedov standard problem including >>> changing the refinement levels. The result is exactly same for all >>> three different computers. Unfortunately my own setup shows >>> differences and it seems the end result changes by the coice of >>> computer/environment/compiler whatever. My own setup has a lot more >>> physics than the simpe test problems. >>> >>> One thing I am thinking about is that I adjusted the refinement >>> criteria to use a reference density instead of looking at the second >>> order derivative. I don't know if this could be the problem, but >>> dt_hydro should not depend on the refinement criteria so I guess >>> there is more to it. >>> >>> Unfortunately it is not easy to change compilers and some other >>> stuff (like cpu/kernel etc.) on the different computers because I >>> have limited rights on those machines. I don't want to find out I >>> have to choose one machine and stick with it. How do I know which >>> machine is the best. Anyway, I am going to try at least to get the >>> libraries the same (hdf5 / mpi) for all machines and see if that >>> helps. Btw the latest hdf5 (1.8.1) doesn't seem to work with Flash2.5. >>> >>> I will also play with compiler options. Try to make them exactly the >>> same and avoid aggressive optimizations. I hope this is where the >>> solution lies. >>> >>> Thanks, >>> Seyit >>> >>> >>> >>> >>> Tomasz Plewa wrote: >>>> Although I prefer a low cost approach to debugging which avoids >>>> dealing with setup/system/compiler information at first and >>>> attempts to eliminate 99% of the source code, here are quick >>>> comments on your systems. >>>> >>>> (1) pgi 6.0 is a quite antique compiler. It's so old that now I do >>>> not even remember what problems we may have had with it. I suspect >>>> quite a few and none of them might be relevant... >>>> >>>> (2) As you might noticed, intel has a newer compiler version. Your >>>> CentOS5 system needs upgrades on both mpich2 and compiler sides. >>>> >>>> (3) Your Ubuntu system seems relatively up-to-date although HDF5 >>>> 1.8.x is out there and seems working fine. >>>> >>>> (4) Compilers are known to do strange things to perfect codes! So >>>> if you run into a trouble on a new system, one of the first things >>>> to try is to lower compiler optimization level. Avoiding -fast or >>>> -ip or -inline is usually a good idea. Simple -O0 may suffice for a >>>> couple of steps. As a bonus, the code compiles fast. Then the trick >>>> is to get it crashing quickly... >>>> >>>> Again, I suggest to simplify your application, isolate the problem, >>>> fix it, then find the best compiler, mpich, options, etc., to run >>>> your production code. >>>> >>>> Enjoy your weekend! >>>> >>>> Tomek >>>> -- >>>> Seyit Hocuk wrote: >>>>> Hi Nathan, >>>>> >>>>> Thats a good Idea, I'll start simple and test those first thing >>>>> monday. It's too late for me now. >>>>> >>>>> To answer your questions: >>>>> >>>>> The problem I am running is my own home made, actually it's a >>>>> modified version of Jeans. Basically including heat, cool, eos >>>>> gamma, compositions, adjusted refinement criteria similar to one >>>>> of the other problems (with delta_ref and delta_deref) and a scala >>>>> of other small things. >>>>> >>>>> 1) >>>>> Centos5 >>>>> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT >>>>> 2008 i686 i686 i386 GNU/Linux >>>>> ifort intel compiler version 8.0 >>>>> hdf5 1.6.6 (serial) >>>>> mpich2 Version: 1.0.3 >>>>> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>>>> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I >>>>> $(MPI_PATH)/include >>>>> >>>>> 2) >>>>> Debian (??) >>>>> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC >>>>> 2005 x86_64 GNU/Linux >>>>> pgf90 6.0-5 32-bit target on x86-64 Linux >>>>> hdf5 1.4.5 (serial) >>>>> mpich2 version 1.0.5 >>>>> gcc version 3.3.5 (Debian 1:3.3.5-13) >>>>> FFLAGS_OPT = -c -fast -r8 -i4 >>>>> >>>>> 3) >>>>> Ubuntu >>>>> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 >>>>> GNU/Linux >>>>> ifort intel compiler version 10.1.012 >>>>> hdf5 1.6.7 (serial) >>>>> mpich2 Version: 1.0.6 >>>>> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >>>>> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >>>>> $(MPI_PATH)/include >>>>> or (only -ipo difference, result seemingly doesn't change) >>>>> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >>>>> >>>>> >>>>> >>>>> >>>>> Nathan Hearn wrote: >>>>>> Hi Seyit, >>>>>> >>>>>> Are you seeing these differences in the standard test problems >>>>>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific >>>>>> to the >>>>>> problem that you are running. (Really, this follows from Tomek's >>>>>> advice to look at the simplest configurations where the differences >>>>>> are seen.) >>>>>> >>>>>> Also, it would be good to include some more information in your >>>>>> messages, for instance: the problem that you are running, the >>>>>> specific >>>>>> architectures and compilers, the optimization flags, and the library >>>>>> versions for HDF5 and MPI that you are using. Details like these >>>>>> can >>>>>> be really helpful in evaluating and fixing the issues you are >>>>>> seeing. >>>>>> >>>>>> >>>>>> - Nathan >>>>>> >>>>>> >>>>>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk >>>>>> wrote: >>>>>> >>>>>>> Oh man, >>>>>>> >>>>>>> I am using 3 different computers (with different architectures, >>>>>>> compilers, >>>>>>> optimizations, environments). All three start differently >>>>>>> immediately from >>>>>>> the beginning (difference in dt_hydro and refinement). >>>>>>> Should I worry very very very much? >>>>>>> I'm now doing a complete simulation and am going to check if my >>>>>>> end results >>>>>>> also change. >>>>>>> >>>>>>> I'm trembling by the thought..... >>>>>>> >>>>> >>>>> >>>> >>> > -------------- next part -------------- A non-text attachment was scrubbed... Name: tplewa.vcf Type: text/x-vcard Size: 297 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080730/2423e30d/attachment.vcf From tplewa at fsu.edu Wed Jul 30 05:57:59 2008 From: tplewa at fsu.edu (Tomasz Plewa) Date: Wed, 30 Jul 2008 06:57:59 -0400 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <489035F5.5020609@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> <488F5D72.1040305@fsu.edu> <48903429.5090007@astro.rug.nl> <489035F5.5020609@astro.rug.nl> Message-ID: <48904937.2080809@fsu.edu> Seyit - The "problem" is already present in a serial model. Also, I noticed that you are using very aggressive compiler optimizations. This frequently affects floating point accuracy. Can you produce a single processor, single patch models with no optimizations? Tomek -- Seyit Hocuk wrote: > BTW... The results also differ when I am using multiple cpu's instead > of 1. > > mpiexec -np 4 ./flash2 (on pc 3 instead of running just ./flash2) > Note: k,kJ,c0 etc. are the same > > > flash: initializing for jeans problem. > > kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 > > k = 1.756203682760181E-019 > kJ = 6.508776532382562E-019 > c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 > velamp = 64881.6252564676 > perturbation is unstable with growth time 8776137041619.21 > min_blocks 1 max_blocks 2 tot_blocks 5 > min_blocks 5 max_blocks 6 tot_blocks 21 > min_blocks 21 max_blocks 22 tot_blocks 85 > min_blocks 85 max_blocks 86 tot_blocks 341 > INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will > send 520 blocks per message. > [CHECKPOINT_WR] Writing checkpoint file jeans_r13vB_hdf5_chk_0000 > Progress: |... > *** Wrote output to jeans_r13vB_hdf5_chk_0000 ( 341 blocks ) *** > number of variables found for plotfile storage = 3 > dens pres temp > *** Wrote output to jeans_r13vB_hdf5_plt_cnt_0000 *** > n t dt | dt_hydro > 1 2.0000E+05 2.0000E+05 | 4.197E+11 > 2 6.0000E+05 4.0000E+05 | 4.197E+11 > 3 1.4000E+06 8.0000E+05 | 4.197E+11 > 4 3.0000E+06 1.6000E+06 | 4.197E+11 > 5 6.2000E+06 3.2000E+06 | 4.197E+11 > 6 1.2600E+07 6.4000E+06 | 4.197E+11 > 7 2.5400E+07 1.2800E+07 | 4.197E+11 > 8 5.1000E+07 2.5600E+07 | 4.197E+11 > 9 1.0220E+08 5.1200E+07 | 4.197E+11 > 10 2.0460E+08 1.0240E+08 | 4.198E+11 > 11 4.0940E+08 2.0480E+08 | 4.198E+11 > 12 8.1900E+08 4.0960E+08 | 4.198E+11 > 13 1.6382E+09 8.1920E+08 | 4.199E+11 > 14 3.2766E+09 1.6384E+09 | 4.201E+11 > 15 6.5534E+09 3.0000E+09 | 4.205E+11 > 16 1.2553E+10 3.0000E+09 | 4.213E+11 > 17 1.8553E+10 3.0000E+09 | 4.221E+11 > 18 2.4553E+10 3.0000E+09 | 4.229E+11 > 19 3.0553E+10 3.0000E+09 | 4.237E+11 > 20 3.6553E+10 3.0000E+09 | 4.246E+11 > 21 4.2553E+10 3.0000E+09 | 4.254E+11 > 22 4.8553E+10 3.0000E+09 | 4.263E+11 > 23 5.4553E+10 3.0000E+09 | 4.272E+11 > 24 6.0553E+10 3.0000E+09 | 4.281E+11 > 25 6.6553E+10 3.0000E+09 | 4.290E+11 > 26 7.2553E+10 3.0000E+09 | 4.300E+11 > 27 7.8553E+10 3.0000E+09 | 4.309E+11 > 28 8.4553E+10 3.0000E+09 | 4.319E+11 > 29 9.0553E+10 3.0000E+09 | 4.328E+11 > 30 9.6553E+10 3.0000E+09 | 4.338E+11 > 31 1.0255E+11 3.0000E+09 | 4.348E+11 > 32 1.0855E+11 3.0000E+09 | 4.358E+11 > 33 1.1455E+11 3.0000E+09 | 4.369E+11 > 34 1.2055E+11 3.0000E+09 | 4.380E+11 > 35 1.2655E+11 3.0000E+09 | 4.393E+11 > 36 1.3255E+11 3.0000E+09 | 4.405E+11 > 37 1.3855E+11 3.0000E+09 | 4.416E+11 > 38 1.4455E+11 3.0000E+09 | 4.429E+11 > 39 1.5055E+11 3.0000E+09 | 4.441E+11 > 40 1.5655E+11 3.0000E+09 | 4.454E+11 > 41 1.6255E+11 3.0000E+09 | 4.465E+11 > 42 1.6855E+11 3.0000E+09 | 4.478E+11 > 43 1.7455E+11 3.0000E+09 | 4.491E+11 > 44 1.8055E+11 3.0000E+09 | 4.503E+11 > 45 1.8655E+11 3.0000E+09 | 4.516E+11 > 46 1.9255E+11 3.0000E+09 | 4.529E+11 > 47 1.9855E+11 3.0000E+09 | 4.541E+11 > 48 2.0455E+11 3.0000E+09 | 4.554E+11 > 49 2.1055E+11 3.0000E+09 | 4.567E+11 > 50 2.1655E+11 3.0000E+09 | 4.576E+11 > > > > > Seyit Hocuk wrote: >> Hi Tomek good morning, >> >> Let me answer your first question. >> >> * /Any luck with a single block/single level runs? / >> >> Nope. Even single level (lref_min = lref_max) runs differ in >> dt_hydro. That means I don't use refinement at all. I guess it is >> also affecting the end results. I have to do a complete simulation >> again to be sure. But the initial files show small differences already. >> >> Here the initial results. >> Note 1: the differences in precision in k, kJ, c0 etc... >> Note 2: pc 1 and 3 are not that much differen, but grow more >> different later on. >> >> >> *pc 1* >> >> flash: initializing for jeans problem. >> >> kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 >> >> k = 1.756203682760181E-019 >> kJ = 6.508776532382562E-019 >> c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 >> velamp = 64881.6252564676 perturbation is unstable with growth >> time 8776137041619.21 min_blocks 5 max_blocks >> 5 tot_blocks 5 >> min_blocks 21 max_blocks 21 tot_blocks 21 >> min_blocks 85 max_blocks 85 tot_blocks 85 >> min_blocks 341 max_blocks 341 tot_blocks 341 >> INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will >> send 520 blocks per message. >> Warning! The HDF5 header files included by this application do not >> match the >> version used by the HDF5 library to which this application is linked. >> Data >> corruption or segmentation faults may occur if the application is >> allowed to continue. You can, at your own risk, disable this check >> by setting >> the environment variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. >> Setting it to 2 will suppress the warning totally. >> Headers are 1.6.6, library is 1.6.7 >> Bye... >> forrtl: info: Fortran error message number is 76. >> forrtl: warning: Could not open message catalog: ifcore_msg.cat. >> forrtl: info: Check environment variable NLSPATH and protection of >> /usr/lib/ifcore_msg.cat. >> Abort >> [1] + Done emacs flash.par >> [seyit at celsius objectT3]$ setenv HDF5_DISABLE_VERSION_CHECK 1 >> [seyit at celsius objectT3]$ ./flash2 >> >> flash: initializing for jeans problem. >> >> kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 >> >> k = 1.756203682760181E-019 >> kJ = 6.508776532382562E-019 >> c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 >> velamp = 64881.6252564676 perturbation is unstable with growth >> time 8776137041619.21 min_blocks 5 max_blocks >> 5 tot_blocks 5 >> min_blocks 21 max_blocks 21 tot_blocks 21 >> min_blocks 85 max_blocks 85 tot_blocks 85 >> min_blocks 341 max_blocks 341 tot_blocks 341 >> INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will >> send 520 blocks per message. >> [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 >> Progress: | >> *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) *** >> number of variables found for plotfile storage = 3 >> dens pres temp >> *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** >> n t dt | dt_hydro >> 1 2.0000E+05 2.0000E+05 | 4.240E+11 >> 2 6.0000E+05 4.0000E+05 | 4.240E+11 >> 3 1.4000E+06 8.0000E+05 | 4.240E+11 >> 4 3.0000E+06 1.6000E+06 | 4.240E+11 >> 5 6.2000E+06 3.2000E+06 | 4.240E+11 >> 6 1.2600E+07 6.4000E+06 | 4.240E+11 >> 7 2.5400E+07 1.2800E+07 | 4.240E+11 >> 8 5.1000E+07 2.5600E+07 | 4.240E+11 >> 9 1.0220E+08 5.1200E+07 | 4.240E+11 >> 10 2.0460E+08 1.0240E+08 | 4.240E+11 >> 11 4.0940E+08 2.0480E+08 | 4.241E+11 >> 12 8.1900E+08 4.0960E+08 | 4.241E+11 >> 13 1.6382E+09 8.1920E+08 | 4.241E+11 >> 14 3.2766E+09 1.6384E+09 | 4.242E+11 >> 15 6.5534E+09 3.0000E+09 | 4.243E+11 >> 16 1.2553E+10 3.0000E+09 | 4.247E+11 >> 17 1.8553E+10 3.0000E+09 | 4.250E+11 >> 18 2.4553E+10 3.0000E+09 | 4.255E+11 >> 19 3.0553E+10 3.0000E+09 | 4.259E+11 >> 20 3.6553E+10 3.0000E+09 | 4.266E+11 >> 21 4.2553E+10 3.0000E+09 | 4.273E+11 >> 22 4.8553E+10 3.0000E+09 | 4.281E+11 >> 23 5.4553E+10 3.0000E+09 | 4.288E+11 >> 24 6.0553E+10 3.0000E+09 | 4.296E+11 >> 25 6.6553E+10 3.0000E+09 | 4.304E+11 >> 26 7.2553E+10 3.0000E+09 | 4.313E+11 >> 27 7.8553E+10 3.0000E+09 | 4.321E+11 >> 28 8.4553E+10 3.0000E+09 | 4.330E+11 >> 29 9.0553E+10 3.0000E+09 | 4.339E+11 >> 30 9.6553E+10 3.0000E+09 | 4.348E+11 >> 31 1.0255E+11 3.0000E+09 | 4.357E+11 >> 32 1.0855E+11 3.0000E+09 | 4.366E+11 >> 33 1.1455E+11 3.0000E+09 | 4.301E+11 >> 34 1.2055E+11 3.0000E+09 | 3.803E+11 >> 35 1.2655E+11 3.0000E+09 | 3.379E+11 >> 36 1.3255E+11 3.0000E+09 | 3.057E+11 >> 37 1.3855E+11 3.0000E+09 | 2.868E+11 >> 38 1.4455E+11 3.0000E+09 | 2.876E+11 >> 39 1.5055E+11 3.0000E+09 | 2.880E+11 >> 40 1.5655E+11 3.0000E+09 | 2.903E+11 >> 41 1.6255E+11 3.0000E+09 | 2.936E+11 >> 42 1.6855E+11 3.0000E+09 | 2.978E+11 >> 43 1.7455E+11 3.0000E+09 | 4.475E+11 >> 44 1.8055E+11 3.0000E+09 | 4.486E+11 >> 45 1.8655E+11 3.0000E+09 | 4.496E+11 >> 46 1.9255E+11 3.0000E+09 | 4.507E+11 >> 47 1.9855E+11 3.0000E+09 | 4.517E+11 >> 48 2.0455E+11 3.0000E+09 | 4.528E+11 >> 49 2.1055E+11 3.0000E+09 | 4.538E+11 >> 50 2.1655E+11 3.0000E+09 | 4.549E+11 >> >> >> >> *pc 2* >> >> flash: initializing for jeans problem. >> >> kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 >> >> k = 1.7562036827601810E-019 >> kJ = 6.5087765323825604E-019 >> c0 = 181807.2941759182 omega^2 = -1.2983542608611234E-026 >> velamp = 64881.62525646757 perturbation is unstable with >> growth time 8776137041619.213 min_blocks 5 >> max_blocks 5 tot_blocks 5 >> min_blocks 21 max_blocks 21 tot_blocks 21 >> min_blocks 85 max_blocks 85 tot_blocks 85 >> min_blocks 341 max_blocks 341 tot_blocks 341 >> INITIAL TIMESTEP = 100000.0000000000 [CHECKPOINT_WR] NOTE: >> will send 520 blocks per message. >> [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 >> Progress: | >> *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) >> *** >> number of variables found for plotfile storage = 3 >> dens pres temp >> *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** >> n t dt | dt_hydro >> 1 2.0000E+05 2.0000E+05 | 4.406E+11 >> 2 6.0000E+05 4.0000E+05 | 4.406E+11 >> 3 1.4000E+06 8.0000E+05 | 4.406E+11 >> 4 3.0000E+06 1.6000E+06 | 4.406E+11 >> 5 6.2000E+06 3.2000E+06 | 4.406E+11 >> 6 1.2600E+07 6.4000E+06 | 4.406E+11 >> 7 2.5400E+07 1.2800E+07 | 4.406E+11 >> 8 5.1000E+07 2.5600E+07 | 4.406E+11 >> 9 1.0220E+08 5.1200E+07 | 4.406E+11 >> 10 2.0460E+08 1.0240E+08 | 4.405E+11 >> 11 4.0940E+08 2.0480E+08 | 4.404E+11 >> 12 8.1900E+08 4.0960E+08 | 4.403E+11 >> 13 1.6382E+09 8.1920E+08 | 4.399E+11 >> 14 3.2766E+09 1.6384E+09 | 4.393E+11 >> 15 6.5534E+09 3.0000E+09 | 4.381E+11 >> 16 1.2553E+10 3.0000E+09 | 4.365E+11 >> 17 1.8553E+10 3.0000E+09 | 4.353E+11 >> 18 2.4553E+10 3.0000E+09 | 4.345E+11 >> 19 3.0553E+10 3.0000E+09 | 4.340E+11 >> 20 3.6553E+10 3.0000E+09 | 4.337E+11 >> 21 4.2553E+10 3.0000E+09 | 4.337E+11 >> 22 4.8553E+10 3.0000E+09 | 4.339E+11 >> 23 5.4553E+10 3.0000E+09 | 4.343E+11 >> 24 6.0553E+10 3.0000E+09 | 4.348E+11 >> 25 6.6553E+10 3.0000E+09 | 4.355E+11 >> 26 7.2553E+10 3.0000E+09 | 4.363E+11 >> 27 7.8553E+10 3.0000E+09 | 4.371E+11 >> 28 8.4553E+10 3.0000E+09 | 4.382E+11 >> 29 9.0553E+10 3.0000E+09 | 4.393E+11 >> 30 9.6553E+10 3.0000E+09 | 4.405E+11 >> 31 1.0255E+11 3.0000E+09 | 4.418E+11 >> 32 1.0855E+11 3.0000E+09 | 4.431E+11 >> 33 1.1455E+11 3.0000E+09 | 4.447E+11 >> 34 1.2055E+11 3.0000E+09 | 4.463E+11 >> 35 1.2655E+11 3.0000E+09 | 4.479E+11 >> 36 1.3255E+11 3.0000E+09 | 4.496E+11 >> 37 1.3855E+11 3.0000E+09 | 4.513E+11 >> 38 1.4455E+11 3.0000E+09 | 4.531E+11 >> 39 1.5055E+11 3.0000E+09 | 4.550E+11 >> 40 1.5655E+11 3.0000E+09 | 4.570E+11 >> 41 1.6255E+11 3.0000E+09 | 4.590E+11 >> 42 1.6855E+11 3.0000E+09 | 4.611E+11 >> 43 1.7455E+11 3.0000E+09 | 4.632E+11 >> 44 1.8055E+11 3.0000E+09 | 4.653E+11 >> 45 1.8655E+11 3.0000E+09 | 4.675E+11 >> 46 1.9255E+11 3.0000E+09 | 4.697E+11 >> 47 1.9855E+11 3.0000E+09 | 4.719E+11 >> 48 2.0455E+11 3.0000E+09 | 4.733E+11 >> 49 2.1055E+11 3.0000E+09 | 4.734E+11 >> 50 2.1655E+11 3.0000E+09 | 4.736E+11 >> >> >> >> *pc 3* >> >> >> flash: initializing for jeans problem. >> >> kx = 7.8539816E-20 ky = 1.5707963E-19 kz = 0.0000000E+00 >> >> k = 1.756203682760181E-019 >> kJ = 6.508776532382562E-019 >> c0 = 181807.294175918 omega^2 = -1.298354260861123E-026 >> velamp = 64881.6252564676 perturbation is unstable with growth >> time 8776137041619.21 min_blocks 5 max_blocks >> 5 tot_blocks 5 >> min_blocks 21 max_blocks 21 tot_blocks 21 >> min_blocks 85 max_blocks 85 tot_blocks 85 >> min_blocks 341 max_blocks 341 tot_blocks 341 >> INITIAL TIMESTEP = 100000.000000000 [CHECKPOINT_WR] NOTE: will >> send 520 blocks per message. >> [CHECKPOINT_WR] Writing checkpoint file jeans_r13vA_hdf5_chk_0000 >> Progress: | >> *** Wrote output to jeans_r13vA_hdf5_chk_0000 ( 341 blocks ) *** >> number of variables found for plotfile storage = 3 >> dens pres temp >> *** Wrote output to jeans_r13vA_hdf5_plt_cnt_0000 *** >> n t dt | dt_hydro >> 1 2.0000E+05 2.0000E+05 | 4.240E+11 >> 2 6.0000E+05 4.0000E+05 | 4.240E+11 >> 3 1.4000E+06 8.0000E+05 | 4.240E+11 >> 4 3.0000E+06 1.6000E+06 | 4.240E+11 >> 5 6.2000E+06 3.2000E+06 | 4.240E+11 >> 6 1.2600E+07 6.4000E+06 | 4.240E+11 >> 7 2.5400E+07 1.2800E+07 | 4.240E+11 >> 8 5.1000E+07 2.5600E+07 | 4.240E+11 >> 9 1.0220E+08 5.1200E+07 | 4.240E+11 >> 10 2.0460E+08 1.0240E+08 | 4.240E+11 >> 11 4.0940E+08 2.0480E+08 | 4.241E+11 >> 12 8.1900E+08 4.0960E+08 | 4.241E+11 >> 13 1.6382E+09 8.1920E+08 | 4.241E+11 >> 14 3.2766E+09 1.6384E+09 | 4.242E+11 >> 15 6.5534E+09 3.0000E+09 | 4.244E+11 >> 16 1.2553E+10 3.0000E+09 | 4.247E+11 >> 17 1.8553E+10 3.0000E+09 | 4.251E+11 >> 18 2.4553E+10 3.0000E+09 | 4.255E+11 >> 19 3.0553E+10 3.0000E+09 | 4.259E+11 >> 20 3.6553E+10 3.0000E+09 | 4.266E+11 >> 21 4.2553E+10 3.0000E+09 | 4.273E+11 >> 22 4.8553E+10 3.0000E+09 | 4.281E+11 >> 23 5.4553E+10 3.0000E+09 | 4.288E+11 >> 24 6.0553E+10 3.0000E+09 | 4.296E+11 >> 25 6.6553E+10 3.0000E+09 | 4.304E+11 >> 26 7.2553E+10 3.0000E+09 | 4.313E+11 >> 27 7.8553E+10 3.0000E+09 | 4.321E+11 >> 28 8.4553E+10 3.0000E+09 | 4.330E+11 >> 29 9.0553E+10 3.0000E+09 | 4.338E+11 >> 30 9.6553E+10 3.0000E+09 | 4.347E+11 >> 31 1.0255E+11 3.0000E+09 | 4.356E+11 >> 32 1.0855E+11 3.0000E+09 | 4.366E+11 >> 33 1.1455E+11 3.0000E+09 | 4.375E+11 >> 34 1.2055E+11 3.0000E+09 | 4.384E+11 >> 35 1.2655E+11 3.0000E+09 | 4.394E+11 >> 36 1.3255E+11 3.0000E+09 | 4.404E+11 >> 37 1.3855E+11 3.0000E+09 | 4.413E+11 >> 38 1.4455E+11 3.0000E+09 | 4.423E+11 >> 39 1.5055E+11 3.0000E+09 | 4.433E+11 >> 40 1.5655E+11 3.0000E+09 | 4.444E+11 >> 41 1.6255E+11 3.0000E+09 | 4.454E+11 >> 42 1.6855E+11 3.0000E+09 | 4.464E+11 >> 43 1.7455E+11 3.0000E+09 | 4.474E+11 >> 44 1.8055E+11 3.0000E+09 | 4.484E+11 >> 45 1.8655E+11 3.0000E+09 | 4.495E+11 >> 46 1.9255E+11 3.0000E+09 | 4.505E+11 >> 47 1.9855E+11 3.0000E+09 | 4.516E+11 >> 48 2.0455E+11 3.0000E+09 | 4.526E+11 >> 49 2.1055E+11 3.0000E+09 | 4.537E+11 >> 50 2.1655E+11 3.0000E+09 | 4.547E+11 >> >> >> >> >> *here again the 3 different pc's* >> >> >> 1) >> Centos5 >> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 >> i686 i686 i386 GNU/Linux >> ifort intel compiler version 8.0 >> hdf5 1.6.6 (serial) *(updated recently to 1.6.7)* >> mpich2 Version: 1.0.3 >> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I $(MPI_PATH)/include >> >> 2) >> Debian (??) >> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC >> 2005 x86_64 GNU/Linux >> pgf90 6.0-5 32-bit target on x86-64 Linux >> hdf5 1.4.5 (serial) >> mpich2 version 1.0.5 >> gcc version 3.3.5 (Debian 1:3.3.5-13) >> FFLAGS_OPT = -c -fast -r8 -i4 >> >> 3) >> Ubuntu >> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux >> ifort intel compiler version 10.1.012 >> hdf5 1.6.7 (serial) >> mpich2 Version: 1.0.6 >> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >> $(MPI_PATH)/include >> or (only -ipo difference, result seemingly doesn't change) >> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >> >> >> >> >> >> >> Tomasz Plewa wrote: >>> Seyit - >>> >>> Avoiding aggressive optimization is certainly good idea when it >>> comes to debugging implementation. Alas, given you experience >>> problems with 3 different compilers it is rather unlilkely that >>> compilers are at fault. >>> >>> If your grid does not resolve structures correctly (persumably due >>> to lax refinement criteria), the code may quite easily produce an >>> unphysical state. But that should not happen during the initial grid >>> generation. >>> >>> Any luck with a single block/single level runs? >>> >>> Tomek >>> -- >>> Seyit Hocuk wrote: >>>> Hi, >>>> >>>> It has been some time now, but to continue the discussion, I have >>>> tested Jeans standard problem and Sedov standard problem including >>>> changing the refinement levels. The result is exactly same for all >>>> three different computers. Unfortunately my own setup shows >>>> differences and it seems the end result changes by the coice of >>>> computer/environment/compiler whatever. My own setup has a lot more >>>> physics than the simpe test problems. >>>> >>>> One thing I am thinking about is that I adjusted the refinement >>>> criteria to use a reference density instead of looking at the >>>> second order derivative. I don't know if this could be the problem, >>>> but dt_hydro should not depend on the refinement criteria so I >>>> guess there is more to it. >>>> >>>> Unfortunately it is not easy to change compilers and some other >>>> stuff (like cpu/kernel etc.) on the different computers because I >>>> have limited rights on those machines. I don't want to find out I >>>> have to choose one machine and stick with it. How do I know which >>>> machine is the best. Anyway, I am going to try at least to get the >>>> libraries the same (hdf5 / mpi) for all machines and see if that >>>> helps. Btw the latest hdf5 (1.8.1) doesn't seem to work with Flash2.5. >>>> >>>> I will also play with compiler options. Try to make them exactly >>>> the same and avoid aggressive optimizations. I hope this is where >>>> the solution lies. >>>> >>>> Thanks, >>>> Seyit >>>> >>>> >>>> >>>> >>>> Tomasz Plewa wrote: >>>>> Although I prefer a low cost approach to debugging which avoids >>>>> dealing with setup/system/compiler information at first and >>>>> attempts to eliminate 99% of the source code, here are quick >>>>> comments on your systems. >>>>> >>>>> (1) pgi 6.0 is a quite antique compiler. It's so old that now I do >>>>> not even remember what problems we may have had with it. I suspect >>>>> quite a few and none of them might be relevant... >>>>> >>>>> (2) As you might noticed, intel has a newer compiler version. Your >>>>> CentOS5 system needs upgrades on both mpich2 and compiler sides. >>>>> >>>>> (3) Your Ubuntu system seems relatively up-to-date although HDF5 >>>>> 1.8.x is out there and seems working fine. >>>>> >>>>> (4) Compilers are known to do strange things to perfect codes! So >>>>> if you run into a trouble on a new system, one of the first things >>>>> to try is to lower compiler optimization level. Avoiding -fast or >>>>> -ip or -inline is usually a good idea. Simple -O0 may suffice for >>>>> a couple of steps. As a bonus, the code compiles fast. Then the >>>>> trick is to get it crashing quickly... >>>>> >>>>> Again, I suggest to simplify your application, isolate the >>>>> problem, fix it, then find the best compiler, mpich, options, >>>>> etc., to run your production code. >>>>> >>>>> Enjoy your weekend! >>>>> >>>>> Tomek >>>>> -- >>>>> Seyit Hocuk wrote: >>>>>> Hi Nathan, >>>>>> >>>>>> Thats a good Idea, I'll start simple and test those first thing >>>>>> monday. It's too late for me now. >>>>>> >>>>>> To answer your questions: >>>>>> >>>>>> The problem I am running is my own home made, actually it's a >>>>>> modified version of Jeans. Basically including heat, cool, eos >>>>>> gamma, compositions, adjusted refinement criteria similar to one >>>>>> of the other problems (with delta_ref and delta_deref) and a >>>>>> scala of other small things. >>>>>> >>>>>> 1) >>>>>> Centos5 >>>>>> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT >>>>>> 2008 i686 i686 i386 GNU/Linux >>>>>> ifort intel compiler version 8.0 >>>>>> hdf5 1.6.6 (serial) >>>>>> mpich2 Version: 1.0.3 >>>>>> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>>>>> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I >>>>>> $(MPI_PATH)/include >>>>>> >>>>>> 2) >>>>>> Debian (??) >>>>>> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC >>>>>> 2005 x86_64 GNU/Linux >>>>>> pgf90 6.0-5 32-bit target on x86-64 Linux >>>>>> hdf5 1.4.5 (serial) >>>>>> mpich2 version 1.0.5 >>>>>> gcc version 3.3.5 (Debian 1:3.3.5-13) >>>>>> FFLAGS_OPT = -c -fast -r8 -i4 >>>>>> >>>>>> 3) >>>>>> Ubuntu >>>>>> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 >>>>>> GNU/Linux >>>>>> ifort intel compiler version 10.1.012 >>>>>> hdf5 1.6.7 (serial) >>>>>> mpich2 Version: 1.0.6 >>>>>> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >>>>>> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >>>>>> $(MPI_PATH)/include >>>>>> or (only -ipo difference, result seemingly doesn't change) >>>>>> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Nathan Hearn wrote: >>>>>>> Hi Seyit, >>>>>>> >>>>>>> Are you seeing these differences in the standard test problems >>>>>>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific >>>>>>> to the >>>>>>> problem that you are running. (Really, this follows from Tomek's >>>>>>> advice to look at the simplest configurations where the differences >>>>>>> are seen.) >>>>>>> >>>>>>> Also, it would be good to include some more information in your >>>>>>> messages, for instance: the problem that you are running, the >>>>>>> specific >>>>>>> architectures and compilers, the optimization flags, and the >>>>>>> library >>>>>>> versions for HDF5 and MPI that you are using. Details like >>>>>>> these can >>>>>>> be really helpful in evaluating and fixing the issues you are >>>>>>> seeing. >>>>>>> >>>>>>> >>>>>>> - Nathan >>>>>>> >>>>>>> >>>>>>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk >>>>>>> wrote: >>>>>>> >>>>>>>> Oh man, >>>>>>>> >>>>>>>> I am using 3 different computers (with different architectures, >>>>>>>> compilers, >>>>>>>> optimizations, environments). All three start differently >>>>>>>> immediately from >>>>>>>> the beginning (difference in dt_hydro and refinement). >>>>>>>> Should I worry very very very much? >>>>>>>> I'm now doing a complete simulation and am going to check if my >>>>>>>> end results >>>>>>>> also change. >>>>>>>> >>>>>>>> I'm trembling by the thought..... >>>>>>>> >>>>>> >>>>>> >>>>> >>>> >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: tplewa.vcf Type: text/x-vcard Size: 297 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080730/038a1f3f/attachment.vcf From seyit at astro.rug.nl Wed Jul 30 07:18:47 2008 From: seyit at astro.rug.nl (Seyit Hocuk) Date: Wed, 30 Jul 2008 14:18:47 +0200 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <48904937.2080809@fsu.edu> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> <488F5D72.1040305@fsu.edu> <48903429.5090007@astro.rug.nl> <489035F5.5020609@astro.rug.nl> <48904937.2080809@fsu.edu> Message-ID: <48905C27.3090506@astro.rug.nl> Tomek, I tried using only " -c -r8 -i4 -I $(MPI_PATH)/include " optimizations on all with single proc, single refinement level. The differences are still there. Including -I $(MPI_PATH)/include has no effect (though pc 1 does not work without it). Is this still agressive optimization? I always need -c otherwise I get errors compiling. I need -r8 otherwise while running I get errors. I guess i4 in not so necessary, but I thought it wouldn't hurt. It also makes no difference by the way in my simulations. Oh and the differences in calculations of k, kJ, c0, omega ... etc. is purely precision. Pc 1 and pc 3 have same values, but pc 2 has one more digit to them. Tomasz Plewa wrote: > Seyit - > > The "problem" is already present in a serial model. > > Also, I noticed that you are using very aggressive compiler > optimizations. This frequently affects floating point accuracy. Can > you produce a single processor, single patch models with no > optimizations? > > Tomek > -- From tplewa at fsu.edu Wed Jul 30 07:31:46 2008 From: tplewa at fsu.edu (Tomasz Plewa) Date: Wed, 30 Jul 2008 08:31:46 -0400 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <48905C27.3090506@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> <488F5D72.1040305@fsu.edu> <48903429.5090007@astro.rug.nl> <489035F5.5020609@astro.rug.nl> <48904937.2080809@fsu.edu> <48905C27.3090506@astro.rug.nl> Message-ID: <48905F32.5030707@fsu.edu> Seyit - Neither of -r8, -i4, -I, or -c option has much to do with optimizations. Please take a look into compiler manual pages. I noticed the difference in formatting the output, which is inconsequential for your problem and likely reflects a difference in implementation of Fortran standard. At the moment I do not see any strong evidence that there is a serious problem with either the code or your computer systems. I believe you can do some good science with it. Tomek -- Seyit Hocuk wrote: > Tomek, > > I tried using only " -c -r8 -i4 -I $(MPI_PATH)/include " optimizations > on all with single proc, single refinement level. The differences are > still there. Including -I $(MPI_PATH)/include has no effect (though > pc 1 does not work without it). Is this still agressive optimization? > I always need -c otherwise I get errors compiling. I need -r8 > otherwise while running I get errors. I guess i4 in not so necessary, > but I thought it wouldn't hurt. It also makes no difference by the way > in my simulations. > > Oh and the differences in calculations of k, kJ, c0, omega ... etc. is > purely precision. Pc 1 and pc 3 have same values, but pc 2 has one > more digit to them. > > > Tomasz Plewa wrote: >> Seyit - >> >> The "problem" is already present in a serial model. >> >> Also, I noticed that you are using very aggressive compiler >> optimizations. This frequently affects floating point accuracy. Can >> you produce a single processor, single patch models with no >> optimizations? >> >> Tomek >> -- > -------------- next part -------------- A non-text attachment was scrubbed... Name: tplewa.vcf Type: text/x-vcard Size: 297 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080730/06991d71/attachment.vcf From nhearn at uchicago.edu Wed Jul 30 09:05:50 2008 From: nhearn at uchicago.edu (Nathan Hearn) Date: Wed, 30 Jul 2008 09:05:50 -0500 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <489038FC.2030005@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> <488F7953.4010802@flash.uchicago.edu> <2467fdc0807291324t5368140ax904909774b1bf582@mail.gmail.com> <489038FC.2030005@astro.rug.nl> Message-ID: <2467fdc0807300705n71a85d08gabfc731daee9c955@mail.gmail.com> Hi Seyit, Forgive me if I am asking a question that has already been addressed, but is it possible that there are uninitialized variables that are being used in your code? It is certainly possible that they could yield different results with each change in compiler, library version, and optimization level. If you haven't done so already, you might want to look through the output of your first checkpoint file (using hdfview or h5dump) to see if any of the datasets have unexpected values. You could also see if your compiler has an option to automatically initialize variables when Flash is run (this should not be used to fix the problem, but rather, to expose it); if possible, see if you can automatically initialize all floating point data to NaN. - Nathan On Wed, Jul 30, 2008 at 4:48 AM, Seyit Hocuk wrote: > Ah thanks Nathan, > > I see there is a good link to the api bindings. That answers the question I > posted a minute ago I guess. > I think I just have to configure hdf5 with a certain flag. " > |--with-default-api-version=v16 |" > > Let me try that out. > > Greetz, > Seyit From richp at flash.uchicago.edu Wed Jul 30 11:38:30 2008 From: richp at flash.uchicago.edu (Paul M. Rich) Date: Wed, 30 Jul 2008 11:38:30 -0500 Subject: [FLASH-USERS] Big Problem?? In-Reply-To: <489037E3.1050400@astro.rug.nl> References: <48441DC4.9050701@astro.rug.nl> <4846C4EE.4040204@astro.rug.nl> <4846CD5A.7090007@flash.uchicago.edu> <4846DF1D.7060705@astro.rug.nl> <20080604144430.h5v67w0v9cw488ks@webmail.scs.fsu.edu> <487C9D03.90800@astro.rug.nl> <2467fdc0807150724m76ed7d35g9b46fab6e669916a@mail.gmail.com> <487CD69C.1060403@astro.rug.nl> <487CE1DC.8080505@flash.uchicago.edu> <4880C5A5.50708@astro.rug.nl> <2467fdc0807181010q52dcbecft6793e7c00dcd1518@mail.gmail.com> <4880D78C.4020708@astro.rug.nl> <4880DC19.7070205@scs.fsu.edu> <488F58E1.9050306@astro.rug.nl> <488F7953.4010802@flash.uchicago.edu> <489037E3.1050400@astro.rug.nl> Message-ID: <48909906.7080805@flash.uchicago.edu> Seyit, These bindings are disabled by default and can be enabled by passing --with-default-api-version=v16 to configure, according to the documentation at http://hdf.ncsa.uiuc.edu/HDF5/doc/RM/APICompatMacros.html. Good luck, Paul Seyit Hocuk wrote: > Hi Paul and good morning, > > Thanks for your reply. To answer your question " Is the version of > HDF5 1.8.1 you are using built with the 1.6 api bindings? ", I don't > know. I didn't check that. I will try that, when I know how to do > that. I just downloaded the source file and did configure-make-make > install. Where does the bindings come in? I will try to find it out. > > Greetz, > Seyit > > > > > Paul M. Rich wrote: >> Seyit, >> >> I can clear up the HDF5 1.8 issue, I think. I've experimented with >> HDF5 1.8.1 in FLASH3 and that will only work if you have built the >> v1.8.x implementation with support for the 1.6 api. I believe that >> FLASH2.5's will work in a similar fashion. Is the version of HDF5 >> 1.8.1 you are using built with the 1.6 api bindings? >> >> Paul Rich >> >> Seyit Hocuk wrote: >>> Hi, >>> >>> It has been some time now, but to continue the discussion, I have >>> tested Jeans standard problem and Sedov standard problem including >>> changing the refinement levels. The result is exactly same for all >>> three different computers. Unfortunately my own setup shows >>> differences and it seems the end result changes by the coice of >>> computer/environment/compiler whatever. My own setup has a lot more >>> physics than the simpe test problems. >>> >>> One thing I am thinking about is that I adjusted the refinement >>> criteria to use a reference density instead of looking at the second >>> order derivative. I don't know if this could be the problem, but >>> dt_hydro should not depend on the refinement criteria so I guess >>> there is more to it. >>> >>> Unfortunately it is not easy to change compilers and some other >>> stuff (like cpu/kernel etc.) on the different computers because I >>> have limited rights on those machines. I don't want to find out I >>> have to choose one machine and stick with it. How do I know which >>> machine is the best. Anyway, I am going to try at least to get the >>> libraries the same (hdf5 / mpi) for all machines and see if that >>> helps. Btw the latest hdf5 (1.8.1) doesn't seem to work with Flash2.5. >>> >>> I will also play with compiler options. Try to make them exactly the >>> same and avoid aggressive optimizations. I hope this is where the >>> solution lies. >>> >>> Thanks, >>> Seyit >>> >>> >>> >>> >>> Tomasz Plewa wrote: >>>> Although I prefer a low cost approach to debugging which avoids >>>> dealing with setup/system/compiler information at first and >>>> attempts to eliminate 99% of the source code, here are quick >>>> comments on your systems. >>>> >>>> (1) pgi 6.0 is a quite antique compiler. It's so old that now I do >>>> not even remember what problems we may have had with it. I suspect >>>> quite a few and none of them might be relevant... >>>> >>>> (2) As you might noticed, intel has a newer compiler version. Your >>>> CentOS5 system needs upgrades on both mpich2 and compiler sides. >>>> >>>> (3) Your Ubuntu system seems relatively up-to-date although HDF5 >>>> 1.8.x is out there and seems working fine. >>>> >>>> (4) Compilers are known to do strange things to perfect codes! So >>>> if you run into a trouble on a new system, one of the first things >>>> to try is to lower compiler optimization level. Avoiding -fast or >>>> -ip or -inline is usually a good idea. Simple -O0 may suffice for a >>>> couple of steps. As a bonus, the code compiles fast. Then the trick >>>> is to get it crashing quickly... >>>> >>>> Again, I suggest to simplify your application, isolate the problem, >>>> fix it, then find the best compiler, mpich, options, etc., to run >>>> your production code. >>>> >>>> Enjoy your weekend! >>>> >>>> Tomek >>>> -- >>>> Seyit Hocuk wrote: >>>>> Hi Nathan, >>>>> >>>>> Thats a good Idea, I'll start simple and test those first thing >>>>> monday. It's too late for me now. >>>>> >>>>> To answer your questions: >>>>> >>>>> The problem I am running is my own home made, actually it's a >>>>> modified version of Jeans. Basically including heat, cool, eos >>>>> gamma, compositions, adjusted refinement criteria similar to one >>>>> of the other problems (with delta_ref and delta_deref) and a scala >>>>> of other small things. >>>>> >>>>> 1) >>>>> Centos5 >>>>> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT >>>>> 2008 i686 i686 i386 GNU/Linux >>>>> ifort intel compiler version 8.0 >>>>> hdf5 1.6.6 (serial) >>>>> mpich2 Version: 1.0.3 >>>>> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>>>> FFLAGS_OPT = -c -r8 -i4 -fast -ipo -ipo_obj -I >>>>> $(MPI_PATH)/include >>>>> >>>>> 2) >>>>> Debian (??) >>>>> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC >>>>> 2005 x86_64 GNU/Linux >>>>> pgf90 6.0-5 32-bit target on x86-64 Linux >>>>> hdf5 1.4.5 (serial) >>>>> mpich2 version 1.0.5 >>>>> gcc version 3.3.5 (Debian 1:3.3.5-13) >>>>> FFLAGS_OPT = -c -fast -r8 -i4 >>>>> >>>>> 3) >>>>> Ubuntu >>>>> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 >>>>> GNU/Linux >>>>> ifort intel compiler version 10.1.012 >>>>> hdf5 1.6.7 (serial) >>>>> mpich2 Version: 1.0.6 >>>>> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) >>>>> FFLAGS_OPT = -c -r8 -i4 -xT -O3 -no-prec-div -static -I >>>>> $(MPI_PATH)/include >>>>> or (only -ipo difference, result seemingly doesn't change) >>>>> FFLAGS_OPT = -c -r8 -i4 -fast -I $(MPI_PATH)/include >>>>> >>>>> >>>>> >>>>> >>>>> Nathan Hearn wrote: >>>>>> Hi Seyit, >>>>>> >>>>>> Are you seeing these differences in the standard test problems >>>>>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific >>>>>> to the >>>>>> problem that you are running. (Really, this follows from Tomek's >>>>>> advice to look at the simplest configurations where the differences >>>>>> are seen.) >>>>>> >>>>>> Also, it would be good to include some more information in your >>>>>> messages, for instance: the problem that you are running, the >>>>>> specific >>>>>> architectures and compilers, the optimization flags, and the library >>>>>> versions for HDF5 and MPI that you are using. Details like these >>>>>> can >>>>>> be really helpful in evaluating and fixing the issues you are >>>>>> seeing. >>>>>> >>>>>> >>>>>> - Nathan >>>>>> >>>>>> >>>>>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk >>>>>> wrote: >>>>>> >>>>>>> Oh man, >>>>>>> >>>>>>> I am using 3 different computers (with different architectures, >>>>>>> compilers, >>>>>>> optimizations, environments). All three start differently >>>>>>> immediately from >>>>>>> the beginning (difference in dt_hydro and refinement). >>>>>>> Should I worry very very very much? >>>>>>> I'm now doing a complete simulation and am going to check if my >>>>>>> end results >>>>>>> also change. >>>>>>> >>>>>>> I'm trembling by the thought..... >>>>>>> >>>>> >>>>> >>>> >> From bernalcg at astroscu.unam.mx Wed Jul 9 17:24:49 2008 From: bernalcg at astroscu.unam.mx (Cristian Giovanny Bernal) Date: Wed, 09 Jul 2008 22:24:49 -0000 Subject: [FLASH-USERS] conserve variable... Message-ID: <20080709215102.M84068@astroscu.unam.mx> Hi people, I have a question. The module MHD of Flash2.5 conserve the variables (like the mass)? I am running a simulation with a rate of injection of mass constant by one of the boundaries (Yr). The other boundaries are periodic (Xl, Xr) and reflecting (Yl). I have two setups (HYDRO & MHD) with exactly the same physic and boundary condition. Nevertheless, when I plotting the flash.dat (using 1:2 "mass Vs time"), I see that HYDRO_setup conserve the mass in the dominion (see the attached plot), but MHD not. I do not why! The magnetic field affect the conservation of the variables? I set the conserved_var = .true. in the flash par, but the result is the same. I see that the dens_profile_MHD is like HYDRO, but with less mass... the injection stays (the red is a analitical solution in the plot). The physics used are: gravity (planepar), EOS (Helmholtz), neutrinoloss (a modified version of radloss), Hydro and MHD. Cartesian coordinates are used in 2D. any idea??? The FLASH3.0 also suffers from this? very thanks for your help... -- Instituto de Astronomia Universidad Nacional Autonoma de Mexico (UNAM) http://www.astroscu.unam.mx -------------- next part -------------- A non-text attachment was scrubbed... Name: mass_hydro_10E28.ps Type: application/postscript Size: 2231864 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080709/f105fb0f/attachment.ps -------------- next part -------------- A non-text attachment was scrubbed... Name: mass_mhd_10E28.ps Type: application/postscript Size: 2416020 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080709/f105fb0f/attachment-0001.ps -------------- next part -------------- A non-text attachment was scrubbed... Name: dens_cheva_hydro_mhd_bucle.gif Type: image/gif Size: 357427 bytes Desc: not available Url : http://flash.uchicago.edu/pipermail/flash-users/attachments/20080709/f105fb0f/attachment.gif