From miller at astro.umd.edu Mon Nov 5 15:13:20 2007 From: miller at astro.umd.edu (Cole Miller) Date: Mon, 5 Nov 2007 16:13:20 -0500 (EST) Subject: [FLASH-USERS] Undefined symbol in FLASH 3.0beta Message-ID: Hi all, I've downloaded FLASH 3.0beta, and although it compiles I have not been able to get the HDF5 version to run (the system details are below). I've compiled this on a single processor, but when I run it I get: > mpirun -np 1 flash3 flash3: symbol lookup error: flash3: undefined symbol: H5P_CLS_DATASET_XFER_g I got a similar error when I tried to run FLASH 2.5 (although the undefined symbol in that case was H5P_CLS_DATASET_CREATE_g). Oddly enough, when I built FLASH 2.5 with HDF4, it ran without complaint, but of course the development is in the direction of HDF5 and so I'd really like to get that working. When I go into the /lib directory of my HDF5 build and type > nm * | grep H5P_CLS_DATASET_XFER_g I get U H5P_CLS_DATASET_XFER_g U H5P_CLS_DATASET_XFER_g U H5P_CLS_DATASET_XFER_g U H5P_CLS_DATASET_XFER_g 0000002c D H5P_CLS_DATASET_XFER_g U H5P_CLS_DATASET_XFER_g U H5P_CLS_DATASET_XFER_g U H5P_CLS_DATASET_XFER_g U H5P_CLS_DATASET_XFER_g U H5P_CLS_DATASET_XFER_g which suggests that the symbol isn't recognized there either, but I don't know how to make it define the symbol or to allow FLASH to bypass this. I see that Tomek Plewa had a similar question a few years ago, but when I contacted him he didn't recall the details of how he resolved it. Any suggestions for what I should try? I'm kind of inexperienced at this sort of installation... Thanks! Cole Miller miller at astro.umd.edu System details: > uname -a Linux nstar.astro.umd.edu 2.6.11-12mdkcustom-amd #1 SMP Tue Oct 18 16:06:53 EDT 2005 i686 AMD Athlon(tm) MP 2000+ unknown GNU/Linux > mpif90 -V Intel(R) Fortran Compiler for 32-bit applications, Version 9.1 Build 20060927Z Package ID: l_fc_c_9.1.039 Copyright (C) 1985-2006 Intel Corporation. All rights reserved. FOR NON-COMMERCIAL USE ONLY > mpicc -V Intel(R) C Compiler for 32-bit applications, Version 9.1 Build 20060816Z Package ID: l_cc_c_9.1.043 Copyright (C) 1985-2006 Intel Corporation. All rights reserved. FOR NON-COMMERCIAL USE ONLY > mpich2version MPICH2 Version: 1.0.6 MPICH2 Release date: Unknown, built on Thu Oct 25 08:14:05 EDT 2007 MPICH2 Device: ch3:sock MPICH2 configure: --prefix=/home/cmiller/mpich2-install --enable-fast --enable-f77 --enable-f90 --enable-cxx --with-pm=mpd MPICH2 CC: icc -O2 -O2 MPICH2 CXX: icc -O2 MPICH2 F77: ifort -O2 MPICH2 F90: ifort -O2 MPICH2 Patch level: none > cat /astromake/opt/hdf5/1.6.6/lib/libhdf5.settings SUMMARY OF THE HDF5 CONFIGURATION ================================= HDF5 Version: 1.6.6 Configured on: Fri Oct 26 13:23:22 EDT 2007 Configured by: vernaleo at thales.astro.umd.edu Configure mode: production Host system: i686-pc-linux-gnu Byte sex: little-endian Libraries: static, shared Parallel support: no Installation point: /astromake/opt/hdf5/1.6.6 Compiler: /astromake/opt/intel/cc/9.1.043/bin/icc (icc-9.1) Compiler switches: -std=c99 -O3 -Wl,-s -UH5_DEBUG_API -DNDEBUG -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_POSIX_SOURCE -D_BSD_SOURCE Extra libraries: -lz -lm Archiver: ar Ranlib: ranlib Debugged Packages: API Tracing: no File addresses: large Configure Summary Compiling Options: Compilation Mode: Production C Compiler: icc CFLAGS: -std=c99 -O3 -Wl,-s CPPFLAGS: -UH5_DEBUG_API -DNDEBUG -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_POSIX_SOURCE -D_BSD_SOURCE LDFLAGS: Debug Mode: None Shared Libraries: Yes Static Libraries: Yes Statically Linked Executables: No Tracing: No Optimization Instrumentation: No Languages: ---------- C++: No Fortran: Yes Fortran Compiler: ifort FFLAGS: -I. -O3 -I. C Compiler: icc CFLAGS: -std=c99 -O3 CPPFLAGS: -D_BSD_SOURCE -D_POSIX_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE LDFLAGS: Features: --------- dmalloc: No Function Stack Tracing: Disabled GASS: No GPFS: No HDF5 v1.4 Compatibility: No hsize_t: Large I/O filters (external): deflate I/O filters (internal): shuffle,fletcher32 Linux Large File Support (LFS): Enabled MPE: No Parallel HDF5: No SRB: No Stream VFD: Enabled Threadsafety: Disabled High Level library: Enabled > cat object/Makefile.h # FLASH makefile definitions for the 64 bit Intel icc/ifort compiler on Linux # # Fedora Core 3 # Intel Linux Fortran 9.0 # hdf5-1.6.6 (icc90) # mpich-1.2.7 (icc90) #---------------------------------------------------------------------------- # Set the HDF/HDF5 library paths -- these need to be updated for your system #---------------------------------------------------------------------------- #MPI_PATH = /usr/local/mpich-intel MPI_PATH = /home/cmiller/mpich2-install HDF4_PATH = /astromake/opt/hdf4/4.2r1 #HDF5_PATH = /usr/local/hdf5-icc HDF5_PATH = /astromake/opt/hdf5/1.6.6 ZLIB_PATH = PAPI_PATH = PAPI_FLAGS = #NCMPI_PATH = /usr/local/pnetcdf-icc NCMPI_PATH = MPE_PATH = TOTALVIEW_PATH = /usr/local/totalview/linux-x86-64 #---------------------------------------------------------------------------- # Compiler and linker commands # # We use the f90 compiler as the linker, so some C libraries may explicitly # need to be added into the link line. #---------------------------------------------------------------------------- FCOMP = ${MPI_PATH}/bin/mpif90 CCOMP = ${MPI_PATH}/bin/mpicc #CPPCOMP = ${MPI_PATH}/bin/mpiCC CPPCOMP = ${MPI_PATH}/bin/mpicc LINK = ${MPI_PATH}/bin/mpif90 # pre-processor flag PP = -D #----------------------------------------------------------------------------- # Compilation flags # # Three sets of compilation/linking flags are defined: one for optimized code # code ("-opt"), one for debugging ("-debug"), and one for testing ("-test"). # Passing these flags to the setup script will cause the value associated with # the corresponding keys (i.e. those ending in "_OPT", "_DEBUG", or "_TEST") to # be incorporated into the final Makefile. For example, passing "-opt" to the # setup script will cause the flags following "FFLAGS_OPT" to be assigned to # "FFLAGS" in the final Makefile. If none of these flags are passed, the default # behavior will match that of the "-opt" flag. # In general, "-opt" is meant to optimize compilation and linking. "-debug" # should enable runtime bounds checking, debugger symbols, and other compiler- # specific debugging options. "-test" is useful for testing different # combinations of compiler flags particular to your individual system. #---------------------------------------------------------------------------- FFLAGS_OPT = -c -r8 -i4 -O3 -real_size 64# -unroll -align -prefetch -pad -ip #FFLAGS_DEBUG = -c -g -r8 -i4 -check all -warn all -real_size 64 FFLAGS_DEBUG = -c -g -r8 -i4 -check bounds -warn all -real_size 64 FFLAGS_TEST = -c -r8 -i4 -O2 -real_size 64 CFLAGS_OPT = -c -O3 -D_LARGEFILE64_SOURCE CFLAGS_DEBUG = -c -g -debug extended -D_LARGEFILE64_SOURCE CFLAGS_TEST = -c -O2 -D_LARGEFILE64_SOURCE CFLAGS_HDF5 = -I $(HDF5_PATH)/include CFLAGS_NCMPI = -I $(NCMPI_PATH)/include CFLAGS_MPI = -I$(MPI_PATH)/include #---------------------------------------------------------------------------- # Linker flags # # There is a seperate version of the linker flags for each of the _OPT, # _DEBUG, and _TEST cases. #---------------------------------------------------------------------------- LFLAGS_OPT = -r8 -i4 -Vaxlib -lsvml -Ur -o LFLAGS_DEBUG = -r8 -i4 -Vaxlib -g -o LFLAGS_TEST = -r8 -i4 -Vaxlib -o #---------------------------------------------------------------------------- # Library specific linking # # If a FLASH module has a 'LIBRARY xxx' line in its Config file, we need to # create a macro in this Makefile.h for LIB_xxx, which will be added to the # link line when FLASH is built. This allows us to switch between different # (incompatible) libraries. We also create a _OPT, _DEBUG, and _TEST # library macro to add any performance-minded libraries (like fast math), # depending on how FLASH was setup. #---------------------------------------------------------------------------- LIB_OPT = LIB_DEBUG = -L$(TOTALVIEW_PATH)/lib -ltvheap_64 -Wl,-rpath,$(TOTALVIEW_PATH)/lib LIB_TEST = LIB_HDF4 = -L $(HDF4_PATH)/lib -lmfhdf -ldf -lz -ljpeg LIB_HDF5 = -L $(HDF5_PATH)/lib -lhdf5 -lz LIB_PNG = -lpng LIB_MPI = -L$(MPI_PATH)/lib -lfmpich -lmpich #LIB_NCMPI = -L$(NCMPI_PATH)/lib -lpnetcdf LIB_MPE = #---------------------------------------------------------------------------- # Additional machine-dependent object files # # Add any machine specific files here -- they will be compiled and linked # when FLASH is built. #---------------------------------------------------------------------------- MACHOBJ = #---------------------------------------------------------------------------- # Additional commands #---------------------------------------------------------------------------- MV = mv -f AR = ar -r RM = rm -f CD = cd RL = ranlib ECHO = echo From sheeler at flash.uchicago.edu Mon Nov 5 15:49:25 2007 From: sheeler at flash.uchicago.edu (Dan Sheeler) Date: Mon, 5 Nov 2007 15:49:25 -0600 (CST) Subject: [FLASH-USERS] Undefined symbol in FLASH 3.0beta In-Reply-To: References: Message-ID: My guess is the hdf5 library was built for only serial io, but you are using the flash parallel io unit. Try resetting up the problem, but explicitly choose the serial io unit. Also, maybe your runtime hdf5 library is different from the compile time, and the compile time has parallel support but the runtime doesn't. In that case, you'd set LD_LIBRARY_PATH to point to the compile time hdf5 library. -- Dan Sheeler ASC Flash Center sheeler at flash.uchicago.edu (773) 834-3236 On Mon, 5 Nov 2007, Cole Miller wrote: > > Hi all, > > I've downloaded FLASH 3.0beta, and although it compiles I have not > been able to get the HDF5 version to run (the system details are > below). I've compiled this on a single processor, but when I run > it I get: >> mpirun -np 1 flash3 > flash3: symbol lookup error: flash3: undefined symbol: H5P_CLS_DATASET_XFER_g > > I got a similar error when I tried to run FLASH 2.5 (although the > undefined symbol in that case was H5P_CLS_DATASET_CREATE_g). Oddly > enough, when I built FLASH 2.5 with HDF4, it ran without complaint, > but of course the development is in the direction of HDF5 and so I'd > really like to get that working. > > When I go into the /lib directory of my HDF5 build and type >> nm * | grep H5P_CLS_DATASET_XFER_g > I get > U H5P_CLS_DATASET_XFER_g > U H5P_CLS_DATASET_XFER_g > U H5P_CLS_DATASET_XFER_g > U H5P_CLS_DATASET_XFER_g > 0000002c D H5P_CLS_DATASET_XFER_g > U H5P_CLS_DATASET_XFER_g > U H5P_CLS_DATASET_XFER_g > U H5P_CLS_DATASET_XFER_g > U H5P_CLS_DATASET_XFER_g > U H5P_CLS_DATASET_XFER_g > which suggests that the symbol isn't recognized there either, but > I don't know how to make it define the symbol or to allow FLASH to > bypass this. I see that Tomek Plewa had a similar question a few > years ago, but when I contacted him he didn't recall the details of > how he resolved it. > > Any suggestions for what I should try? I'm kind of inexperienced at > this sort of installation... > > Thanks! > Cole Miller > miller at astro.umd.edu > > > System details: > > >> uname -a > Linux nstar.astro.umd.edu 2.6.11-12mdkcustom-amd #1 SMP > Tue Oct 18 16:06:53 EDT 2005 i686 AMD Athlon(tm) MP 2000+ unknown GNU/Linux > >> mpif90 -V > Intel(R) Fortran Compiler for 32-bit applications, Version 9.1 > Build 20060927Z Package ID: l_fc_c_9.1.039 > Copyright (C) 1985-2006 Intel Corporation. All rights reserved. > FOR NON-COMMERCIAL USE ONLY > >> mpicc -V > Intel(R) C Compiler for 32-bit applications, Version 9.1 > Build 20060816Z Package ID: l_cc_c_9.1.043 > Copyright (C) 1985-2006 Intel Corporation. All rights reserved. > FOR NON-COMMERCIAL USE ONLY > >> mpich2version > MPICH2 Version: 1.0.6 > MPICH2 Release date: Unknown, built on Thu Oct 25 08:14:05 EDT 2007 > MPICH2 Device: ch3:sock > MPICH2 configure: --prefix=/home/cmiller/mpich2-install > --enable-fast --enable-f77 --enable-f90 --enable-cxx --with-pm=mpd > MPICH2 CC: icc -O2 -O2 > MPICH2 CXX: icc -O2 > MPICH2 F77: ifort -O2 > MPICH2 F90: ifort -O2 > MPICH2 Patch level: none > >> cat /astromake/opt/hdf5/1.6.6/lib/libhdf5.settings > SUMMARY OF THE HDF5 CONFIGURATION > ================================= > > HDF5 Version: 1.6.6 > Configured on: Fri Oct 26 13:23:22 EDT 2007 > Configured by: vernaleo at thales.astro.umd.edu > Configure mode: production > Host system: i686-pc-linux-gnu > Byte sex: little-endian > Libraries: static, shared > Parallel support: no > Installation point: /astromake/opt/hdf5/1.6.6 > Compiler: /astromake/opt/intel/cc/9.1.043/bin/icc (icc-9.1) > Compiler switches: -std=c99 -O3 -Wl,-s -UH5_DEBUG_API -DNDEBUG > -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 > -D_POSIX_SOURCE -D_BSD_SOURCE > Extra libraries: -lz -lm > Archiver: ar > Ranlib: ranlib > Debugged Packages: > API Tracing: no > File addresses: large > > Configure Summary > Compiling Options: > Compilation Mode: Production > C Compiler: icc > CFLAGS: -std=c99 -O3 -Wl,-s > CPPFLAGS: -UH5_DEBUG_API -DNDEBUG -D_LARGEFILE_SOURCE > -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_POSIX_SOURCE -D_BSD_SOURCE > LDFLAGS: > Debug Mode: None > Shared Libraries: Yes > Static Libraries: Yes > Statically Linked Executables: No > Tracing: No > Optimization Instrumentation: No > > Languages: > ---------- > C++: No > Fortran: Yes > Fortran Compiler: ifort > FFLAGS: -I. -O3 -I. > C Compiler: icc > CFLAGS: -std=c99 -O3 > CPPFLAGS: -D_BSD_SOURCE -D_POSIX_SOURCE > -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE > LDFLAGS: > > > Features: > --------- > dmalloc: No > Function Stack Tracing: Disabled > GASS: No > GPFS: No > HDF5 v1.4 Compatibility: No > hsize_t: Large > I/O filters (external): deflate > I/O filters (internal): shuffle,fletcher32 > Linux Large File Support (LFS): Enabled > MPE: No > Parallel HDF5: No > SRB: No > Stream VFD: Enabled > Threadsafety: Disabled > High Level library: Enabled > > >> cat object/Makefile.h > > # FLASH makefile definitions for the 64 bit Intel icc/ifort compiler on Linux > # > # Fedora Core 3 > # Intel Linux Fortran 9.0 > # hdf5-1.6.6 (icc90) > # mpich-1.2.7 (icc90) > > #---------------------------------------------------------------------------- > # Set the HDF/HDF5 library paths -- these need to be updated for your system > #---------------------------------------------------------------------------- > > #MPI_PATH = /usr/local/mpich-intel > MPI_PATH = /home/cmiller/mpich2-install > HDF4_PATH = /astromake/opt/hdf4/4.2r1 > #HDF5_PATH = /usr/local/hdf5-icc > HDF5_PATH = /astromake/opt/hdf5/1.6.6 > > ZLIB_PATH = > > PAPI_PATH = > PAPI_FLAGS = > > #NCMPI_PATH = /usr/local/pnetcdf-icc > NCMPI_PATH = > MPE_PATH = > > TOTALVIEW_PATH = /usr/local/totalview/linux-x86-64 > > #---------------------------------------------------------------------------- > # Compiler and linker commands > # > # We use the f90 compiler as the linker, so some C libraries may explicitly > # need to be added into the link line. > #---------------------------------------------------------------------------- > > FCOMP = ${MPI_PATH}/bin/mpif90 > CCOMP = ${MPI_PATH}/bin/mpicc > #CPPCOMP = ${MPI_PATH}/bin/mpiCC > CPPCOMP = ${MPI_PATH}/bin/mpicc > LINK = ${MPI_PATH}/bin/mpif90 > > # pre-processor flag > > PP = -D > > #----------------------------------------------------------------------------- > # Compilation flags > # > # Three sets of compilation/linking flags are defined: one for optimized > code > # code ("-opt"), one for debugging ("-debug"), and one for testing > ("-test"). > # Passing these flags to the setup script will cause the value associated > with > # the corresponding keys (i.e. those ending in "_OPT", "_DEBUG", or "_TEST") > to > # be incorporated into the final Makefile. For example, passing "-opt" to > the > # setup script will cause the flags following "FFLAGS_OPT" to be assigned to > # "FFLAGS" in the final Makefile. If none of these flags are passed, the > default > # behavior will match that of the "-opt" flag. > # In general, "-opt" is meant to optimize compilation and linking. "-debug" > # should enable runtime bounds checking, debugger symbols, and other > compiler- > # specific debugging options. "-test" is useful for testing different > # combinations of compiler flags particular to your individual system. > #---------------------------------------------------------------------------- > > FFLAGS_OPT = -c -r8 -i4 -O3 -real_size 64# -unroll -align -prefetch -pad > -ip > #FFLAGS_DEBUG = -c -g -r8 -i4 -check all -warn all -real_size 64 > FFLAGS_DEBUG = -c -g -r8 -i4 -check bounds -warn all -real_size 64 > FFLAGS_TEST = -c -r8 -i4 -O2 -real_size 64 > > CFLAGS_OPT = -c -O3 -D_LARGEFILE64_SOURCE > CFLAGS_DEBUG = -c -g -debug extended -D_LARGEFILE64_SOURCE > CFLAGS_TEST = -c -O2 -D_LARGEFILE64_SOURCE > > CFLAGS_HDF5 = -I $(HDF5_PATH)/include > CFLAGS_NCMPI = -I $(NCMPI_PATH)/include > CFLAGS_MPI = -I$(MPI_PATH)/include > > #---------------------------------------------------------------------------- > # Linker flags > # > # There is a seperate version of the linker flags for each of the _OPT, > # _DEBUG, and _TEST cases. > #---------------------------------------------------------------------------- > > LFLAGS_OPT = -r8 -i4 -Vaxlib -lsvml -Ur -o > LFLAGS_DEBUG = -r8 -i4 -Vaxlib -g -o > LFLAGS_TEST = -r8 -i4 -Vaxlib -o > > #---------------------------------------------------------------------------- > # Library specific linking > # > # If a FLASH module has a 'LIBRARY xxx' line in its Config file, we need to > # create a macro in this Makefile.h for LIB_xxx, which will be added to the > # link line when FLASH is built. This allows us to switch between different > # (incompatible) libraries. We also create a _OPT, _DEBUG, and _TEST > # library macro to add any performance-minded libraries (like fast math), > # depending on how FLASH was setup. > #---------------------------------------------------------------------------- > LIB_OPT = > LIB_DEBUG = -L$(TOTALVIEW_PATH)/lib -ltvheap_64 > -Wl,-rpath,$(TOTALVIEW_PATH)/lib > LIB_TEST = > > LIB_HDF4 = -L $(HDF4_PATH)/lib -lmfhdf -ldf -lz -ljpeg > LIB_HDF5 = -L $(HDF5_PATH)/lib -lhdf5 -lz > LIB_PNG = -lpng > > LIB_MPI = -L$(MPI_PATH)/lib -lfmpich -lmpich > #LIB_NCMPI = -L$(NCMPI_PATH)/lib -lpnetcdf > LIB_MPE = > > > #---------------------------------------------------------------------------- > # Additional machine-dependent object files > # > # Add any machine specific files here -- they will be compiled and linked > # when FLASH is built. > #---------------------------------------------------------------------------- > > MACHOBJ = > > #---------------------------------------------------------------------------- > # Additional commands > #---------------------------------------------------------------------------- > > MV = mv -f > AR = ar -r > RM = rm -f > CD = cd > RL = ranlib > ECHO = echo > From miller at astro.umd.edu Tue Nov 6 14:36:20 2007 From: miller at astro.umd.edu (Cole Miller) Date: Tue, 6 Nov 2007 15:36:20 -0500 (EST) Subject: [FLASH-USERS] Solved: Undefined symbol in FLASH 3.0beta In-Reply-To: References: Message-ID: Hi all, I'm writing this so there exists a specific record of the solution to the problem I ran into, which is that I was getting runtime errors such as > flash3: symbol lookup error: flash3: undefined symbol: H5P_CLS_DATASET_XFER_g or H5P_CLS_DATASET_CREATE_g in the case of FLASH2.5. Dan Sheeler suggested a couple of things, and the one that worked in my case is an explicit setting of LD_LIBRARY_PATH to point to the compile time hdf5 library: setenv LD_LIBRARY_PATH /astromake/opt/hdf5/1.6.6/lib:$LD_LIBRARY_PATH A way of testing this that Dan suggested is to type "ldd flash3" in the object/ directory. With LD_LIBRARY_PATH defined I get linux-gate.so.1 => (0xffffe000) libsvml.so => /astromake/opt/intel/cc/9.1.043/lib/libsvml.so (0xb7f8c000) libhdf5.so.0 => /astromake/opt/hdf5/1.6.6/lib/libhdf5.so.0 (0xb7e85000) libz.so.1 => /lib/libz.so.1 (0xb7e5a000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7e49000) librt.so.1 => /lib/tls/librt.so.1 (0xb7e35000) libm.so.6 => /lib/tls/libm.so.6 (0xb7e12000) libc.so.6 => /lib/tls/libc.so.6 (0xb7cf3000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7cea000) libdl.so.2 => /lib/libdl.so.2 (0xb7ce6000) libirc.so => /astromake/opt/intel/cc/9.1.043/lib/libirc.so (0xb7ca5000) libimf.so => /astromake/opt/intel/cc/9.1.043/lib/libimf.so (0xb7a67000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0xb7fec000) and without I get linux-gate.so.1 => (0xffffe000) libsvml.so => /astromake/opt/intel/cc/9.1.043/lib/libsvml.so (0xb7f8c000) libhdf5.so.0 => /usr/lib/libhdf5.so.0 (0xb7e9c000) libz.so.1 => /lib/libz.so.1 (0xb7e89000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7e78000) librt.so.1 => /lib/tls/librt.so.1 (0xb7e64000) libm.so.6 => /lib/tls/libm.so.6 (0xb7e41000) libc.so.6 => /lib/tls/libc.so.6 (0xb7d22000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7d19000) libdl.so.2 => /lib/libdl.so.2 (0xb7d15000) libirc.so => /astromake/opt/intel/cc/9.1.043/lib/libirc.so (0xb7cd4000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0xb7fec000) The differences are (1) libhdf5.so.0 is defined differently when LD_LIBRARY_PATH points directly to the hdf5 library, and (2) when this is defined we have an extra line, related to libimf.so. Thanks to Dan for his help! Cole From lia at astro.columbia.edu Wed Nov 7 12:40:38 2007 From: lia at astro.columbia.edu (Lia Corrales) Date: Wed, 07 Nov 2007 13:40:38 -0500 Subject: [FLASH-USERS] Problem with h5_wrappers, idl 6.2 Message-ID: <1194460838.6997.21.camel@juliet.astro.columbia.edu> Hi! I am new to flash and working through some of the example problems. I am using Flash2.3. I configured the h5_wrappers appropriately, and xflash ran just fine in idl, until I ran the example "orbit" problem. When I try to make a plot, I get the following error: % XMANAGER: Caught unexpected error from client application. Message follows... % CALL_EXTERNAL: Error loading sharable executable. Symbol: get_numProps, File = h5_wrappers.so h5_wrappers.so: cannot open shared object file: No such file or directory I did not run into problems like this with other examples like sedov or isentropic_vortex. I found that get_numProps is provided by the h5_wrappers.c code, but can't figure out why it fails. I am working on a system that has IDL 6.2 and a recent version of HDF5. If anyone has had this problem and can help me out, thank you! - Lia - From flash-users at flash.uchicago.edu Wed Nov 7 13:47:25 2007 From: flash-users at flash.uchicago.edu (VIAGRA ® Official Site) Date: Wed, 7 Nov 2007 13:47:25 -0600 (CST) Subject: [FLASH-USERS] [SPAM] November 79% OFF Message-ID: <20071107095158.3327.qmail@snbw01600204> An HTML attachment was scrubbed... URL: http://flash.uchicago.edu/pipermail/flash-users/attachments/20071107/f20d7a83/attachment.html From miller at astro.umd.edu Sat Nov 17 10:17:35 2007 From: miller at astro.umd.edu (Cole Miller) Date: Sat, 17 Nov 2007 11:17:35 -0500 (EST) Subject: [FLASH-USERS] Question about direct manipulation of Flash HDF5 data In-Reply-To: References: Message-ID: Hi all, I'd appreciate some help in terms of processing HDF5 output files from FLASH3.0 beta. I've been able to run several of the test problems successfully and have used the xflash3 widget to produce images, do 1-d readout, et cetera. So far, so good. However, for serious processing I need to be able to do a number of other things. For example: 1. At a given x,y,z point in the simulation region, what is the time history of the pressure? 2. What is the temperature over a uniform grid at a given time slice, resampled from a nonuniform AMR grid? Basically, I'd like more direct control over the data. In principle I suppose that since an IDL widget can access this information, there should be a way to print it out directly from IDL. However, I haven't used IDL before; any suggestions? I have also tried to use the h5dump feature on individual files, but I have no experience with the output format (DDL or XML), and it looks like it would be a major endeavor to extract the required information from the ASCII output. I've also tried, briefly, to look through some of the C routines on the HDF site that are supposed to be able to read HDF5 files, but have yet to surmount difficulties related to various libraries to which I need to link (maybe this can be solved by setting PATH variables in some way?). Anyway, any help would be great! Thanks, Cole From hudson at mcs.anl.gov Sat Nov 17 19:43:01 2007 From: hudson at mcs.anl.gov (Randy Hudson) Date: Sat, 17 Nov 2007 19:43:01 -0600 Subject: [FLASH-USERS] Question about direct manipulation of Flash HDF5 data In-Reply-To: References: Message-ID: <473F98A5.3030708@mcs.anl.gov> Cole, ------------------------------ Re (2) If you're not tied to IDL, you have a machine on which the executable of VisIt (http://www.llnl.gov/visit/executables.html) can run, and you want to visualize the temp you mention in (2), I'm pretty sure visit can resample the flash amr over a regular grid. See the section on the "Resample" operator on p. 115 of the VisIt manual (http://www.llnl.gov/visit/1.5/VisItUsersManual1.5.pdf). ------------------------------ Re (1) P. 222 of the VisIt manual (with the help of the previous page or so) describes how to create a new lineout curve in the same window for every step of a time series, so you can see a cascade of curves for a time series. You can pass the lineout sample line through the location you want to monitor over time. Probably better: p. 214 of the same manual states that any query that can be plotted over time contains a "Time Curve" button in the query's control panel. The "Pick" query contains such a button, and is used to pick values, by location, frmo the data domain. Cole Miller wrote: > > Hi all, > > I'd appreciate some help in terms of processing HDF5 output > files from FLASH3.0 beta. I've been able to run several of > the test problems successfully and have used the xflash3 widget > to produce images, do 1-d readout, et cetera. So far, so good. > However, for serious processing I need to be able to do a number > of other things. For example: > > 1. At a given x,y,z point in the simulation region, what is the > time history of the pressure? > > 2. What is the temperature over a uniform grid at a given time slice, > resampled from a nonuniform AMR grid? > > Basically, I'd like more direct control over the data. > > In principle I suppose that since an IDL widget can access this > information, there should be a way to print it out directly from > IDL. However, I haven't used IDL before; any suggestions? I have > also tried to use the h5dump feature on individual files, but I > have no experience with the output format (DDL or XML), and it > looks like it would be a major endeavor to extract the required > information from the ASCII output. I've also tried, briefly, to > look through some of the C routines on the HDF site that are supposed > to be able to read HDF5 files, but have yet to surmount difficulties > related to various libraries to which I need to link (maybe this can > be solved by setting PATH variables in some way?). > > Anyway, any help would be great! > > Thanks, > Cole > > -- Randy. From flash-users at flash.uchicago.edu Mon Nov 19 01:29:06 2007 From: flash-users at flash.uchicago.edu (VIAGRA ® Official Site) Date: Mon, 19 Nov 2007 01:29:06 -0600 (CST) Subject: [FLASH-USERS] [SPAM] November 71% OFF Message-ID: <20071119112907.4323.qmail@user10> An HTML attachment was scrubbed... URL: http://flash.uchicago.edu/pipermail/flash-users/attachments/20071119/639be745/attachment.html From Lloyd at Viagra.com Sat Nov 24 04:38:51 2007 From: Lloyd at Viagra.com (Lloyd at Viagra.com) Date: Sat, 24 Nov 2007 04:38:51 -0600 (CST) Subject: [FLASH-USERS] [SPAM] November 75% OFF Message-ID: <20071124124028.11771.qmail@catv-5985c769.catv.broadband.hu> An HTML attachment was scrubbed... URL: http://flash.uchicago.edu/pipermail/flash-users/attachments/20071124/78855922/attachment.html From kdere at gmu.edu Tue Nov 27 07:44:25 2007 From: kdere at gmu.edu (Ken Dere) Date: Tue, 27 Nov 2007 08:44:25 -0500 Subject: [FLASH-USERS] Question about direct manipulation of Flash HDF5 data In-Reply-To: References: Message-ID: <474C1F39.7020504@gmu.edu> Another alternative is to use the Python PyTables module. It is pretty straightforward. If you are interested, I can send a short Python function that just lists the nodes in an hdf file. I also shows the basic mechanics of just opening an hdf file. Ken Cole Miller wrote: > > Hi all, > > I'd appreciate some help in terms of processing HDF5 output > files from FLASH3.0 beta. I've been able to run several of > the test problems successfully and have used the xflash3 widget > to produce images, do 1-d readout, et cetera. So far, so good. > However, for serious processing I need to be able to do a number > of other things. For example: > > 1. At a given x,y,z point in the simulation region, what is the > time history of the pressure? > > 2. What is the temperature over a uniform grid at a given time slice, > resampled from a nonuniform AMR grid? > > Basically, I'd like more direct control over the data. > > In principle I suppose that since an IDL widget can access this > information, there should be a way to print it out directly from > IDL. However, I haven't used IDL before; any suggestions? I have > also tried to use the h5dump feature on individual files, but I > have no experience with the output format (DDL or XML), and it > looks like it would be a major endeavor to extract the required > information from the ASCII output. I've also tried, briefly, to > look through some of the C routines on the HDF site that are supposed > to be able to read HDF5 files, but have yet to surmount difficulties > related to various libraries to which I need to link (maybe this can > be solved by setting PATH variables in some way?). > > Anyway, any help would be great! > > Thanks, > Cole -- Kenneth P. Dere Research Professor of Solar Physics Dept. of Computation and Data Sciences MS 6A2 George Mason University 4400 University Drive Fairfax VA 22030 703-993-4555 703-993-1992 FAX kdere at gmu.edu From mdas at umich.edu Tue Nov 27 09:40:32 2007 From: mdas at umich.edu (Mousumi Das) Date: Tue, 27 Nov 2007 10:40:32 -0500 (EST) Subject: [FLASH-USERS] 3D simulation Message-ID: Hello All, I am using FLASH 2.4 and trying to run 3-dimensional simulation with higher resolution by increasing zones in setup. ./setup example -auto -3d -maxblocks=1500 -nxb=36 -nyb=36 -nzb=36 with refinement-level =6 The code compiles and generate excutable flash2. But when I am trying to run it came with an error: "2 processes killed (possibly by Open MPI)" It happens when I am trying to run high resolution zones stated above. If I use the following set up: ./setup example -auto -3d -maxblocks=2500 -nxb=16 -nyb=16 -nzb=16 It compiles and runs well. For higher number of zones set up it came out with the above error message. Can anyone of you came accross with such thing, can you help me or giving an idea what could be the possible reason for the above error mesg. regards, Mousumi Das Research Associate University of Michigan From dubey at flash.uchicago.edu Tue Nov 27 12:35:30 2007 From: dubey at flash.uchicago.edu (Anshu Dubey) Date: Tue, 27 Nov 2007 12:35:30 -0600 (CST) Subject: [FLASH-USERS] 3D simulation In-Reply-To: References: Message-ID: <34863.76.217.49.138.1196188530.squirrel@flash.uchicago.edu> That is probably just too big for the loader on the machine. I don't know of any platform on which that size problem will load. You should try to build it with fewer maxblocks. Anshu > Hello All, > I am using FLASH 2.4 and trying to run 3-dimensional simulation with > higher > resolution by increasing zones in setup. > > ./setup example -auto -3d -maxblocks=1500 -nxb=36 -nyb=36 -nzb=36 > with refinement-level =6 > > The code compiles and generate excutable flash2. > > But when I am trying to run it came with an error: > > "2 processes killed (possibly by Open MPI)" > > It happens when I am trying to run high resolution zones stated above. > If I use the following set up: > > ./setup example -auto -3d -maxblocks=2500 -nxb=16 -nyb=16 -nzb=16 > > It compiles and runs well. > For higher number of zones set up it came out with the above error > message. > Can anyone of you came accross with such thing, can you help me or giving > an > idea what could be the possible reason for the above error mesg. > > regards, > Mousumi Das > Research Associate > University of Michigan > From dubey at flash.uchicago.edu Tue Nov 27 12:37:04 2007 From: dubey at flash.uchicago.edu (Anshu Dubey) Date: Tue, 27 Nov 2007 12:37:04 -0600 (CST) Subject: [FLASH-USERS] 3D simulation In-Reply-To: References: Message-ID: <34866.76.217.49.138.1196188624.squirrel@flash.uchicago.edu> For further clarification, maxblocks is the number of blocks per processor, not the global number of blocks. Anshu > Hello All, > I am using FLASH 2.4 and trying to run 3-dimensional simulation with > higher > resolution by increasing zones in setup. > > ./setup example -auto -3d -maxblocks=1500 -nxb=36 -nyb=36 -nzb=36 > with refinement-level =6 > > The code compiles and generate excutable flash2. > > But when I am trying to run it came with an error: > > "2 processes killed (possibly by Open MPI)" > > It happens when I am trying to run high resolution zones stated above. > If I use the following set up: > > ./setup example -auto -3d -maxblocks=2500 -nxb=16 -nyb=16 -nzb=16 > > It compiles and runs well. > For higher number of zones set up it came out with the above error > message. > Can anyone of you came accross with such thing, can you help me or giving > an > idea what could be the possible reason for the above error mesg. > > regards, > Mousumi Das > Research Associate > University of Michigan > From mdas at umich.edu Thu Nov 29 09:43:30 2007 From: mdas at umich.edu (Mousumi Das) Date: Thu, 29 Nov 2007 10:43:30 -0500 (EST) Subject: [FLASH-USERS] 3D simulation In-Reply-To: <34863.76.217.49.138.1196188530.squirrel@flash.uchicago.edu> References: <34863.76.217.49.138.1196188530.squirrel@flash.uchicago.edu> Message-ID: Hello All, I was trying to build up the set up for 3D simulation with smaller maxblocks. ./setup example -auto -3d -maxblocks=400 -nxb=36 -nyb=36 -nzb=36 with refinement-level =6 so that the executable starts to run. It was running over 16 processors for one and half days and goes up to the follwoing point, suddenly it stops showing no error mesg , even amr_log file did not show any error mesg. Can anybody has any idea what was gone wrong. regards, Mousumi [ 11-27-2007 19:16.58 ] [AMR_REFINE_DEREFINE]: refinement complete [ 11-27-2007 19:17.00 ] [INIT_FROM_SCRATCH]: create level=6 [ 11-28-2007 11:42.21 ] [AMR_REFINE_DEREFINE]: refinement initiated at 11:42.21 [ 11-28-2007 11:44.00 ] [AMR_REFINE_DEREFINE] blocks all: min=169 max=172 tot=2729 [ 11-28-2007 11:44.01 ] [AMR_REFINE_DEREFINE] blocks valid: min=148 max=151 tot=2388 [ 11-28-2007 11:44.01 ] [AMR_REFINE_DEREFINE]: refinement complete [ 11-28-2007 11:44.08 ] [INIT_FROM_SCRATCH]: create level=6 On Tue, 27 Nov 2007, Anshu Dubey wrote: > That is probably just too big for the loader on the machine. I don't > know of any platform on which that size problem will load. You should > try to build it with fewer maxblocks. > > Anshu > >> Hello All, >> I am using FLASH 2.4 and trying to run 3-dimensional simulation with >> higher >> resolution by increasing zones in setup. >> >> ./setup example -auto -3d -maxblocks=1500 -nxb=36 -nyb=36 -nzb=36 >> with refinement-level =6 >> >> The code compiles and generate excutable flash2. >> >> But when I am trying to run it came with an error: >> >> "2 processes killed (possibly by Open MPI)" >> >> It happens when I am trying to run high resolution zones stated above. >> If I use the following set up: >> >> ./setup example -auto -3d -maxblocks=2500 -nxb=16 -nyb=16 -nzb=16 >> >> It compiles and runs well. >> For higher number of zones set up it came out with the above error >> message. >> Can anyone of you came accross with such thing, can you help me or giving >> an >> idea what could be the possible reason for the above error mesg. >> >> regards, >> Mousumi Das >> Research Associate >> University of Michigan >> > > > > From dubey at flash.uchicago.edu Thu Nov 29 10:35:42 2007 From: dubey at flash.uchicago.edu (Anshu Dubey) Date: Thu, 29 Nov 2007 10:35:42 -0600 (CST) Subject: [FLASH-USERS] 3D simulation In-Reply-To: References: <34863.76.217.49.138.1196188530.squirrel@flash.uchicago.edu> Message-ID: <4362.76.217.55.234.1196354142.squirrel@flash.uchicago.edu> Please tell us the specifics of the machine you are running on, specifically what kind of platform is it and what is the memory per processor. My suspicion is that you are still very close to exhausting the available memory on the machine with the sizes you are using. Anshu > Hello All, > > I was trying to build up the set up for 3D simulation with smaller > maxblocks. > > ./setup example -auto -3d -maxblocks=400 -nxb=36 -nyb=36 -nzb=36 > with refinement-level =6 so that the executable starts to run. > > It was running over 16 processors for one and half days and goes up to > the follwoing point, suddenly it stops showing no error mesg , even > amr_log file did not show any error mesg. Can anybody has any idea > what was gone wrong. > regards, > Mousumi > > [ 11-27-2007 19:16.58 ] [AMR_REFINE_DEREFINE]: refinement complete > [ 11-27-2007 19:17.00 ] [INIT_FROM_SCRATCH]: create level=6 > [ 11-28-2007 11:42.21 ] [AMR_REFINE_DEREFINE]: refinement initiated at > 11:42.21 > [ 11-28-2007 11:44.00 ] [AMR_REFINE_DEREFINE] blocks all: min=169 > max=172 tot=2729 > [ 11-28-2007 11:44.01 ] [AMR_REFINE_DEREFINE] blocks valid: min=148 > max=151 tot=2388 > [ 11-28-2007 11:44.01 ] [AMR_REFINE_DEREFINE]: refinement complete > [ 11-28-2007 11:44.08 ] [INIT_FROM_SCRATCH]: create level=6 > > > > > On Tue, 27 Nov 2007, Anshu Dubey wrote: > >> That is probably just too big for the loader on the machine. I don't >> know of any platform on which that size problem will load. You should >> try to build it with fewer maxblocks. >> >> Anshu >> >>> Hello All, >>> I am using FLASH 2.4 and trying to run 3-dimensional simulation >>> with >>> higher >>> resolution by increasing zones in setup. >>> >>> ./setup example -auto -3d -maxblocks=1500 -nxb=36 -nyb=36 -nzb=36 >>> with refinement-level =6 >>> >>> The code compiles and generate excutable flash2. >>> >>> But when I am trying to run it came with an error: >>> >>> "2 processes killed (possibly by Open MPI)" >>> >>> It happens when I am trying to run high resolution zones stated above. >>> If I use the following set up: >>> >>> ./setup example -auto -3d -maxblocks=2500 -nxb=16 -nyb=16 -nzb=16 >>> >>> It compiles and runs well. >>> For higher number of zones set up it came out with the above error >>> message. >>> Can anyone of you came accross with such thing, can you help me or >>> giving >>> an >>> idea what could be the possible reason for the above error mesg. >>> >>> regards, >>> Mousumi Das >>> Research Associate >>> University of Michigan >>> >> >> >> >> > From mdas at umich.edu Thu Nov 29 10:58:10 2007 From: mdas at umich.edu (Mousumi Das) Date: Thu, 29 Nov 2007 11:58:10 -0500 (EST) Subject: [FLASH-USERS] 3D simulation In-Reply-To: <4362.76.217.55.234.1196354142.squirrel@flash.uchicago.edu> References: <34863.76.217.49.138.1196188530.squirrel@flash.uchicago.edu> <4362.76.217.55.234.1196354142.squirrel@flash.uchicago.edu> Message-ID: Hello, The details of the nodes the job ran on are: amd Opteron 250's each node has 4 GB ram, The compiler used was pgi/6.1 The MPI library is OpenMPI-1.0.2 We had to recompile flash+hdf5 with the option -mcmodel=medium because of the following error: relocation truncated to fit: R_X86_64_PC32 but fftw and openmpi were NOT compiled with the option -mcmodel=medium thanks, Mousumi On Thu, 29 Nov 2007, Anshu Dubey wrote: > Please tell us the specifics of the machine you are running on, > specifically what kind of platform is it and what is the memory > per processor. My suspicion is that you are still very close to > exhausting the available memory on the machine with the sizes > you are using. > > Anshu > >> Hello All, >> >> I was trying to build up the set up for 3D simulation with smaller >> maxblocks. >> >> ./setup example -auto -3d -maxblocks=400 -nxb=36 -nyb=36 -nzb=36 >> with refinement-level =6 so that the executable starts to run. >> >> It was running over 16 processors for one and half days and goes up to >> the follwoing point, suddenly it stops showing no error mesg , even >> amr_log file did not show any error mesg. Can anybody has any idea >> what was gone wrong. >> regards, >> Mousumi >> >> [ 11-27-2007 19:16.58 ] [AMR_REFINE_DEREFINE]: refinement complete >> [ 11-27-2007 19:17.00 ] [INIT_FROM_SCRATCH]: create level=6 >> [ 11-28-2007 11:42.21 ] [AMR_REFINE_DEREFINE]: refinement initiated at >> 11:42.21 >> [ 11-28-2007 11:44.00 ] [AMR_REFINE_DEREFINE] blocks all: min=169 >> max=172 tot=2729 >> [ 11-28-2007 11:44.01 ] [AMR_REFINE_DEREFINE] blocks valid: min=148 >> max=151 tot=2388 >> [ 11-28-2007 11:44.01 ] [AMR_REFINE_DEREFINE]: refinement complete >> [ 11-28-2007 11:44.08 ] [INIT_FROM_SCRATCH]: create level=6 >> >> >> >> >> On Tue, 27 Nov 2007, Anshu Dubey wrote: >> >>> That is probably just too big for the loader on the machine. I don't >>> know of any platform on which that size problem will load. You should >>> try to build it with fewer maxblocks. >>> >>> Anshu >>> >>>> Hello All, >>>> I am using FLASH 2.4 and trying to run 3-dimensional simulation >>>> with >>>> higher >>>> resolution by increasing zones in setup. >>>> >>>> ./setup example -auto -3d -maxblocks=1500 -nxb=36 -nyb=36 -nzb=36 >>>> with refinement-level =6 >>>> >>>> The code compiles and generate excutable flash2. >>>> >>>> But when I am trying to run it came with an error: >>>> >>>> "2 processes killed (possibly by Open MPI)" >>>> >>>> It happens when I am trying to run high resolution zones stated above.. >>>> If I use the following set up: >>>> >>>> ./setup example -auto -3d -maxblocks=2500 -nxb=16 -nyb=16 -nzb=16 >>>> >>>> It compiles and runs well. >>>> For higher number of zones set up it came out with the above error >>>> message. >>>> Can anyone of you came accross with such thing, can you help me or >>>> giving >>>> an >>>> idea what could be the possible reason for the above error mesg. >>>> >>>> regards, >>>> Mousumi Das >>>> Research Associate >>>> University of Michigan >>>> >>> >>> >>> >>> >> > > > > From brockp at umich.edu Fri Nov 30 07:54:44 2007 From: brockp at umich.edu (Brock Palen) Date: Fri, 30 Nov 2007 08:54:44 -0500 Subject: [FLASH-USERS] Flash memory usage. Message-ID: <699B1EE1-6FD1-46E4-A28F-1859FFA18F79@umich.edu> How does flash do memory allocation? If I am running into memory problems but my only option for more memory is to run on more nodes does flash subdivide the amount of memory in use evenly? For example: a model when running on 4 cpus, each process takes 10GB of real memory. If I run that same model on 8 how close will each process get to 5 gb of use? Brock Palen Center for Advanced Computing brockp at umich.edu (734)936-1985 From gbp at phys.soton.ac.uk Fri Nov 30 08:29:12 2007 From: gbp at phys.soton.ac.uk (Georgi Pavlovski) Date: Fri, 30 Nov 2007 14:29:12 +0000 Subject: [FLASH-USERS] Flash memory usage. In-Reply-To: <699B1EE1-6FD1-46E4-A28F-1859FFA18F79@umich.edu> References: <699B1EE1-6FD1-46E4-A28F-1859FFA18F79@umich.edu> Message-ID: <47501E38.70801@phys.soton.ac.uk> Hi Brock, I think the answer is no. Each processor allocates unk array (at least in flash 2.x), which is (nvar)x(2*nguard+nxb)x(2*nguard+nyb)x(2*nguard+nzb)x(maxblocks) big in double precision. The rest of the variables are small compared to unk. This sets your memory requirements. If you have a single fluid nvar=11. - Georgi Brock Palen wrote: > How does flash do memory allocation? If I am running into memory > problems but my only option for more memory is to run on more nodes does > flash subdivide the amount of memory in use evenly? > For example: > > a model when running on 4 cpus, each process takes 10GB of real memory. > > If I run that same model on 8 how close will each process get to 5 gb of > use? > > Brock Palen > Center for Advanced Computing > brockp at umich.edu > (734)936-1985 > > > > > -- =================================================================== Dr Georgi B. Pavlovski School of Physics and Astronomy Tel: +44-(0)-23-8059-2089 University of Southampton Fax: +44-(0)-23-8059-3910 Southampton SO17 1BJ, U.K. Email: gbp at phys.soton.ac.uk Web: www.astro.soton.ac.uk/~gbp =================================================================== From dubey at flash.uchicago.edu Fri Nov 30 08:33:07 2007 From: dubey at flash.uchicago.edu (Anshu Dubey) Date: Fri, 30 Nov 2007 08:33:07 -0600 (CST) Subject: [FLASH-USERS] Flash memory usage. In-Reply-To: <699B1EE1-6FD1-46E4-A28F-1859FFA18F79@umich.edu> References: <699B1EE1-6FD1-46E4-A28F-1859FFA18F79@umich.edu> Message-ID: <1772.76.217.55.234.1196433187.squirrel@flash.uchicago.edu> The amount of static memory allocated is directly related to the number of maxblocks you specify at compile time. Then there is local storage in modules that goes on the stack and is usually dependent on the size of the individual block. Hdf5_serial IO has relatively large dynamic allocation, roughly the size of the local mesh on the processor, and paramesh does its own dynamic allocation during communication. This allocation depends upon the amount of data that needs to move between processors. So the only way allocation is really influenced by spreading work out on more nodes is by cutting down the maxblocks proportionately. Maxblocks itself is a local quantity, it is the maximum number of blocks you anticipate having on any one processor at any time during your simulation. I hope that answers your question, if you need further clarification please let me know. Anshu > How does flash do memory allocation? If I am running into memory > problems but my only option for more memory is to run on more nodes > does flash subdivide the amount of memory in use evenly? > For example: > > a model when running on 4 cpus, each process takes 10GB of real memory. > > If I run that same model on 8 how close will each process get to 5 gb > of use? > > Brock Palen > Center for Advanced Computing > brockp at umich.edu > (734)936-1985 > > From brockp at umich.edu Fri Nov 30 09:49:49 2007 From: brockp at umich.edu (Brock Palen) Date: Fri, 30 Nov 2007 10:49:49 -0500 Subject: [FLASH-USERS] Flash memory usage. In-Reply-To: <1772.76.217.55.234.1196433187.squirrel@flash.uchicago.edu> References: <699B1EE1-6FD1-46E4-A28F-1859FFA18F79@umich.edu> <1772.76.217.55.234.1196433187.squirrel@flash.uchicago.edu> Message-ID: <02EE32CD-20DA-4AF0-B233-B3E9D20C7E92@umich.edu> That does help allot, but being a admin of a system ware a user is running flash, I only compiled it and ran the test cases I do not understand what its doing. The max blocks, are blocks a physical property of the model? maxblocks is the maximum number of blocks on any one cpus any time during the run, but if block represent physical space in the model, and block are distributed among cpus, using more cpus (more nodes to get more ram) and having fewer blocks per cpu would allow us to slower maxblocks thus lowering the memory needed by each cpu right? There is a 95% chance I am completely wrong. The user is running a 3d problem. The formula given to me by Georgi Pavlovski was: (nvar)x(2*nguard+nxb)x(2*nguard+nyb)x(2*nguard+nzb)x(maxblocks) So the memory use is proportional to maxblocks. Brock Palen Center for Advanced Computing brockp at umich.edu (734)936-1985 On Nov 30, 2007, at 9:33 AM, Anshu Dubey wrote: > The amount of static memory allocated is directly related to the > number of > maxblocks you specify at compile time. Then there is local storage in > modules that goes on the stack and is usually dependent on the size > of the individual block. Hdf5_serial IO has relatively large dynamic > allocation, roughly the size of the local mesh on the processor, and > paramesh does its own dynamic allocation during communication. This > allocation depends upon the amount of data that needs to move between > processors. So the only way allocation is really influenced by > spreading > work out on more nodes is by cutting down the maxblocks > proportionately. > Maxblocks itself is a local quantity, it is the maximum number > of blocks you anticipate having on any one processor at any time > during > your simulation. > > I hope that answers your question, if you need further > clarification please > let me know. > > Anshu > > >> How does flash do memory allocation? If I am running into memory >> problems but my only option for more memory is to run on more nodes >> does flash subdivide the amount of memory in use evenly? >> For example: >> >> a model when running on 4 cpus, each process takes 10GB of real >> memory. >> >> If I run that same model on 8 how close will each process get to 5 gb >> of use? >> >> Brock Palen >> Center for Advanced Computing >> brockp at umich.edu >> (734)936-1985 >> >> > > > From gbp at phys.soton.ac.uk Fri Nov 30 10:12:39 2007 From: gbp at phys.soton.ac.uk (Georgi Pavlovski) Date: Fri, 30 Nov 2007 16:12:39 +0000 Subject: [FLASH-USERS] Flash memory usage. In-Reply-To: <02EE32CD-20DA-4AF0-B233-B3E9D20C7E92@umich.edu> References: <699B1EE1-6FD1-46E4-A28F-1859FFA18F79@umich.edu> <1772.76.217.55.234.1196433187.squirrel@flash.uchicago.edu> <02EE32CD-20DA-4AF0-B233-B3E9D20C7E92@umich.edu> Message-ID: <47503677.9080808@phys.soton.ac.uk> maxblocks is the maximum amount of blocks allowed per cpu. flash (paramesh module) creates a grid at the start of the simulation. Individual blocks are distributed among available cpus. The goal is to make the load as equal as possible. When refinement/derefinement happens the blocks are getting reshuffled among cpus. The number of active (valid) blocks can grow/decrease during the run. The storage (memory requirements) are constant for a run. They are defined by the maximum allowed number of blocks per cpu during the compilation. At any one time every cpu has unk array of the quoted size. Some of that space will never be used if local number of blocks per cpu (lnblocks changes after each refinement/derefinement event) is less than the maxblocks. I hope that helps. Brock Palen wrote: > That does help allot, but being a admin of a system ware a user is > running flash, I only compiled it and ran the test cases I do not > understand what its doing. > > The max blocks, are blocks a physical property of the model? maxblocks > is the maximum number of blocks on any one cpus any time during the > run, but if block represent physical space in the model, and block are > distributed among cpus, using more cpus (more nodes to get more ram) > and having fewer blocks per cpu would allow us to slower maxblocks thus > lowering the memory needed by each cpu right? > > There is a 95% chance I am completely wrong. The user is running a 3d > problem. > > The formula given to me by Georgi Pavlovski was: > > (nvar)x(2*nguard+nxb)x(2*nguard+nyb)x(2*nguard+nzb)x(maxblocks) > > So the memory use is proportional to maxblocks. > > Brock Palen > Center for Advanced Computing > brockp at umich.edu > (734)936-1985 > > > On Nov 30, 2007, at 9:33 AM, Anshu Dubey wrote: > >> The amount of static memory allocated is directly related to the >> number of >> maxblocks you specify at compile time. Then there is local storage in >> modules that goes on the stack and is usually dependent on the size >> of the individual block. Hdf5_serial IO has relatively large dynamic >> allocation, roughly the size of the local mesh on the processor, and >> paramesh does its own dynamic allocation during communication. This >> allocation depends upon the amount of data that needs to move between >> processors. So the only way allocation is really influenced by spreading >> work out on more nodes is by cutting down the maxblocks proportionately. >> Maxblocks itself is a local quantity, it is the maximum number >> of blocks you anticipate having on any one processor at any time during >> your simulation. >> >> I hope that answers your question, if you need further clarification >> please >> let me know. >> >> Anshu >> >> >>> How does flash do memory allocation? If I am running into memory >>> problems but my only option for more memory is to run on more nodes >>> does flash subdivide the amount of memory in use evenly? >>> For example: >>> >>> a model when running on 4 cpus, each process takes 10GB of real memory. >>> >>> If I run that same model on 8 how close will each process get to 5 gb >>> of use? >>> >>> Brock Palen >>> Center for Advanced Computing >>> brockp at umich.edu >>> (734)936-1985 >>> >>> >> >> >> > > > > -- =================================================================== Dr Georgi B. Pavlovski School of Physics and Astronomy Tel: +44-(0)-23-8059-2089 University of Southampton Fax: +44-(0)-23-8059-3910 Southampton SO17 1BJ, U.K. Email: gbp at phys.soton.ac.uk Web: www.astro.soton.ac.uk/~gbp ===================================================================