From miller at astro.umd.edu Wed Dec 5 06:46:40 2007 From: miller at astro.umd.edu (Cole Miller) Date: Wed, 5 Dec 2007 07:46:40 -0500 (EST) Subject: [FLASH-USERS] Gravity, GPOL_VAR, and GPOT_VAR Message-ID: <20071205124640.24D2085864@nstar.astro.umd.edu> Hi all, I've been trying to modify the Cellular test problem to include gravity (ultimately with success, I think, but see below). Along the way I ran into a couple of things that might be useful to have on record. I'm using a version of FLASH 3.0 beta that I downloaded on September 11, 2007, so perhaps these have been changed already. 1. There is a typo in Simulation_initBlock.F90 in source/Simulation/SimulationMain/Cellular. In this particular test problem, one sets up an ambient medium with a perturbation in it, and watches the burning progress. In the Simulation_initBlock.F90 code we have the lines else if (NDIM .EQ. 2) then if (sim_usePseudo1d) then dist = xx - sim_xCenterPerturb else dist = sqrt((xx - sim_xCenterPerturb)**2 + & & (xx - sim_yCenterPerturb)**2) endif Clearly, the second to last line should have (yy - sim_yCenterPerturb)**2 instead. 2. On a much more complicated note, for a long time I was unsuccessful in adding constant gravity to this problem. I did all the obvious things, e.g., adding REQUIRES physics/Gravity/ to the Config file, along with gconst and gdirec, making sure that there was a line "useGravity = .true." in flash.par, running it in three dimensions instead of two, and so on. No luck. Eventually, with the help of Sean O'Neill here at Maryland, I tracked down the problem. In source/physics/Hydro/HydroMain/split/PPM/PPMKernel/ there is a program called hydro_1d.F90. This program reads the gravity and applies the force, if any, to the fluid. In it, we have the following lines: #ifdef GPOL_VAR call Gravity_accelOneRow (pos, xyzswp, blockID, numIntCells8,ograv,GPOL_VAR) #ifndef GPOT_VAR call Driver_abortFlash("Shouldn't have gpol defined without gpot") #endif #endif #ifdef GPOT_VAR call Gravity_accelOneRow (pos, xyzswp, blockID, numIntCells8,grav,GPOT_VAR) #endif dtfac = dt/dt_old do i = 1,numIntCells8 dg = dtfac*(grav(i) - ograv(i)) hgrav(i) = grav(i) + 0.5e0*dg ngrav(i) = grav(i) + dg enddo and then hgrav is passed to the Riemann solver as a force. The problem is that nowhere that I saw is GPOL_VAR or GPOT_VAR defined. Not in Flash.h, not in any of the files after gmake is run. I tried several variants of the setup script, with the same result. Therefore, since prior to the lines quoted above the arrays grav() and ograv() are both set to zero, hgrav is as well, so there is no gravity. However, on further examination, in the program Gravity_accelOneRow the last variable (GPOL_VAR or GPOT_VAR here) is optional, and is in fact not used at all for constant gravity. Those two variables aren't used anywhere else, either. Therefore, just before the dtfac line I added: #ifndef GPOT_VAR call Gravity_accelOneRow (pos, xyzswp, blockID, numIntCells8,grav,-1) #endif This appears to work. Is there a more elegant solution? Is there, for example, a specific option in ./setup that will define GPOL_VAR and/or GPOT_VAR? If not, perhaps it would be useful to look into this. Cole From klaus at flash.uchicago.edu Wed Dec 5 08:50:47 2007 From: klaus at flash.uchicago.edu (Klaus Weide) Date: Wed, 5 Dec 2007 08:50:47 -0600 (CST) Subject: [FLASH-USERS] Gravity, GPOL_VAR, and GPOT_VAR In-Reply-To: <20071205124640.24D2085864@nstar.astro.umd.edu> References: <20071205124640.24D2085864@nstar.astro.umd.edu> Message-ID: Cole, Thanks for your message. I am responding here only to the second part: On Wed, 5 Dec 2007, Cole Miller wrote: > 2. On a much more complicated note, for a long time I was unsuccessful > in adding constant gravity to this problem. I did all the obvious > things, e.g., adding REQUIRES physics/Gravity/ to the Config file, > along with gconst and gdirec, making sure that there was a line > "useGravity = .true." in flash.par, running it in three dimensions > instead of two, and so on. No luck. Eventually, with the help of > Sean O'Neill here at Maryland, I tracked down the problem. > > In source/physics/Hydro/HydroMain/split/PPM/PPMKernel/ > there is a program called hydro_1d.F90. This program reads the > gravity and applies the force, if any, to the fluid. In it, we have the > following lines: > #ifdef GPOL_VAR > call Gravity_accelOneRow (pos, xyzswp, blockID, numIntCells8,ograv,GPOL_VAR) > #ifndef GPOT_VAR > call Driver_abortFlash("Shouldn't have gpol defined without gpot") > #endif > #endif > #ifdef GPOT_VAR > call Gravity_accelOneRow (pos, xyzswp, blockID, numIntCells8,grav,GPOT_VAR) > #endif > dtfac = dt/dt_old > > do i = 1,numIntCells8 > dg = dtfac*(grav(i) - ograv(i)) > hgrav(i) = grav(i) + 0.5e0*dg > ngrav(i) = grav(i) + dg > enddo > > and then hgrav is passed to the Riemann solver as a force. I am sorry you had to spend time on this. We have also found after the beta release the code did not work properly with constant gravity. In the current FLASH3 development code, the relevant section in hydro_1d.F90 loos like this: ................................................................................ if (useGravity) then pos(1)=jCell; pos(2)=kCell #ifdef GPOL_VAR call Gravity_accelOneRow (pos, xyzswp, blockID, numIntCells8,ograv,GPOL_VAR) #ifndef GPOT_VAR call Driver_abortFlash("Shouldn't have gpol defined without gpot") #endif #endif #ifdef GPOT_VAR call Gravity_accelOneRow (pos, xyzswp, blockID, numIntCells8,grav,GPOT_VAR) dtfac = dt/dt_old do i = 1,numIntCells8 dg = dtfac*(grav(i) - ograv(i)) hgrav(i) = grav(i) + 0.5e0*dg ngrav(i) = grav(i) + dg enddo #else ! Neither GPOT_VAR nor GPOL_VAR defined -> assume time-independent gravity field - KW call Gravity_accelOneRow (pos, xyzswp, blockID, numIntCells8,grav) hgrav = grav ngrav = grav #endif else ograv(:) = 0.e0 hgrav(:) = 0.e0 ngrav(:) = 0.e0 end if ................................................................................ This should esssentially do the same as your fix (while saving soem unnecessary operations). > The problem is that nowhere that I saw is GPOL_VAR or GPOT_VAR defined. > Not in Flash.h, not in any of the files after gmake is run. I tried > several variants of the setup script, with the same result. Therefore, > since prior to the lines quoted above the arrays grav() and ograv() are > both set to zero, hgrav is as well, so there is no gravity. GPOT_VAR should be defined (automatically, in Flash.h) if and only if there is a "VARIABLE gpot" declaration in a Config file that is included in the setup. That is currently only the case if one uses a "Poisson" implementation of gravity, where a variable in UNK to hold the gravitational potential is actually needed. Klaus Weide, FLASH Center Code Group From flash-users at flash.uchicago.edu Fri Dec 7 00:50:01 2007 From: flash-users at flash.uchicago.edu (VIAGRA ® Official Site) Date: Fri, 7 Dec 2007 00:50:01 -0600 (CST) Subject: [FLASH-USERS] [SPAM] ***SPAM*** December 72% OFF Message-ID: <20071207125003.18982.qmail@dual> An HTML attachment was scrubbed... URL: http://flash.uchicago.edu/pipermail/flash-users/attachments/20071207/6300b25d/attachment.html From tangsk at astro.umass.edu Tue Dec 11 10:18:09 2007 From: tangsk at astro.umass.edu (Shikui Tang) Date: Tue, 11 Dec 2007 11:18:09 -0500 Subject: [FLASH-USERS] performance of parallel hdf5 on Bigben at PSC Message-ID: <475EB841.1080909@astro.umass.edu> Hi all, I have a serious problem when using the parallel hdf5 module to restart from a checkpoint file. It takes about half an hour to read a single variable to each node for a 1GB file. The same problem was reported two year ago (http://flash.uchicago.edu/pipermail/flash-users/2005-April/001938.html) but I cannot find the follow-ups. Did anyone succeed in using the parallel hdf5 module so far? Will the Flash center continue support such issue on Flash2.5? Thanks! Bests, Shikui From dubey at flash.uchicago.edu Mon Dec 17 07:56:00 2007 From: dubey at flash.uchicago.edu (Anshu Dubey) Date: Mon, 17 Dec 2007 07:56:00 -0600 (CST) Subject: [FLASH-USERS] performance of parallel hdf5 on Bigben at PSC In-Reply-To: <475EB841.1080909@astro.umass.edu> References: <475EB841.1080909@astro.umass.edu> Message-ID: <39762.75.3.145.172.1197899760.squirrel@flash.uchicago.edu> People have used the parallel hdf5 module in Flash2.5 at the center and still do, but on many platforms the restart is extremely slow. We haven't been able to determine the cause. However, fortunately, the problem didn't get carried over to Flash3, so we strongly recommend switching over to Flash3 if it is possible for you. Anshu > Hi all, > > I have a serious problem when using the parallel hdf5 module to > restart from a checkpoint file. It takes about half an hour to read a > single variable to each node for a 1GB file. The same problem was > reported two year ago > (http://flash.uchicago.edu/pipermail/flash-users/2005-April/001938.html) > but I cannot find the follow-ups. Did anyone succeed in using the > parallel hdf5 module so far? Will the Flash center continue support such > issue on Flash2.5? Thanks! > > Bests, > Shikui > From mdas at umich.edu Mon Dec 17 10:31:25 2007 From: mdas at umich.edu (Mousumi Das) Date: Mon, 17 Dec 2007 11:31:25 -0500 (EST) Subject: [FLASH-USERS] Restart In-Reply-To: <39762.75.3.145.172.1197899760.squirrel@flash.uchicago.edu> References: <475EB841.1080909@astro.umass.edu> <39762.75.3.145.172.1197899760.squirrel@flash.uchicago.edu> Message-ID: Hi All, I am using FLASH2.5. My programme stops while writing a checkpint file after outputing several plot files, but it has not completed the run till tstop value. [ 12-14-2007 21:45.22 ] step: n=364 t=5.630710E-09 dt=1.014811E-11 [ 12-14-2007 21:48.17 ] [AMR_REFINE_DEREFINE]: refinement initiated at 21:48.1 7 [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE] blocks all: min=1326 max=1336 tot=13321 [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE] blocks valid: min=1163 max=1167 tot=11656 [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE]: refinement complete [ 12-14-2007 21:49.32 ] file_wr_open: type=checkpoint name=Ni_Tem_hdf5_chk_000 2 I want to restart the simulation with previous checkpointpoint file Ni_Tem_hdf5_chk_0001. According to FLASH manual I change the restart logical variable to .true. cpunumber I specfied 0001 ( last written checkpointfile) pltnumber 0001 ( last written plotfile number after 0001 check point) Although the last written pltnumber is 0056. But there is gap between checkpoint file chk_0001 and chk_0002. But with this I am not able to restart the simulation. How do I restart the simulation run if I want to get the plotfile (0057) after the last plotfile generated (0056). thanks, Mousumi On Mon, 17 Dec 2007, Anshu Dubey wrote: > People have used the parallel hdf5 module in Flash2.5 at the center > and still do, but on many platforms the restart is extremely slow. > We haven't been able to determine the cause. However, fortunately, > the problem didn't get carried over to Flash3, so we strongly > recommend switching over to Flash3 if it is possible for you. > > Anshu > >> Hi all, >> >> I have a serious problem when using the parallel hdf5 module to >> restart from a checkpoint file. It takes about half an hour to read a >> single variable to each node for a 1GB file. The same problem was >> reported two year ago >> (http://flash.uchicago.edu/pipermail/flash-users/2005-April/001938.html) >> but I cannot find the follow-ups. Did anyone succeed in using the >> parallel hdf5 module so far? Will the Flash center continue support such >> issue on Flash2.5? Thanks! >> >> Bests, >> Shikui >> > > > > From dubey at flash.uchicago.edu Mon Dec 17 11:29:54 2007 From: dubey at flash.uchicago.edu (Anshu Dubey) Date: Mon, 17 Dec 2007 11:29:54 -0600 (CST) Subject: [FLASH-USERS] Restart In-Reply-To: References: <475EB841.1080909@astro.umass.edu> <39762.75.3.145.172.1197899760.squirrel@flash.uchicago.edu> Message-ID: <55000.75.3.145.172.1197912594.squirrel@flash.uchicago.edu> The checkpoint and plotfile numbers are not related to each other. For the plotfile, the number only provides the number from where the plotfile count starts upon restart. But if your second checkpoint didn't get written correctly, I am afraid you will have to restart from the first checkpoint, and repeat all the steps in between. You might want to increase the frequence of checkpointing if you are running into IO problems, that way you won't have to rollback as much. Anshu > > I am using FLASH2.5. My programme stops while writing a checkpint file > after outputing several plot files, but it has not completed the run till > tstop value. > > [ 12-14-2007 21:45.22 ] step: n=364 t=5.630710E-09 dt=1.014811E-11 > [ 12-14-2007 21:48.17 ] [AMR_REFINE_DEREFINE]: refinement initiated at > 21:48.1 7 > [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE] blocks all: min=1326 > max=1336 tot=13321 > [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE] blocks valid: min=1163 > max=1167 tot=11656 > [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE]: refinement complete > [ 12-14-2007 21:49.32 ] file_wr_open: type=checkpoint > name=Ni_Tem_hdf5_chk_000 2 > > > I want to restart the simulation with previous checkpointpoint file > Ni_Tem_hdf5_chk_0001. According to FLASH manual I change the restart > logical variable to .true. > cpunumber I specfied 0001 ( last written checkpointfile) > pltnumber 0001 ( last written plotfile number after 0001 check point) > > Although the last written pltnumber is 0056. But there is gap between > checkpoint file chk_0001 and chk_0002. > > But with this I am not able to restart the simulation. How do I restart > the simulation run if I want to get the plotfile (0057) after the last > plotfile generated (0056). > thanks, > Mousumi > > On Mon, 17 Dec 2007, Anshu Dubey wrote: > >> People have used the parallel hdf5 module in Flash2.5 at the center >> and still do, but on many platforms the restart is extremely slow. >> We haven't been able to determine the cause. However, fortunately, >> the problem didn't get carried over to Flash3, so we strongly >> recommend switching over to Flash3 if it is possible for you. >> >> Anshu >> >>> Hi all, >>> >>> I have a serious problem when using the parallel hdf5 module to >>> restart from a checkpoint file. It takes about half an hour to read a >>> single variable to each node for a 1GB file. The same problem was >>> reported two year ago >>> (http://flash.uchicago.edu/pipermail/flash-users/2005-April/001938.html) >>> but I cannot find the follow-ups. Did anyone succeed in using the >>> parallel hdf5 module so far? Will the Flash center continue support >>> such >>> issue on Flash2.5? Thanks! >>> >>> Bests, >>> Shikui >>> >> >> >> >> > From mdas at umich.edu Fri Dec 28 10:21:17 2007 From: mdas at umich.edu (Mousumi Das) Date: Fri, 28 Dec 2007 11:21:17 -0500 (EST) Subject: [FLASH-USERS] Restart In-Reply-To: <55000.75.3.145.172.1197912594.squirrel@flash.uchicago.edu> References: <475EB841.1080909@astro.umass.edu> <39762.75.3.145.172.1197899760.squirrel@flash.uchicago.edu> <55000.75.3.145.172.1197912594.squirrel@flash.uchicago.edu> Message-ID: Hello All, Even if I increase the frequency of the checkpointing, that is not going to help, bacause the code stops running while writting the 2nd checkpoint file always, chk_0002. Log file shows the follwoing error mesg. [ 12-28-2007 01:45.36 ] step: n=163 t=1.969343E-09 dt=8.117542E-12 [ 12-28-2007 01:47.06 ] step: n=164 t=1.985578E-09 dt=8.124520E-12 [ 12-28-2007 01:48.50 ] file_wr_open: type=checkpoint name=Ni_Tem_hdf5_chk_0002 If I see the output file it gives the error mesg [CHECKPOINT_WR] Writing checkpoint file Ni_Tem_hdf5_chk_0002 Progress: |......... Signal:11 info.si_errno:0(Success) si_code:1(SEGV_MAPERR) Failing at addr:0x2abd18f1e8 *** End of error message *** I have checked there is enough space for wrting files. Did anyone of you face such problem before? thanks, Mousumi On Mon, 17 Dec 2007, Anshu Dubey wrote: > The checkpoint and plotfile numbers are not related to each other. > For the plotfile, the number only provides the number from where > the plotfile count starts upon restart. > > But if your second checkpoint didn't get written correctly, I am afraid > you will have to restart from the first checkpoint, and repeat all the > steps in between. You might want to increase the frequence of > checkpointing if you are running into IO problems, that way you won't > have to rollback as much. > > Anshu >> >> I am using FLASH2.5. My programme stops while writing a checkpint file >> after outputing several plot files, but it has not completed the run till >> tstop value. >> >> [ 12-14-2007 21:45.22 ] step: n=364 t=5.630710E-09 dt=1.014811E-11 >> [ 12-14-2007 21:48.17 ] [AMR_REFINE_DEREFINE]: refinement initiated at >> 21:48.1 7 >> [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE] blocks all: min=1326 >> max=1336 tot=13321 >> [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE] blocks valid: min=1163 >> max=1167 tot=11656 >> [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE]: refinement complete >> [ 12-14-2007 21:49.32 ] file_wr_open: type=checkpoint >> name=Ni_Tem_hdf5_chk_000 2 >> >> >> I want to restart the simulation with previous checkpointpoint file >> Ni_Tem_hdf5_chk_0001. According to FLASH manual I change the restart >> logical variable to .true. >> cpunumber I specfied 0001 ( last written checkpointfile) >> pltnumber 0001 ( last written plotfile number after 0001 check point) >> >> Although the last written pltnumber is 0056. But there is gap between >> checkpoint file chk_0001 and chk_0002. >> >> But with this I am not able to restart the simulation. How do I restart >> the simulation run if I want to get the plotfile (0057) after the last >> plotfile generated (0056). >> thanks, >> Mousumi >> >> On Mon, 17 Dec 2007, Anshu Dubey wrote: >> >>> People have used the parallel hdf5 module in Flash2.5 at the center >>> and still do, but on many platforms the restart is extremely slow. >>> We haven't been able to determine the cause. However, fortunately, >>> the problem didn't get carried over to Flash3, so we strongly >>> recommend switching over to Flash3 if it is possible for you. >>> >>> Anshu >>> >>>> Hi all, >>>> >>>> I have a serious problem when using the parallel hdf5 module to >>>> restart from a checkpoint file. It takes about half an hour to read a >>>> single variable to each node for a 1GB file. The same problem was >>>> reported two year ago >>>> (http://flash.uchicago.edu/pipermail/flash-users/2005-April/001938.html) >>>> but I cannot find the follow-ups. Did anyone succeed in using the >>>> parallel hdf5 module so far? Will the Flash center continue support >>>> such >>>> issue on Flash2.5? Thanks! >>>> >>>> Bests, >>>> Shikui >>>> >>> >>> >>> >>> >> > > > > From dubey at flash.uchicago.edu Fri Dec 28 10:27:28 2007 From: dubey at flash.uchicago.edu (Anshu Dubey) Date: Fri, 28 Dec 2007 10:27:28 -0600 (CST) Subject: [FLASH-USERS] Restart In-Reply-To: References: <475EB841.1080909@astro.umass.edu> <39762.75.3.145.172.1197899760.squirrel@flash.uchicago.edu> <55000.75.3.145.172.1197912594.squirrel@flash.uchicago.edu> Message-ID: <40347.75.3.130.101.1198859248.squirrel@flash.uchicago.edu> Try switching the IO mode between hdf5_serial and hdf5_parallel and see if you do any better. > Hello All, > Even if I increase the frequency of the checkpointing, that is not > going to help, bacause the code stops running while writting the 2nd > checkpoint file always, chk_0002. Log file shows the follwoing error mesg. > > [ 12-28-2007 01:45.36 ] step: n=163 t=1.969343E-09 dt=8.117542E-12 > [ 12-28-2007 01:47.06 ] step: n=164 t=1.985578E-09 dt=8.124520E-12 > [ 12-28-2007 01:48.50 ] file_wr_open: type=checkpoint > name=Ni_Tem_hdf5_chk_0002 > > If I see the output file it gives the error mesg > [CHECKPOINT_WR] Writing checkpoint file Ni_Tem_hdf5_chk_0002 > Progress: |......... > Signal:11 info.si_errno:0(Success) si_code:1(SEGV_MAPERR) > Failing at addr:0x2abd18f1e8 > *** End of error message *** > > I have checked there is enough space for wrting files. Did anyone of you > face such problem before? > thanks, > Mousumi > > > On Mon, 17 Dec 2007, Anshu Dubey wrote: > >> The checkpoint and plotfile numbers are not related to each other. >> For the plotfile, the number only provides the number from where >> the plotfile count starts upon restart. >> >> But if your second checkpoint didn't get written correctly, I am afraid >> you will have to restart from the first checkpoint, and repeat all the >> steps in between. You might want to increase the frequence of >> checkpointing if you are running into IO problems, that way you won't >> have to rollback as much. >> >> Anshu >>> >>> I am using FLASH2.5. My programme stops while writing a checkpint >>> file >>> after outputing several plot files, but it has not completed the run >>> till >>> tstop value. >>> >>> [ 12-14-2007 21:45.22 ] step: n=364 t=5.630710E-09 dt=1.014811E-11 >>> [ 12-14-2007 21:48.17 ] [AMR_REFINE_DEREFINE]: refinement initiated >>> at >>> 21:48.1 7 >>> [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE] blocks all: >>> min=1326 >>> max=1336 tot=13321 >>> [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE] blocks valid: >>> min=1163 >>> max=1167 tot=11656 >>> [ 12-14-2007 21:49.17 ] [AMR_REFINE_DEREFINE]: refinement complete >>> [ 12-14-2007 21:49.32 ] file_wr_open: type=checkpoint >>> name=Ni_Tem_hdf5_chk_000 2 >>> >>> >>> I want to restart the simulation with previous checkpointpoint file >>> Ni_Tem_hdf5_chk_0001. According to FLASH manual I change the restart >>> logical variable to .true. >>> cpunumber I specfied 0001 ( last written checkpointfile) >>> pltnumber 0001 ( last written plotfile number after 0001 check point) >>> >>> Although the last written pltnumber is 0056. But there is gap between >>> checkpoint file chk_0001 and chk_0002. >>> >>> But with this I am not able to restart the simulation. How do I restart >>> the simulation run if I want to get the plotfile (0057) after the last >>> plotfile generated (0056). >>> thanks, >>> Mousumi >>> >>> On Mon, 17 Dec 2007, Anshu Dubey wrote: >>> >>>> People have used the parallel hdf5 module in Flash2.5 at the center >>>> and still do, but on many platforms the restart is extremely slow. >>>> We haven't been able to determine the cause. However, fortunately, >>>> the problem didn't get carried over to Flash3, so we strongly >>>> recommend switching over to Flash3 if it is possible for you. >>>> >>>> Anshu >>>> >>>>> Hi all, >>>>> >>>>> I have a serious problem when using the parallel hdf5 module to >>>>> restart from a checkpoint file. It takes about half an hour to read a >>>>> single variable to each node for a 1GB file. The same problem was >>>>> reported two year ago >>>>> (http://flash.uchicago.edu/pipermail/flash-users/2005-April/001938.html) >>>>> but I cannot find the follow-ups. Did anyone succeed in using the >>>>> parallel hdf5 module so far? Will the Flash center continue support >>>>> such >>>>> issue on Flash2.5? Thanks! >>>>> >>>>> Bests, >>>>> Shikui >>>>> >>>> >>>> >>>> >>>> >>> >> >> >> >> >