From woitke at strw.leidenuniv.nl  Wed Sep  8 06:24:57 2004
From: woitke at strw.leidenuniv.nl (Peter Woitke)
Date: Wed, 8 Sep 2004 13:24:57 +0200 (MEST)
Subject: [FLASH-BUGS] How to define mass scalars in FLASH 2.4?
Message-ID: <Pine.LNX.4.58.0409081225190.12677@berkel.strw.leidenuniv.nl>

Dear developers,

the option to include "mass scalars" is new in FLASH 2.4, which obey
the conservation equations

  d/dt(\rho\phi_i) + \nabla(\rho\phi_i u) = 0

with the option to include user-defines source terms. Fine. However, I
could not find much information about the usage of mass scalars in the
user manual. (The EOS=0-species I used before to advect my variables in
FLASH 2.3 are no longer supported in 2.4).  After defining the number of
mass scalars in Config by e.g.

NUMMASSSCALARS 5

I need to specify the names of these quantities, don't I?
(Otherwise I don't know how to get an output of these variables
in the plotfiles)

- How can I do this?


Further questions:

- How can I avoid/circumvent the 0<=\phi_i<=1 condition? What is
  the CMA-flag in hydro_1d?

- How are these variables "prolonged"? Are they part of the 1...nvar
  variables in the unk-vector? Can I use a CONSERVED treatment in
  amr_prolong_gen_unk_fun.F90?

Thank you for your help,

Peter Woitke


From tomek at flash.uchicago.edu  Wed Sep  8 11:49:42 2004
From: tomek at flash.uchicago.edu (Tomasz Plewa)
Date: Wed, 8 Sep 2004 11:49:42 -0500
Subject: [FLASH-BUGS] How to define mass scalars in FLASH 2.4?
In-Reply-To: <Pine.LNX.4.58.0409081225190.12677@berkel.strw.leidenuniv.nl>; from woitke@strw.leidenuniv.nl on Wed, Sep 08, 2004 at 01:24:57PM +0200
References: <Pine.LNX.4.58.0409081225190.12677@berkel.strw.leidenuniv.nl>
Message-ID: <20040908114942.A16870@flash.uchicago.edu>

Peter -

On Wed, Sep 08, 2004 at 01:24:57PM +0200, Peter Woitke wrote:
[...]
> FLASH 2.3 are no longer supported in 2.4).  After defining the number of
> mass scalars in Config by e.g.
> 
> NUMMASSSCALARS 5
> 
> I need to specify the names of these quantities, don't I?
> (Otherwise I don't know how to get an output of these variables
> in the plotfiles)
>
> - How can I do this?

Check

	source/io/getvarlabels.F90

Mass scalars are named "ms_1", "ms_2", ..., "ms_9".

Apparently one cannot have more than 9 unless that (local) convention
is modified by user.

> Further questions:
> 
> - How can I avoid/circumvent the 0<=\phi_i<=1 condition? What is
>   the CMA-flag in hydro_1d?

CMA takes care about mass fractions; mass scalars are treated independent
of each other and are not subject to limiting. You may want to extend
or modify CMA parts if you need this additional functionality.

> - How are these variables "prolonged"? Are they part of the 1...nvar
>   variables in the unk-vector? Can I use a CONSERVED treatment in
>   amr_prolong_gen_unk_fun.F90?

Mass scalars are evolved by solving advection equation in the conservation
form. So these are conserved and convervative prolongation should preserve
that property.

Hope this helps.

Tomek
-- 
Wed, 11:49 CDT (16:49 GMT), Sep-08-2004
_______________________________________________________________________________

   Tomasz Plewa                                      www:   flash.uchicago.edu
   Computational Physics and Validation Group        email: tomek at uchicago.edu
   The ASC FLASH Center, The University of Chicago   phone: 773.834.3227
   5640 South Ellis, RI 475, Chicago, IL 60637       fax:   773.834.3230
_______________________________________________________________________________


From shawn at flash.uchicago.edu  Thu Sep 23 14:23:09 2004
From: shawn at flash.uchicago.edu (Shawn Needham)
Date: Thu, 23 Sep 2004 14:23:09 -0500 (CDT)
Subject: [FLASH-BUGS] Non-member submission from [Erik-Jan Rijkhorst <rijkhorst@strw.leidenuniv.nl>]
 (fwd)
Message-ID: <Pine.LNX.4.44.0409231420170.15720-100000@flash.uchicago.edu>

Date: Thu, 23 Sep 2004 17:56:11 +0200 (MEST)
From: Erik-Jan Rijkhorst <rijkhorst at strw.leidenuniv.nl>
X-X-Sender: rijkhors at laak.strw.leidenuniv.nl
To: flash-bugs at flash.uchicago.edu
cc: Peter Woitke <woitke at strw.leidenuniv.nl>
Subject: sedov_sph with Flash2.4

Dear developers,

We are having trouble to run the 1D spherical sedov_sph setup on the 
parallel machines we have at our disposal. The machines are:

Aster with Intel efc/ecc version 8.0 compilers
An SGI Altix 3700 system, consisting of 416 CPUs (Intel Itanium 2, 1,3 GHz]

and

Teras with MIPSpro SGI f90/cc compilers
1024-CPU system consisting of two 512-CPU SGI Origin 3800 systems

(http://www.sara.nl for more info)

The problem we encounter is that at some timestep, the variables undergo 
sudden gross changes, e.g. creating new velocity jumps at locations where 
nothing should happen.

We have used simple testing compilation flags, e.g. without optimisation.

These problems occur if (and only if) the following three conditions are 
met:

  1) refinement is allowed, i.e. lrefinmax > lrefine_min. We used
     lrefine_min=1 and lrefine_max=8
  2) the module /mesh/amr/paramesh2.0/quadratic_spherical is involved
     (we have similar problems with other applications using this module)
  3) a certain number of processors is used, e.g.

  4 PE  =>  ok
  5 PE  =>  ok
  6 PE  =>  ok
  7 PE  =>  ok
  8 PE  =>  sudden variable jumps at timestep 1666
  9 PE  =>  ok
10 PE  =>  ok
12 PE  =>  ok
14 PE  =>  ok
16 PE  =>  sudden variable jumps at timestep 444

The errors are reproducable and similar on Aster and Teras (e.g. occur at 
the same timestep).

(However, on Aster with 8 PEs, we get a "Nonconvergence in subroutine 
rieman!"-error at timestep 1666.)

Because of the 3 conditions mentioned above needed to trigger this 
behaviour, we currently believe that there probably is a bug somewhere in 
the prolongation (or possibly guard cell exchange routines?) that go with 
the new quadratic/spherical routines.

We have checked some of the 'Flash daily test results' that are on the web 
but couldn't find a test with the new spherical module (like sod_sph or 
sedov_sph) done with more than 2 processors. Have such test with for 
example 8 processors been done on similar systems as ours?

If needed we can send you more information about the architecture, 
compiler and/or compilation flags used etc. Just tell us what you need.

Thank you very much for your help,

Peter Woitke and Erik-Jan Rijkhorst



From woitke at strw.leidenuniv.nl  Wed Sep  8 06:24:57 2004
From: woitke at strw.leidenuniv.nl (Peter Woitke)
Date: Wed, 8 Sep 2004 13:24:57 +0200 (MEST)
Subject: [FLASH-BUGS] How to define mass scalars in FLASH 2.4?
Message-ID: <Pine.LNX.4.58.0409081225190.12677@berkel.strw.leidenuniv.nl>

Dear developers,

the option to include "mass scalars" is new in FLASH 2.4, which obey
the conservation equations

  d/dt(\rho\phi_i) + \nabla(\rho\phi_i u) = 0

with the option to include user-defines source terms. Fine. However, I
could not find much information about the usage of mass scalars in the
user manual. (The EOS=0-species I used before to advect my variables in
FLASH 2.3 are no longer supported in 2.4).  After defining the number of
mass scalars in Config by e.g.

NUMMASSSCALARS 5

I need to specify the names of these quantities, don't I?
(Otherwise I don't know how to get an output of these variables
in the plotfiles)

- How can I do this?


Further questions:

- How can I avoid/circumvent the 0<=\phi_i<=1 condition? What is
  the CMA-flag in hydro_1d?

- How are these variables "prolonged"? Are they part of the 1...nvar
  variables in the unk-vector? Can I use a CONSERVED treatment in
  amr_prolong_gen_unk_fun.F90?

Thank you for your help,

Peter Woitke


From tomek at flash.uchicago.edu  Wed Sep  8 11:49:42 2004
From: tomek at flash.uchicago.edu (Tomasz Plewa)
Date: Wed, 8 Sep 2004 11:49:42 -0500
Subject: [FLASH-BUGS] How to define mass scalars in FLASH 2.4?
In-Reply-To: <Pine.LNX.4.58.0409081225190.12677@berkel.strw.leidenuniv.nl>; from woitke@strw.leidenuniv.nl on Wed, Sep 08, 2004 at 01:24:57PM +0200
References: <Pine.LNX.4.58.0409081225190.12677@berkel.strw.leidenuniv.nl>
Message-ID: <20040908114942.A16870@flash.uchicago.edu>

Peter -

On Wed, Sep 08, 2004 at 01:24:57PM +0200, Peter Woitke wrote:
[...]
> FLASH 2.3 are no longer supported in 2.4).  After defining the number of
> mass scalars in Config by e.g.
> 
> NUMMASSSCALARS 5
> 
> I need to specify the names of these quantities, don't I?
> (Otherwise I don't know how to get an output of these variables
> in the plotfiles)
>
> - How can I do this?

Check

	source/io/getvarlabels.F90

Mass scalars are named "ms_1", "ms_2", ..., "ms_9".

Apparently one cannot have more than 9 unless that (local) convention
is modified by user.

> Further questions:
> 
> - How can I avoid/circumvent the 0<=\phi_i<=1 condition? What is
>   the CMA-flag in hydro_1d?

CMA takes care about mass fractions; mass scalars are treated independent
of each other and are not subject to limiting. You may want to extend
or modify CMA parts if you need this additional functionality.

> - How are these variables "prolonged"? Are they part of the 1...nvar
>   variables in the unk-vector? Can I use a CONSERVED treatment in
>   amr_prolong_gen_unk_fun.F90?

Mass scalars are evolved by solving advection equation in the conservation
form. So these are conserved and convervative prolongation should preserve
that property.

Hope this helps.

Tomek
-- 
Wed, 11:49 CDT (16:49 GMT), Sep-08-2004
_______________________________________________________________________________

   Tomasz Plewa                                      www:   flash.uchicago.edu
   Computational Physics and Validation Group        email: tomek at uchicago.edu
   The ASC FLASH Center, The University of Chicago   phone: 773.834.3227
   5640 South Ellis, RI 475, Chicago, IL 60637       fax:   773.834.3230
_______________________________________________________________________________


From shawn at flash.uchicago.edu  Thu Sep 23 14:23:09 2004
From: shawn at flash.uchicago.edu (Shawn Needham)
Date: Thu, 23 Sep 2004 14:23:09 -0500 (CDT)
Subject: [FLASH-BUGS] Non-member submission from [Erik-Jan Rijkhorst <rijkhorst@strw.leidenuniv.nl>]
 (fwd)
Message-ID: <Pine.LNX.4.44.0409231420170.15720-100000@flash.uchicago.edu>

Date: Thu, 23 Sep 2004 17:56:11 +0200 (MEST)
From: Erik-Jan Rijkhorst <rijkhorst at strw.leidenuniv.nl>
X-X-Sender: rijkhors at laak.strw.leidenuniv.nl
To: flash-bugs at flash.uchicago.edu
cc: Peter Woitke <woitke at strw.leidenuniv.nl>
Subject: sedov_sph with Flash2.4

Dear developers,

We are having trouble to run the 1D spherical sedov_sph setup on the 
parallel machines we have at our disposal. The machines are:

Aster with Intel efc/ecc version 8.0 compilers
An SGI Altix 3700 system, consisting of 416 CPUs (Intel Itanium 2, 1,3 GHz]

and

Teras with MIPSpro SGI f90/cc compilers
1024-CPU system consisting of two 512-CPU SGI Origin 3800 systems

(http://www.sara.nl for more info)

The problem we encounter is that at some timestep, the variables undergo 
sudden gross changes, e.g. creating new velocity jumps at locations where 
nothing should happen.

We have used simple testing compilation flags, e.g. without optimisation.

These problems occur if (and only if) the following three conditions are 
met:

  1) refinement is allowed, i.e. lrefinmax > lrefine_min. We used
     lrefine_min=1 and lrefine_max=8
  2) the module /mesh/amr/paramesh2.0/quadratic_spherical is involved
     (we have similar problems with other applications using this module)
  3) a certain number of processors is used, e.g.

  4 PE  =>  ok
  5 PE  =>  ok
  6 PE  =>  ok
  7 PE  =>  ok
  8 PE  =>  sudden variable jumps at timestep 1666
  9 PE  =>  ok
10 PE  =>  ok
12 PE  =>  ok
14 PE  =>  ok
16 PE  =>  sudden variable jumps at timestep 444

The errors are reproducable and similar on Aster and Teras (e.g. occur at 
the same timestep).

(However, on Aster with 8 PEs, we get a "Nonconvergence in subroutine 
rieman!"-error at timestep 1666.)

Because of the 3 conditions mentioned above needed to trigger this 
behaviour, we currently believe that there probably is a bug somewhere in 
the prolongation (or possibly guard cell exchange routines?) that go with 
the new quadratic/spherical routines.

We have checked some of the 'Flash daily test results' that are on the web 
but couldn't find a test with the new spherical module (like sod_sph or 
sedov_sph) done with more than 2 processors. Have such test with for 
example 8 processors been done on similar systems as ours?

If needed we can send you more information about the architecture, 
compiler and/or compilation flags used etc. Just tell us what you need.

Thank you very much for your help,

Peter Woitke and Erik-Jan Rijkhorst