#+TITLE:     TM5-MP USER MANUAL (r860)
#+AUTHOR:    Ph. Le Sager, KNMI
#+EMAIL:     sager@knmi.nl
#+DATE:      2018-07-18
#+DESCRIPTION:
#+KEYWORDS:
#+LANGUAGE:  en
#+OPTIONS:   H:3 num:t toc:t \n:nil @:t ::t |:t ^:nil -:t f:t *:t <:t
#+OPTIONS:   TeX:t LaTeX:t skip:nil d:nil todo:t pri:nil tags:not-in-toc
#+INFOJS_OPT: view:nil toc:t ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js
#+EXPORT_SELECT_TAGS: export
#+EXPORT_EXCLUDE_TAGS: noexport
#+LINK_UP:
#+LINK_HOME:
#+XSLT:

* Introduction and conventions
This document is a user guide to TM5-MP, which differs from its predecessor
TM5 (referred as TM5-zoom from now on) in its MPI implementation and by the
absence of zoom options.
All shell commands in the text assume BASH shell.

* TM5-MP subversion repository cheat-sheet
Here is a very short guide for using TM5-MP subversion repository. It is not an
introduction to subversion (svn), but the collection of svn commands used in
this manual, gathered here for quick reference.

#+BEGIN_SRC sh
# environment
export repo='https://svn.knmi.nl/svn/TM5-MP'

# start with a frozen release...
svn co $repo/tags/1.0b

# ...or a branch for development
svn cp $repo/trunk $repo/branches/mybranch -m "create my fancy branch"
svn co $repo/branches/mybranch

# save your work
cd mybranch
svn mkdir DIR         # create DIR and schedule for addition
svn add NEWFILE       # schedule NEWFILE for addition
svn del OLDFILE       # schedule OLDFILE for removal
svn ci -m "message"   # commit
svn up

# get updates committed by other developers of the same branch
svn up

# incorporate changes from the trunk into my clean (without edits) branch
cd mybranch
svn merge ^/trunk

# get information
svn info
svn log
svn stat
svn diff
svn annotate

cd mybranch
svn mergeinfo ^/trunk

# undo
svn merge ^/branches/mybranch -c -NUMBER  # undo locally the commit NUMBER
svn revert FILENAME                       # undo local edit of FILENAME

# reintegrate my branch into the trunk [Administrators only]
cd trunk
svn merge ^/branches/mybranch

# resolving conflict
# TODO

# move your local edits to a new branch
svn copy  $repo/trunk  $repo/branches/new-branch -m "created new branch"
svn switch ^/branches/new-branch
svn ci -m "message"
#+END_SRC

* Getting the code
** Preliminaries
For convenience, the SVN repository root address is exported to the environment:
#+BEGIN_SRC sh
 export repo='https://svn.knmi.nl/svn/TM5-MP'
#+END_SRC
The /repo/ variable is used throughout the manual.

** Getting access to the repository
*** Credentials
    To get access, you need a username and password that can be
    obtained after contacting the repository administrator. These
    credentials will let you
    - visit the repository through a web interface, and
    - use the subversion software to get a copy of the code, and
      commit your own code to the repository.

    At some location, subversion accesses internet over a proxy. Ask
    your administrator for details on how to set up your connection.
    One proxy example is the ECMWF gateway. See hereafter.

*** Proxy set up: example of ecgate at ECMWF
To access the subversion server at KNMI from the 'ecgate' computer at ECMWF,
first check if the following file exists:
#+BEGIN_SRC sh
 ~/.subversion/servers
#+END_SRC

If not, run 'svn help' to create one. Add the following lines at the end of
the file (in the [global] section) :
#+BEGIN_SRC sh
 http-proxy-host = proxy.ecmwf.int
 http-proxy-port = 3333
#+END_SRC

*** First time connection to KNMI-SVN server
The first time you try to use svn (for example on ECMWF ecgate
machine), you may get an "error" like :
#+BEGIN_EXAMPLE
Error validating server certificate for
'https://svn.knmi.nl:443':
- The certificate is not issued by a trusted authority. Use the fingerprint to validate the certificate manually!
Certificate information:
- Hostname: svn.knmi.nl
- Valid: from Nov 30 00:00:00 2009 GMT until Nov 29 23:59:59 2012 GMT
- Issuer: TERENA, NL
- Fingerprint: ab:47:53:94:cb:de:7b:5.........
(R)eject, accept (t)emporarily or accept (p)ermanently?
#+END_EXAMPLE
You must accept permanently. Then enter your password. However,
the server first assumes that your KNMI-SVN user id is the same
as your client id (ECMWF id for example). So it may not work, and
will ask a second time, now asking for both user id and password.
Enter your KNMI-SVN ones, and you are all set.

*** Web interfaces
You can have a look at the TM5 code in the repository through a
user-friendly web interface:
- http://dev.knmi.nl/projects/tm5mp/repository
- https://svn.knmi.nl/usvn/project/TM5-MP

Note that the web interface is convenient for peeking at the code and the
latest commits, but *cannot* be used to get a copy of the code or run any svn
command.

** Get a copy for testing or production runs
You need to checkout the version XYZ you want to use:
#+BEGIN_SRC sh
 svn co $repo/tags/XYZ  MYNAME
#+END_SRC
where MYNAME is not mandatory, and defaults to "XYZ".

** Get a copy for development
You need to create and checkout a branch. Most of the time you branch out from
the stable trunk, i.e. your branch starts as a copy of it:
#+BEGIN_SRC sh
 svn cp $repo/trunk  $repo/branches/mybranch -m "create my fancy branch"
 svn co $repo/branches/mybranch MYNAME
#+END_SRC
where MYNAME is not mandatory, and defaults to "mybranch".

* Get the input data
** Full sets
*** Meteo
    See http://tm.knmi.nl/index.php/ERA_Interim_meteo for a
    description of the driving met fields, and to find out where to
    retrieve them from.
*** Emissions for chemistry projects
    The large emissions inventories for year $YYYY can be retrieved
    from the following locations at ECMWF. For all these 4
    inventories, once unpacked, specify their path in your
    "chem.input.rc". It is suggested to put each of them in its own
    directory below a common directory: EMISSDIR/MEGAN,
    EMISSDIR/EDGAR, EMISSDIR/AR5,...

**** CMIP6
     The CMIP6 emissions cover a long period and are stored in several archives:
#+BEGIN_SRC sh
   ec:/nm6/EC-EARTH/ECEARTH3.2b/INPUT/ece-data-tm5-cmip6-anthro.tgz
   ec:/nm6/EC-EARTH/ECEARTH3.2b/INPUT/ece-data-tm5-cmip6-biomass.tgz
   ec:/nm6/EC-EARTH/ECEARTH3.2b/INPUT/ece-data-tm5-cmip6-solid-biofuel.tgz
   ec:/nm6/EC-EARTH/ECEARTH3.2b/INPUT/ece-data-tm5-cmip6-speciated-1850-1899.tgz
   ec:/nm6/EC-EARTH/ECEARTH3.2b/INPUT/ece-data-tm5-cmip6-speciated-1900-1999.tgz
   ec:/nm6/EC-EARTH/ECEARTH3.2b/INPUT/ece-data-tm5-cmip6-speciated-2000-2014.tgz
   ec:/nm6/EC-EARTH/ECEARTH3.2b/INPUT/ece-data-tm5-cmip6-air-histo.tgz
#+END_SRC

**** MACC-City (1998-2010, anthropogenic + natural)
#+BEGIN_SRC sh
 ec:/nm6/EMISSIONS/MACCity/${YYYY}.tar
 ec:/nm6/EMISSIONS/MACCity/gridbox_area.nc
#+END_SRC
**** MEGAN (1980-2010, biogenic)
#+BEGIN_SRC sh
 # For 2000-2010: 
 ec:/nm6/EMISSIONS/MEGAN/MEGAN_${YYYY}.tar
 # For 1980-1999:
 ec:/nks/EMISSIONS/MEGAN/MEGAN_${YYYY}.tar
#+END_SRC
**** EDGAR 4.x
EDGAR 4.2 (1970-2008, anthropogenic sectors except transport) along EDGAR 4.1
(2005,transport sector only) can replace MACC-City for the anthropogenic
emissions. But it is used in any case for CH4, which is missing in MACC-City.
#+BEGIN_SRC sh
 ec:/nm6/EMISSIONS/EDGAR4/gridbox_area.nc
 ec:/nm6/EMISSIONS/EDGAR4/EDGAR41_2005.tar
#+END_SRC
#+BEGIN_SRC sh
 ec:/nm6/EMISSIONS/EDGAR4/EDGAR42_${YYYY}.tar
#+END_SRC
After extraction, put all the EDGAR files in the same TGT target directory (TM5 requirement):
#+BEGIN_SRC sh
 mv EDGAR41_2005/*    TGT
 mv EDGAR42_${YYYY}/* TGT
#+END_SRC
**** GFED3 (1998-2010, biomass burning)
#+BEGIN_SRC sh
 ec:/nks/EMISSIONS/GFED3/GFED_BCOC.tar # for OC/BC
 ec:/nm6/EMISSIONS/GFED3/GFED3_$YYYY.tar # for the other species
#+END_SRC
**** AR5
     The AR5 inventory provides BC and OC emissions when running with
     the M7 aerosols module. Other species (anthropogenic and biomass
     burning) emissions are used when running TM5 coupled to EC-Earth,
     and will soon be replaced with CMIP6 inventories.

#+BEGIN_SRC sh
 ec:/nks/EMISSIONS/AR5/AR5_$YYYY.tar # for historical period (up to 2000 included)
 ec:/nks/nmi_ECFS/EMIS/AR5/IPCC_AR5_emissions_$RCP_$YYYY.tar.gz
#+END_SRC

*** Other inputs
    The "common input dataset" (land-sea mask, photolysis data,
    aerosols, LPJ emissions inventories...) is used by all projects
    and found in:
    #+BEGIN_SRC sh
    ec:/nm6/EMISSIONS/MISC/TM5_INPUT_20180719.tar.gz
    #+END_SRC
    Set the key "my.data.dir" in your 'machine.rc' file to the path
    where you unpacked this archive. That is:
    #+BEGIN_SRC sh
    my.data.dir : /YOUR/PATH/TO/TM5_INPUT
    #+END_SRC
*** Restart files
    Initial values of the state variables are available for testing
    the chemistry. Several restart files options are available for
    testing purposes in the "input data" from the previous section:
    #+BEGIN_SRC sh
     /YOUR/PATH/TO/TM5_INPUT/testdata_restart
    #+END_SRC
    You can test restart options 31 (obsolete), 32 (partial restart),
    33 (full restart), and 5 (mmix) with those data. See the README
    therein for details, and the [[https://dev.knmi.nl/projects/tm5mp/wiki/Restart_options][wiki page about restart]].

** Benchmark subsets
   For testing purposes, a subset of the larger datasets is available.
   You still need to retrieve the [[Other inputs]["common input data set"]]. You can get
   a subset of 2005-2006 data from MACC-City, MEGAN, GFED3, AR5,
   EDGAR:
   #+BEGIN_SRC sh
   ec:/nm6/EMISSIONS/MISC/TM5_EMISS_20160923.tar.gz
   #+END_SRC
   For the meteo, the following sets are enough to simulate January
   2006:
   #+BEGIN_SRC sh
   ec:/nm6/EMISSIONS/MISC/bench_v4_meteo_20130711_mfuvw.tar.gz
   ec:/nm6/EMISSIONS/MISC/bench_v4_meteo_20130711_ml60.tar.gz
   ec:/nm6/EMISSIONS/MISC/bench_v4_meteo_20130711_sfc.tar.gz
   ec:/nm6/EMISSIONS/MISC/bench_v4_meteo_20140319_conv.tar.gz
   #+END_SRC
* First things after checkout
** TM5 rc files
   For a general introduction to the rc files used by TM5, visit the wiki:
   https://dev.knmi.nl/projects/tm5mp/wiki/Introducing_the_RC_files.
   To get started, copy a template from the rc sub-directory to the top
   directory:
   #+BEGIN_SRC sh
   cd MYNAME
    
   cp rc/main-base.rc.tmpl  my.rc           # for the base only
    
   cp rc/main-chem.rc.tmpl  my.rc                   # for the chemistry
   cp rc/chem-input-default.rc.tmpl  chem-input.rc  # for the chemistry
   #+END_SRC
   Modify these copies for your environment (next section). On the
   wiki, you will find an [[https://dev.knmi.nl/projects/tm5mp/wiki/TM5_rc_files][introduction to the various TM5 rc files]]
   required to compile and run the model. Pages that describes the
   keys of the [[https://dev.knmi.nl/projects/tm5mp/wiki/Main_RC_file][main RC file]], and specifically the [[https://dev.knmi.nl/projects/tm5mp/wiki/Restart_options][restart options]] and
   [[https://dev.knmi.nl/projects/tm5mp/wiki/Output_setup][output options]] are also available. The /rc/README/ file in the
   source code provides also some useful information.

   Some information that you will not find on these wiki pages:
   - The flag "with_optics" is used to computed AOD from M7. It has
     then no effect if "with_m7" is not set. Note that if "with_m7" is
     set but not "with_optics", then M7 tracers are transported but
     not used to compute AODs.

** Environment setup
*** Requirements
    You need to have HDF4 and netCDF4 libraries installed. The netCDF4
    and the underlying HDF5 must be installed with parallel I/O
    enabled if you want to output chemistry time-series. You also need
    python, subversion (1.6 or above), GNU make (sometimes installed
    as /gmake/,and sometimes as /make/), and a FORTRAN compiler. To
    find out the F90 dependencies, we use the /makedepf90/ program. It
    is available at
    http://personal.inet.fi/private/erikedelmann/makedepf90/. Make
    sure that it is on your $PATH.
*** Machine, expert, and compiler rcfiles
    There is a set of rcfiles with advanced settings that users rarely
    modified. They mostly deal with hardware and software specifics,
    and are found in the 'rc' sub-directory. For porting TM5 to a new
    platform, start by copying the /machine/ template:

    : cp machine-platform.rc.tmpl machine-<institute>-<platform>.rc

    and follow the explanations in the file.

    All rcfiles that need to be ported from the TM5-zoom model still
    have the "pycasso-" prefix, and are found in the /rc/not-ported/
    directory. Make the needed modifications, following files that are
    already ported and the tips hereafter, and rename them by removing
    the "pycasso-" prefix.

*** Tip for chem-input rcfile
    By putting all the emissions inventories under the same common
    directory you just have to specify:
    : topdir.dynamic  : /path/to/parent/dir/of/emissions/inventories
    in your /chem.input.rc/. This will automatically set the path to
    the various inventories:
   : input.emis.dir.ED41
   : input.emis.dir.gfed
   : input.emis.dir.MACC
   : input.emis.dir.MEGAN
   : input.emis.dir.AR5

*** Tips for machine file
    To modify a machine file from TM5-zoom, there are three (3)
    potential changes to look for:
- *my.meteo.dir* has been removed (it is now in the main rc file).
  This was motivated by the fact that several users on the same
  machine were using a totally different runtime meteo dir, and we
  want to keep one machine file per machine.
- check path in *#include* statements, if any (typically to a compiler or
  queue rc file)
- libraries: udunits 1.x (udunits for FORTRAN) and GRIBEX are not maintained
  anymore, and become more and more difficult to install on recent
  machines/compilers. We added an interface to udunits v2.x (udunits for C),
  and to the grib_api.
  + *grib_api* has been added. This is needed for those who process
    meteo in grib format. GRIBEX is not supported anymore in TM5.
  + udunits is obsolete, and should be replace by *udunits1*
    or *udunits2*, depending on which version is available on your system.
    However udunits, whic only applied to netcdf met fields, is now optional.

*** Tips for job manager
    The job manager rcfiles have not changed from TM5-zoom. If yours
    is not available, you need to add it to the
    "bin/submit_tm5_tools.py" script. Beware of /qsub/: there are
    several of them around, and not all are handled in the script.

** Differences with TM5-zoom
   The basic tree structure with a base, levels, and proj dirs, the
   later with a collection of project dirs, will be familiar to users
   of TM5-zoom. However there are few changes in the user interface,
   which are listed here.
*** Moved keys
- move *region.rc* out of expert.rc into the main.rc. Motivation: the reduced
  grid definition may differ and user should be aware of it. For example,
  there are two versions of the region-glb100x100.rc with two totally
  different reduced grids.
- move *meteo dir* (too much user dependent) into main.rc, out of the machine.rc
- move *my.source.dirs* to expert.rc. In your main.rc, you just provide the
  list of proj for the *my.source.proj* key.
  key. See the templates. This simplification is feasible, because we now have
  only one base and automatic selection of levels code.
- move a set of keys *(jobstep, jobstep.timerange.start,
  jobstep.timerange.end, prev.output.dir)* into expert rcfile: no reason to
  keep those in the main rcfile, since they are setup by the script.
- move *par.ntask* at the end of the main rcfile. It is now set by the script,
  but must stay in that file.
- *par.openmp* and *par.nthread* have moved to the expert rcfile. OpenMP is
  not used yet in TM5-MP.

*** New keys in the main rcfile
- *par.nx*, *par.ny*: number of cores along longitudes and latitudes. You
  should set the maximum of cores in the latitudinal direction first (/par.ny/).
  Then try to increase /par.nx/. For 3x2 runs, you should use
  /par.ny:45/ and set /par.nx/ between 1 and 4.
- *my.source.proj* : list of project directories with code to gather.
- *convective.fluxes.tiedtke* : to switch between EI convective fluxes and
  those computed from Tiedtke scheme. It is needed to correctly scale the
  Lightning NOx emissions. It should be set to F for using ERA-interim
  convective fluxes. It has no effect (ignored) if you use OD met fields.
- *write.column.budgets* : T by default, so this is a key to switch off
  some output, namely the Non-Horizontally-Aggregated-Budgets. Most
  budgets are computed for spatial bins that encompass several grid boxes,
  vertically and horizontally. The definition of these regions is done in
  /budget.F90/ and there are currently 2 versions of this file with
  different budget regions definition. Some budgets are not aggregated,
  and end up using a lot of disk space. It is possible to skip writing of
  these budgets by setting this key to F. If not present or empty it
  defaults to T. The difference is only on the horizontal binning,
  since the vertical binning is the same for all budgets. In pseudo-code:
#+BEGIN_SRC sh
 (if true (default), writes [im,jm,nbud_vg] budgets)
#+END_SRC
  It is recommended to use the default value (T).

*** Key with new behavior
- *output.pdump.vmr.???.dhour*: time step of the chemistry time series can be
  less than an hour. It is now entered as fractional hour (0.5 for example),
  and converted to int(frac_hour*3600).
*** Compilation through the job manager
    It is now possible to submit the compilation job through the job
    manager. This assumes bash is available on your machine, and
    requires few keys in your machine rcfile. The feature was required
    on cca @ECMWF. So, if you need something similar, look into its
    corresponding machine rcfile.
*** New macros (cpp)
- tropomi : to use different format/conventions for timeseries output
  in the modified CB05 chemistry project.
- with_parallel_io_meteo : use to read netCDF meteo in parallel
- with grib_api : to use grib_api. Used only in pre-processing of the
  original meteo fields from MARS archive.
- with_udunits1 & with_udunits2 : replace with_udunits and provide choice for
  udunits lib. Not required for any configuration, but can be use to
  check the netcdf met fields.
*** Lightning NOx emissions
    Total has been added to the budget file, and is not printed
    anymore into the log file.
*** New libraries
    Switch to udunits v2 and grib_api in place of udunits v1 and gribex. See new
    macros.
* Compile and/or run: the Pycasso wrapper
** Basics
The pycasso script for TM5 is linked into the top directory as setup_tm5.
It lets you:
1. gather and compile the code source,
2. setup a run, and
3. submit a run.

The call is:
#+BEGIN_SRC sh
 ./setup_tm5 [options] my.rc
#+END_SRC

You can see the list of available options with -h:
#+BEGIN_SRC sh
 ./setup_tm5 -h
#+END_SRC

** Gather and compile code
TM5 is modular. The main consequence of this modularity is that you cannot
compile the code in place. The first step of the setup script consists in
gathering the code from the //base/ and (optionally) additional projects, which
are below the //proj/ dir, into one location, the "my.project.dir/build" dir.
The later is created if needed, and "my.project.dir" is defined in your rc
file. Gathering the code and compiling it are done by default.

Note that nothing prevents you to use the checkout dir as "my.project.dir".

There are three rules when building the code source:
1. The files in each project dir listed in the rcfile key *proj* overwrite
   those from the /base/ (or a previously added project) if they happen to
   have the same name.

2. The string that follows a double underscore in the file name are removed.
   This is not mandatory when naming files, but a way to help identifying
   files provenance. For example, ebischeme__dummy.F90 becomes ebischeme.F90
   before being copied to the build directory.

3. Defined in the expert rcfile, some macros remove unneeded files.

Finally, for sake of completion, there are a couple of files that are
generated by the script. They are the *.h include files and the /dims_grid.F90/.

** Setup and submit run
The run is also setup by default. It consists in creating a run dir and a
runtime rc file, and in copying some scripts. Automatic submission of your run
is determined by the value of /submit.auto/ key in your rc file. It can also
be triggered by the /-s/ option at the command line.

A run can be submitted to the foreground, background or through a job manager.
The default is set in the /expert.rc/ file, and can be overwritten with
command line options.

** Examples
Most used commands are:
- compile and setup run (will also submit run, with default submit method, if /submit.auto/ is T)
#+BEGIN_SRC sh
 setup_tm5 file.rc
#+END_SRC
- compile, setup and submit (-s) run with the default submit method
#+BEGIN_SRC sh
 setup_tm5 -s file.rc
#+END_SRC
- compile, setup and run through job scheduler ("submit in the queue",
  -sq)
#+BEGIN_SRC sh
 setup_tm5 -sq file.rc
#+END_SRC
- recompile everything (will also submit run if /submit.auto/ is T)
#+BEGIN_SRC sh
 setup_tm5 -n file.rc
#+END_SRC

Less frequently used command:
- submit run only (without gather & compile & setup run)
#+BEGIN_SRC sh
 cd rundir
 submit_tm5 file.rc
#+END_SRC
- setup and submit run in the queue (*without* gather & compile)
#+BEGIN_SRC sh
 setup_tm5 -rsq file.rc
#+END_SRC
- gather and compile in the queue *without* setting-up/submitting the run
#+BEGIN_SRC sh
 setup_tm5 -mq file.rc
#+END_SRC

The last two options (-r for run, -m for make) are useful when the
code is compiled through a job manager ("in the queue"),
since pycasso has then no way for checking when compilation is done
and successful. In other words, -r is truly a "skip compilation"
option, while -m is a "skip run setup and submission".

Note that "using -r without -s" lets you setup the rundir without
running, if submit.auto is F. This feature may turn out useful for
debugging, since we have a way of doing "setup rundir without
compiling and without running". However, since you most likely would
like to have the run submitted when using -r, just set auto.submit to
True in your main rc to not have to type -s all the time.

* Developing TM
Once you have created and checked out your branch, you can start modifying
and/or adding code. Such activity is part of a working cycle, which is
presented first, followed by a description of each steps involved.
** Working cycle
The repository consists of a /trunk/, several /branches/ and several /tags/.
Below each of those, you will find a "copy" of the code.

The /trunk/ is the main line of development and, unlike /branches/, should be
stable. It is used to gather developments from various branches, when they are
deemed mature and stable and of interest for others. Only core developers
maintain the trunk.

For a branch, a typical development cycle is:
- synchronize: get update from the /trunk/ and/or your branch
- develop: write new code, modify old code, test, validate,...
- commit
These steps should be repeated as often as needed, and are describe in more
details in the following sections. Once the steering committee agrees, a
branch can be merged into the trunk. This last step is done by the core
developers.

** Synchronize
If there is more than one developers working with the same branch, the best way to
collaborate is to commit and update frequently (and do not forget to communicate!).

#+BEGIN_SRC sh
# it is better and easier to work from the top dir, but if you only want part
# of a commit from a colleague you could navigate the tree directory. So:

cd MYNAME
svn up

# That's it. Let's finish with a tip. You can check if a colleague has committed
# something by issuing:
svn status -u
#+END_SRC

You should also keep up-to-date with the /trunk/. Here are the steps to
follow to sync your branch.

#+BEGIN_SRC sh
# check that you have no edit:
cd MYNAME   # it is better and easier to work from the top dir
svn status

# if you have some changes (typically indicated by a M, A, or D for Modified,
# Added, Deleted - ignore all unknown files indicated with a ?), then commit
# them before going further:
svn ci -m "message"

# always update:
svn up

# check if merging will create any conflict:
svn merge ^/trunk --dry-run

# and merge
svn merge ^/trunk

# The last commands may not work if your version of svn is somewhat old. Then
# you need to use the full repository path:
svn merge ${repo}/trunk

# You end up with uncommitted changes in your code. Test them, and commit if
# you have no problem with them (see 'Commit' section hereafter).

# Again, let's finish with a couple of tips. You can check if there is
# something in the /trunk/ that could be merged into your branch:
svn mergeinfo ^/trunk --show-revs eligible

# and try:
svn mergeinfo ^/trunk
#+END_SRC

** Develop
   So, you have checked out your branch, and wonder where you should
   start coding? Typically, you will work in an existing project or
   create a new one. Projects are sub-directories below the /proj/
   directory. You include them to the final code sources by listing
   them in your main rcfile (look for the /my.source.proj/ key).

   Few things to keep in mind when writing new code:
   - follow the TM5 coding standards. See: [[http://tm.knmi.nl/index.php/Coding_standard][TM5 coding standards]]
   - *Remember* that the /base/ is used by everybody. To keep it
     usable for everybody, modifications to the base should be of
     interest for every users and not break existing functionalities.
     If it does, it is likely that the branch will not be merged back
     to the /trunk/ to be shared with other developments and projects.

*** About parallelization 
    There are two type of MPI decompositions considered in TM5-MP, one
    being related (and the other not) to the lat-lon grids. Here is an
    overview of the MPI handling in TM5:

    - The *general mpi* routines to initialize and finalize MPI,
      access its constant parameters, broadcasting/reduction of any
      variable,... are in /tm5_partools.F90/.

    - The *domain decomposition* for data arrays attached to the
      geophysical grids (tracer, meteo, ..): The distributed grid
      object (DGO) and its methods are in the /domain_decomp.F90/
      module, which you should not use it directly. DGO in TM5 are
      defined for each region, and handled in /distributed_grids.F90/,
      which inherits all DGO methods and should be used in your code:
      #+BEGIN_SRC fortran
      use tm5_distgrid, only : dgrid, Get_DistGrid, ...
      #+END_SRC

      Note that collective communications (gather and scatter of 2D, 3D
      and 4D arrays) *assume* that global array are without halo.

    - It is possible to decompose across all processes, and along its
      first dimension, any 1D or 2D array. This can help speed up
      computations that are done only on root and you would like to
      speed up. The dedicated routines are found in /arr_decomp.F90/.
      
    - The code contains some *openPM* directives to speed up some
      loop. However these are from the old TM5-zoom, and have not been
      tested in TM5-MP. They would requires fixing and testing to
      work. So considered openMP parallelization as /not implemented/.
      Ideally all I,J,L,K loops (at least the expensive ones) should
      be parallelized with openMP.

   To summarize:
   - Communications routines for data *attached* to distributed grids
     are available through /distributed_grids.F90/.
   - More general communications routines are in /tm5_partools.F90/
     (and /arr_decomp.F90/ for alternate decomposition).

*** Common task #1 - traceback
- Error handling through a TRACEBACK macro:
  Each subroutine declares a parameter holding its name:
  #+BEGIN_SRC fortran
  character(len=*), parameter  :: rname ='abc'
  #+END_SRC
  for the TRACEBACK macro to work. You must declare /rname/ if your
  routine uses IF_NOT_OK_RETURN(), defined at the top of the module.
  See code for examples. *Note*: the use of these macros is not
  possible with pure subroutine.
- For increase verbosity when setting /okdedug:T/ in your main rcfile, you can
  add:
  #+BEGIN_SRC fortran
  call goLabel(rname) ! at begin of subroutine
  ...
  call goLabel()      ! on exit
  #+END_SRC
  This code will print /<rname>/ when entering a subroutine, and
  /(rname)/ when exiting, in the log file.

*** Common task #2 - reading and scattering a global data set
The strategy is to read on one core (root) and scatter the data. The method
works with any file format, and there is no penalty in using it as long as you
do not read the file too often.

The following example read the 3D field 'z0' from a netcdf4 file. The data are on
a 1x1 lon-lat grid, which is identified as /iglbsfc/, and /nlon360, nlat180/ in TM5.

  #+BEGIN_SRC fortran
! get local bounds of 1x1
CALL get_distgrid( dgrid(iglbsfc), I_STRT=i1, I_STOP=i2, J_STRT=j1, J_STOP=j2 )

! allocate target array
ALLOCATE( z0( i1:i2, j1:j2, nz0 ) )

! open file
CALL MDF_Open( TRIM(fileName), MDF_NETCDF4, MDF_READ, FileID, status )
IF_NOTOK_RETURN(status=1)

! allocate array to read
IF (isRoot) THEN
    ALLOCATE( global_3d( nlon360, nlat180, nz0) )
ELSE
    ALLOCATE( global_3d(1,1,1) )
ENDIF

! read the data
IF (isRoot) THEN
    CALL MDF_Inq_VarID ( FileID,   'z0',    var_id, status ) ; IF_NOTOK_MDF()
    CALL MDF_Get_Var   ( FileID, var_id, global_3d, status ) ; IF_NOTOK_MDF()
ENDIF

! distribute the data
CALL SCATTER ( dgrid(iglbsfc), z0, global_3d, 0, status)
IF_NOTOK_RETURN(status=1)

! clean
DEALLOCATE( global_3d )
IF (isRoot) THEN
    CALL MDF_Close( dust_FileID, status )
    IF_NOTOK_RETURN(status=1)
ENDIF
  #+END_SRC


Note that often there is an intermediate step, when user want the data on a
different grid. Something along those lines:

  #+BEGIN_SRC fortran
! get local bounds of 1x1
CALL get_distgrid( dgrid(iglbsfc), I_STRT=i01, I_STOP=i02, J_STRT=j01, J_STOP=j02 )

! get local bounds of TM5 run resolution
CALL get_distgrid( dgrid(1), I_STRT=i1, I_STOP=i2, J_STRT=j1, J_STOP=j2 )

! allocate target array
ALLOCATE( z0( i1:i2, j1:j2, nz0 ) )

! open file
CALL MDF_Open( TRIM(fileName), MDF_NETCDF4, MDF_READ, FileID, status )
IF_NOTOK_RETURN(status=1)

! allocate array to read
IF (isRoot) THEN
    ALLOCATE( global_3d_src( nlon360, nlat180, nz0) )
    ALLOCATE( global_3d_tgt( nlon, nlat, nz0) )
ELSE
    ALLOCATE( global_3d_tgt(1,1,1) )
ENDIF

! read and remap the data
IF (isRoot) THEN
    CALL MDF_Inq_VarID ( FileID,   'z0',        var_id, status )
    IF_NOTOK_MDF()
    CALL MDF_Get_Var   ( FileID, var_id, global_3d_src, status )
    IF_NOTOK_MDF()

    ! ADD REMAP FROM global_3d_src TO global_3d_tgt
    <ADD CODE HERE>
ENDIF

! distribute the regridded data
CALL SCATTER ( dgrid(1), z0, global_3d_tgt, 0, status)
IF_NOTOK_RETURN(status=1)

! clean
DEALLOCATE( global_3d_tgt )
IF (isRoot) THEN
    DEALLOCATE( global_3d_src )
    CALL MDF_Close( FileID, status )
    IF_NOTOK_RETURN(status=1)
ENDIF
  #+END_SRC
*** Common task #3 - remapping data without gathering/scattering
    This is about involving every cores into remapping a dataset. It is
    discussed in details in [[#on_core_remap][this section]].

** Commit
This is the easy part. Just do:

#+BEGIN_SRC sh
# it is better and easier to work from the top dir
cd MYNAME
svn ci -m "message"
#+END_SRC

You can do this as often as you want, since it is not required to have
a stable branch. Actually it is recommended to do it often, since it
makes it easier to find when bugs have been introduced, and to undo changes.

** Debug
*** rc file keys
The following keys can help debugging:
- Your machine rcfile (or its compiler rcfile) should provide a debug option
  for the *my.build.configure.flags* key. Just use it, along the /-n/ option
  at the command line.
- And the *okdebug*, and *go.print* keys let you increase the verbosity of the
  program.
*** How do you follow a box?
    In tm5_tracer_data.F90, set the /ii,jj,ll,kk/ at the beginning of
    the TRACER_PRINT subroutine to the indices of the box and tracer
    you want to follow. Every call to TRACER_PRINT will print the
    tracer mass in that box. Some of these call are already in the
    code within if-okdebug-then statements.

*** About bit reproducibility
The /mmix/, /restart/, and (from chemistry only) /j_stat/ output should be
bit-reproducible, when switching MPI layout. Budgets are not, because they are
aggregated.

Currently, one subroutine cannot guarantee bit reproducibility between
different MPI layout: /boundary.F90/ (in /proj/cbm4/ and /proj/cb05/). This is due
to the way nudging factors are calculated. The order of sum will change with
/par.nx/. If your test required bit reproducibility, you need to use
the *without_boundary* flag.

Also, be sure to use the correct compiler flags. For example, for the CRAY
compiler, "-hflex_mp=conservative" flag should be replaced with
"-hflex_mp=intolerant".

*** Non-existing MPI wrapper
If you have an error message like:
#+BEGIN_EXAMPLE
'No specific match can be found for the generic subprogram call
"PAR_BROADCAST".'
#+END_EXAMPLE
It means that the interface does not cover your input type/size. You need to
add a subroutine. See the many interfaces examples in
/base/tm5_partools.F90/. For example:
#+BEGIN_SRC fortran
  INTERFACE Par_Broadcast
     MODULE PROCEDURE Par_Broadcast_i
     MODULE PROCEDURE Par_Broadcast_s
     MODULE PROCEDURE Par_Broadcast_l0
     MODULE PROCEDURE Par_Broadcast_l1
     MODULE PROCEDURE Par_Broadcast_r0
     MODULE PROCEDURE Par_Broadcast_r1
     MODULE PROCEDURE Par_Broadcast_r2
     MODULE PROCEDURE Par_Broadcast_r3
  END INTERFACE
#+END_SRC
does not cover vector or arrays of integer, only the case of a single integer.
* Miscellaneous
** Performance
*** Lapack
    It is possible to use LAPACK in TM5. Just specify the location and
    the library flags for linking in your machine rc file (see for an example,
    //rc/machine-ecmwf-cca-ifort.rc/ file), and add /with_lapack/ to
    the /my.defs_misc/ key in your main rc file. LAPACK is used in the
    convection scheme, and can speed up that part of the code by 10%,
    although overall speed up is a lot less.
** Output size
- W/o the restart files, @3x2-34L, with the current timeseries required for
  the benchmarking of CB05, we got around 50G of data for one year of output.
- Restart files are 820M, so 12 of them gives: 9.6G.
- total=60G/y
- At 1x1: 6 times more, ie 360G

** Remapping and number of cores
Remapping data between different grids is done in several places in
the code. It is usually performed on a single core to remap an input
file, like emissions or a restart file with /istart=32/. Once remapped
to the good resolution, the dataset is scattered. However, it is
possible that the data are already scattered on the source grid, and
you do not want to gather-remap-scatter but instead would like to just
remap. It would save a lot of MPI communication, and make the code
more scalable, particularly if you have to often repeat the operation
during a run. This so-called on-core remapping can be easily
implemented. The method has one (small) drawback: it limits the number
of cores you can use, although not the maximum (see below). Note that
the run will stop if the restrictions on the core number are not
fulfilled. You will get an error message like:
#+BEGIN_SRC sh
ERROR - do not know how to relate east bounds:
ERROR -   lli  bound, spacing, number : -126.0000    3.0000    18
ERROR -   lliX bound, spacing, number : -128.0000    1.0000    52
ERROR in grid_type_ll/Relate
ERROR in grid_type_ll/FillGrid
#+END_SRC
The TM5-MP code already does on-core remapping when reading netCDF
meteo in parallel at a different-from-simulation resolution. It is
examined first to show what are the limitations. Directions to apply
something similar in your code is then presented. We finish with the
specific core limitation for reading restart files with istart=32,
which are due to the reading itself and not the remapping.

*** Reading netCDF meteo in parallel
When reading the netCDF met fields in parallel, and the meteo and the
run are at different resolutions, a remapping is performed on each
core assuming that the data on the other cores are not needed. In
other words, the MPI decomposition for both resolutions must share
boundaries. This makes the remapping efficient, but the
number-of-boxes-per-core (which is different for each resolution)
should be multiple of each other, and implies (by design) that the
grid boxes be evenly distributed in both resolutions.

**** Example: reading meteo at 1x1 for runs at 3x2.
     For a run at 3x2, you can use up to 45 cores in the latitudinal
     direction. Normally any number between 2 and 45 would work, but
     if you read the meteo at 1x1, you must evenly distribute the grid
     boxes. Evenly decompositions at 3x2 can be obtained from prime
     numbers:
 #+BEGIN_SRC sh
  120 long. boxes = 2x2x2x3x5
  90 lat. boxes = 2x3x3x5
 #+END_SRC
 we are then limited to these numbers of core in each direction:
 #+BEGIN_SRC sh
 NX = 2, 3, 4, 5, 6,  8, 10, 12, 15, 20, 24, 30, 40 or 60
 NY = 2, 3, 5, 6, 9, 10, 15, 18, 30, or 45
 #+END_SRC
 which also divide 360 and 180, the number of boxes at 1x1 in each
 direction. Since the model does not scale very well in the NX
 direction, the limit is on NY: you cannot use any number between 31
 and 44 for example. But you can still use 45 cores!

*** Implementing on-core remapping
    :PROPERTIES:
    :CUSTOM_ID: on_core_remap
    :END:
    The method is straightforward, and build around these few calls for a
    remapping between the run resolution (region=1) and 1x1
    (region=iglbsfc):
    #+BEGIN_SRC fortran

    USE METEODATA, ONLY : lli

    ! get local bounds of 1x1
    CALL get_distgrid( dgrid(iglbsfc), I_STRT=i1, I_STOP=i2, J_STRT=j1, J_STOP=j2 )
    allocate( Arr1x1( i1:i2, j1:j2 ) )  ! no halo

    ! get local bounds of TM5 run resolution
    CALL get_distgrid( dgrid(1), I_STRT=i1, I_STOP=i2, J_STRT=j1, J_STOP=j2 )
    allocate( Arr3x2( i1:i2, j1:j2 ) )  ! no halo

    ! ...do some work with Arr1x1...

    ! remap Arr1x1 into Arr3x2
    ! Replace 'sum' with 'area-average' or 'average', depending on the nature
    ! of the data
    call fillgrid( lli(1), 'n', Arr3x2, lli(igblsfc), 'n', Arr1x1, 'sum', rc)
    IF_ERROR_RETURN(status=1)

    ! check rc
    if (status==-1)then
        write(gol,*) "Only part of target array was filled!"; call goErr
        write(gol,*) "Make sure that the number of processors"; call goErr
        write(gol,*) "divides the number of grid boxes."; call goErr
        TRACEBACK; status=1; return
    endif
    #+END_SRC
    Ideally you want to check if the run is not done at 1x1 to avoid
    unneeded work. This can be achieved with:
    #+BEGIN_SRC fortran
    if (igblsfc==1) then ...
    #+END_SRC

*** Restart with istart=32
When reading a restart, all cores are involved. Since the maximum
number of cores you can use depends on the resolution, you are limited
by the lower resolution: restart or run. More specifically, the input
data resolution (number of boxes) and the number of cores should
fulfill the following requirement, in both longitude and latitude
directions:
#+BEGIN_SRC sh
 nb-of-boxes/nb-of-cores > max(1,halo)
#+END_SRC
When remapping a restart file at a resolution coarser than the model
resolution, this constraint decreases the maximum number of cores you can use.
For example, reading a 3x2 or 6x4 restart file for a 1x1 run, you have:
  - run at 1x1: max number of cores is 180x90 (halo=2)
  - restart at 3x2: max number of cores is 120x90 (halo=0)
  - restart at 6x4: max number of cores is 60x45 (halo=0)
If these limits are a problem, you can make a very short run (3hr) that writes
an extra restart at time step 0.

* Frequently Asked Questions
** Could you give me some pointers regarding station output?
   Although made available in TM5-MP, /station output/ is definitively
   an undocumented and untested feature. It still uses HDF4 for
   output, which we do not recommend. (All new code should use
   netCDF.) Here are some information to get you started:

   /Station output/ is sampled in three ways: 
   - grid (GRD), simple mixing ration of the gird box
   - SLP: interpolation to exact location using slopes of the advection scheme
   - INT: lineair interpolation. *The latter scheme will not work for TM5-MP.*

   What you need? A station input file looks like this:
   :       site        lat        long  height/alt   land  0=masl,1=mag             
   : 280                                                                            
   :  AIA00500    -40.53      144.30         500    0     1                         
   :  AIA01500    -40.53      144.30        1500    0     1                         
   :  AIA02500    -40.53      144.30        2500    0     1                         
   :  AIA03500    -40.53      144.30        3500    0     1                         
   :  AIA04500    -40.53      144.30        4500    0     1                         
   :  AIA05500    -40.53      144.30        5500    0     1                         
   :  AIA06500    -40.53      144.30        6500    0     1                         
   :     ALT       82.45      -62.52         210    1     0                         
   :     AMS      -37.95       77.53         150    0     0                         
   :     ASC       -7.92      -14.42          54    0     0                         
   :     ASK       23.18        5.42        2728    1     0                         
   :     ASKSFC    23.18        5.42           0    1     1                         
   :     AVI       17.75      -64.75           3    0     0                         
   :     AZR       38.77      -27.38          40    0     0  Terceira Is., Portugal 
   :     BAL       55.50       16.67           7    0     0                         
   :     BGU       41.83        3.33          30    1     0                         
   :     BGUOCN    41.83        3.33          30    0     0                         
   :     BHD      -41.41      174.87          80    1     0                         
   :     BHDOCN   -41.41      174.87          80    0     0                         
   :     BME       32.37      -64.65          30    0     0  St. David's Head       
   :     BMW       32.27      -64.88          30    0     0  Tudor Hill             
   :     BRW       71.32     -156.60          11    1     0                         

    From here, you should go through the /station_output.F90/ file to
    see what is actually done.

** How do I calculate deposition fields?
   Do not use the /output.pdump.depositions/ feature. It is broken.
   Instead, you can calculate the deposition fluxes from the budget files
   and/or USER_OUTPUT_AEROCOM. The latter can write out hourly, daily
   or monthly accumulated deposition fields. It is possible to add new
   deposition fields as well.
** Can I find the height of the layers at a specific grid point without re-running the model and writing it out?
   One way to estimate the layer heights, is to calculate them from
   the air mass, pressure and temperature fields (provided e.g. in the
   AeroCom output file), asssuming dry air:
   : density = pressure / (Rgas * temperature) with Rgas = 8.3144 / (28.964e-3)
   : layer_thickness = air_mass / density
   The height of a layer (denoted l) can then be calculated by adding
   the thicknesses of the layers below plus half the thickness of
   layer l. A better way would be to use the geopotential heights
   calculated in TM5, which includes moisture effects, but these are
   not (as of r561) part of the output.
** Combining six 1x1 grid cells into one 3x2 grid cell
   Q: Do you have already a piece of code to calculate the land-use
   fraction on the 3x2 grid based on the 1x1 input grid? If not, where
   does TM5 start to count to put the 6 1x1 grid cells together (e.g.
   lower left corner)?

   A: We do not have that code, but it is straightforward to coarsen
   the input land-sea mask file. At both resolutions, the lower left
   corner of the first cell is at 90S / 180W.

** Why does "makedepf90" crash or get stuck?
   Troubles with makedepf90 have been reported before. To avoid
   problem, check that there is no:
   - conflict markers in the code left from a previous merge (e.g.
     line starting with "<<<<<<<<")
   - extra files with a name similar to an existing one (e.g.
     /user_output.F90_old/ or an unsaved modified file in your editor
     that creates some temporary file) in your source directory
** Is the thermal and/or dynamical tropopause computed in the model?
   You can find the /ltropo/ function in the /toolbox.F90/ module. It
   returns the index of the tropopause level based on the temperature.
   For the dynamical tropopause, you need the potential vorticity,
   which is not available in TM5 in its standard version. It is
   mentioned in some places, and there is a /vod2uv/ routine which
   transforms VOrticity and Divergence into pu and pv. But this is
   used only when the code is driven by EC-Earth or by met fields from
   MARS ERA-Interim retrievals (i.e. raw pre-processed ERA-Interim).

** How do I run my script before or after a run?
   There are pre- and post-processing sections in the main rc file:

   : 01  !======================================================!
   : 02  ! PRE-Processing
   : 03  !======================================================!
   : 04  ...
   : 05  ! II - User scripts to setup run
   : 06  input.user.scripts   : <bindir>/tm5-tmm-setup <rcfile>
   : 07
   : 08  !======================================================!
   : 09  ! POST-Processing
   : 10  !======================================================!
   : 11  ...
   : 12  ! III - Run user scripts
   : 13  output.user.scripts.condition : True
   : 14  output.user.scripts           : /path/to/my/script

   Line 6 is an example of a pre-prosssing script to submit before the
   run. It refers to two special variables /<bindir>/ and /<rcfile>/, that
   are expanded as /../bin/ and the current rc file name by the python
   script. See next Frequently Asked Questions for how useful that can be.

   Line 13 is a conditional test to run the script(s) on line 14. Set
   to /True/, the script is run at the end of ever leg. But to run you
   script only at the end of the last leg, you would use the following
   condition (two lines for readability, but should be on one line):
   : output.user.scripts.condition : \
   :          "%{jobstep.timerange.end}" == "%{timerange.end}"
   There is no conditional test for pre-processing, since you can run
   you script outside the TM5 workflow if needed only once.

   Note that these scripts are submitted to the queue requesting a
   single core, i.e. they are serial jobs. It is possible:
    - to pass the rc file as an argument to the script, so all keys
      are available in your script
    - to use intermediate key for readability, if the path to your
      script is quite long for example
    - to submit more than one script (use ";")
   Some of these features are illustrated in this example:

   : your.script        : /scratch/rundir/get_some_data
   : my.script          : /scratch/rundir/get_my_data
   : input.user.scripts : ${my.script} <rcfile> ; ${your.script}

** How do I read one of the rc file key in my python or bash script?
*** case of sh or bash script
    You can read any of your rcfile keys value by calling the
    /go_readrc/ script available in the /bin/ subdirectory. You can see
    example calls in the /tm5-tmm-store/ script. You just need to pass
    the rc file path to your script. If your script is triggered
    through the rc file in post- or pre-processing you can refer to it
    with /<rcfile>/, and to the /bin/ subdirectory with /<bin>/.
*** case of python script
    With a python script you can simple use the /bin/rc.py/ module.
    See its header doc string for how-to.

** What is the content of the MMIX output files exactly?
   The /mmix/ files contains monthly averages of volume mixing ratio,
   temperature and pressure. In the global attributes, you will find:
   - names : names of the species in the order they are in the
     /fscale/ and /ra/ arrays
   - fscale : scaling factor for conversion of mixing ratios in kg of
     tracer per kg of air to practical mixing ratio units (e.g. ppm),
     the later being saved in the mmix file. Fscale is just the ratio
     of molecular weight of air to that of tracer: mw(air)/mw(tracer)
   - ra : molecular weight of species (See /chem_param.inc/ and
     /chem_param.F90/ files for details)
   - at, bt : A's and B's hybrid coefficients to compute level
     pressures.

   So the following relation hold:

   : mass_tracer * fscale = mixing_ratio * mass_of_air

   For aerosols, the units in the mmix files indeed deserve some
   explanation. The masses of the various aerosol components are given
   as mixing ratios in kg aerosol/kg air, where the aerosol mass is
   calculated using the ra values given in m7_chem_param.inc and
   chem_param.F90:

   - For SO4, xmso4=xms+4*xmo, so the unit is kg SO4/kg air.
     Similarly, for nitrate the unit is kg NO3/kg air and for ammonium
     it is kg NH4/kg air.
   - For BC, xmbc=xmc, so the unit is kg C/kg air.
   - For POM, sea salt, and dust, the unit is kg (total particle
     mass)/kg air. In the case of sea salt, xmnacl=xmna+xmcl;
     therefore all sea salt is assumed to be NaCl. For POM and dust,
     xmpom and xmdust are used. These are arbitrarily set to xmair.
   - The mode number concentrations in the mmix file are given in
     number of particles/kg air.

** How do I make a meteo field available on 1x1?
   Q: I want to emit my biological particles depending on specific
   humidity on the 1x1 degree grid. However, /humid_dat/ seems to live
   only on the coarser 3x2 grid. Is there a way to have this data on
   the 1x1 grid?

   A: Yes. You just need to tell TM5 to used it with a call like this one:
   #+BEGIN_SRC fortran
    call Set( u10m_dat(iglbsfc), status, used=.true. )
   #+END_SRC
   The iglbsfc is the 1x1 grid. This example for u10 is in several
   places, you do the the same with /humid_dat/ in one of the init
   routine (normally the one in the module where you will use the
   data) and you are all set.
** Which part of the code ensures global mass conservation?
   Q1: It is my impression that when TM5 reads in era interim wind
   fields and produces the mfu/mfv/mfw fields, it ensures global
   airmass conservation. Could you point me to the part of the code
   that does that?

   A1: the mass balancing is done in:
   : TM5/base/trunk/src/meteo.F90
   in routine:
   : Meteo_Setup_Mass
   by a call to:
   : BalanceMassFluxes
   The input (unbalanced) mass fluxes are: /mfu, mfv, mfw/
   which are copied to new arrays: /pu, pv, pw/
   from these, /pu/pv/ are then changed to ensure mass conservation.

   Q2: Is this the routine that makes sure the global air mass is
   constant?

   A2: Global mass conservation is ensured by scaling the global
   surface pressure fields (/sp1_dat/ and /sp2_dat/) with a single-cell
   global surface pressure field /sp_region0/. Search for the later in
   meteo.F90
** Can I create daily mmix files?
   - you could (but should not) set the /job.step.length/ to 1 in your
     rc file. But this is not efficient, since the model stops and
     restarts after each day. This is particularly true within
     EC-Earth (where you cannot set /job.step.length/, but should use
     the leg length in the XML settings file for that).
   - better is to use USER_OUTPUT_AEROCOM to get daily mean (or
     monthly mean or hourly) output. It writes out a lot of aerosol
     related variables. Available only with the full chemistry project.
   - you can also use USER_OUTPUT_PDUMP (also only avaialble with the
     full chemistry project). Use a time step of 1h for the timeseries
     (3h may be enough for getting daily averages), and select only
     the species you need. This is the same philosophy as IFS output;
     you average yourself the output afterwards.
** Is it possible to output the mixing ratios at a higher temporal frequency than monthly?
   - There is an output formatting option, /output.mix.dhour/, which
     was returning the average mixing ratio over "dhour" of some
     tracers. But it a dummy routine in TM5-MP, the focus being on
     timeseries, which are available through USER_OUTPUT_PDUMP or
     USER_OUTPUT_AEROCOM (option to produce monthly mean, daily mean
     or hourly output).
   - For snapshot, you can also write extra restart every day with:
   : restart.write.extra       : T
   : restart.write.extra.hour  : 00
   : restart.write.extra.dhour : 24
** What is the difference between the pm6 and rw/rwd radii?
   The rw_mode and rwd_mode are the ambient (wet) and dry median
   number radii of the modes in TM5. The variables pm6rp and pm6dry
   are used inside the M7 routines. These were included into TM5
   without attempting to remove redundancies.

* Known Issues
  We try to keep track of issues in the development portal [[https://dev.knmi.nl/projects/tm5mp/issues][here]].
  Additionally here are some known problems.

** Bug in output.pdump.depositions
   There is a section in the rc file (output.pdump.depositions.apply),
   which lets you output deposition fluxes. The resulting fluxes look
   really strange. There is a peak at the beginning of every day in
   all grid cells. This is a bug. The first record of every day is
   wrong. This feature has not been used since before the MP version
   of the model, and used only by the initial developer. What is known
   so far: the output.pdump.dep* fields depend on the budgets. One
   consequence is that they are not defined at the same time step as
   the other timeseries. But also, they are build from differences.
   Around l. 5735 of pdump.F90:

   : ! deposition flux ~ (current budget - previous budget)/dt
   : depflux = ( budget_loc - RF%wdep_budget(:,:,k) ) / dt_sec    ! mole/s
   : call AreaOper( lli(region), depflux, '/', 'm2', status )     ! mole/m2/s
   : IF_NOTOK_RETURN(status=1)
   :
   : ! save current budget & store record
   : RF%wdep_budget(:,:,k)       = budget_loc
   : RF%data2d_wet(:,:,RF%trec,k)= depflux

   The later (RF%data2d_wet) is stored, and show up in your plot. The
   re-initialization of RF%wdep_budget(:,:,k) seems to be the problem.
   Since these are daily files, RF_DEPS_Init is called every day, and
   set:
   : ! allocate storage: > allocate(
   : RF%ddep_budget(imr,jmr,RF%ntr) ) ; RF%ddep_budget = 0.0 > allocate(
   : RF%wdep_budget(imr,jmr,RF%ntr) ) ; RF%wdep_budget = 0.0

   I think the solution is to separate the allocate/deallocate from
   the init/done, probably with additional arrays. The tricky part of
   the code is to get the proper number of time records (background:
   we do not want to have an unlimited dimension for the time axis in
   the netcdf files, it slows down a lot the IO).
** Interpolating model at observation location
   Let say the domain on one processor is indicated by the blue area.
   When needed, information from the neighboring processors (in
   yellow) is fetched. This is typically done in the code with calls to
   update_halo functions, and is useful for transport and some
   interpolations. However, if your interpolation scheme requires one
   of the "corner neighbors" /C/,  it will not work, and you must either
   change interpolation method, or interpolate off-line in a
   post-processing step.
   | C | x | x | x | C |
   |---+---+---+---+---|
   | x | . | . | . | x |
   |---+---+---+---+---|
   | x | . | . | . | x |
   |---+---+---+---+---|
   | x | . | . | . | x |
   |---+---+---+---+---|
   | C | x | x | x | C |

/Last saved on 2018-06-11/
