ecearth3/sources/tm5mp/proj/domino/version_history_readme

This code:
DOMINO NO2 retrieval/assimilation, DOMINO v3.1.8, 25-08-2016

Version history DOMINO NO2 retrieval:

3.1.8 DOMINO v3 with TROPOMI interfaces and de-striping, Henk Eskes, KNMI, August 2016
  - Bug fix MStripeCorrection: make sure one file is generated per day, and one file is read at startup 
    (stripe corr for orbit at 00:06 is written to file for the previous day)
  - Update to read the latest AMF files, which contain AMF/AMFgeo
    o assim.F90: read .rc flag "domino.amf.divide.by.amgeo", passing on to "ReadAmfLut"
    o MAmfRead.F90: added flag "divideByAMFGeo"
    o MAmfGet.F90: use simple AMFgeo definition instead of Leue expression
  - MStripeCorrection.F90
    o Debugging of the destriping code
    o Stripe input/output : netcdf files
    o Number of across track points generalised (no longer 60), to support e.g. TROPOMI

3.1.7 DOMINO v3 with TROPOMI interfaces, Henk Eskes, KNMI, August 2016
  - MOmF_output.F90:
      o major changes: to account for different number of superobs in fc and an
  - MStripeCorrection.F90: undef -> nf_fill_float 
  - MObservationOperator.F90: 
      o undef -> nf_fill_float
      o cloud radiance fraction: extra output to measure difference between input and computed values "crf_m0" etc
      o cloud radiance fraction: in case CRF is computed (changed), it is also replaced in the L2 file
  - TM5IF_module.F90:
      Now supports "cloud_radiance_fraction_no2" output (only when computed internally)
  - MTObsFcInfo.F90: extra flag "cloudRadFraction_computed" added
  - MOmFSuper.F90: 
      o "usePixel" is used (kept in storage) for both forecast and analysis step
      o Extra flag check included to allow less observations in the analysis case, update of "usePixel" 
      o bug fix: "MOmFSuper.F90" re-written to force the number of obs in analysis and forecast to be the same
  - MAmFGet: warnings have been commented to avoid very long listings
  - bug fix: "MObsRead.F90"; one (undefined) line too many was read from the list file 
  - rc-file: extra "domino.<flag>" settings added
  - "assim_interf_mod.F90"
    extra flag "domino.doNudgeOdinHNO3" read from the .rc file
  - Re-tuning of the "strat_error" and "forecastError", leading to a stronger forcing to the observations:
    "MOmFSuper.F90" and "MNO2RetrievalError.F90"
  - "assim.F90"
    o Calls to "MOmF_output" routine
    o Extra flags read from the .rc file: domino.compute.retrieval.for.analysis, domino.write.OmF, domino.debug.mode
    o call to "terrain_read" removed
    o extra determination of total column: "no2col_fc" and "no2col_an"
  - "terrain.F90" module removed; call in "assim.F90" removed
  - "TM5IF_module.F90" - now reads also the orbit number
  - New module "MOmF_output.F90" 
    extra observation-minus-forecast output to netcdf
    extra calls in "assim.F90"
  - "assim.F90" now includes a flag "computeRetrievalForAnalysis" which makes it 
    possible to skip the retrieval for the analysis
  - "MOmFSuper.F90", now assimilates half of the superobservations for speed increase

3.1.6  DOMINO v3 with TROPOMI interfaces, Henk Eskes, KNMI, July 2016
  - "MObsRead.F90" is now reading all orbit file names at start, and 
       advances through the list testing only one file at a time
    "TM5IF_module.F90": provides an extra flag "before_or_after" to signal
       if orbit time is before, at, or after time interval
  - Added timer for assimilation step in "tm5.F90"
  - Adding "tm5.inc" to all files
  - LAPACK-based "oi_module.F90"  (speed-up of analysis by about factor 5)
  - Bug fix: TM5IF_module.F90
       order of dimensions of "tm5_pressure_level_a" and "tm5_pressure_level_b" (reverse order of fortran and netcdf!)
  - "assim_interf_mod.F90": ODIN HNO3/O3 nudging switched off 
       ODIN nudging is now fixed in TM5, and TM5 implementation is default

3.1.5  DOMINO v3 with TROPOMI interfaces, Henk Eskes, KNMI, May 2016
  - Bug fix: MAmfGet.F90, 
       "flag": warnings are now processed
  - Snow and Ice: Switch to scene retrieval has been implemented
    MObservationOperator.F90
       introduced array "isSnowIce", 
          set cloud (radiance) fraction to = 1 when snow/ice cover > 50 %
  - "TM5IF_module.F90"
        The write subroutine has been largely rewritten to be consistent with the 
        latest output definition (Jos van Geffen, May 2016)
	Few changes in read part, 
            including reading of snowIceFlag, sceneAlbedo, scenePressure and "use_scd_qa4ecv_units"
     MTObsFcInfo.F90:
        Added "amfclear" (+ allocate/deallocate)
     MTObsTrack
        Added "snowIceFlag" (+ allocate/deallocate)
        Added "sceneAlbedo" (+ allocate/deallocate)
        Added "scenePressure" (+ allocate/deallocate)
     MObservationOperator.F90:
        Added variables "no2slcfctropclear" and "amfclear"
        Fill "ObsFcInfo%amfclear"       
     assim.F90: pass TM5 hybrid layer info to writing routine:
        call TM5IF_obsfcinfo_write( no2Tr, obsFcInfo, TM5Data )"
  - Added new ODIN HNO3/O3 nudging capability
        A new file "odin_hno3_o3_ratio.nc" was created based on the original ODIN climatology files
           In these files there are hno3/o3 fields explicitly processed for the top 6 layers of TM5 
           (Odin integrated over the TM5 layers)
        "MOdinHNO3.F90": This new module was added to read the new ODIN climatology file, 
           and to compute the nudging adjustment for HNO3. Nudging formula has been updated (explicit exponential)
        "assim_interf_mod.F90": This interface was extended to gather the zonal means of O3 and HNO3, 
           and to update the HNO3 concentration after nudging
        "boundary.F90": bug fix on reals being used for array indices
           If "use_ODIN" (input.climat.ODIN) is False, nothing will be done (default option was 10hPa nudging)

3.1.4  DOMINO v3 with TROPOMI interfaces, Henk Eskes, KNMI, April 2016
  28 April:
    - Upgrade of the TM5 code to revision 334, date: 2016-04-25 
      The previous domino versions were using an old TM5-MP version of April 2015
    - Upgrade of "tm5.F90" to revision 334
  22 April:
    - Adjustment of the file reading routine in "TM5IF_module.F90"
      to read the QA4ECV files:
      o "processing_error_flag" is no longer used, instead we read "processing_quality_flags" which is more detailed
      o "solar_zenith_angle_surf" is now "solar_zenith_angle", same for "viewing_zenith_angle" and "relative_azimuth_angle"
      o different units on SCD are accounted for, see "use_scd_qa4ecv_units" 
      o the SCD units are read and used: "TM5IF_get_var_att_str("scd_no2", "units", scd_unit_string)"
        unit conversion is now depending on this attribute
      o added "function TM5IF_get_var_att_str(varname, name, value)"
    - Adjustment of the error checking in "MObservationOperator.F90"
      o subroutine "CheckValueRanges" re-written:
        nr of errors are now counted, instead of reporting individual errors
      o same for main routine:
        counters added: "nPixelsWithErrors", "nSmallAmftroptotal"
	reporting only total number of pixels with errors, at end of the routine
	'GetOMFHx: WARNING, nr of pixels with error flag =', nPixelsWithErrors
    - Adding stripe correction calls in "MObservationOperator.F90"
      o "doApplyStripeCorrection" flag added
      o "use MStripeCorrection"
        calls to "ReadOMIStripeCorrection" and "ComputeOMIStripeCorrection" added
        use "stripeCorr" and "stripeCorrAvailable"
	   satscd = no2Tr%no2SLC(iObs)
           if ( doApplyStripeCorrection .and. stripeCorrAvailable ) then
              ! apply the OMI slant column stripe corrections to the DOAS slant column 
              satscd = satscd - stripeCorr( no2Tr%pixelIndex(iObs) ) 
      o Better support for the TROPOMI flagging: warnings vs errors
        "ObsFcInfo%flag(iObs) > 0" replaced by "iand(ObsFcInfo%flag(iObs), 255) /=  0" etc
  7 April:
    - Include the new cross section temperature correction factor
      Update by Marina Zara, TN-OMIE-KNMI-982, March 2016, for DOMINO-3  
      correction factor = 1 - 0.00316 (T-Tdoasfit) + 3.39e-6 (T-Tdoasfit)^2  
        "real, parameter  :: Tcorrection_Acoeff = -0.00316"
        "real, parameter  :: Tcorrection_Bcoeff = 3.39e-6"
        "dTemp = Tlayer(iObs,l) - Tdoasfit"
        "sigmacorrlev(iObs,l) = 1.0 + Tcorrection_Acoeff * dTemp + Tcorrection_Bcoeff * dTemp * dTemp"
    - Re-write of "MAmfGet.F90" by Martien de Haan, to avoid problems at exact edges of the LUT.

3.1.3  DOMINO v3 with TROPOMI interfaces, Henk Eskes, KNMI, 18 Nov 2015
    Henk Eskes, KNMI, 7 Apr 2015
    - Bug fix "MAmfGet.F90", routine "GetAmf_vectorised"
    - Extra debug output for "MAmfGet.F90" and "MObservationOperator.F90" 	 
    - replaced 3 flags: PQF_E_GENERIC_EXCEPTION -> PQF_E_GENERIC_RANGE_ERROR
    Henk Eskes, KNMI, 18 Nov 2015
         Extracted from the DOMINO 3.1.1 and 3.1.2 code
    Main changes: This version includes the error messages implemented 
       for 3.1.2 and adds them to the 3.1.1 code.
    - MTAmfLut, MAmfGet.F90: copy of version 3.1.2
    - MNO2RetrievalError.F90: copy of version 3.1.2
    - MObservationOperator.F90: copy of version 3.1.2
         includes extra option “useTerrainPressureFromFile” and extra parameter “amfTropCutOff”
    - MTObsFcInfo:  copy of version 3.1.2
         includes init with fill values, “fltrop” -> “flag” has new meaning
    - MTObsTrack:  copy of version 3.1.2
         “terrainPressureFactor” replaced by “terrainPressure” (hPa)
    - “pqf_module” added, with error/warning flag definitions
    Other changes:
    - assim: “debugMode” added
    - TM5IF_module.F90: “terrainPressureFactor” replaced by “terrainPressure” (hPa)

3.1.2  TROPOMI DOMINO NRT code,     Henk Eskes, KNMI, 23 Oct 2015
    Extracted from the DOMINO 3.1.1 code
    - Setting change: useCloudRadianceFractionFromFile = .true. (MObservationOperator.F90)
    - Extra flag: "useTerrainPressureFromFile" in MObservationOperator.F90
    - Extra flag: "amfTropCutOff" in MObservationOperator.F90
    - Extra routine: "CheckValueRanges" in MObservationOperator.F90
    - "NO2ErrEstimate" now returns a status
    - "fltrop" replaced by "flag" in "MTObsFcInfo"
    - Initialise variables with netcdf fill values in "MTObsFcInfo"
    - Clear albedo "albclear" and cloud radiance fraction is restricted to [0,1] in  MObservationOperator.F90
    - "GetAmf_vectorised": extra check on flag > 0 performed
    - Now using the "pqf_module" for warning/error flags
    - "MTObsTrack.F90": replace "terrainPressureFactor" by "terrainPressure"
    - Error handling has been revised in a major way: aborts have been replaced by setting the error flag
      e.g. new routine "CheckValueRanges" 
      e.g. "GetAmf_vectorised" now has error flag as input/output
      e.g. flag is used to skip code in case of error, in "GetOMFHx"
    - Printing through call to "fortranlog" (TROPOMI)
    - Debug option introduced

3.1.1   DOMINO v3 with TROPOMI interfaces,  13 October 2015,  Henk Eskes, KNMI
   Changes wrt 3.1.0: 
   - New interface to updated AMF Lookup Tables  (Alba Lorente, Folkert Boersma, QA4ECV, Sep-Oct 2015)
     "MAmfGet.F90"  "MAmfRead.F90"  "MTAmf.F90"
   - Added interface module for AMF: "MTAmf.F90"   
   - Remove "newmonth" and "month" input parameters from call to "GetOMFHx" (no impact on results),
     updated explanatory text.
     This involves modules "MObservationOperator.F90" and "MNO2RetrievalError.F90"

3.1.0    28 September 2015, DOMINO v3 with TROPOMI interfaces
   Henk Eskes, Mark ter Linden, KNMI
   Addition of TROPOMI netcdf interfaces
   Changes wrt 3.0.0:
   - Interface modules "MTObsTrack.F90" and "MTObsFcInfo.F90" were updated / simplified
     the "TObsTrack" structure now has allocatable arrays
   - Modules "MObsRead.F90" and "TM5IF_module.f90" developed by Mark ter Linden
     For reading slant columns retrieval L2 files in TROPOMI netcdf format
     For writing (replacing) vertical column data in the same L2 files
   - The O2-O2 cloud retrieval correction modules of Bram Maasakkers have been removed
   - Paths are now read from the TM5 rc file
   - Bug fixes:
      o call to "NO2ErrEstimate": there was a bug in passing the arrays to the routine 
             (only first data block in the arrays was used)

3.0.0    21 August 2015, beta version DOMINO v3
   Henk Eskes, KNMI
   With new interfaces to the TM5-CB05-mp parallel version, release early 2015
   --> project: proj/domino
   Based on DOMINO-3 discussed in thesis Maasakkers, see below (v2.1.0.0) 
   Changes wrt v2.1:
   - call in "tm5.F90" has been adjusted (two calls replaced by one, different location in time stepping sequence)
   - interface to the parallel TM5 code has been developed: module "assim_interf_mod.F90" 
      o introducing "gather" and "scatter" mechanisms
      o interface has an improved way of extracting the NO2 field, using the ratio of "mixing NOx" and "chemistry NOx" 
      o scaling of NOx concentration and slopes 
   - introduce "MTM5Data.F90", which is a generic interface to TM5 (also for the NRT code, based on TM files)
   - introduce the generic structure "MTObsTrack.F90" for the interface to the input slant column data files (L2-slc)
   - introduce the generic structure "MTObsFcInfo.F90" for the interface to the output vertical column data files (L2) 
   - "assim" now computes the grid cell columns: "GetNO2PartialColumnField"
   - re-written netcdf interfaces for the various input files (lookup-tables) 
   - clean-up of all modules
   - replaced fixed storage by allocatable storage in several places
   - removal of 32-bit additional arrays (which are not needed)
   - ...

2.1.0.0   3 July prototype DOMINO v3
       Discussed in: 
          Maasakkers J.D., Vital improvements to the retrieval of tropospheric NO2 columns from the 
             Ozone Monitoring Instrument, Master thesis, University of Technology Eindhoven, 
             Eindhoven, The Netherlands
          Maasakkers J.D., Overview of changes to the DOMINO retrieval of tropospheric NO2 columns from 
             the Ozone Monitoring Instrument, July 2 2013

2.0.0.0   7 September 2010, Folkert Boersma,: DOMINO v2.0  with:
        * improved AMF LUT (15 layers + below-surface extrapolation),
        * hi-res terrain height following Zhou et al., AMT, 2009, 
        * improved surface albedo (Kleipool et al., JGR, 2008) for NO2 AMF, and O2-O2 cloud parameters,
        * explicit transportation of NO2 (along with NO, NO3, N2O5, and HNO4) instead of NOx in TM4
          to repair 30-minute 'lagging' in NO-NO2 partitioning, leading to insufficient vertical mixing
          of boundary-layer NO2 (and too sharply peaked profiles).
        * weekly cycle for NOx emissions included in TM4 (sources_sinks.f90) for Europe and North America. 

1.0.0.12a 12 June 2009:
          Test version for Ruud Dirksen to test influence of 15-layer AMF calculation


1.0.0.12 29 september 2008:
          Repaired small error in calculation of the pixelcornerpoints. This was not done correctly 
          for the last row of the file

1.0.0.11a 5 June 2008:
          Special version for Jennifer, ignore the eclipse flag. 
          Only used to process data of 22 sep 2006


1.0.0.11 29 April 2008:    
          2 Bugs solved. Bug #1: the fillvalues for the copied parameter SlantcolumnamountNo2 where not properly treated.
          All values of this field were multiplied by  1e15, including the fillvalues. This has been solved now.
          Bug #2: integer overflow when storing averaging kernel. This is solved by changing the scale factor from 1e-4 to 1e-3
          This can handling kernel element values up to 32 instead of 3.2 as before.
          Raise outputflag when the groundpixelqualityflag show that the surface of the pixel is covered with
          snow. If so, the cloudfraction and cloudpressure are unreliable. 
          Surface albedo is set to 0.6 in the L2 slant column product
          Data version promoted to 1.0.2 29 April 2008

1.0.0.10 25 March 2008:    Changed description of TM4TerrainHeight and TM4SurfacePressure field (minor)
                              Changed dataype of TroposphericColumnFlag to signed byte. (Dataversion 1.0.1, 25 March 2008)
                              Include averaging kernel in DOMINO product. Remove it from the PROFILE product

1.0.0.9 20 March 2008:     Change outputformat for the purpose to minimise the resulting filesize. Some datafields were removed (SpacecraftAltitude,
                              SpacecraftLatitude,SpacecraftLongitude,OriginalSlantcolumnAmountNo2,AssimilatedStratosphericVerticalColumnError)
                              The links in geolocationdata to TM4PressurelevelA & TM4PressurelevelB were removed
                              Furthermore was the datatype of some fields adjusted to int16 in order to reduce space
                              SurfaceAlbedo         int32 -> int16 + scaling 0.0001
                              CloudRadianceFraction int32 -> int16 + scaling 0.01
                              TM4TerrainHeight      int32 -> int16 
                              TM4TropoPauseLevel    int32 -> UCHAR

                              In the profile file the following datatype modifications were implemented:
                              AveragingKernel       float -> int16 + scaling  0.0001
                              TemperatureProfile    float -> uint16 + scaling 0.01
                              The links in geolocationdata to TM4PressurelevelA & TM4PressurelevelB were removed

                              The dataversion is now set to 1.0.1


1.0.0.8a 20 March 2008:    Special version for Jennifer, ignore the eclipse flag. Only used to process data of 22 sep 2006


1.0.0.8 16 March 2008:     Change in write procedure. Resolve bug because of which the int32 fields TM4Terrainheight, TM4Tropopauselevel
                              and TroposphericColumnflag were not written. Major impact on data usability for e.g. Yipin
                              This affects data of 26 sep 2006 and onwards

1.0.0.7 10 March 2008:     Minor change in write procedure. Write correct filename to L2 inputfile swathattribute 
                              Added NumTimes swathattribute

1.0.0.6 20 February 2008:  Minor change in write procedure. Now the scalefactor of the cloudfraction is correctly written to file

1.0.0.5 19 February 2008:  Changed Title attribute of SurfaceAlbedo datafield to "..GOME database.." instead of "..Koelemeijer database.."

1.0.0.4 15 February 2008:  Keep on processing with the Koelemeijer albedo database. Dataversion = 1.0.0


1.0.0.3 14 february 2008:  Attribute DataVersion added to output. This is to 
                              distinguish between (minor) program changes that do not
                              affect the algorithm and changes that do involve the 
                              algorithm and thereby affect the results.
                              Also write temperature profile to the NO2 profile/av kernel file.
                              IMPORTANT !!!!!!!!!
                              Bug fix: so far the albedo from the L2 file was used (Koelemijer database).
                              This is the result of a test for using the OMI albedo data from the L2 file
                              which will be implemented in the future. This test setting was overlooked and
                              not restored for the reprocessing. The data until ?? August 2006 are affected by this bug.
                              In principle this dataset is an interesting testcase to study the effect of the surface albedo
                              on the retrieval.


1.0.0.2 12 february 2008:  Scaling of cloudfraction & TM4surfacepressure solved.
                              Writing NO2 profile & averaging kernel to separate files

1.0.0.1  4 february 2008:  Write mmix data at 13h30 (for Yipin)
                              Solved system stop for zero column values in ass_Hx
                              IMPORTANT: because of change of ECMWF meteodata format at 31 jan 2006 there is a 
                              transition in the savefile. (35 -> 34 sigma layers). It was not possible to interpolate
                              the old-format file of 30 jan to the new-format so a save file was created by a free model 
                              run starting on 20 jan 2006 with a 10 feb 2006 savefile (new format) that still was around from
                              the NRT processing.


1.0.0.0  4 January  2008:  The version for reprocessingFix pixelcornerpoints issue for time jumps in granule.
                              Forgotten to activate fitflag filtering!!
                              Issue with pixel cornerpoints
                              Mistake in scaling and/or units of cloudfraction & TM4surfacepressure

0.9.9.4  18 December 2007: Changes in output file. Separate file for DOMINO output, (no append) reduction of # fields

0.9.9.3  15 December 2007: Parallellisation of convection, chemistry & AMF calculation implemented

0.9.9.2  3 December 2007:  Vectorized version of AMF calculation

0.9.9.1  3 December 2007:  Major upgrade of the program. This upgrade coincides with the reprocessing of the OMI data.
                              As a result of the reprocessing the striping of the NO2 data is greatly reduced. 
                              The upgrade includes
                                -HDF-EOS5 output
                                -different destriping algorithm
                                -proper usage of the flags in the L2 product


0.9.3.6  14 may 2007:      Exclude orbits in case of solar eclipse

0.9.3.5  15 april 2007:    Store OmF (Observation minus Forecast) results in he5 file

0.9.3.4  20 March 2007:    Added slant column error to output product

0.9.3.3  26 February 2007: Bug solved that caused incorrect ltropo values in the output product

0.9.3.2  15 February 2007: Added Model predicted trop column to output product
                              Fieldname fcvcdtr in ANC_ vdata

0.9.3.1  26 January 2007:  Added initial SCD to output product. This the slant
                              column calculated by the OMNO2A algorithm
                              Fieldname scdorig in ANC_ vdata

0.9.3    1 December 2006:  Ground pixel quality flag added to geolocation data

0.9.2   16 November 2006:  Inputpointers added to metadata (global attributes)

0.9.1  11 September 2006:  pixel nr & image nr added to geolocation data

0.9        9 August 2006:  Stripe correction polynomial fit instead of FFT