This code: DOMINO NO2 retrieval/assimilation, DOMINO v3.1.8, 25-08-2016 Version history DOMINO NO2 retrieval: 3.1.8 DOMINO v3 with TROPOMI interfaces and de-striping, Henk Eskes, KNMI, August 2016 - Bug fix MStripeCorrection: make sure one file is generated per day, and one file is read at startup (stripe corr for orbit at 00:06 is written to file for the previous day) - Update to read the latest AMF files, which contain AMF/AMFgeo o assim.F90: read .rc flag "domino.amf.divide.by.amgeo", passing on to "ReadAmfLut" o MAmfRead.F90: added flag "divideByAMFGeo" o MAmfGet.F90: use simple AMFgeo definition instead of Leue expression - MStripeCorrection.F90 o Debugging of the destriping code o Stripe input/output : netcdf files o Number of across track points generalised (no longer 60), to support e.g. TROPOMI 3.1.7 DOMINO v3 with TROPOMI interfaces, Henk Eskes, KNMI, August 2016 - MOmF_output.F90: o major changes: to account for different number of superobs in fc and an - MStripeCorrection.F90: undef -> nf_fill_float - MObservationOperator.F90: o undef -> nf_fill_float o cloud radiance fraction: extra output to measure difference between input and computed values "crf_m0" etc o cloud radiance fraction: in case CRF is computed (changed), it is also replaced in the L2 file - TM5IF_module.F90: Now supports "cloud_radiance_fraction_no2" output (only when computed internally) - MTObsFcInfo.F90: extra flag "cloudRadFraction_computed" added - MOmFSuper.F90: o "usePixel" is used (kept in storage) for both forecast and analysis step o Extra flag check included to allow less observations in the analysis case, update of "usePixel" o bug fix: "MOmFSuper.F90" re-written to force the number of obs in analysis and forecast to be the same - MAmFGet: warnings have been commented to avoid very long listings - bug fix: "MObsRead.F90"; one (undefined) line too many was read from the list file - rc-file: extra "domino." settings added - "assim_interf_mod.F90" extra flag "domino.doNudgeOdinHNO3" read from the .rc file - Re-tuning of the "strat_error" and "forecastError", leading to a stronger forcing to the observations: "MOmFSuper.F90" and "MNO2RetrievalError.F90" - "assim.F90" o Calls to "MOmF_output" routine o Extra flags read from the .rc file: domino.compute.retrieval.for.analysis, domino.write.OmF, domino.debug.mode o call to "terrain_read" removed o extra determination of total column: "no2col_fc" and "no2col_an" - "terrain.F90" module removed; call in "assim.F90" removed - "TM5IF_module.F90" - now reads also the orbit number - New module "MOmF_output.F90" extra observation-minus-forecast output to netcdf extra calls in "assim.F90" - "assim.F90" now includes a flag "computeRetrievalForAnalysis" which makes it possible to skip the retrieval for the analysis - "MOmFSuper.F90", now assimilates half of the superobservations for speed increase 3.1.6 DOMINO v3 with TROPOMI interfaces, Henk Eskes, KNMI, July 2016 - "MObsRead.F90" is now reading all orbit file names at start, and advances through the list testing only one file at a time "TM5IF_module.F90": provides an extra flag "before_or_after" to signal if orbit time is before, at, or after time interval - Added timer for assimilation step in "tm5.F90" - Adding "tm5.inc" to all files - LAPACK-based "oi_module.F90" (speed-up of analysis by about factor 5) - Bug fix: TM5IF_module.F90 order of dimensions of "tm5_pressure_level_a" and "tm5_pressure_level_b" (reverse order of fortran and netcdf!) - "assim_interf_mod.F90": ODIN HNO3/O3 nudging switched off ODIN nudging is now fixed in TM5, and TM5 implementation is default 3.1.5 DOMINO v3 with TROPOMI interfaces, Henk Eskes, KNMI, May 2016 - Bug fix: MAmfGet.F90, "flag": warnings are now processed - Snow and Ice: Switch to scene retrieval has been implemented MObservationOperator.F90 introduced array "isSnowIce", set cloud (radiance) fraction to = 1 when snow/ice cover > 50 % - "TM5IF_module.F90" The write subroutine has been largely rewritten to be consistent with the latest output definition (Jos van Geffen, May 2016) Few changes in read part, including reading of snowIceFlag, sceneAlbedo, scenePressure and "use_scd_qa4ecv_units" MTObsFcInfo.F90: Added "amfclear" (+ allocate/deallocate) MTObsTrack Added "snowIceFlag" (+ allocate/deallocate) Added "sceneAlbedo" (+ allocate/deallocate) Added "scenePressure" (+ allocate/deallocate) MObservationOperator.F90: Added variables "no2slcfctropclear" and "amfclear" Fill "ObsFcInfo%amfclear" assim.F90: pass TM5 hybrid layer info to writing routine: call TM5IF_obsfcinfo_write( no2Tr, obsFcInfo, TM5Data )" - Added new ODIN HNO3/O3 nudging capability A new file "odin_hno3_o3_ratio.nc" was created based on the original ODIN climatology files In these files there are hno3/o3 fields explicitly processed for the top 6 layers of TM5 (Odin integrated over the TM5 layers) "MOdinHNO3.F90": This new module was added to read the new ODIN climatology file, and to compute the nudging adjustment for HNO3. Nudging formula has been updated (explicit exponential) "assim_interf_mod.F90": This interface was extended to gather the zonal means of O3 and HNO3, and to update the HNO3 concentration after nudging "boundary.F90": bug fix on reals being used for array indices If "use_ODIN" (input.climat.ODIN) is False, nothing will be done (default option was 10hPa nudging) 3.1.4 DOMINO v3 with TROPOMI interfaces, Henk Eskes, KNMI, April 2016 28 April: - Upgrade of the TM5 code to revision 334, date: 2016-04-25 The previous domino versions were using an old TM5-MP version of April 2015 - Upgrade of "tm5.F90" to revision 334 22 April: - Adjustment of the file reading routine in "TM5IF_module.F90" to read the QA4ECV files: o "processing_error_flag" is no longer used, instead we read "processing_quality_flags" which is more detailed o "solar_zenith_angle_surf" is now "solar_zenith_angle", same for "viewing_zenith_angle" and "relative_azimuth_angle" o different units on SCD are accounted for, see "use_scd_qa4ecv_units" o the SCD units are read and used: "TM5IF_get_var_att_str("scd_no2", "units", scd_unit_string)" unit conversion is now depending on this attribute o added "function TM5IF_get_var_att_str(varname, name, value)" - Adjustment of the error checking in "MObservationOperator.F90" o subroutine "CheckValueRanges" re-written: nr of errors are now counted, instead of reporting individual errors o same for main routine: counters added: "nPixelsWithErrors", "nSmallAmftroptotal" reporting only total number of pixels with errors, at end of the routine 'GetOMFHx: WARNING, nr of pixels with error flag =', nPixelsWithErrors - Adding stripe correction calls in "MObservationOperator.F90" o "doApplyStripeCorrection" flag added o "use MStripeCorrection" calls to "ReadOMIStripeCorrection" and "ComputeOMIStripeCorrection" added use "stripeCorr" and "stripeCorrAvailable" satscd = no2Tr%no2SLC(iObs) if ( doApplyStripeCorrection .and. stripeCorrAvailable ) then ! apply the OMI slant column stripe corrections to the DOAS slant column satscd = satscd - stripeCorr( no2Tr%pixelIndex(iObs) ) o Better support for the TROPOMI flagging: warnings vs errors "ObsFcInfo%flag(iObs) > 0" replaced by "iand(ObsFcInfo%flag(iObs), 255) /= 0" etc 7 April: - Include the new cross section temperature correction factor Update by Marina Zara, TN-OMIE-KNMI-982, March 2016, for DOMINO-3 correction factor = 1 - 0.00316 (T-Tdoasfit) + 3.39e-6 (T-Tdoasfit)^2 "real, parameter :: Tcorrection_Acoeff = -0.00316" "real, parameter :: Tcorrection_Bcoeff = 3.39e-6" "dTemp = Tlayer(iObs,l) - Tdoasfit" "sigmacorrlev(iObs,l) = 1.0 + Tcorrection_Acoeff * dTemp + Tcorrection_Bcoeff * dTemp * dTemp" - Re-write of "MAmfGet.F90" by Martien de Haan, to avoid problems at exact edges of the LUT. 3.1.3 DOMINO v3 with TROPOMI interfaces, Henk Eskes, KNMI, 18 Nov 2015 Henk Eskes, KNMI, 7 Apr 2015 - Bug fix "MAmfGet.F90", routine "GetAmf_vectorised" - Extra debug output for "MAmfGet.F90" and "MObservationOperator.F90" - replaced 3 flags: PQF_E_GENERIC_EXCEPTION -> PQF_E_GENERIC_RANGE_ERROR Henk Eskes, KNMI, 18 Nov 2015 Extracted from the DOMINO 3.1.1 and 3.1.2 code Main changes: This version includes the error messages implemented for 3.1.2 and adds them to the 3.1.1 code. - MTAmfLut, MAmfGet.F90: copy of version 3.1.2 - MNO2RetrievalError.F90: copy of version 3.1.2 - MObservationOperator.F90: copy of version 3.1.2 includes extra option “useTerrainPressureFromFile” and extra parameter “amfTropCutOff” - MTObsFcInfo: copy of version 3.1.2 includes init with fill values, “fltrop” -> “flag” has new meaning - MTObsTrack: copy of version 3.1.2 “terrainPressureFactor” replaced by “terrainPressure” (hPa) - “pqf_module” added, with error/warning flag definitions Other changes: - assim: “debugMode” added - TM5IF_module.F90: “terrainPressureFactor” replaced by “terrainPressure” (hPa) 3.1.2 TROPOMI DOMINO NRT code, Henk Eskes, KNMI, 23 Oct 2015 Extracted from the DOMINO 3.1.1 code - Setting change: useCloudRadianceFractionFromFile = .true. (MObservationOperator.F90) - Extra flag: "useTerrainPressureFromFile" in MObservationOperator.F90 - Extra flag: "amfTropCutOff" in MObservationOperator.F90 - Extra routine: "CheckValueRanges" in MObservationOperator.F90 - "NO2ErrEstimate" now returns a status - "fltrop" replaced by "flag" in "MTObsFcInfo" - Initialise variables with netcdf fill values in "MTObsFcInfo" - Clear albedo "albclear" and cloud radiance fraction is restricted to [0,1] in MObservationOperator.F90 - "GetAmf_vectorised": extra check on flag > 0 performed - Now using the "pqf_module" for warning/error flags - "MTObsTrack.F90": replace "terrainPressureFactor" by "terrainPressure" - Error handling has been revised in a major way: aborts have been replaced by setting the error flag e.g. new routine "CheckValueRanges" e.g. "GetAmf_vectorised" now has error flag as input/output e.g. flag is used to skip code in case of error, in "GetOMFHx" - Printing through call to "fortranlog" (TROPOMI) - Debug option introduced 3.1.1 DOMINO v3 with TROPOMI interfaces, 13 October 2015, Henk Eskes, KNMI Changes wrt 3.1.0: - New interface to updated AMF Lookup Tables (Alba Lorente, Folkert Boersma, QA4ECV, Sep-Oct 2015) "MAmfGet.F90" "MAmfRead.F90" "MTAmf.F90" - Added interface module for AMF: "MTAmf.F90" - Remove "newmonth" and "month" input parameters from call to "GetOMFHx" (no impact on results), updated explanatory text. This involves modules "MObservationOperator.F90" and "MNO2RetrievalError.F90" 3.1.0 28 September 2015, DOMINO v3 with TROPOMI interfaces Henk Eskes, Mark ter Linden, KNMI Addition of TROPOMI netcdf interfaces Changes wrt 3.0.0: - Interface modules "MTObsTrack.F90" and "MTObsFcInfo.F90" were updated / simplified the "TObsTrack" structure now has allocatable arrays - Modules "MObsRead.F90" and "TM5IF_module.f90" developed by Mark ter Linden For reading slant columns retrieval L2 files in TROPOMI netcdf format For writing (replacing) vertical column data in the same L2 files - The O2-O2 cloud retrieval correction modules of Bram Maasakkers have been removed - Paths are now read from the TM5 rc file - Bug fixes: o call to "NO2ErrEstimate": there was a bug in passing the arrays to the routine (only first data block in the arrays was used) 3.0.0 21 August 2015, beta version DOMINO v3 Henk Eskes, KNMI With new interfaces to the TM5-CB05-mp parallel version, release early 2015 --> project: proj/domino Based on DOMINO-3 discussed in thesis Maasakkers, see below (v2.1.0.0) Changes wrt v2.1: - call in "tm5.F90" has been adjusted (two calls replaced by one, different location in time stepping sequence) - interface to the parallel TM5 code has been developed: module "assim_interf_mod.F90" o introducing "gather" and "scatter" mechanisms o interface has an improved way of extracting the NO2 field, using the ratio of "mixing NOx" and "chemistry NOx" o scaling of NOx concentration and slopes - introduce "MTM5Data.F90", which is a generic interface to TM5 (also for the NRT code, based on TM files) - introduce the generic structure "MTObsTrack.F90" for the interface to the input slant column data files (L2-slc) - introduce the generic structure "MTObsFcInfo.F90" for the interface to the output vertical column data files (L2) - "assim" now computes the grid cell columns: "GetNO2PartialColumnField" - re-written netcdf interfaces for the various input files (lookup-tables) - clean-up of all modules - replaced fixed storage by allocatable storage in several places - removal of 32-bit additional arrays (which are not needed) - ... 2.1.0.0 3 July prototype DOMINO v3 Discussed in: Maasakkers J.D., Vital improvements to the retrieval of tropospheric NO2 columns from the Ozone Monitoring Instrument, Master thesis, University of Technology Eindhoven, Eindhoven, The Netherlands Maasakkers J.D., Overview of changes to the DOMINO retrieval of tropospheric NO2 columns from the Ozone Monitoring Instrument, July 2 2013 2.0.0.0 7 September 2010, Folkert Boersma,: DOMINO v2.0 with: * improved AMF LUT (15 layers + below-surface extrapolation), * hi-res terrain height following Zhou et al., AMT, 2009, * improved surface albedo (Kleipool et al., JGR, 2008) for NO2 AMF, and O2-O2 cloud parameters, * explicit transportation of NO2 (along with NO, NO3, N2O5, and HNO4) instead of NOx in TM4 to repair 30-minute 'lagging' in NO-NO2 partitioning, leading to insufficient vertical mixing of boundary-layer NO2 (and too sharply peaked profiles). * weekly cycle for NOx emissions included in TM4 (sources_sinks.f90) for Europe and North America. 1.0.0.12a 12 June 2009: Test version for Ruud Dirksen to test influence of 15-layer AMF calculation 1.0.0.12 29 september 2008: Repaired small error in calculation of the pixelcornerpoints. This was not done correctly for the last row of the file 1.0.0.11a 5 June 2008: Special version for Jennifer, ignore the eclipse flag. Only used to process data of 22 sep 2006 1.0.0.11 29 April 2008: 2 Bugs solved. Bug #1: the fillvalues for the copied parameter SlantcolumnamountNo2 where not properly treated. All values of this field were multiplied by 1e15, including the fillvalues. This has been solved now. Bug #2: integer overflow when storing averaging kernel. This is solved by changing the scale factor from 1e-4 to 1e-3 This can handling kernel element values up to 32 instead of 3.2 as before. Raise outputflag when the groundpixelqualityflag show that the surface of the pixel is covered with snow. If so, the cloudfraction and cloudpressure are unreliable. Surface albedo is set to 0.6 in the L2 slant column product Data version promoted to 1.0.2 29 April 2008 1.0.0.10 25 March 2008: Changed description of TM4TerrainHeight and TM4SurfacePressure field (minor) Changed dataype of TroposphericColumnFlag to signed byte. (Dataversion 1.0.1, 25 March 2008) Include averaging kernel in DOMINO product. Remove it from the PROFILE product 1.0.0.9 20 March 2008: Change outputformat for the purpose to minimise the resulting filesize. Some datafields were removed (SpacecraftAltitude, SpacecraftLatitude,SpacecraftLongitude,OriginalSlantcolumnAmountNo2,AssimilatedStratosphericVerticalColumnError) The links in geolocationdata to TM4PressurelevelA & TM4PressurelevelB were removed Furthermore was the datatype of some fields adjusted to int16 in order to reduce space SurfaceAlbedo int32 -> int16 + scaling 0.0001 CloudRadianceFraction int32 -> int16 + scaling 0.01 TM4TerrainHeight int32 -> int16 TM4TropoPauseLevel int32 -> UCHAR In the profile file the following datatype modifications were implemented: AveragingKernel float -> int16 + scaling 0.0001 TemperatureProfile float -> uint16 + scaling 0.01 The links in geolocationdata to TM4PressurelevelA & TM4PressurelevelB were removed The dataversion is now set to 1.0.1 1.0.0.8a 20 March 2008: Special version for Jennifer, ignore the eclipse flag. Only used to process data of 22 sep 2006 1.0.0.8 16 March 2008: Change in write procedure. Resolve bug because of which the int32 fields TM4Terrainheight, TM4Tropopauselevel and TroposphericColumnflag were not written. Major impact on data usability for e.g. Yipin This affects data of 26 sep 2006 and onwards 1.0.0.7 10 March 2008: Minor change in write procedure. Write correct filename to L2 inputfile swathattribute Added NumTimes swathattribute 1.0.0.6 20 February 2008: Minor change in write procedure. Now the scalefactor of the cloudfraction is correctly written to file 1.0.0.5 19 February 2008: Changed Title attribute of SurfaceAlbedo datafield to "..GOME database.." instead of "..Koelemeijer database.." 1.0.0.4 15 February 2008: Keep on processing with the Koelemeijer albedo database. Dataversion = 1.0.0 1.0.0.3 14 february 2008: Attribute DataVersion added to output. This is to distinguish between (minor) program changes that do not affect the algorithm and changes that do involve the algorithm and thereby affect the results. Also write temperature profile to the NO2 profile/av kernel file. IMPORTANT !!!!!!!!! Bug fix: so far the albedo from the L2 file was used (Koelemijer database). This is the result of a test for using the OMI albedo data from the L2 file which will be implemented in the future. This test setting was overlooked and not restored for the reprocessing. The data until ?? August 2006 are affected by this bug. In principle this dataset is an interesting testcase to study the effect of the surface albedo on the retrieval. 1.0.0.2 12 february 2008: Scaling of cloudfraction & TM4surfacepressure solved. Writing NO2 profile & averaging kernel to separate files 1.0.0.1 4 february 2008: Write mmix data at 13h30 (for Yipin) Solved system stop for zero column values in ass_Hx IMPORTANT: because of change of ECMWF meteodata format at 31 jan 2006 there is a transition in the savefile. (35 -> 34 sigma layers). It was not possible to interpolate the old-format file of 30 jan to the new-format so a save file was created by a free model run starting on 20 jan 2006 with a 10 feb 2006 savefile (new format) that still was around from the NRT processing. 1.0.0.0 4 January 2008: The version for reprocessingFix pixelcornerpoints issue for time jumps in granule. Forgotten to activate fitflag filtering!! Issue with pixel cornerpoints Mistake in scaling and/or units of cloudfraction & TM4surfacepressure 0.9.9.4 18 December 2007: Changes in output file. Separate file for DOMINO output, (no append) reduction of # fields 0.9.9.3 15 December 2007: Parallellisation of convection, chemistry & AMF calculation implemented 0.9.9.2 3 December 2007: Vectorized version of AMF calculation 0.9.9.1 3 December 2007: Major upgrade of the program. This upgrade coincides with the reprocessing of the OMI data. As a result of the reprocessing the striping of the NO2 data is greatly reduced. The upgrade includes -HDF-EOS5 output -different destriping algorithm -proper usage of the flags in the L2 product 0.9.3.6 14 may 2007: Exclude orbits in case of solar eclipse 0.9.3.5 15 april 2007: Store OmF (Observation minus Forecast) results in he5 file 0.9.3.4 20 March 2007: Added slant column error to output product 0.9.3.3 26 February 2007: Bug solved that caused incorrect ltropo values in the output product 0.9.3.2 15 February 2007: Added Model predicted trop column to output product Fieldname fcvcdtr in ANC_ vdata 0.9.3.1 26 January 2007: Added initial SCD to output product. This the slant column calculated by the OMNO2A algorithm Fieldname scdorig in ANC_ vdata 0.9.3 1 December 2006: Ground pixel quality flag added to geolocation data 0.9.2 16 November 2006: Inputpointers added to metadata (global attributes) 0.9.1 11 September 2006: pixel nr & image nr added to geolocation data 0.9 9 August 2006: Stripe correction polynomial fit instead of FFT