#27 Potential major source of error when fetching restarts

Hi @pbarriat @fklein

When an EC-Earth simulation is initialized from restarts of another simulation, there is potentially a big problem, and it is likely that the wrong initial restarts will be fetched.

Here, I am in the following case, where an experiment has been specified to restart from, and a special restart date has been provided

force_run_from_scratch=false
special_restart=true
special_restart_from=ap00
special_restart_date="2012-05-01"

In that case, there is a sed command used at around line 275 of the submission script, that is used to identify the leg number to restart from, by modifying the ece.info that was beforehand copied from the mother experiment:

sed -i "/$special_year/q" ${run_dir}/ece.info

The problem is that this ece.info file has, in many occasions, four-digit numbers that look like years, but are not:

fmasson@frontal1 10:18:26 ~/modeles/ecearth/ecearth_3.3.0/runtime/classic 62 > tail /SCRATCH/acad/ecearth/fmasson/run/ap00/ece.info
# CPMIP performance: 4.76 SYPD   2054 CHPSY
leg_number=855
leg_start_date="Thu, 01 Nov 2012 00:00:00 +0000"
leg_end_date="Sat, 01 Dec 2012 00:00:00 +0000"
#
# Finished leg at 2020-09-24 01:44:40 after 00:25:38 (hh:mm:ss)
# CPMIP performance: 4.77 SYPD   2052 CHPSY
leg_number=856
leg_start_date="Sat, 01 Dec 2012 00:00:00 +0000"
leg_end_date="Tue, 01 Jan 2013 00:00:00 +0000"

Here, 2052 and 2054 appear but they are not years, they are rather performance indicators (I think).

The fact that the sed commands bases itself on the special year only is problematic, because it can give totally unwanted results. Here is an example of successful sed command (with the special year 2008):

fmasson@frontal1 10:20:44 ~/modeles/ecearth/ecearth_3.3.0/runtime/classic 65 > sed -e /2008/q /SCRATCH/acad/ecearth/fmasson/run/ap00/ece.info | tail
# CPMIP performance: 3.55 SYPD   2752 CHPSY
leg_number=795
leg_start_date="Thu, 01 Nov 2007 00:00:00 +0000"
leg_end_date="Sat, 01 Dec 2007 00:00:00 +0000"
#
# Finished leg at 2020-09-13 00:35:48 after 00:26:17 (hh:mm:ss)
# CPMIP performance: 4.65 SYPD   2104 CHPSY
leg_number=796
leg_start_date="Sat, 01 Dec 2007 00:00:00 +0000"
leg_end_date="Tue, 01 Jan 2008 00:00:00 +0000"

but here is an example of the command not working properly (with the special year 2012):

fmasson@frontal1 10:20:51 ~/modeles/ecearth/ecearth_3.3.0/runtime/classic 66 > sed -e /2012/q /SCRATCH/acad/ecearth/fmasson/run/ap00/ece.info | tail
leg_end_date="Mon, 01 Jan 1934 00:00:00 +0000"
#
# Finished leg at 2020-07-16 11:41:05 after 04:58:19 (hh:mm:ss)
# CPMIP performance: 4.82 SYPD   2028 CHPSY
leg_number=85
leg_start_date="Mon, 01 Jan 1934 00:00:00 +0000"
leg_end_date="Tue, 01 Jan 1935 00:00:00 +0000"
#
# Finished leg at 2020-07-16 16:37:22 after 04:56:01 (hh:mm:ss)
# CPMIP performance: 4.86 SYPD   2012 CHPSY

This is because 2012 appears not as a year, and the sed command does not look further. Here, the run will start from 1934 instead of 2012!

Fix

Add "00:00:00" after special year to make sure the file is correctly cropped.

fmasson@frontal1 10:23:46 ~/modeles/ecearth/ecearth_3.3.0/runtime/classic 70 > sed -e "/2012 00:00:00/q" /SCRATCH/acad/ecearth/fmasson/run/ap00/ece.info | tail
# CPMIP performance: 4.76 SYPD   2054 CHPSY
leg_number=843
leg_start_date="Tue, 01 Nov 2011 00:00:00 +0000"
leg_end_date="Thu, 01 Dec 2011 00:00:00 +0000"
#
# Finished leg at 2020-09-22 22:40:13 after 00:25:38 (hh:mm:ss)
# CPMIP performance: 4.77 SYPD   2052 CHPSY
leg_number=844
leg_start_date="Thu, 01 Dec 2011 00:00:00 +0000"
leg_end_date="Sun, 01 Jan 2012 00:00:00 +0000"

This means changing in the submission script the line

sed -i "/$special_year/q" ${run_dir}/ece.info

into

sed -i "/$special_year 00:00:00/q" ${run_dir}/ece.info

Question: was this line coded by you @pbarriat (in which case the problem is limited to UCLouvain runs) or is it present in the default submission script that comes along the code (in which case I'll open a ticket on the EC-Earth dev portal)?

Cheers,

François

Edit

The fix above does not even work when the initial date of year is not 1st of January. Here is a more general piece of code that will fetch the exact leg folder, even when monthly or daily restarts are saved:

        special_year=${special_restart_date:0:4}
        special_month=${special_restart_date:5:2}
        special_day=${special_restart_date:8:2}
        special_date=`date -d ${special_year}-${special_month}-${special_day} +%a,\ %d\ %b\ %Y`
        grepString="${special_date} 00:00:00 +0000"
        sed -i "/${grepString}/q" ${run_dir}/ece.info