# EC-Earth_patch

EC-Earth is a global coupled climate model. It is developed by a consortium of European research institutions.
The goal of this project is to adapt this model for our needs in TECLIM: install, run management, etc.

### Important Notes

**YOU MUST HAVE YOUR `~/.ssh/id_rsa.pub` FILE BEFORE STARTING TO WORK WITH GIT ON ZENOBE**.

```
mv $HOME/.ssh/config $HOME/.ssh/config_old
cat config | sed -e 's/pbarriat/'$USER'/g' > $HOME/.ssh/config

[ ! -f ~/.ssh/id_rsa.pub ] && ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub gwceci.cism.ucl.ac.be
```

## Install

First, clone the `pbarriat/ecearth_patch` repository:
```bash
mkdir ~/modeles
cd ~/modeles
mkdir ecearth
git clone ssh://regit/pbarriat/ecearth_patch.git
```

Now, update your shell startup script (`~/.bashrc`) to add ecearth globals variables to your environment. 

For instance **on Zenobe**, you need to add:
```bash
configure_ecearth()
{
  if [ `hostname` != "frontal1" ]; then
    echo "First, we move to frontal1..."
    touch ~/.configure_ecearth
    ssh frontal1
  fi
  export EC_DEP=/projects/acad/ecearth/opt
  if [ -f $EC_DEP/modules.load ]; then
    source $EC_DEP/modules.load
  fi
  [[ -f ~/.configure_ecearth ]] && rm ~/.configure_ecearth
}
if [ `hostname` = "frontal1" ]; then
  [[ -f ~/.configure_ecearth ]] && configure_ecearth
fi
if [ `hostname` = "frontal3" ]; then
  PATH="/projects/acad/ecearth/opt/apps/lua/lua/bin:$PATH"
  . /projects/acad/ecearth/opt/lmod/lmod/init/profile
  ulimit -u 8192
  export OMP_NUM_THREADS=2
fi
```

Then, checkout the EC-Earth SVN repository. Four alternatives:
```bash
cd ecearth

svn co https://svn.ec-earth.org/ecearth3/tags/3.2.2 ecearth_3.2.2

svn co https://svn.ec-earth.org/ecearth3/tags/3.2.3 ecearth_3.2.3

svn co https://svn.ec-earth.org/ecearth3/branches/projects/primavera ecearth_primavera

svn co https://svn.ec-earth.org/ecearth3/trunk ecearth_trunk
```

Final step, go to the `pbarriat/ecearth_patch` repository and run the `install.sh` script:
```
cd ecearth_patch
./install.sh
```

## Compile

### Lemaitre3

* Go to `ecearth sources` and create all the Makefile's with lemaitre3 configuration:

```bash
cd ecearth_3.2.2/sources
./util/ec-conf/ec-conf -p lemaitre3-intelmpi config-build.xml

module purge
module load 2016a netCDF-Fortran/4.4.4-intel-2016a grib_api/1.16.0-intel-2016a openjpeg/1.5.2-intel-2016a
```

* Compile **oasis3**:

```bash
cd oasis3-mct/util/make_dir
make BUILD_ARCH=ecconf -f TopMakefileOasis3
cd ../../..
```

* Compile **IFS**:

```bash
cd ifs-36r4
./makeifs -j 4
cd ..
```

* Compile **xios-2**:

```bash
cd xios-2
./make_xios --prod --arch ecconf --use_oasis oasis3_mct --netcdf_lib netcdf4_par -j 4
cd ..
```

* Compile **NEMO**:

```bash
cd nemo-3.6/CONFIG
./makenemo -n ORCA1L75_LIM3 -m ecconf -j4
cd ../..
```

* Compile **runoff-mapper**:

```bash
cd runoff-mapper/src
make
cd ../..
```

Primavera context:

* Compile **amip-forcing**:

```bash
cd amip-forcing/src
make
cd ../..
```

* Configure **ELPiN**:

```bash
cd util/ELPiN
mkdir bathymetries
ln -s /scratch/ucl/elic/pbarriat/DATA/EC-EARTH/nemo/initial/ORCA1L75/bathy_meter.nc bathymetries/ece_bathy_meter_ORCA1.nc
ln -s /scratch/ucl/elic/pbarriat/DATA/EC-EARTH/nemo/initial/ORCA025L75/bathy_meter.nc bathymetries/ece_bathy_meter_ORCA025.nc
sed -i 's/ORCA025/ORCA1/g' src/namelist_mpp
./Configure
cd ../..
```

### Zenobe

First, **sign out of Zenobe** and sign in again. Then, type `configure_ecearth`.

Compile the model:

1. Go to `~/modeles/ecearth_$tag/sources` and:
```
./util/ec-conf/ec-conf -p zenobe-intelmpi config-build.xml
```
2. Go to `./oasis3-mct/util/make_dir` and:
```
make BUILD_ARCH=ecconf -f TopMakefileOasis3
```
3. Go to `./ifs-36r4` and:
```
./makeifs -j 4
```
4. Go to `./xios-2` and:
```
./make_xios --prod --arch ecconf --use_oasis oasis3_mct --netcdf_lib netcdf4_par -j 4
```
5. Go to `./nemo-3.6/CONFIG` and:
```
./makenemo -n ORCA1L75_LIM3 -m ecconf -j4
```
6. Go to `./runoff-mapper/src` and:
```
make
```

Primavera context:

7. Go to `./amip-forcing/src` and:
```
make
```
8. Go to `./util/ELPiN` and:

```
mkdir bathymetries
ln -s /SCRATCH/acad/ecearth/DATA/nemo/initial/ORCA1L75/bathy_meter.nc bathymetries/ece_bathy_meter_ORCA1.nc
ln -s /SCRATCH/acad/ecearth/DATA/nemo/initial/ORCA025L75/bathy_meter.nc bathymetries/ece_bathy_meter_ORCA025.nc
sed -i 's#"$min_num"#"${min_num/./,}"#g' ELPiNv2.cmd
sed -i 's/ORCA025/ORCA1/g' src/namelist_mpp
./Configure
```

## Run

### Lemaitre3

```bash
cd ../runtime/classic
../../sources/util/ec-conf/ec-conf -p lemaitre3 config-run.xml
vi ece-ifs+nemo.sh
sbatch ece-ifs+nemo.sh
```

### Zenobe

For a quick classical run test:
- Go to `~/modeles/ecearth_$tag/runtime/classic` and:
```
../../sources/util/ec-conf/ec-conf -p zenobe config-run.xml
```
- Submit the script `ece-ifs+nemo.sh` to the PBS batch system:
```
qsub ece-ifs+nemo.sh
```

## Benchmarks

### Lemaitre3

| XIOS | NEMO | IFS | Runoff | WTIME per YEAR | #CORES |
| ---- | ---- | --- | ------ | -------------- | ------ |
| 1    |  64  |  78 |    1   |    **3h56**    |   144  |
| 22   |  72  | 288 |    1   |                |   384  |
| 22   | 144  | 432 |    1   |                |   600  |

### Zenobe

| XIOS | NEMO | IFS | Runoff | WTIME per YEAR | #CORES |
| ---- | ---- | --- | ------ | -------------- | ------ |
| 1    |  64  |  78 |    1   |      **5h**    |   144  |
| 22   |  72  | 288 |    1   |    **3h15**    |   384  |
| 22   | 144  | 432 |    1   |                |   600  |

## Contributors

- ELIC members [@pbarriat](http://www.climate.be:3000/pbarriat)

## License

This project is under the Creative Commons CC0 1.0 Universal License. See the [LICENSE](http://www.climate.be:3000/TECLIM/LOVECLIM/src/master/LICENSE) file for the full license text.