# Coral - simulation manager for NEMO
## Purpose

Coral is a collaborative job submission script  manager for the NEMO (Nucleus for European Modelling of the Ocean) modeling framework [Documentation](http://www.nemo-ocean.eu). With Coral, you can use and re-use job submission scripts designed and tested by your colleagues. You can ammend them in an easy way, and generate a submission script tailored to your needs. Then, you can share your script with your colleages.

## Overview

- Run 'coral -h' to get a list of Coral commands, and 'coral \<command\> -h' to get help for a specific \<command\>
- Please see the [Documentation](https://gogs.elic.ucl.ac.be/Coral/docs) for common usages and change log.
- Having trouble? Get help with [Troubleshooting](https://gogs.elic.ucl.ac.be/Coral/docs/troubleshooting.html)

## Features

Once Nemo and Coral are properly installed, you can create a working submission script in three commands:
 - coral init: initiate a set of configuration files (with working default values on the cluster) for the experiment features, data location, restart options, etc.
 - coral build: create a submission script from the configuration files
 - coral submit: submit the job, with options to run tests locally, ignore automatic job resubmission, restart from scratch, etc.
  
At each stage, you can act manually to adapt the workflow to your liking. 
  
Once the job is submitted, you can simply run 'coral status' in the run directory to get an overview of what is happening. No need to look for job ids, or output files.

## Requirements

NEMO & XIOS source code: see [How to get NEMO from scratch](https://gogs.elic.ucl.ac.be/TECLIM/Coral/wiki/NEMO-from-scratch)

## Install

Clone the Coral repository (for instance in the EXTERNAL directory of your NEMO install) 

```bash
cd $HOME/modeles/nemo/3.6/EXTERNAL
git clone ssh://git@www.climate.be:3022/TECLIM/Coral.git coral
```

and update your shell startup script to source the 'setup-coral.sh' file to add 'coral' to your PATH variable.

```bash
configure_nemo()
{
  [...]
  source $HOME/modeles/nemo/3.6/EXTERNAL/coral/setup-coral.sh
}
```

## Getting started

NEMO & XIOS must be compiled: see [How to compile NEMO](https://gogs.elic.ucl.ac.be/TECLIM/Coral/wiki/NEMO-compilation)

### Create a NEMO config experiment

```bash
cd $HOME/modeles/nemo/3.6/CONFIG
# Create new experiment
./makenemo -n ORCA1_LIM3_REF -d "OPA_SRC LIM_SRC_3" -m myARCHfile -j0
# Check your CPP keys
vi ORCA1_LIM3_REF/cpp_ORCA1_LIM3_REF.fcm
# Compile
./makenemo -n ORCA1_LIM3_REF -m myARCHfile -j4
```

### Create a Coral instance

```bash
cd ORCA1_LIM3_REF
mkdir CORAL00
cd CORAL00
coral -h
coral init -h
coral list
coral init lemaitre3
```

### Benchmarks

#### Lemaitre3 

> Blocking Switch = 2 x 576 cores & 1 x 768 cores

* Compiler: intel-2016a
* Flags: -O2 -i4 -r8 -fp-model precise -xHost

| XIOS |  NEMO  | WTIME per YEAR | #CORES | MEM/CPU |
| ---- | ------ | -------------- | ------ | ------- |
|   2  |    22  |  **133min17**  |    24  |   3072  |
|   2  |    46  |  **77min14**   |    48  |   3072  |
|   2  |   140  |  **44min20**   |   142  |   3072  |
|   4  |   260  |  **43min15**   |   264  |   3584  |
|   4  |   520  |  **36min09**   |   524  |   3584  |
|   8  |   760  |  **40min31**   |   768  |   3584  |
|   8  |  1048  |  **50min51**   |  1056  |   3584  |
|   8  |  1912  |  **53min49**   |  1920  |   3584  |

#### Zenobe

* Compiler: intel-2015b
* Flags: -O2 -i4 -r8 -fp-model precise -xHost

| XIOS |  NEMO  | WTIME per YEAR | #CORES | MEM/CPU |
| ---- | ------ | -------------- | ------ | ------- |
|   2  |    22  |  **229min28**  |    24  |   2625  |
|   2  |    46  |  **146min**    |    48  |   2625  |
|   2  |   140  |  **58min29**   |   142  |   2625  |
|   4  |   260  |  **48min30**   |   264  |   2625  |
|   4  |   520  |  **34min54**   |   524  |   2625  |
|   4  |   760  |  **38min46**   |   768  |   2625  |
|   8  |  1048  |  **43min**     |  1056  |   2625  |
|   8  |  1936  |  **36min26**   |  1944  |   2625  |

![bench_nemo.jpeg](https://gogs.elic.ucl.ac.be/TECLIM/Coral/raw/master/utils/bench_nemo.jpeg)

## Acknowledgments

- Thanks [@dfr](https://gogs.elic.ucl.ac.be/dfr) for initializing this project.

## Contributors

- [@dfr](https://gogs.elic.ucl.ac.be/dfr), [@pbarriat](https://gogs.elic.ucl.ac.be/pbarriat).

## License

This project is under the Creative Commons CC0 1.0 Universal License. See the [LICENSE](https://gogs.elic.ucl.ac.be/pbarriat/Coral/src/master/LICENSE) file for the full license text.