8. Compiling#

The files that you need to edit for compilation are:

cppdefs.h	CPP-keys* allowing to select configuration, numerical schemes, parameterizations, forcing and boundary conditions * CROCO extensively uses the C preprocessor (cpp) during compilation to replace code statements, insert files into the code, and select relevant parts of the code depending on its directives.
param.h	Grid settings: the values of the model grid size are: LLm0 points in the X direction MMm0 points in the Y direction N vertical levels For realistic regional cases, LLm0 and MMm0 are given by running `make_grid.m`, and N is defined in `crocotools_param.m` `param.h` also contains: Parallelisation settings Tides, Wetting-Drying, Point sources, Floats, Stations specifications
jobcomp	The compilation script (including settings for paths, compilers, libraries, etc)

Warning

CROCO needs to be compiled for each configuration (domain, coupled, uncoupled, parameterizations…), i.e., each time you change something in cppdefs.h or param.h

Let’s explore, check, and edit the 3 aforementionned files:

8.1. cppdefs.h#

Let’s explore, check, and edit: cppdefs.h

First section of cppdefs.h defines your configuration (test case or realistic regional case):

#undef  BASIN           /* Basin Example */
#undef  CANYON          /* Canyon Example */
#undef  EQUATOR         /* Equator Example  */
#undef  INNERSHELF      /* Inner Shelf Example */
#undef  RIVER           /* River run-off Example */
#undef  OVERFLOW        /* Graviational/Overflow Example */
#undef  SEAMOUNT        /* Seamount Example */
#undef  SHELFRONT       /* Shelf Front Example */
#undef  SOLITON         /* Equatorial Rossby Wave Example */
#undef  THACKER         /* Thacker wetting-drying Example */
#undef  UPWELLING       /* Upwelling Example */
#undef  VORTEX          /* Baroclinic Vortex Example */
#undef  INTERNAL        /* Internal Tide Example */
#undef  IGW             /* COMODO Internal Tide Example */
#undef  JET             /* Baroclinic Jet Example */
#undef  SHOREFACE       /* Shoreface Test Case on a Planar Beach */
#undef  RIP             /* Rip Current Test Case */
#undef  SANDBAR         /* Bar-generating Flume Example */
#undef  SWASH           /* Swash Test Case on a Planar Beach */
#undef  TANK            /* Tank Example */
#undef  ACOUSTIC        /* Acoustic wave Example */
#undef  GRAV_ADJ        /* Graviational Adjustment Example */
#undef  ISOLITON        /* Internal Soliton Example */
#undef  KH_INST         /* Kelvin-Helmholtz Instability Example */
#undef  TS_HADV_TEST    /* Horizontal tracer advection Example */
#define REGIONAL        /* REGIONAL Applications */

For the BENGUELA_LR case we are running, you should have:

#define REGIONAL        /* REGIONAL Applications */

Then, in cppdefs.h, you have one section for each case. Let’s explore the REGIONAL case section:

First is the name of your configuration:

#if defined REGIONAL
/*
!====================================================================
!               REGIONAL (realistic) Configurations
!====================================================================
!
!----------------------
! BASIC OPTIONS
!----------------------
!
*/
                    /* Configuration Name */
# define BENGUELA_LR

Then, you can set parallelization option (you can set define MPI if you want to run in parallel)
```
/* Parallelization */
# undef  OPENMP
# undef  MPI
```
Then, you can set I/O options (XIOS server, netcdf 4 parallel option, NB: we will have a dedicated tutorial on XIOS)
```
/* I/O server */
# undef  XIOS
```

Non-hydrostatic option

/* Non-hydrostatic option */
# undef  NBQ

Nesting settings

/* Nesting */
# undef  AGRIF
# undef  AGRIF_2WAY

Coupling with other models (atmosphere, waves)

/* OA and OW Coupling via OASIS (MPI) */
# undef  OA_COUPLING
# undef  OW_COUPLING

Including wave-current interactions

/* Wave-current interactions */
# undef  MRL_WCI

Managing open boundaries (you can choose to close one of the boundaries, useful in coastal cases)

/* Open Boundary Conditions */
# undef  TIDES
# define OBC_EAST
# define OBC_WEST
# define OBC_NORTH
# define OBC_SOUTH

Activating applications

/* Applications */
# undef  BIOLOGY
# undef  FLOATS
# undef  STATIONS
# undef  PASSIVE_TRACER
# undef  SEDIMENT
# undef  BBL

Defining a dedicated log file for CROCO standard output (default is undef but you can define LOGFILE to facilitate the reading of model output, particularly useful for coupled simulations)
```
/* dedicated croco.log file */
# undef  LOGFILE
```
Warning

Keep undef LOGFILE is you use Plurimonth run scritps as: run_croco_inter.bash because it already re-direct the CROCO output, and check it…
Time reference setting:

Warning

By default no reference time is used, and time is referred to the beginning of the simulation only
```
/* Calendar */
# undef USE_CALENDAR
```

Then you have detailed settings (you can find a description of all cpp keys in Contents and Architecture section of the Tutorials). Let’s just highlight a few ones:
- In grid configuration
```
/* Grid configuration */
# define CURVGRID
# define SPHERICAL
# define MASKING
# undef  WET_DRY
# define NEW_S_COORD
```
  Warning
  
  you should check that the vertical coordinate setting NEW_S_COORD is in adequation with your pre-processing setting (vtransform=2 in crocotools_param.m)
- In surface forcing subsection:
  - if you have prepared croco_frc.nc file (using make_frc.m)
    /* Surface Forcing */ # undef BULK_FLUX
  - if you have prepared croco_blk.nc file (using make_blk.m)
    /* Surface Forcing */ # define BULK_FLUX
- Then, you have to set your lateral forcing according to your pre-processing as well:
  - If you have prepared croco_clm.nc file (using make_clim.m)
    /* Lateral Forcing */ # define CLIMATOLOGY
    and
    # undef FRC_BRY
  - Or, if you have prepared croco_bry.nc file (using make_bry.m)
    /* Lateral Forcing */ # undef CLIMATOLOGY
    and
    # define FRC_BRY
  The other CPP-keys will be explored in other tutorials.

8.2. param.h#

param.h is composed of the following sections:

Dimensions of Physical Grid and array dimensions
MPI related variables
Number maximum of weights for the barotropic mode
OA-Coupling, Tides, Wetting-Drying, Point sources, Floast, Stations
Derived dimension parameters
I/O : flag for type sigma vertical transformation
Number of tracers
Tracer identification indices

Most of the time you only need to check/edit the 2 first sections:

Check the grid settings:
```
#  elif defined  BENGUELA_LR
      parameter (LLm0=41,   MMm0=42,   N=32)   ! BENGUELA_LR
```
- LLm0: Dimension (ghost points included) in the $\xi$ direction.
- MMm0: Dimension (ghost points included) in the $\eta$ direction.
- N: Number of $\rho$-vertical points, in the vertical grid.
Check and eventually edit the parallelization settings:
```
#ifdef MPI
      integer NP_XI, NP_ETA, NNODES
      parameter (NP_XI=1,  NP_ETA=4,  NNODES=NP_XI*NP_ETA)
      parameter (NPP=1)
      parameter (NSUB_X=1, NSUB_E=1)
#elif defined OPENMP
      parameter (NPP=4)
```
- In the case of OpenMP parallelization, NPP is the number of cpu used in the computation
- In the case of MPI parallelization, it is equal to to NNODES.
- AUTOTILING (implemented by L. Debreu): cpp-key that enable to compute the optimum subdomains partition in terms of computation time.
Note

MPI tiles should be at least 20x20 points.

8.3. jobcomp#

Now that your input files are set up, you can proceed to compilation:

Here we assume that you have set a few environment variables for compilers and libraries. Here is an example with Intel compilers and a netcdf library located in $HOME/softs/netcdf. Adapt these to your own settings (in your .bashrc file):

# compilers
export CC=icc
export FC=ifort
export F90=ifort
export F77=ifort
export MPIF90=mpiifort

# netcdf library
export NETCDF=$HOME/softs/netcdf
export PATH=$NETCDF/bin:${PATH}
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${NETCDF}/lib

Edit the compilation script jobcomp:

# set source, compilation and run directories
#
SOURCE=~/croco/croco/OCEAN
SCRDIR=./Compile
RUNDIR=`pwd`
ROOT_DIR=$SOURCE/..
#
# determine operating system
#
OS=`uname`
echo "OPERATING SYSTEM IS: $OS"

#
# compiler options
#
FC=$FC

#
# set MPI directories if needed
#
MPIF90=$MPIF90
MPIDIR=$(dirname $(dirname $(which $MPIF90) ))
MPILIB="-L$MPIDIR/lib -lmpi -limf -lm"
MPIINC="-I$MPIDIR/include"

# set NETCDF directories
#
#-----------------------------------------------------------
# Use :
#-lnetcdf           : version netcdf-3.6.3                --
#-lnetcdff -lnetcdf : version netcdf-4.1.2                --
#-lnetcdff          : version netcdf-fortran-4.2-gfortran --
#-----------------------------------------------------------
#
#NETCDFLIB="-L/usr/local/lib -lnetcdf"
#NETCDFINC="-I/usr/local/include"
NETCDFLIB=$(nf-config --flibs)
NETCDFINC=-I$(nf-config --includedir)

Compile the model:
```
./jobcomp > jobcomp.log
```
If compilation is successful, you should have a croco executable in your directory.

You will also find a Compile directory containing the model source files:
- .F files: original model source files that have been copied from ~/croco/croco/OCEAN
- _.f files: pre-compiled files in which only parts defined by cpp-keys are kept
- .o object files

8.4. Compilation options#

A very summarized information on compilation options is given here. For further details, search information on the web, or with your cluster assistance team. Useful informations can also be found on this page: http://www.idris.fr/jean-zay/cpu/jean-zay-cpu-comp_options.html

Optimization options:
- -O0, -O1, -O2, -O3, -fast : optimization level. -O0 is no optimization, use it for debug. -O3 and -fast are more agressive optimization options that can lead to problems in reproducibility of your run (especially it is better to avoid -fast).
- -xCORE-AVX2 : vectorization option, very agressive optimization => non-reproducibility of CROCO
- -fno-alias, -no-fma, -ip : other optimization options, commonly used
- -ftz: set to 0 denormal very small numbers. It is set by default with -O1, -O2, -O3 (can be a problem in calculation precision)
Debug options: -O0 -g -debug -fpe-all=0 -no-ftz -traceback -check all -fbacktrace -fbounds-check -finit-real=nan -finit-integer=8888
Precision and writing options:
- -fp-model precise: important to have good precision and reproducibility of your calculations
- -assume byterecl: way of writing: byte instead of bit
- -convert big_endian: way of writing binaries (important for avoiding huge negative numbers)
- -i4, -r8``: way of writing integers and reals (important also for reproducibility between different clusters)
- -72: specifies that the statement field of each fixed-form source line ends at column 72.
- -mcmodel=medium -shared-intel : do not limit memory to 2Go for data (useful for writing large output files)

8.5. Tips in case of errors during compilation#

In case of strange errors during compilation (e.g. “catastrophic error: could not find …”), try one of these solutions:

check your home space is not full ;-)
check your paths to compilers and libraries (especially Netcdf library)
check that you have the good permissions, and check that your executable files (configure, make…) do are executable
check that your shell scripts headers are correct or add them if necessary (e.g. for bash: #!/bin/bash)
try to exit/log out the machine, log in back, clean and restart compilation

Errors and tips related to netcdf library:

with netcdf 4.3.3.1: need to add the following compilation flag for all models: -mt_mpi

The error associated to a missing -mt_mpi flag is of this type: “ /opt/intel//impi/4.1.1.036/intel64/lib/libmpi_mt.so.4: could not read symbols: Bad value “
with netcdf 4.1.3: do NOT add -mt_mpi flag
with netcdf4, need to place hdf5 library path in your environment:

export LD_LIBRARY_PATH=YOUR_HDF5_DIR/lib:$LD_LIBRARY_PATH

with netcdf 4, if you use the library splitted in 2: C part and Fortran part, you need to place links to C library before links to Fortran library and need to put both path in this same order in your LD_LIBRARY_PATH

In case of ‘segmentation fault’ error:

try to allocate more memory with “unlimited -s unlimited”
try to launch the compilation as a job (batch) with more allocated memory

Compiling

Contents

8. Compiling#

8.1. cppdefs.h#

8.2. param.h#

8.3. jobcomp#

8.4. Compilation options#

8.5. Tips in case of errors during compilation#