Compiling ========= The files that you need to edit for compilation are: +---------------+-----------------------------------------------------------------------------------+ | **cppdefs.h** | CPP-keys* allowing to select configuration, numerical schemes, parameterizations, | | | forcing and boundary conditions. | | | * *CROCO extensively uses the C preprocessor (cpp) during compilation to* | | | *replace code statements, insert files into the code, and select relevant* | | | *parts of the code depending on its directives.* | +---------------+-----------------------------------------------------------------------------------+ | **param.h** | Grid settings, the values of the model grid size are: | | | | | | * LLm0 points in the X direction | | | * MMm0 points in the Y direction | | | * N vertical levels | | | | | | For realistic regional cases, LLm0 and MMm0 are given : | | | | | | * in croco_tools : by running ``make_grid.m`` | | | * in croco_pytools : by running ``make_grid.py`` | | | | | | N is defined in ``crocotools_param.m`` or ibc.ini (IBC_Sigma_params) | | | | | | ``param.h`` also contains: Parallelisation settings | | | Tides, Wetting-Drying, Point sources, Floats, Stations specifications | +---------------+-----------------------------------------------------------------------------------+ | **jobcomp** | The compilation script (including settings for paths, compilers, libraries, etc) | +---------------+-----------------------------------------------------------------------------------+ .. warning:: CROCO needs to be compiled for each configuration (domain, coupled, uncoupled, parameterizations...), *i.e.*, each time you change something in ``cppdefs.h`` or ``param.h`` Let's explore, check, and edit the 3 aforementionned files: cppdefs.h ^^^^^^^^^ Let's explore, check, and edit: ``cppdefs.h`` #. First section of ``cppdefs.h`` defines your configuration (test case or realistic regional case): :: #undef BASIN /* Basin Example */ #undef CANYON /* Canyon Example */ #undef EQUATOR /* Equator Example */ #undef INNERSHELF /* Inner Shelf Example */ #undef SINGLE_COLUMN /* 1DV vertical mixing Example */ #undef RIVER /* River run-off Example */ #undef OVERFLOW /* Gravitational/Overflow Example */ #undef SEAMOUNT /* Seamount Example */ #undef SHELFRONT /* Shelf Front Example */ #undef SOLITON /* Equatorial Rossby Wave Example */ #undef THACKER /* Thacker wetting-drying Example */ #undef UPWELLING /* Upwelling Example */ #undef VORTEX /* Baroclinic Vortex Example */ #undef INTERNAL /* Internal Tide Example */ #undef IGW /* COMODO Internal Tide Example */ #undef JET /* Baroclinic Jet Example */ #undef SHOREFACE /* Shoreface Test Case on a Planar Beach */ #undef RIP /* Rip Current Test Case */ #undef SANDBAR /* Bar-generating Flume Example */ #undef SWASH /* Swash Test Case on a Planar Beach */ #undef TANK /* Tank Example */ #undef MOVING_BATHY /* Moving Bathymetry Example */ #undef ACOUSTIC /* Acoustic wave Example */ #undef GRAV_ADJ /* Graviational Adjustment Example */ #undef ISOLITON /* Internal Soliton Example */ #undef KH_INST /* Kelvin-Helmholtz Instability Example */ #undef TS_HADV_TEST /* Horizontal tracer advection Example */ #undef DUNE /* Dune migration Example */ #undef SED_TOY /* 1DV sediment toy Example */ #undef TIDAL_FLAT /* 2DV tidal flat Example */ #undef ESTUARY /* 3D tidal estuary Example */ #undef KILPATRICK /* 2D sst front*/ #undef SEAGRASS /* 2DV over seagrass using OBSTRUCTION module*/ For the BENGUELA_LR case we are running, you should have: :: #define REGIONAL /* REGIONAL Applications */ #. Then, in ``cppdefs.h``, you have one section for each case. Let's explore the REGIONAL case section: * First is the name of your configuration: :: #if defined REGIONAL /* !==================================================================== ! REGIONAL (realistic) Configurations !==================================================================== ! !---------------------- ! BASIC OPTIONS !---------------------- ! */ /* Configuration Name */ # define BENGUELA_LR * Then, you can set parallelization option (you can set ``define MPI`` if you want to run in parallel) :: /* Parallelization */ # undef OPENMP # undef MPI * Then, you can set I/O options (XIOS server, netcdf 4 parallel option, NB: we will have a dedicated tutorial on XIOS) :: /* I/O server */ # undef XIOS * Non-hydrostatic option :: /* Non-hydrostatic option */ # undef NBQ * Nesting settings :: /* Nesting */ # undef AGRIF # undef AGRIF_2WAY * Coupling with other models (atmosphere, waves) :: /* OA and OW Coupling via OASIS (MPI) */ # undef OA_COUPLING # undef OW_COUPLING * Including wave-current interactions :: /* Wave-current interactions */ # undef MRL_WCI * Managing open boundaries (you can choose to close one of the boundaries, useful in coastal cases) :: /* Open Boundary Conditions */ # undef TIDES # define OBC_EAST # define OBC_WEST # define OBC_NORTH # define OBC_SOUTH * Activating applications :: /* Applications */ # undef BIOLOGY # undef FLOATS # undef STATIONS # undef PASSIVE_TRACER # undef SEDIMENT # undef BBL * Defining a dedicated log file for CROCO standard output (default is undef but you can ``define LOGFILE`` to facilitate the reading of model output, particularly useful for coupled simulations) :: /* dedicated croco.log file */ # undef LOGFILE .. warning:: Keep ``undef LOGFILE`` is you use Plurimonth run scritps as: ``run_croco_inter.bash`` because it already re-direct the CROCO output, and check it... * Time reference setting: .. warning:: By default no reference time is used, and time is referred to the beginning of the simulation only :: /* Calendar */ # undef USE_CALENDAR #. Then you have detailed settings (you can find a description of available cpp keys in :ref:`cppdefs.h `). Let's just highlight a few ones: * In grid configuration :: /* Grid configuration */ # define CURVGRID # define SPHERICAL # define MASKING # undef WET_DRY # define NEW_S_COORD .. warning:: you should check that the vertical coordinate setting ``NEW_S_COORD`` is in adequation with your pre-processing setting (``vtransform=2`` in ``crocotools_param.m``). In croco_pytools, using ``NEW_S_COORD`` is mandatory. * In surface forcing subsection: * if you have prepared ``croco_frc.nc`` file (using ``make_frc.m``) :: /* Surface Forcing */ # undef BULK_FLUX * if you have prepared ``croco_blk.nc`` file (using ``make_blk.m``) :: /* Surface Forcing */ # define BULK_FLUX * Then, you have to set your lateral forcing according to your pre-processing as well: * If you have prepared ``croco_clm.nc`` file (using ``make_clim.m``) :: /* Lateral Forcing */ # define CLIMATOLOGY and :: # undef FRC_BRY * Or, if you have prepared ``croco_bry.nc`` file (using ``make_bry.m``) :: /* Lateral Forcing */ # undef CLIMATOLOGY and :: # define FRC_BRY The other CPP-keys will be explored in other tutorials. param.h ^^^^^^^ ``param.h`` is composed of the following sections: * Dimensions of Physical Grid and array dimensions * MPI related variables * Number maximum of weights for the barotropic mode * OA-Coupling, Tides, Wetting-Drying, Point sources, Floast, Stations * Derived dimension parameters * I/O : flag for type sigma vertical transformation * Number of tracers * Tracer identification indices Most of the time you only need to check/edit the 2 first sections: #. Check the grid settings: :: # elif defined BENGUELA_LR parameter (LLm0=41, MMm0=42, N=32) ! BENGUELA_LR * **LLm0**: Dimension (ghost points included) in the :math:`\xi` direction. * **MMm0**: Dimension (ghost points included) in the :math:`\eta` direction. * **N**: Number of :math:`\rho`-vertical points, in the vertical grid. #. Check and eventually edit the parallelization settings: :: #ifdef MPI integer NP_XI, NP_ETA, NNODES parameter (NP_XI=1, NP_ETA=4, NNODES=NP_XI*NP_ETA) parameter (NPP=1) parameter (NSUB_X=1, NSUB_E=1) #elif defined OPENMP parameter (NPP=4) * In the case of OpenMP parallelization, NPP is the number of cpu used in the computation * In the case of MPI parallelization, it is equal to to NNODES. * AUTOTILING (implemented by L. Debreu): cpp-key that enable to compute the optimum subdomains partition in terms of computation time. .. note:: MPI tiles should be at least 20x20 points. Compilation using jobcomp ^^^^^^^^^^^^^^^^^^^^^^^^^ Now that your input files are set up, you can proceed to compilation. You have to copy the jobcomp script in your working directory. Here we assume that you have set a few environment variables for compilers and libraries. Here is an example with Intel compilers and a netcdf library located in $HOME/softs/netcdf. Adapt these to your own settings (in your .bashrc file): .. code-block:: bash # compilers export CC=icc export FC=ifort export F90=ifort export F77=ifort export MPIF90=mpiifort # netcdf library export NETCDF=$HOME/softs/netcdf export PATH=$NETCDF/bin:${PATH} export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${NETCDF}/lib To use the jobcomp script, you can first explore the available option through : .. code-block:: bash ./jobcomp -h Available option are : * --src : Specify the CROCO source directory (default: ../croco/OCEAN). * --fc : Specify the Fortran compiler (default: gfortran). Available = gfortran, ifort, ifc, nvfortran, pgfortran * --mpif90 : Specify the MPI Fortran compiler (default: mpif90). * --fflags : Specify the compilation flags * default: if fc=ifort or ifx : '-O2 -mcmodel=medium -fno-alias -i4 -r8 -fp-model precise' . * default: if fc=gfortran : '-O2 -mcmodel=medium -fdefault-real-8 -fdefault-double-8 -std=legacy' . * default: if fc=nvfortran or pgfortran : '-g -fast -r8 -i4 -mcmodel=medium -Mbackslash' . * --jobs : Number of processes for compilation (default: 1). * --netcdf-inc : Specify the NetCDF include directory. (default: nf-config --includedir) * --netcdf-lib : Specify the NetCDF library directory. (default: nf-config --flibs) * --xios-dir : Specify the XIOS root directory. (default: /root/xios) * --prism-dir : Specify the OASIS-MCT root directory. (default: ../../../oasis3-mct/compile_oa3-mct) If you use default options, you can compile with : .. code-block:: bash ./jobcomp > jobcomp.log Or you can for example use ifort instead of gfortran : .. code-block:: bash ./jobcomp --fc ifort --mpif90 mpiifort > jobcomp.log If compilation is successful, you should have a ``croco`` executable in your directory. You will also find a ``Compile`` directory containing the model source files: * ``.F`` files: original model source files that have been copied from ``~/croco/croco/OCEAN`` * ``_.f`` files: pre-compiled files in which only parts defined by cpp-keys are kept * ``.o`` object files Compilation options ^^^^^^^^^^^^^^^^^^^ A very summarized information on compilation options is given here. For further details, search information on the web, or with your cluster assistance team. Useful informations can also be found on this page: http://www.idris.fr/jean-zay/cpu/jean-zay-cpu-comp_options.html * Optimization options: * ``-O0``, ``-O1``, ``-O2``, ``-O3``, ``-fast`` : optimization level. ``-O0`` is no optimization, use it for debug. ``-O3`` and ``-fast`` are more agressive optimization options that can lead to problems in reproducibility of your run (especially it is better to avoid -fast). * ``-xCORE-AVX2`` : vectorization option, very agressive optimization => non-reproducibility of CROCO * ``-fno-alias``, ``-no-fma``, ``-ip`` : other optimization options, commonly used * ``-ftz``: set to 0 denormal very small numbers. It is set by default with ``-O1``, ``-O2``, ``-O3`` (can be a problem in calculation precision) * Debug options: ``-O0 -g -debug -fpe-all=0 -no-ftz -traceback -check all -fbacktrace -fbounds-check -finit-real=nan -finit-integer=8888`` * Precision and writing options: * ``-fp-model precise``: important to have good precision and reproducibility of your calculations * ``-assume byterecl``: way of writing: byte instead of bit * ``-convert big_endian``: way of writing binaries (important for avoiding huge negative numbers) * ``-i4``, -r8``: way of writing integers and reals (important also for reproducibility between different clusters) * ``-72``: specifies that the statement field of each fixed-form source line ends at column 72. * ``-mcmodel=medium -shared-intel`` : do not limit memory to 2Go for data (useful for writing large output files) Tips in case of errors during compilation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In case of strange errors during compilation (*e.g.* "catastrophic error: could not find ..."), try one of these solutions: * check your home space is not full ;-) * check your paths to compilers and libraries (especially Netcdf library) * check that you have the good permissions, and check that your executable files (configure, make...) do are executable * check that your shell scripts headers are correct or add them if necessary (*e.g.* for bash: ``#!/bin/bash``) * try to exit/log out the machine, log in back, clean and restart compilation Errors and tips related to netcdf library: * with netcdf 4.3.3.1: need to add the following compilation flag for all models: ``-mt_mpi`` The error associated to a missing ``-mt_mpi`` flag is of this type: " /opt/intel//impi/4.1.1.036/intel64/lib/libmpi_mt.so.4: could not read symbols: Bad value " * with netcdf 4.1.3: do NOT add ``-mt_mpi`` flag * with netcdf4, need to place hdf5 library path in your environment: :: export LD_LIBRARY_PATH=YOUR_HDF5_DIR/lib:$LD_LIBRARY_PATH * with netcdf 4, if you use the library splitted in 2: C part and Fortran part, you need to place links to C library before links to Fortran library and need to put both path in this same order in your LD_LIBRARY_PATH In case of 'segmentation fault' error: * try to allocate more memory with "unlimited -s unlimited" * try to launch the compilation as a job (batch) with more allocated memory