18.2. Compiling in coupled mode

Warning

You need to compile OASIS before compiling the models

Note

In case of error during compilation, refer to the “Tips in case or error during compilation” below

18.2.1. Compiling OASIS

  1. You need to have dowloaded the OASIS sources (see Download section). They are assumed, in the following, to be under: $HOME/oasis/oasis3-mct

  2. Then, explore the oasis3-mct directory, you will find:

    • doc: oasis documentation

    • lib: mct, psmile, and scrip libraries folders

    • util: with notably make_dir folder containing TopMakefileOasis3, make.inc, and several make.* for different machines

    • examples

  3. Enter the make_dir directory:

    cd ~/oasis/oasis3-mct/util/make_dir

  4. Edit the configure file for your machine make.*.

  5. Edit the make.inc file to point to your make.yourmachine (examples for machines we have tested can be found here: $HOME/croco/croco/SCRIPTS/SCRIPTS_COUPLING/OASIS_IN/make.*)

    include  $(home)/oasis/oasis3-mct/util/make_dir/make.YOURMACHINE

    Warning

    Absolute paths are mandatory in make.* files!

  6. Clean your directory and launch compilation:

    make realclean -f TopMakefileOasis3 > oasis_clean.out
make -f TopMakefileOasis3 > oasis_make.out

    If compilation is successful, you should find in ~/oasis/ a compile_oasis3-mct directory including:

    • lib containing libmct.a  libmpeu.a  libpsmile.MPI1.a  libscrip.a

    • build

    Note

    In case of error during compilation, note that classical errors are associated to:

    • files missing executable permission

    • issues in the paths given in make.yourmachine

    • compilation options that have to be set carefully (in make.YOURMACHINE)

18.2.2. Compiling CROCO

  1. To work in coupled mode you need to activate OA_COUPLING and/or OW_COUPLING in cppdefs.h:

    #define OA_COUPLING
#define OW_COUPLING

    You also need to define MPI:

    #define MPI

    Warning

    MPI is mandatory for coupling, even if the run is launched on 1 CPU. Indeed the MPI communicator is used to communicate with OASIS.

  2. Edit all the usual paths, compilers, libraries in jobcomp, and notably OASIS path PRISM_ROOT_DIR:

    # set OASIS-MCT (or OASIS3) directories if needed
#
PRISM_ROOT_DIR=~/oasis/compile_oasis3-mct

    You may also eventually need to set/change compilation options.

    Warning

    -O3 compilation option is quite agressive and may result in some errors on some machines and with some compilers during coupled run (e.g. stokes velocities set to 0). To avoid such errors, set optimization to -02.

  3. And compile:

    ./jobcomp >& compile_coupled.log

    If compilation aborts (netcdf errors in oasis functions), you may need to change the following lines to:

    LDFLAGS1="$LDFLAGS1 $LIBPSMILE $NETCDFLIB"
CPPFLAGS1="$CPPFLAGS1 ${PSMILE_INCDIR} $NETCDFINC"
FFLAGS1="$FFLAGS1 ${PSMILE_INCDIR} $NETCDFINC"

    Then try to compile again.

18.2.3. Compiling the TOY model

A toy model is available in the croco/SCRIPTS/SCRIPTS_COUPLING/TOY_IN. It consists of a few fortran routines, that exchange variables with OASIS to mimic a wave or atmosphere model. The toy model is compiled using a Makefile. See the readme in croco/SCRIPTS/SCRIPTS_COUPLING/TOY_IN for instructions.

  1. Copy the TOY model in your configuration directory and check/edit the Makefile.YOURMACHINE for your machine (examples for a few clusters are provided), and link it to Makefile:

    cp -r ~/croco/croco/SCRIPTS/SCRIPTS_COUPLING/TOY_IN ~/CONFIGS/BENGUELA_LR_cpl/.
cd ~/CONFIGS/BENGUELA_LR_cpl/TOY_IN
ln -sf Makefile.YOURMACHINE Makefile

  2. Then clean and compile:

    make clean &> toy_clean.out
make &> toy_make.out

If the compilation is successfull you should have the TOY executable toy_model

18.2.4. Compiling WRF

Note

Currently the distributed version of WRF does not include coupling with waves, if you want to use such functionality you can use the fork including modifications for coupling with WW3 and CROCO through the OASIS coupler, but note that this is a development version… https://github.com/wrf-croco/WRF/tree/WRF-CROCO

WRF needs to be compiled both in forced and coupled modes.

  1. Enter WRF directory, and configure your compilation:

    cd ~/wrf/WRFV4.2.1
# cleaning before configure (must be done if you re-compile)
./clean -a
# Then launch configure
./configure

    Choose distributed memory option (dm) and compiler option in adequation with your machine setup (in our case it will be #24).

    Note

    • For creating model output files larger than 2Go, you should consider using netcdf large file support function. It is activated through the WRFIO_NCD_LARGE_FILE_SUPPORT environment variable (set to 1).

    • WRF is strict on netcdf dependencies, meaning that problems during compilation are often due to netcdf settings. WRF uses:

      • NETCDF environment variable that can be set before launching configure, otherwise configure will ask you to provide your netcdf full path

      • NETCDF4 environment variable that can be set to 1 if you want to use netcdf 4 facilities (if your netcdf library allows it). When using netcdf4 library, check if all dependencies are properly set, they are usually found with nf- config --flibs command

      • always check all the lines associated to netcdf library and dependencies in the generated configure.wrf: NETCDF4_IO_OPTS, NETCDF4_DEP_LIB, INCLUDE_MODULES (last line should be netcdf inlcude path), LIB_EXTERNAL (last line should be netcdf library and its dependencies).

  2. Check and edit the generated configure.wrf file. Notably edit the parallel compiler lines:

    DM_FC           =       mpiifort
DM_CC           =       mpiicc

  3. First compile in uncoupled mode:

    ./compile em_real >& compile_uncoupled.log

    Note

    WRF supports using multiple processors for compilation. The default number of processors used is 2. But you can compile with more processors by using the J environment variable set (example for 8 processors: J=-j 8).

    Note

    WRF compilation will take a while (about 1h) and may take a lot of memory. You may need to launch compilation in a job. Examples for a few machines are provided here, along with a script to help you compile:

    ~/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WRF_IN/*.compile.wrf.*
~/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WRF_IN/make_WRF_compil

    If compilation is successful, you will find in main the following executables:

    wrf.exe real.exe ndown.exe tc.exe

  4. Copy them to dedicated directory (as well as your configure.wrf, in case you need to recompile):

    mkdir exe_uncoupled
cp configure.wrf exe_uncoupled/.
cp main/*.exe exe_uncoupled/.
cp compile_uncoupled.log exe_uncoupled/.

  5. To compile in coupled mode, you need to edit configure.wrf, first copy it to configure.wrf.coupled:

    cp configure.wrf configure.wrf.coupled

    And then edit configure.wrf.coupled:

    # Just before: #### Architecture specific settings ####, add for OASIS:
OA3MCT_ROOT_DIR = $(OASISDIR)

# In: #### Architecture specific settings ####, add -Dkey_cpp_oasis3 :
ARCH_LOCAL      =       -DNONSTANDARD_SYSTEM_FUNC -DWRF_USE_CLM $(NETCDF4_IO_OPTS) -Dkey_cpp_oasis3

# In: # POSTAMBLE, add includes and libraries associated to OASIS before netcdf ones, as follows:
INCLUDE_MODULES =       $(MODULE_SRCH_FLAG) \
                        $(ESMF_MOD_INC) $(ESMF_LIB_FLAGS) \
                        -I$(WRF_SRC_ROOT_DIR)/main \
                        -I$(WRF_SRC_ROOT_DIR)/external/io_netcdf \
                        -I$(WRF_SRC_ROOT_DIR)/external/io_int \
                        -I$(WRF_SRC_ROOT_DIR)/frame \
                        -I$(WRF_SRC_ROOT_DIR)/share \
                        -I$(WRF_SRC_ROOT_DIR)/phys \
                        -I$(WRF_SRC_ROOT_DIR)/chem -I$(WRF_SRC_ROOT_DIR)/inc \
                        -I$(OA3MCT_ROOT_DIR)/build/lib/mct \
                        -I$(OA3MCT_ROOT_DIR)/build/lib/psmile.MPI1 \
                        -I$(NETCDFPATH)/include \

 LIB_EXTERNAL    = \
                      -L$(WRF_SRC_ROOT_DIR)/external/io_netcdf -lwrfio_nf \
                      -L$(OA3MCT_ROOT_DIR)/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip \
                      -L$(NETCDF)/lib -lnetcdff -lnetcdf

    Examples of configure.wrf.uncoupled and configure.wrf.coupled are provided in $HOME/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WRF_IN/CONFIGURE_WRF/.

    Warning

    Compiling WRF in coupled mode required a lot of memory (>3.5Go). If needed, submit a job with extra-memory to compile.

  6. To compile:

    ./clean -a # clean before compilation
cp configure.wrf.coupled configure.wrf
./compile em_real >& compile.coupled.log

    If compilation is successful, you will find in main the following executables:

    wrf.exe real.exe ndown.exe tc.exe

  7. Copy them to dedicated directory (as well as your configure.wrf, in case you need to recompile):

    mkdir exe_coupled
cp configure.wrf exe_coupled/.
cp main/*.exe exe_coupled/.
cp compile.coupled.log exe_coupled/.

18.2.4.1. WPS compilation

  1. Enter WPS directory, and configure your compilation:

    cd ~/wrf/WPS # note that you should use the WPS version consistent with your WRF version!
./clean -a
./configure

    Choose distributed memory option (dm) and compiler option in adequation with your machine setup.

  2. Check and edit configure.wps, notably WRF_DIR and compilers:

    WRF_DIR                 =       ../WRF

DM_FC               = mpiifort
DM_CC               = mpiicc

  3. Compile WPS:

    ./compile >& compile_wps.log

    If compilation is successful, you will find in your WPS directory:

    geogrid.exe
ungrib.exe
metgrid.exe

Alternatively, a compile_wps.bash and examples of configure.wps are provided in the Coupling_tools/WRF_WPS.

18.2.5. Compiling WW3

  1. Go to the model bin directory to perform the compilation:

    cd ~/ww3/model/bin

    WW3 compilation requests 3 files:

    • a switch file which contains the parallelisation, and the numerical parameterization setting. These swhiches are keywords listed in a so-called switch file. Many templates are provided by institutions with a suffix switch_*. This file is used during compilation. Open one of the coupled example switch file: switch_OASOCM (for coupling with an ocean model) or switch_OASACM (for coupling with an atmospheric model)

      • For running in coupled mode, some switches are mandatory:

        DIST MPI COU OASIS and OASOCM (for coupling with an ocean model) and/or OASACM (for coupling with an atmospheric model)

      • Also, the switches to interpolate in time current or wind need to be set to 0 in coupled case mode (and forced cases used to compare to coupled mode):

        CRT0 WNT0

    • a comp.COMPILER file

    • a link.COMPILER file

    The 2 later files contain useful options and links for compilation. You therefore need to check the ones that you will use depending on you compiler and machine settings.

    In this tutorial, let’s take the example of comp.Intel and link.Intel files.

  2. You can edit the compilation options in comp.Intel, for instance:

    opt="-c $list -O3 -ip -xHost -no-fma -fp-model precise -assume byterecl -fno-alias -fno-fnalias -module $path_m"

  3. First we will compile WW3 in uncoupled mode. To do that, create an equivalent switch file than switch_OASOCM but without coupling switches:

    cp switch_OASOCM switch_UNCOUPLED

    In switch_UNCOUPLED, erase the following switches: COU OASIS OASOCM

  4. Now you are ready to setup and compile WW3:

    ./w3_setup .. -c Intel -s UNCOUPLED
./w3_automake

    If compilation is successful, you will find your executables in ../exe, you should move these executables to a dedicated directory:

    mkdir ../exe_UNCOUPLED
mv ../exe/* ../exe_UNCOUPLED/.

  5. To compile in coupled mode, check that the $OASISDIR variable correctly refers to your OASIS compile directory, and re-setup and re-launch your compilation:

    For coupling with the ocean:

    ./w3_clean -c
./w3_setup .. -c Intel -s OASOCM
./w3_automake

    If compilation is successful, you should move your executable to a proper directory:

    mkdir ../exe_OASOCM
mv ../exe/* ../exe_OASOCM/.

    For coupling with the atmosphere:

    ./w3_clean -c
./w3_setup .. -c Intel -s OASACM
./w3_automake

    If compilation is successful, you should move your executable to a proper directory:

    mkdir ../exe_OASACM
mv ../exe/* ../exe_OASACM/.

    For coupling with both the ocean and the atmosphere, first create a switch_OASOCM_OASACM:

    cp switch_OASOCM switch_OASOCM_OASACM

    Edit it to have both OASOCM and OASACM switches:

    F90 NOGRB NC4 TRKNC DIST MPI PR3 UQ FLX0 LN1 ST4 STAB0 NL1 BT4 DB1 MLIM TR0 BS0 IC2 IS0 REF1 XX0 WNT2 WNX1 RWND CRT0 CRX1 TIDE COU OASIS OASOCM OASACM O0 O1 O2 O2a O2b O2c O3 O4 O5 O6 O7

    And compile:

    ./w3_clean -c
./w3_setup .. -c Intel -s OASOCM_OASACM
./w3_automake

    If compilation is successful, you should move your executable to a proper directory:

    mkdir ../exe_OASOCM_OASACM
mv ../exe/* ../exe_OASOCM_OASACM/.

    Note

    a script to help you compile the various mode is also available in: $HOME/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WW3_IN/make_WW3_compil

18.2.6. Tips in case of errors during compilation

In case of strange errors during compilation (e.g. “catastrophic error: could not find …”), try one of these solutions:

  • check your home space is not full ;-)

  • check your paths to compilers and libraries (especially Netcdf library)

  • check that you have the good permissions, and check that your executable files (configure, make…) do are executable

  • check that your shell scripts headers are correct or add them if necessary (e.g. for bash: #!/bin/bash)

  • try to exit/log out the machine, log in back, clean and restart compilation

Errors and tips related to netcdf library:

  • with netcdf 4.3.3.1: need to add the following compilation flag for all models: -mt_mpi

    The error associated to a missing -mt_mpi flag is of this type: “ /opt/intel//impi/4.1.1.036/intel64/lib/libmpi_mt.so.4: could not read symbols: Bad value “

  • with netcdf 4.1.3: do NOT add -mt_mpi flag

  • with netcdf4, need to place hdf5 library path in your environment:

export LD_LIBRARY_PATH=YOUR_HDF5_DIR/lib:$LD_LIBRARY_PATH

  • with netcdf 4, if you use the library splitted in 2: C part and Fortran part, you need to place links to C library before links to Fortran library and need to put both path in this same order in your LD_LIBRARY_PATH

In case of ‘segmentation fault’ error:

  • try to allocate more memory with “unlimited -s unlimited”

  • try to launch the compilation as a job (batch) with more allocated memory