Ctrl+K

Compiling in coupled mode

19.2. Compiling in coupled mode#

Warning

You need to compile OASIS before compiling the models

Note

In case of error during compilation, refer to the “Tips in case or error during compilation” below

19.2.1. Compiling OASIS#

  1. You need to have dowloaded the OASIS sources (see Download section). They are assumed, in the following, to be under: $HOME/oasis/oasis3-mct

  2. Then, explore the oasis3-mct directory, you will find:

    • doc: oasis documentation

    • lib: mct, psmile, and scrip libraries folders

    • util: with notably make_dir folder containing TopMakefileOasis3, make.inc, and several make.* for different machines

    • examples

  3. Enter the make_dir directory

    cd ~/oasis/oasis3-mct/util/make_dir

  4. Edit the configure file for your machine make.*.

  5. Edit the make.inc file to point to your make.YOURMACHINE (examples for machines we have tested can be found here: $HOME/croco/croco/SCRIPTS/SCRIPTS_COUPLING/OASIS_IN/make.*)

    include  $(home)/oasis/oasis3-mct/util/make_dir/make.YOURMACHINE

    Warning

    Absolute paths are mandatory in make.* files!

  6. Clean your directory and launch compilation

    make realclean -f TopMakefileOasis3 > oasis_clean.out
make -f TopMakefileOasis3 > oasis_make.out

    If compilation is successful, you should find in ~/oasis/ a compile_oasis3-mct directory including:

    • lib containing libmct.a  libmpeu.a  libpsmile.MPI1.a  libscrip.a

    • build

    Note

    In case of error during compilation, note that classical errors are associated to:

    • files missing executable permission

    • issues in the paths given in make.yourmachine

    • compilation options that have to be set carefully (in make.YOURMACHINE)

19.2.2. Compiling CROCO#

Note

CROCO needs to be compiled for each configuration and parallel settings. Thus the scripts provided in the croco/SCRIPTS/SCRIPTS_COUPLING toolbox provide an option to compile CROCO “online” when the run is launched according to the configuration, coupled options and parallel settings. Here is just a roadmap on how to do it “by hand”.

  1. To work in coupled mode you need to activate OA_COUPLING and/or OW_COUPLING in cppdefs.h:

    #define OA_COUPLING
#define OW_COUPLING

    You also need to define MPI:

    #define MPI

    Warning

    MPI is mandatory for coupling, even if the run is launched on 1 CPU. Indeed the MPI communicator is used to communicate with OASIS.

  2. Edit all the usual paths, compilers, libraries in jobcomp, and notably OASIS path PRISM_ROOT_DIR:

    # set OASIS-MCT (or OASIS3) directories if needed
#
PRISM_ROOT_DIR=~/oasis/compile_oasis3-mct

    You may also eventually need to set/change compilation options.

    Warning

    -O3 compilation option is quite agressive and may result in some errors on some machines and with some compilers during coupled run (e.g. stokes velocities set to 0). To avoid such errors, set optimization to -O2.

  3. And compile:

    ./jobcomp >& compile_coupled.log

    If compilation aborts (netcdf errors in oasis functions), you may need to change the following lines to:

    LDFLAGS1="$LDFLAGS1 $LIBPSMILE $NETCDFLIB"
CPPFLAGS1="$CPPFLAGS1 ${PSMILE_INCDIR} $NETCDFINC"
FFLAGS1="$FFLAGS1 ${PSMILE_INCDIR} $NETCDFINC"

    Then try to compile again.

19.2.3. Compiling the TOY model#

A toy model is available in the croco/SCRIPTS/SCRIPTS_COUPLING/TOY_IN. It consists of a few fortran routines, that exchange variables with OASIS to mimic another model (wave, atmosphere, ocean). The toy model is compiled using a Makefile. See the readme in croco/SCRIPTS/SCRIPTS_COUPLING/TOY_IN for instructions.

  1. Copy the TOY model in your configuration directory and check/edit the Makefile.YOURMACHINE for your machine (examples for a few clusters are provided), and link it to Makefile:

    cp -r ~/croco/croco/SCRIPTS/SCRIPTS_COUPLING/TOY_IN ~/CONFIGS/BENGUELA_LR_cpl/.
cd ~/CONFIGS/BENGUELA_LR_cpl/TOY_IN
ln -sf Makefile.YOURMACHINE Makefile

  2. Then use the compilation script:

    ./make_toy_compil.sh

If the compilation is successfull you should have the TOY executables toy_wav toy_atm toy_oce

19.2.4. Compiling WRF#

Note

Currently the distributed version of WRF does not include coupling with waves, if you want to use such functionality you can use the fork including modifications for coupling with WW3 and CROCO through the OASIS coupler, but note that this is a development version… wrf-croco/WRF

WRF needs to be compiled both in forced and coupled modes.

  1. Enter WRF directory, and configure your compilation

    cd ~/wrf/WRFV4.2.1
# cleaning before configure (must be done if you re-compile)
./clean -a
# Then launch configure
./configure

    Choose distributed memory option (dm) and compiler option in adequation with your machine setup (in our case it will be #24).

    Note

    • For creating model output files larger than 2Go, you should consider using netcdf large file support function. It is activated through the WRFIO_NCD_LARGE_FILE_SUPPORT environment variable (set to 1).

    • WRF is strict on netcdf dependencies, meaning that problems during compilation are often due to netcdf settings. WRF uses:

    • NETCDF environment variable that can be set before launching configure, otherwise configure will ask you to provide your netcdf full path

    • NETCDF4 environment variable that can be set to 1 if you want to use netcdf 4 facilities (if your netcdf library allows it). When using netcdf4 library, check if all dependencies are properly set, they are usually found with nf-config --flibs command

    • always check all the lines associated to netcdf library and dependencies in the generated configure.wrf: NETCDF4_IO_OPTS, NETCDF4_DEP_LIB, INCLUDE_MODULES (last line should be netcdf inlcude path), LIB_EXTERNAL (last line should be netcdf library and its dependencies).

  2. Check and edit the generated configure.wrf file. Notably edit the parallel compiler lines

    DM_FC           =       mpiifort
DM_CC           =       mpiicc

  3. First compile in uncoupled mode

    ./compile em_real >& compile_uncoupled.log

    Note

    WRF supports using multiple processors for compilation. The default number of processors used is 2. But you can compile with more processors by using the J environment variable set (example for 8 processors: J=-j 8).

    Note

    WRF compilation will take a while (about 1h) and may take a lot of memory. You may need to launch compilation in a job. Examples for a few machines are provided here, along with a script to help you compile:

    ~/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WRF_IN/*.compile.wrf.*
~/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WRF_IN/make_WRF_compil

    If compilation is successful, you will find in WRF main directory the following executables:

    • wrf.exe

    • real.exe

    • ndown.exe

    • tc.exe

  4. Copy them to dedicated directory (as well as your configure.wrf, in case you need to recompile)

    mkdir exe_uncoupled
cp configure.wrf exe_uncoupled/.
cp main/*.exe exe_uncoupled/.
cp compile_uncoupled.log exe_uncoupled/.

  5. To compile in coupled mode, you need to edit configure.wrf, first copy it to configure.wrf.coupled

    cp configure.wrf configure.wrf.coupled

    And then edit configure.wrf.coupled

    # Just before: #### Architecture specific settings ####, add for OASIS:
OA3MCT_ROOT_DIR = $(OASISDIR)

# In: #### Architecture specific settings ####, add -Dkey_cpp_oasis3 :
ARCH_LOCAL      =       -DNONSTANDARD_SYSTEM_FUNC -DWRF_USE_CLM $(NETCDF4_IO_OPTS) -Dkey_cpp_oasis3

# In: # POSTAMBLE, add includes and libraries associated to OASIS before netcdf ones, as follows:
INCLUDE_MODULES =   $(MODULE_SRCH_FLAG) \
      $(ESMF_MOD_INC) $(ESMF_LIB_FLAGS) \
      -I$(WRF_SRC_ROOT_DIR)/main \
      -I$(WRF_SRC_ROOT_DIR)/external/io_netcdf \
      -I$(WRF_SRC_ROOT_DIR)/external/io_int \
      -I$(WRF_SRC_ROOT_DIR)/frame \
      -I$(WRF_SRC_ROOT_DIR)/share \
      -I$(WRF_SRC_ROOT_DIR)/phys \
      -I$(WRF_SRC_ROOT_DIR)/chem -I$(WRF_SRC_ROOT_DIR)/inc \
      -I$(OA3MCT_ROOT_DIR)/build/lib/mct \
      -I$(OA3MCT_ROOT_DIR)/build/lib/psmile.MPI1 \
      -I$(NETCDFPATH)/include \

   LIB_EXTERNAL    = \
                        -L$(WRF_SRC_ROOT_DIR)/external/io_netcdf -lwrfio_nf \
                        -L$(OA3MCT_ROOT_DIR)/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip \
                        -L$(NETCDF)/lib -lnetcdff -lnetcdf

    Examples of configure.wrf.uncoupled and configure.wrf.coupled are provided in croco/SCRIPTS/SCRIPTS_COUPLING/WRF_IN/CONFIGURE_WRF/.

    Warning

    Compiling WRF in coupled mode required a lot of memory (>3.5Go). If needed, submit a job with extra-memory to compile.

  6. To compile

    ./clean -a # clean before compilation
cp configure.wrf.coupled configure.wrf
./compile em_real >& compile.coupled.log

    If compilation is successful, you will find in WRF main directory the following executables:

    • wrf.exe

    • real.exe

    • ndown.exe

    • tc.exe

  7. Copy them to dedicated directory (as well as your configure.wrf, in case you need to recompile)

    mkdir exe_coupled
cp configure.wrf exe_coupled/.
cp main/*.exe exe_coupled/.
cp compile.coupled.log exe_coupled/.

Note

Using the WRF moving nest in coupled mode is possible, but only the parent static model can be coupled through OASIS. Feedback between the static parent domain and the moving nest are used to update fields computed at high-resolution in the moving nest on one hand, and coupled to the ocean or wave model in the static parent domain on the other hand. To use this functionnality, WRF has to be compiled with the moving nest option, and a dedicated Registry.EM is available in WRF_IN/FOR_MOVING_NEST to allow the moving nest to receive surface updates from the parent static domain, that is coupled to the ocean or wave model. Copy this Registry.EM in your WRF/Registry directory before compiling with the moving nest option. Warning, this Registry.EM should not be used in “normal” mode (no moving nest).

19.2.5. Compiling WPS#

Note

Note that you should use the WPS version consistent with your WRF version!

  1. Enter WPS directory, and configure your compilation

    cd ~/wrf/WPS
./clean -a
./configure

    Choose distributed memory option (dm) and compiler option in adequation with your machine setup.

  2. Check and edit configure.wps, notably WRF_DIR and compilers

    WRF_DIR                 =       ../WRF

DM_FC               = mpiifort
DM_CC               = mpiicc

  3. Compile WPS

    ./compile >& compile_wps.log

    If compilation is successful, you will find in your WPS directory

    geogrid.exe
ungrib.exe
metgrid.exe

Alternatively, a compile_wps.bash and examples of configure.wps are provided in the Coupling_tools/WRF_WPS.

19.2.6. Compiling WW3#

Warning

Currently the following compilation procedure, and coupling tools provided in croco are designed to work with the WW3 6.07.1 release plus some additional changes in a few coupled routines in WW3 which are provided in the croco/SCRIPTS/SCRIPTS_COUPLING/WW3_IN directory. See readme_ww3_version in this directory. More recent versions of WW3 contains these modifications, but as these more recent versions are currently not tagged as “releases”, we prefer to stick to the latest official release and just add the few modified routines.

  1. First copy the modified routines into WW3 6.07.1 directory:

    cp ~/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WW3_IN/modified_ftn/*.ftn  ~/ww3/model/ftn/.

  2. Go to the model bin directory to perform the compilation:

    cd ~/ww3/model/bin

    WW3 compilation requests 3 files:

    • a switch file which contains the parallelisation, and the numerical parameterization setting. These swhiches are keywords listed in a so-called switch file. Many templates are provided by institutions with a suffix switch_*. This file is used during compilation. Open one of the coupled example switch file: switch_OASOCM (for coupling with an ocean model) or switch_OASACM (for coupling with an atmospheric model)

      • For running in coupled mode, some switches are mandatory: DIST MPI COU OASIS and OASOCM (for coupling with an ocean model) and/or OASACM (for coupling with an atmospheric model)

      • Also, the switches to interpolate in time current or wind need to be set to 0 in coupled case mode (and forced cases used to compare to coupled mode): CRT0 WNT0

    • a comp.COMPILER file

    • a link.COMPILER file

    The 2 later files contain useful options and links for compilation. You therefore need to check the ones that you will use depending on you compiler and machine settings.

    In this tutorial, let’s take the example of comp.Intel and link.Intel files.

  3. You can edit the compilation options in comp.Intel, for instance:

    opt="-c $list -O3 -ip -xHost -no-fma -fp-model precise -assume byterecl -fno-alias -fno-fnalias -module $path_m"

  4. First we will compile WW3 in uncoupled mode. To do that, create an equivalent switch file than switch_OASOCM but without coupling switches:

    cp switch_OASOCM switch_UNCOUPLED

    In switch_UNCOUPLED, erase the following switches: COU OASIS OASOCM

  5. Now you are ready to setup and compile WW3:

    ./w3_setup .. -c Intel -s UNCOUPLED
./w3_automake

    If compilation is successful, you will find your executables in ../exe, you should move these executables to a dedicated directory:

    mkdir ../exe_UNCOUPLED
mv ../exe/* ../exe_UNCOUPLED/.

  6. To compile in coupled mode, check that the $OASISDIR variable correctly refers to your OASIS compile directory, and re-setup and re-launch your compilation:

    For coupling with the ocean

    ./w3_clean -c
./w3_setup .. -c Intel -s OASOCM
./w3_automake

    If compilation is successful, you should move your executable to a proper directory

    mkdir ../exe_OASOCM
mv ../exe/* ../exe_OASOCM/.

    For coupling with the atmosphere

    ./w3_clean -c
./w3_setup .. -c Intel -s OASACM
./w3_automake

    If compilation is successful, you should move your executable to a proper directory

    mkdir ../exe_OASACM
mv ../exe/* ../exe_OASACM/.

    For coupling with both the ocean and the atmosphere, first create a switch_OASOCM_OASACM

    cp switch_OASOCM switch_OASOCM_OASACM

    Edit it to have both OASOCM and OASACM switches

    F90 NOGRB NC4 TRKNC DIST MPI PR3 UQ FLX0 LN1 ST4 STAB0 NL1 BT4 DB1 MLIM TR0 BS0 IC2 IS0 REF1 XX0 WNT2 WNX1 RWND CRT0 CRX1 TIDE COU OASIS OASOCM OASACM O0 O1 O2 O2a O2b O2c O3 O4 O5 O6 O7

    And compile

    ./w3_clean -c
./w3_setup .. -c Intel -s OASOCM_OASACM
./w3_automake

    If compilation is successful, you should move your executable to a proper directory

    mkdir ../exe_OASOCM_OASACM
mv ../exe/* ../exe_OASOCM_OASACM/.

    Note

    a script to help you compile the various mode is also available in: $HOME/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WW3_IN/make_WW3_compil

19.2.7. Tips in case of errors during compilation#

In case of strange errors during compilation (e.g. “catastrophic error: could not find …”), try one of these solutions:

  • check your home space is not full ;-)

  • check your paths to compilers and libraries (especially Netcdf library)

  • check that you have the good permissions, and check that your executable files (configure, make…) do are executable

  • check that your shell scripts headers are correct or add them if necessary (e.g. for bash: #!/bin/bash)

  • try to exit/log out the machine, log in back, clean and restart compilation

Errors and tips related to netcdf library:

  • with netcdf 4.3.3.1: need to add the following compilation flag for all models: -mt_mpi The error associated to a missing -mt_mpi flag is of this type: “ /opt/intel//impi/4.1.1.036/intel64/lib/libmpi_mt.so.4: could not read symbols: Bad value “

  • with netcdf 4.1.3: do NOT add -mt_mpi flag

  • with netcdf4, need to place hdf5 library path in your environment:

export LD_LIBRARY_PATH=YOUR_HDF5_DIR/lib:$LD_LIBRARY_PATH

  • with netcdf 4, if you use the library splitted in 2: C part and Fortran part, you need to place links to C library before links to Fortran library and need to put both path in this same order in your LD_LIBRARY_PATH

In case of ‘segmentation fault’ error:

  • try to allocate more memory with “unlimited -s unlimited”

  • try to launch the compilation as a job (batch) with more allocated memory

Contents