19.2. Compiling in coupled mode#
Warning
You need to compile OASIS before compiling the models
Note
In case of error during compilation, refer to the “Tips in case or error during compilation” below
19.2.1. Compiling OASIS#
You need to have dowloaded the OASIS sources (see Download section). They are assumed, in the following, to be under:
$HOME/oasis/oasis3-mct
Then, explore the
oasis3-mct
directory, you will find:doc
: oasis documentationlib
:mct
,psmile
, andscrip
libraries foldersutil
: with notablymake_dir
folder containingTopMakefileOasis3
,make.inc
, and severalmake.*
for different machinesexamples
Enter the
make_dir
directorycd ~/oasis/oasis3-mct/util/make_dir
Edit the configure file for your machine
make.*
.Edit the
make.inc
file to point to yourmake.YOURMACHINE
(examples for machines we have tested can be found here:$HOME/croco/croco/SCRIPTS/SCRIPTS_COUPLING/OASIS_IN/make.*
)include $(home)/oasis/oasis3-mct/util/make_dir/make.YOURMACHINE
Warning
Absolute paths are mandatory in
make.*
files!Clean your directory and launch compilation
make realclean -f TopMakefileOasis3 > oasis_clean.out make -f TopMakefileOasis3 > oasis_make.out
If compilation is successful, you should find in
~/oasis/
acompile_oasis3-mct
directory including:lib
containinglibmct.a libmpeu.a libpsmile.MPI1.a libscrip.a
build
Note
In case of error during compilation, note that classical errors are associated to:
files missing executable permission
issues in the paths given in
make.yourmachine
compilation options that have to be set carefully (in
make.YOURMACHINE
)
19.2.2. Compiling CROCO#
Note
CROCO needs to be compiled for each configuration and parallel settings.
Thus the scripts provided in the croco/SCRIPTS/SCRIPTS_COUPLING
toolbox
provide an option to compile CROCO “online” when the run is launched
according to the configuration, coupled options and parallel settings.
Here is just a roadmap on how to do it “by hand”.
To work in coupled mode you need to activate
OA_COUPLING
and/orOW_COUPLING
incppdefs.h
:#define OA_COUPLING #define OW_COUPLING
You also need to define
MPI
:#define MPI
Warning
MPI
is mandatory for coupling, even if the run is launched on 1 CPU. Indeed the MPI communicator is used to communicate with OASIS.Edit all the usual paths, compilers, libraries in
jobcomp
, and notably OASIS pathPRISM_ROOT_DIR
:# set OASIS-MCT (or OASIS3) directories if needed # PRISM_ROOT_DIR=~/oasis/compile_oasis3-mct
You may also eventually need to set/change compilation options.
Warning
-O3
compilation option is quite agressive and may result in some errors on some machines and with some compilers during coupled run (e.g. stokes velocities set to 0). To avoid such errors, set optimization to-O2
.And compile:
./jobcomp >& compile_coupled.log
If compilation aborts (netcdf errors in oasis functions), you may need to change the following lines to:
LDFLAGS1="$LDFLAGS1 $LIBPSMILE $NETCDFLIB" CPPFLAGS1="$CPPFLAGS1 ${PSMILE_INCDIR} $NETCDFINC" FFLAGS1="$FFLAGS1 ${PSMILE_INCDIR} $NETCDFINC"
Then try to compile again.
19.2.3. Compiling the TOY model#
A toy model is available in the croco/SCRIPTS/SCRIPTS_COUPLING/TOY_IN
.
It consists of a few fortran routines, that exchange variables with OASIS to
mimic another model (wave, atmosphere, ocean).
The toy model is compiled using a Makefile. See the readme
in
croco/SCRIPTS/SCRIPTS_COUPLING/TOY_IN
for instructions.
Copy the TOY model in your configuration directory and check/edit the
Makefile.YOURMACHINE
for your machine (examples for a few clusters are provided), and link it toMakefile
:cp -r ~/croco/croco/SCRIPTS/SCRIPTS_COUPLING/TOY_IN ~/CONFIGS/BENGUELA_LR_cpl/. cd ~/CONFIGS/BENGUELA_LR_cpl/TOY_IN ln -sf Makefile.YOURMACHINE Makefile
Then use the compilation script:
./make_toy_compil.sh
If the compilation is successfull you should have the TOY executables
toy_wav toy_atm toy_oce
19.2.4. Compiling WRF#
Note
Currently the distributed version of WRF does not include coupling with waves, if you want to use such functionality you can use the fork including modifications for coupling with WW3 and CROCO through the OASIS coupler, but note that this is a development version… wrf-croco/WRF
WRF needs to be compiled both in forced and coupled modes.
Enter WRF directory, and configure your compilation
cd ~/wrf/WRFV4.2.1 # cleaning before configure (must be done if you re-compile) ./clean -a # Then launch configure ./configure
Choose distributed memory option (dm) and compiler option in adequation with your machine setup (in our case it will be #24).
Note
For creating model output files larger than 2Go, you should consider using netcdf large file support function. It is activated through the
WRFIO_NCD_LARGE_FILE_SUPPORT
environment variable (set to 1).WRF is strict on netcdf dependencies, meaning that problems during compilation are often due to netcdf settings. WRF uses:
NETCDF
environment variable that can be set before launching configure, otherwise configure will ask you to provide your netcdf full pathNETCDF4
environment variable that can be set to 1 if you want to use netcdf 4 facilities (if your netcdf library allows it). When using netcdf4 library, check if all dependencies are properly set, they are usually found withnf-config --flibs
commandalways check all the lines associated to netcdf library and dependencies in the generated
configure.wrf
:NETCDF4_IO_OPTS
,NETCDF4_DEP_LIB
,INCLUDE_MODULES
(last line should be netcdf inlcude path),LIB_EXTERNAL
(last line should be netcdf library and its dependencies).
Check and edit the generated
configure.wrf
file. Notably edit the parallel compiler linesDM_FC = mpiifort DM_CC = mpiicc
First compile in uncoupled mode
./compile em_real >& compile_uncoupled.log
Note
WRF supports using multiple processors for compilation. The default number of processors used is 2. But you can compile with more processors by using the
J
environment variable set (example for 8 processors:J=-j 8
).Note
WRF compilation will take a while (about 1h) and may take a lot of memory. You may need to launch compilation in a job. Examples for a few machines are provided here, along with a script to help you compile:
~/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WRF_IN/*.compile.wrf.* ~/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WRF_IN/make_WRF_compil
If compilation is successful, you will find in WRF
main
directory the following executables:wrf.exe
real.exe
ndown.exe
tc.exe
Copy them to dedicated directory (as well as your configure.wrf, in case you need to recompile)
mkdir exe_uncoupled cp configure.wrf exe_uncoupled/. cp main/*.exe exe_uncoupled/. cp compile_uncoupled.log exe_uncoupled/.
To compile in coupled mode, you need to edit
configure.wrf
, first copy it toconfigure.wrf.coupled
cp configure.wrf configure.wrf.coupled
And then edit
configure.wrf.coupled
# Just before: #### Architecture specific settings ####, add for OASIS: OA3MCT_ROOT_DIR = $(OASISDIR) # In: #### Architecture specific settings ####, add -Dkey_cpp_oasis3 : ARCH_LOCAL = -DNONSTANDARD_SYSTEM_FUNC -DWRF_USE_CLM $(NETCDF4_IO_OPTS) -Dkey_cpp_oasis3 # In: # POSTAMBLE, add includes and libraries associated to OASIS before netcdf ones, as follows: INCLUDE_MODULES = $(MODULE_SRCH_FLAG) \ $(ESMF_MOD_INC) $(ESMF_LIB_FLAGS) \ -I$(WRF_SRC_ROOT_DIR)/main \ -I$(WRF_SRC_ROOT_DIR)/external/io_netcdf \ -I$(WRF_SRC_ROOT_DIR)/external/io_int \ -I$(WRF_SRC_ROOT_DIR)/frame \ -I$(WRF_SRC_ROOT_DIR)/share \ -I$(WRF_SRC_ROOT_DIR)/phys \ -I$(WRF_SRC_ROOT_DIR)/chem -I$(WRF_SRC_ROOT_DIR)/inc \ -I$(OA3MCT_ROOT_DIR)/build/lib/mct \ -I$(OA3MCT_ROOT_DIR)/build/lib/psmile.MPI1 \ -I$(NETCDFPATH)/include \ LIB_EXTERNAL = \ -L$(WRF_SRC_ROOT_DIR)/external/io_netcdf -lwrfio_nf \ -L$(OA3MCT_ROOT_DIR)/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip \ -L$(NETCDF)/lib -lnetcdff -lnetcdf
Examples of
configure.wrf.uncoupled
andconfigure.wrf.coupled
are provided incroco/SCRIPTS/SCRIPTS_COUPLING/WRF_IN/CONFIGURE_WRF/
.Warning
Compiling WRF in coupled mode required a lot of memory (>3.5Go). If needed, submit a job with extra-memory to compile.
To compile
./clean -a # clean before compilation cp configure.wrf.coupled configure.wrf ./compile em_real >& compile.coupled.log
If compilation is successful, you will find in WRF
main
directory the following executables:wrf.exe
real.exe
ndown.exe
tc.exe
Copy them to dedicated directory (as well as your
configure.wrf
, in case you need to recompile)mkdir exe_coupled cp configure.wrf exe_coupled/. cp main/*.exe exe_coupled/. cp compile.coupled.log exe_coupled/.
Note
Using the WRF moving nest in coupled mode is possible, but only the parent
static model can be coupled through OASIS. Feedback between the static
parent domain and the moving nest are used to update fields computed at
high-resolution in the moving nest on one hand, and coupled to the ocean or
wave model in the static parent domain on the other hand. To use this
functionnality, WRF has to be compiled with the moving nest option, and a
dedicated Registry.EM
is available in WRF_IN/FOR_MOVING_NEST
to
allow the moving nest to receive surface updates from the parent static
domain, that is coupled to the ocean or wave model. Copy this Registry.EM
in your WRF/Registry
directory before compiling with the moving nest
option. Warning, this Registry.EM
should not be used in “normal” mode
(no moving nest).
19.2.5. Compiling WPS#
Note
Note that you should use the WPS version consistent with your WRF version!
Enter WPS directory, and configure your compilation
cd ~/wrf/WPS ./clean -a ./configure
Choose distributed memory option (dm) and compiler option in adequation with your machine setup.
Check and edit
configure.wps
, notablyWRF_DIR
and compilersWRF_DIR = ../WRF DM_FC = mpiifort DM_CC = mpiicc
Compile WPS
./compile >& compile_wps.log
If compilation is successful, you will find in your WPS directory
geogrid.exe ungrib.exe metgrid.exe
Alternatively, a compile_wps.bash
and examples of configure.wps
are
provided in the Coupling_tools/WRF_WPS
.
19.2.6. Compiling WW3#
Warning
Currently the following compilation procedure, and coupling tools provided
in croco are designed to work with the WW3 6.07.1 release plus some
additional changes in a few coupled routines in WW3 which are provided in
the croco/SCRIPTS/SCRIPTS_COUPLING/WW3_IN
directory. See
readme_ww3_version
in this directory. More recent versions of WW3
contains these modifications, but as these more recent versions are
currently not tagged as “releases”, we prefer to stick to the latest
official release and just add the few modified routines.
First copy the modified routines into WW3 6.07.1 directory:
cp ~/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WW3_IN/modified_ftn/*.ftn ~/ww3/model/ftn/.
Go to the model bin directory to perform the compilation:
cd ~/ww3/model/bin
WW3 compilation requests 3 files:
a switch file which contains the parallelisation, and the numerical parameterization setting. These swhiches are keywords listed in a so-called switch file. Many templates are provided by institutions with a suffix
switch_*
. This file is used during compilation. Open one of the coupled example switch file:switch_OASOCM
(for coupling with an ocean model) orswitch_OASACM
(for coupling with an atmospheric model)For running in coupled mode, some switches are mandatory:
DIST MPI COU OASIS
andOASOCM
(for coupling with an ocean model) and/orOASACM
(for coupling with an atmospheric model)Also, the switches to interpolate in time current or wind need to be set to 0 in coupled case mode (and forced cases used to compare to coupled mode):
CRT0
WNT0
a comp.COMPILER file
a link.COMPILER file
The 2 later files contain useful options and links for compilation. You therefore need to check the ones that you will use depending on you compiler and machine settings.
In this tutorial, let’s take the example of
comp.Intel
andlink.Intel
files.You can edit the compilation options in
comp.Intel
, for instance:opt="-c $list -O3 -ip -xHost -no-fma -fp-model precise -assume byterecl -fno-alias -fno-fnalias -module $path_m"
First we will compile WW3 in uncoupled mode. To do that, create an equivalent switch file than
switch_OASOCM
but without coupling switches:cp switch_OASOCM switch_UNCOUPLED
In
switch_UNCOUPLED
, erase the following switches:COU
OASIS
OASOCM
Now you are ready to setup and compile WW3:
./w3_setup .. -c Intel -s UNCOUPLED ./w3_automake
If compilation is successful, you will find your executables in
../exe
, you should move these executables to a dedicated directory:mkdir ../exe_UNCOUPLED mv ../exe/* ../exe_UNCOUPLED/.
To compile in coupled mode, check that the
$OASISDIR
variable correctly refers to your OASIS compile directory, and re-setup and re-launch your compilation:For coupling with the ocean
./w3_clean -c ./w3_setup .. -c Intel -s OASOCM ./w3_automake
If compilation is successful, you should move your executable to a proper directory
mkdir ../exe_OASOCM mv ../exe/* ../exe_OASOCM/.
For coupling with the atmosphere
./w3_clean -c ./w3_setup .. -c Intel -s OASACM ./w3_automake
If compilation is successful, you should move your executable to a proper directory
mkdir ../exe_OASACM mv ../exe/* ../exe_OASACM/.
For coupling with both the ocean and the atmosphere, first create a
switch_OASOCM_OASACM
cp switch_OASOCM switch_OASOCM_OASACM
Edit it to have both OASOCM and OASACM switches
F90 NOGRB NC4 TRKNC DIST MPI PR3 UQ FLX0 LN1 ST4 STAB0 NL1 BT4 DB1 MLIM TR0 BS0 IC2 IS0 REF1 XX0 WNT2 WNX1 RWND CRT0 CRX1 TIDE COU OASIS OASOCM OASACM O0 O1 O2 O2a O2b O2c O3 O4 O5 O6 O7
And compile
./w3_clean -c ./w3_setup .. -c Intel -s OASOCM_OASACM ./w3_automake
If compilation is successful, you should move your executable to a proper directory
mkdir ../exe_OASOCM_OASACM mv ../exe/* ../exe_OASOCM_OASACM/.
Note
a script to help you compile the various mode is also available in:
$HOME/croco/croco/SCRIPTS/SCRIPTS_COUPLING/WW3_IN/make_WW3_compil
19.2.7. Tips in case of errors during compilation#
In case of strange errors during compilation (e.g. “catastrophic error: could not find …”), try one of these solutions:
check your home space is not full ;-)
check your paths to compilers and libraries (especially Netcdf library)
check that you have the good permissions, and check that your executable files (configure, make…) do are executable
check that your shell scripts headers are correct or add them if necessary (e.g. for bash:
#!/bin/bash
)try to exit/log out the machine, log in back, clean and restart compilation
Errors and tips related to netcdf library:
with netcdf 4.3.3.1: need to add the following compilation flag for all models:
-mt_mpi
The error associated to a missing-mt_mpi
flag is of this type: “ /opt/intel//impi/4.1.1.036/intel64/lib/libmpi_mt.so.4: could not read symbols: Bad value “with netcdf 4.1.3: do NOT add
-mt_mpi
flagwith netcdf4, need to place hdf5 library path in your environment:
export LD_LIBRARY_PATH=YOUR_HDF5_DIR/lib:$LD_LIBRARY_PATH
with netcdf 4, if you use the library splitted in 2: C part and Fortran part, you need to place links to C library before links to Fortran library and need to put both path in this same order in your LD_LIBRARY_PATH
In case of ‘segmentation fault’ error:
try to allocate more memory with “unlimited -s unlimited”
try to launch the compilation as a job (batch) with more allocated memory