22. XIOS

As a start point for this tutorial, we will use the BASIN test case (see section 5.1)

cd ~/CONFIGS/BASIN

Is everything ok ? Compiling ? Running ? Are the 2 files basin_rst.nc and basin_his.nc created ?

What is the walltime (or real time)?

Now add the XIOS functionnality in the croco executable:

If the XIOS is installed on your target machine (it is the case on Datarmor), there are only 2 new simple steps to follow :

1. Edit cppdef.h:

   Need to define 2 news cpp keys fot this test case:

   /*
   !                       Basin Example
   !                       ===== =======
   */
   # define XIOS
   # undef  OPENMP
   .....
   

2. Edit the compilation script jobcomp:

   Need to add the XIOS library path

   #
   # set XIOS directory if needed
   #
   XIOS_ROOT_DIR=$HOME/xios-2.5
   #
   

   For this tutorial, we need to comment three lines (217, 218 and 219) in jobcomp:

   #        $CPP1 -P -traditional -imacros cppdefs.h  ${ROOT_DIR}/XIOS/field_def.xml_full $RUNDIR/field_def.xml
   #        $CPP1 -P -traditional -imacros cppdefs.h  ${ROOT_DIR}/XIOS/domain_def.xml $RUNDIR/domain_def.xml
   #        $CPP1 -P -traditional -imacros cppdefs.h  ${ROOT_DIR}/XIOS/iodef.xml $RUNDIR/iodef.xml
   

   For this tutorial, we need to modify the routine send_xios_diags.F :

   cp ~/croco/croco/XIOS/send_xios_diags.F .
   

   Edit send_xios_diags.F and comment lines 2077, 2078 and 2133:

   !      call xios_send_field("uwnd",uwnd)
   !      call xios_send_field("vwnd",vwnd)
   
   !      call xios_send_field("bvf",bvf)
   

   Compile the model again:

   ./jobcomp
   

Before running the model with XIOS module, we need three xml files (field_def.xml, domain_def.xml and iodef.xml) :

cp /home/datawork-croco/datarmor-only/CONFIGS/TUTO20/BASIN_WITH_XIOS/field_def.xml .
cp /home/datawork-croco/datarmor-only/CONFIGS/TUTO20/BASIN_WITH_XIOS/domain_def.xml .
cp /home/datawork-croco/datarmor-only/CONFIGS/TUTO20/BASIN_WITH_XIOS/iodef.xml.OneFile iodef.xml

Have a look at iodef.xml file :

• selected fields to output,

• output frequency output_freq,

• what kind of output (instantaneous, average) operation,

• …

At the end of the iodef.xml file, look at the line

<variable id="using_server" type="bool">false</variable>

The boolean false means that croco will run with XIOS in “attached mode”. Each computing processor will write in the output file. In this “attached mode”, XIOS behaves like a netcdf4 layer.

In this iodef.xml file, the configuration for outputs is the same as in croco.in file.

Run the model in “attached mode”:

qsub job_croco_mpi.pbs

Compare the new file Basin_Example_10d_inst_0001-01-01-0001-04-30.nc with the previous one basin_his.nc :

ncview basin_his.nc & ; ncview Basin_Example_10d_inst_0001-01-01-0001-04-30.nc

Note

If your output file start with a ?, it is due to a tab before the configuration title in croco.in: Basin Example. Just replace the tab by a blank space.

—> For large configuration, XIOS is very efficient in netcdf parallel writting.

Edit iodef.xml file and add new 2D and 3D fields to be written in the output file by uncommenting lines :

<field field_ref="w" name="w" />
<field field_ref="salt" name="salt" />
<field field_ref="sustr" name="sustr" />
<field field_ref="svstr" name="svstr" />
<field field_ref="rho" name="rho" />

Run the model:

qsub job_croco_mpi.pbs

Have a look at the new file Basin_Example_10d_inst_0001-01-01-0001-04-30.nc

Add an extra file for average output in editing iodef.xml (or you can get an example there):

cp /home/datawork-croco/datarmor-only/CONFIGS/TUTO20/BASIN_WITH_XIOS/iodef.xml.Twofiles iodef.xml

Have a look at the iodef.xml file to understand how to simply add a new output file

run the model:

qsub job_croco_mpi.pbs

Have a look at the new netcdf file Basin_Example_5d_aver_0001-01-01-0001-04-30.nc

What is the walltime (or real time)?

Run the model in “detached mode”:

edit iodef.xml and modify boolean at ``true`` in line:

<variable id="using_server" type="bool">true</variable>

The boolean true means that croco will run with XIOS in “detached mode”. Each computing processor will send fields to one or several XIOS servers which will be in charge of writing the outputs files.

Edit job_croco_mpi.pbs to add one XIOS server

##PBS -l select=1:ncpus=28:mpiprocs=4:mem=8g
#PBS -l select=1:ncpus=28:mpiprocs=5:mem=8g

#time $MPI_LAUNCH croco croco.in >& croco.out
time $MPI_LAUNCH -n 4 croco croco.in : -n 1 xios_server.exe  >& croco.out

There will be 4 computing processors sending fields to 1 xios server writting in output files.

Run the model:

qsub job_croco_mpi.pbs

Theorically, computing processors will run faster (keep in mind that reading and writting files is slow, computing is fast!).

What is the walltime (or real time)?

Is it worth to use detached mode in this case?

Adding an online dignostic using ONLY xios:

In the output file, we need to have a new variable computed from already defined variables. For instance, we want to have zeta*zeta …

Edit field_def.xml and add the new variable zeta2:

<field id="zeta" long_name="free-surface" unit="meter" />
<field id="zeta2" long_name="squared free-surface" unit="meter2" > (zeta*zeta) </field>

Then edit iodef.xml and add the new variable to be written in the output file:

<field_group id="inst_fields" operation="instant">
<field field_ref="zeta" name="zeta" />
<field field_ref="zeta2" name="zeta2" />

No need to compile, just run the model:

qsub job_croco_mpi.pbs

If you have time, add xios in the previous BENGUELA_LR

cd $confs/Run_BENGUELA_LR
cp /home/datawork-croco/datarmor-only/CONFIGS/TUTO20/BENGUELA_LR_XIOS/* .

Compile once:

./jopbcomp

Run :

qsub job_croco_mpi.pbs

Explore files, edit and modify iodef.xml, and run again …