XIOS
=========================

**As a start point for this tutorial, we will use the BASIN test case (see section 5.1)**

::

    cd ~/CONFIGS/BASIN

Is everything ok ? Compiling ? Running ? Are the 2 files basin_rst.nc and basin_his.nc created ?

**What is the walltime (or real time)?**

**Now add the XIOS functionnality in the croco executable:**

If the XIOS is installed on your target machine (it is the case on Datarmor), 
there are only 2 new simple steps to follow :

#. Edit ``cppdef.h``:

   Need to define 2 news cpp keys fot this test case:
   ::
 
        /*
        !                       Basin Example
        !                       ===== =======
        */
        # define XIOS
        # undef  OPENMP
        .....


#. Edit the compilation script ``jobcomp``:

   Need to add the XIOS library path
   ::

        #
        # set XIOS directory if needed
        #
        XIOS_ROOT_DIR=$HOME/xios-2.5
        #

   For this tutorial, we need to comment three lines (217, 218 and 219) in ``jobcomp``:
   ::


        #        $CPP1 -P -traditional -imacros cppdefs.h  ${ROOT_DIR}/XIOS/field_def.xml_full $RUNDIR/field_def.xml
        #        $CPP1 -P -traditional -imacros cppdefs.h  ${ROOT_DIR}/XIOS/domain_def.xml $RUNDIR/domain_def.xml
        #        $CPP1 -P -traditional -imacros cppdefs.h  ${ROOT_DIR}/XIOS/iodef.xml $RUNDIR/iodef.xml

   For this tutorial, we need to modify the routine ``send_xios_diags.F`` :
   ::

        cp ~/croco/croco/XIOS/send_xios_diags.F .

   Edit send_xios_diags.F and comment lines 2077, 2078 and 2133:
   ::

        !      call xios_send_field("uwnd",uwnd)
        !      call xios_send_field("vwnd",vwnd)

        !      call xios_send_field("bvf",bvf)

   Compile the model again:

   ::
        
        ./jobcomp

**Before running the model  with XIOS module, we need three xml files 
(field_def.xml, domain_def.xml and iodef.xml):**

::

    cp /home/datawork-croco/datarmor-only/CONFIGS/TUTO20/BASIN_WITH_XIOS/field_def.xml .
    cp /home/datawork-croco/datarmor-only/CONFIGS/TUTO20/BASIN_WITH_XIOS/domain_def.xml .
    cp /home/datawork-croco/datarmor-only/CONFIGS/TUTO20/BASIN_WITH_XIOS/iodef.xml.OneFile iodef.xml

Have a look at iodef.xml file :

* selected fields to output,
* output frequency ``output_freq``,
* what kind of output (instantaneous, average) ``operation``,
* ...

At the end of the iodef.xml file, look at the line
::

    <variable id="using_server" type="bool">false</variable>

The boolean ``false`` means that croco will run with XIOS in "attached mode". 
**Each computing processor will write in the output file**. 
In this "attached mode", XIOS behaves like a netcdf4 layer.

In this ``iodef.xml`` file, the configuration for outputs is the same as in croco.in file.

**Run the model in "attached mode":**

::

    qsub job_croco_mpi.pbs

Compare the new file ``Basin_Example_10d_inst_0001-01-01-0001-04-30.nc`` with the previous one ``basin_his.nc`` :

::

    ncview basin_his.nc & ; ncview Basin_Example_10d_inst_0001-01-01-0001-04-30.nc

.. note:: If your output file start with a ``?``, it is due to a tab before the configuration title in croco.in: ``Basin Example``. Just replace the tab by a blank space. 

--> For large configuration, XIOS is very efficient in netcdf parallel writting.

**Edit iodef.xml file and add new 2D and 3D fields to be written in the output file by uncommenting lines :**
::

    <field field_ref="w" name="w" />
    <field field_ref="salt" name="salt" />
    <field field_ref="sustr" name="sustr" />
    <field field_ref="svstr" name="svstr" />
    <field field_ref="rho" name="rho" />

Run the model:
::

    qsub job_croco_mpi.pbs

Have a look at the new file ``Basin_Example_10d_inst_0001-01-01-0001-04-30.nc``


**Add an extra file for average output in editing iodef.xml (or you can get an example there):**
::

    cp /home/datawork-croco/datarmor-only/CONFIGS/TUTO20/BASIN_WITH_XIOS/iodef.xml.Twofiles iodef.xml

Have a look at the iodef.xml file to understand how to simply add a new output file

run the model:
::

    qsub job_croco_mpi.pbs

Have a look at the new netcdf file ``Basin_Example_5d_aver_0001-01-01-0001-04-30.nc``

What is the walltime (or real time)?


**Run the model in "detached mode":**

**Edit iodef.xml and modify boolean at ``true`` in line:**
::

    <variable id="using_server" type="bool">true</variable>

The boolean ``true`` means that croco will run with XIOS in "detached mode". **Each computing processor will send fields to one or several XIOS servers which will be in charge of writing the outputs files**.

**Edit job_croco_mpi.pbs to add one XIOS server**
::

    ##PBS -l select=1:ncpus=28:mpiprocs=4:mem=8g
    #PBS -l select=1:ncpus=28:mpiprocs=5:mem=8g
    
    #time $MPI_LAUNCH croco croco.in >& croco.out
    time $MPI_LAUNCH -n 4 croco croco.in : -n 1 xios_server.exe  >& croco.out

There will be 4 computing processors sending fields to 1 xios server writting in output files.

Run the model:
::

    qsub job_croco_mpi.pbs

Theorically, computing processors will run faster (**keep in mind that reading and writting files is slow, computing is fast!**).

What is the walltime (or real time)?

Is it worth to use detached mode in this case?

**Adding an online dignostic using ONLY xios:**

In the output file, we need to have a new variable computed from already defined variables. For instance, we want to have zeta*zeta ...

Edit ``field_def.xml`` and add the new variable zeta2:
::

    <field id="zeta" long_name="free-surface" unit="meter" />
    <field id="zeta2" long_name="squared free-surface" unit="meter2" > (zeta*zeta) </field>

Then edit ``iodef.xml`` and add the new variable to be written in the output file:
::

    <field_group id="inst_fields" operation="instant">
    <field field_ref="zeta" name="zeta" />
    <field field_ref="zeta2" name="zeta2" />

No need to compile, just run the model:
::

    qsub job_croco_mpi.pbs


**If you have time, add xios in the previous BENGUELA_LR**

::

    cd $confs/Run_BENGUELA_LR
    cp /home/datawork-croco/datarmor-only/CONFIGS/TUTO20/BENGUELA_LR_XIOS/* .

Compile once:
::

    ./jobcomp

Run :
::

    qsub job_croco_mpi.pbs

Explore files, edit and modify iodef.xml, and run again ...