16.1. Test Script#
Note
This section describes how to use the script for testing the CROCO code on different configurations and hosts.
16.1.1. Introduction#
This script allows executing and evaluating the performance of the CROCO code on various setups:
Execution modes: Sequential, MPI, OpenMP, debug or standard optimizations
Hosts: Runs on different architectures
Tests performed:
Checking parallel reproducibility
Checking restartability
Non-regression testing by comparing with a reference version
Performance analysis (scalability, execution time)
Visualization of physical results
It uses the concepts of :
cases : The cases to run (BASIN, SOLITON…) and some case size changing the grid size (eg. BASIN-LARGE, BASIN-HUGE).
variants : The parallelization mode (openmp, mpi, gpu) with specification on the resources to use because CROCO needs to be rebuilt for every case du to some currently static definitions in the CROCO code on the use of those resources.
hosts : The specific adaptation to make for the host you are running on.
All the options are passed through json files wholly configurable.
The benchmarking script is used to compare CROCO on various tests configurations including the ones listed at the beginning of cppdefs.h file and in the review of test cases.
The benchmarking script is in BENCH folder : BENCH/bench-croco.py. A lot of
options are available and detailed in this documentation.
16.1.2. Requirements#
As it will build CROCO for you, you will need to have the basic CROCO dependencies available (see CROCO system requirements). Including MPI, OPENMP or OPENACC if you want to use it.
Additionaly, you will have to install python packages listed in
requirements.txt.
If you already have an activated venv
pip install -r requirements.txt
Otherwise you first needs to create a virtualenv :
# only the first time
python3 -m venv venv
# for every terminal used to run the bench script
source venv/bin/activate
# only once
pip install -r requirement.txt
16.1.3. Usage#
16.1.3.1. Running the tests#
To execute the script with a predefined set of configurations:
# simplest way
./bench-croco.py
# use your own config file
./bench-croco.py -c config-mine.jsonc
It will run the cases and variants combination defined in default_selection of the config file.
You can also choose by hand some of those :
# just a single
./bench-croco.py --case BASIN --variants sequential
# or many
./bench-croco.py --case BASIN,BASIN-LARGE --variants sequential,openmp-4,mpi-4
16.1.3.2. Script options#
The testing script comes with a lot of options, you can list them all with the following command:
./bench-croco.py -h
More details on each capability are given in the following sections.
16.1.3.2.1. Use a specific configuration file#
The default configuration file is BENCH/config.jsonc. It contains all the
available cases, variants and host. If you want to run a new test case
configuration, you will have to add it in this file.
A specific configuration file can be provided.
Example :
# use your own config file
./bench-croco.py -C config-mine.jsonc
# or equivalent
./bench-croco.py --config config-mine.jsonc
Details on the configuration file are in configuration file section.
Tip
Configuration file can be splitted in subdirectory mode by merging several subfiles.
In the main config.jsonc file, the inclusion is described by the imports section.
See also imports section.
16.1.3.2.2. Select cases, variants, host#
The default cases run are BASIN and BASIN-LARGE. You can select the case you want to run.
Example :
# use a specific case
./bench-croco.py -c CANYON
# or equivalent
./bench-croco.py --cases CANYON
The default variants run are @cpu which is a meta_variant defined in the configuration file. You can select a specific variant you want to run.
Example for the variant mpi-2 (MPI using 2 cores) :
# use a specific variant
./bench-croco.py -v mpi-2
# or equivalent
./bench-croco.py --variants mpi-2
If you want to adjust options to your host, you can specify a host. If you want to add your specific host, you can add it in the configuration file or in a new file in the hosts directory.
Example to use “bigfoot” host options:
# use your own config file
./bench-croco.py --host bigfoot
Tip
Wildcards
You can also use wildcards for case and variant to select them. For example below to run all BASIN* cases and all OpenMP variants :
./bench-croco.py --cases 'BASIN*' --variants 'openmp-*'
Tip
Meta variants
You will notice in the config.jsonc file the definition of meta_variants which allow you to define sets of variants which you might use often for your run.
For example :
"meta_variants": {
"@debug": ["sequential", "mpi-2", "mpi-4", "openmp-2", "openmp-4"],}
Then you can just use them easily:
./bench-croco.py --cases BASIN --variants @debug
And you can remove some of the elements by using ‘-’ :
./bench-croco.py --cases BASIN --variants @debug,-openmp-4
Tip
Meta case
If you want to simply run all available cases and all variants, you can just:
./bench-croco.py --cases @all --variants @all
# you can remove some of the elements by using '-' :
./bench-croco.py --cases @all,-BASIN-LARGE --variants @all
# you can also use wildcard in the expr for removing :
./bench-croco.py --cases '@all,-BASIN-*' --variants '@all,-openmp-*'
You can also defined your own meta-case in your specific configuration file.
16.1.3.2.3. Select the modes to run#
The script run in steps. The available steps are :
build : code compilation for each case and variant
run : run simulation for each case and variant
check : perform comparison with a reference
plotperf : performance plot to show runtime of each variant
plotphy : physical plot using python script specified in case config
plotraw : plot raw map of variables at start/middle/end of simulation.
animraw : same plot as mesh with an animation over the simulation.
The default steps are build,run,check,plotperf.
You can specify the steps you want to run.
Example:
./bench-croco.py --cases BASIN -s build,run
#or
./bench-croco.py --cases BASIN --steps build,run
16.1.3.2.4. Build options#
The first step run by the script is the build one.
For each case and each variant, CROCO will be compiled in the corresponding rundir.
The script use jobcomp script to build croco executable. Example:
./bench-croco.py --cases BASIN -s build
If you rebuild a case you have already built, the script won’t rebuild it.
To force a rebuild, you can add -r or --rebuild in the command line.
Example:
./bench-croco.py --cases BASIN -s build --rebuild
By default, the script use -j 4 option to compile in parallel.
You can adjust this behavior with the option -j or --jobs in the
command line.
Example:
./bench-croco.py --cases BASIN -s build --jobs 1
Warning
Note that with AGRIF activated, code won’t compile in parallel as -j option is set to 1 in jobcomp.
16.1.3.2.5. Run options#
By default, each run is done only one time. To do performance tests, you can
perform each run several time by specifying the number of times you need in
the command line using -R or --runs options.
Example:
./bench-croco.py --cases BASIN -R 5
If you run tests on an host without sufficient resources to run a specific
variant, this one can be automatically skipped by adding the option -a
or --auto-skip in the command line.
Example:
./bench-croco.py --cases @all --variants @all -r --auto-skip
Some tests requires external datasets (mostly large Netcdf files). This is the case for the predefined configurations REGIONAL and COASTAL. You can download the needed datasets to run test in the Download section: https://www.croco-ocean.org/download/.
Once you have the files on the host you are running, you can specify its path
in the command line using --data-root-path.
In this case, specified path should contain cases specific directories
(croco_files_path in case config file).
Example:
./bench-croco.py --cases BENGUELA_LR* --data-root-path /my/data/path
Note
The option --data-root-path is not required for all analytical cases
which does not need large external datasets.
Tip
You can run BENGUELA_LR* and VILAINE configurations using datasets available here : https://data-croco.ifremer.fr/ForCI-INRIA.tar.gz
Once you have download the archive, extract it and put in
--data-root-path the path toward the directory
16.1.3.2.6. Check options#
When all cases and variants are done, you can perform the check step.
For each case, this step will compare netcdf file resulting of each variant with the file resulting of the reference variant (by default sequential).
You can specify another variant than sequential using --compare-to option.
Example:
./bench-croco.py -v sequential-ifort,mpi-ifort-4 --compare-to sequential-ifort
By default, comparison is made over netcdf results file. A more precise option
is available through the option --cvtk. This enable usage of CVTK_DEBUG
cppkey for comparison with a reference and track difference directly in the
code.
For some reason, the reference sequential run can cost a lot to produce if the case is large. In order not to redo it all the time, you can use two approach :
The simpler one consist in making the run a first time to get the reference data generated then to run without sequential:
Example:
# first time
./bench-croco.py --cases @all --variants @all
# others
./bench-croco.py --cases '@all,-sequential' --variants @all
The other approach consist in producing a reference directory which can be
zipped and uploaded somewhere. This can be done using --build-ref and then
--use-ref options.
Example:
# first time
./bench-croco.py --cases @all --variants @all --build-ref ./croco-refs-2025-02
# others
./bench-croco.py --cases @all --variants @all --use-ref ./croco-refs-2025-02
16.1.3.2.7. Performance options#
In order to perform correct performance measurement you can ask bench-croco.py to repeat several times the run to get some error bars.
By default it makes only a single run per case/variant.
You can set it to 5 by :
./bench-croco.py --case BASIN --variants @all --runs 5
By default, when plotting performance results, the scripts will search for all
results corresponding to the case in the result directory. You can avoid loading
previous results by using -N or --no-previous.
16.1.3.2.8. Working and results directories options#
The script will build CROCO for each case/variant in the workdir (./rundir
by default). You can specify another directory using -w or
--workdir option.
The results are stored in ./results (by default).
You can specify another directory using --results option.
The script will keep the results of each run by {hostname}-{date} but you can also ask the script to add a given name if you are making different tries {title}-{hostname}-{date} .
# while you are developping your optimizations (trash dir)
./bench-croco.py --title dev
# Produce clean final perf ready for paper
./bench-croco.py --title final-perf
You can also remove date from the folder name by using
--no_date_in_result_dir option.
./bench-croco.py --title mytest --no_date_in_result_dir`
16.1.3.2.9. Reporting#
If you want to run bench-croco.py for validation and get a summary report on what is valid or not, you can simply add ``–report``to the command line :
./bench-croco.py --case @all --variants @all -r --report
You will get an output like :
------------------------------- REPORT SUMMARY ---------------------------------
CASE-VARIANT Build Run Check Plotphy
SOLITON-sequential None True True None
SOLITON-openmp-2 None True True None
SOLITON-LARGE-sequential None True True None
SOLITON-LARGE-openmp-2 None True True None
Plus the results stored in status_report.json.
Note
The option --report automatically add the option --continue which
allows the script to continue instead of stopping on first error.
You can therefore used --continue without --report option.
16.1.3.2.11. Debug option#
You can add --debug option to your command line. This will add debug
compilation option defined in the host config file.
To avoid long unnecessary simulations, this will also limit the number of time steps to 6 and modify the CROCO output to write every timesteps.
This option is helpfull for CROCO developpers.
16.1.3.2.12. Restart option#
You can add --restart option to your command line. This will run the
reference variant once and after all the variant in restarted mode using the
restart file generatad by the reference variant.
The check steps will then verify the restartability of the case.
To avoid long unnecessary simulations, this will also limit the number of time steps to 6.
16.1.4. Configuration files structures#
The configuration file config.jsonc contains all the available cases,
variants and hosts known by bench-croco.py script.
If you want to add a new case, a new variant or a new host, you must add it in the configuration.
16.1.4.1. Imports#
You will notice at the head of the configuration file the imports section which allow to split the config file in multiple sub files and merge them at loading time.
///////////////////////////// IMPORTS ///////////////////////////////////////
"imports": {
"hosts": ["config.d/hosts/*"],
"cases": ["config.d/cases/*"],
"variants": ["config.d/variants/*"],
},
This is usefull for exemple for the hosts and cases definitions as it permits a user to define its own hosts without requirement to patch the files tracked by git and to keep its definition for himself until he finally decide to commit it.
16.1.4.2. Case#
For each case configuration, a specific file is added in the config.d/cases/ directory. You can browse this directory to explore how cases are configured.
Tip
For one configuration, you can put several cases to change cppkeys or domain definition.
For example, for REGIONAL configuration, several cases are defined :
BENGUELA_LR_TIDES_BULK
BENGUELA_LR_AGRIF_1WAY
…
Each case can contains the following keys :
title: string, the title you will have in plot
case: string, the cppkeys activated for the case
splitting: dict, used with mpi or openmp variants to distribute resources along xi/eta axis. Note that if a splitting is not available for the number of processors you ask, the variant will be automatically skipped
patches: dict, see Case Configuration patching
check_outputs: list of strings, list of files used for check step (example : [“basin_his.nc”])
croco_files_path: string[optionnal], path toward external dataset in the given data_root_path
plotphy : dict, [optionnal], used with plotphy step to produce a physical plot, must contains plotphy_script (string, name of the python script to use) and plotphy_script_input (string, name of the file tu use in script arg)
input_file: string[optionnal], default is TEST_CASES/croco.in.<capitalised_case>
unsupported: list of strings[optionnal], list of unsupported option or variant (example : [“restart”,”openmp”]).
cppkeys: dict[optionnal], to list cppkeys to activate or deactivate for this case.
16.1.4.3. Variant#
Each variant can contains the following keys :
configure: variant options to add to the configure command line
tuning_familly: “gnu”, “intel” or “nvfortran”
environ : if a specific environment variable is needed
command_prefix : if a prefix is needed
unpack : to avoid to repeat for each threads number variables can be unpacked at runtime (See also Variables)
requires : to defined the number of cpu/gpu needed
Sequential, mpi-, openmp- and openacc variants are already defined, in config.d/variants directory.
16.1.4.4. Host#
Each host configuration can contains the following keys :
resources: to list the cpu and gpu available on the host
tuning: to list options [extra, standard, debug, mpi_option] for gnu, intel and nvfortran
environ[optionnal]: if you need specific variable
hwloc_bind[optionnal]: if you need options for openmp or mpi
A default configuraion is given in config.jsonc in host @default.
16.1.4.5. Case Configuration patching#
You will notice in the cases configuration files (config.d/cases/*) that there is a patching part which explain how to modify the CROCO configuration files to get the case setted up.
As an example, the patching to increase the domain size can be done by :
"patches": {
// Define a file to patch
"param_override.h": {
// Patch mode
"mode": "replace",
// What to replace (expected to be present)
"what": " parameter (LLm0=60, MMm0=50, N=10)",
// By what to replace
"by": " parameter (LLm0=200, MMm0=200, N=50)",
// Description to be printed in the terminal when applied
"descr": "Set mesh size to (200 x 200 x 50)"
},
},
Tip
It can also contains an array of patch elements if needs to change several things in the same file.
Example :
"patches": {
// Change the number of steps and the output frequency
"TEST_CASES/croco.in.Basin": [{
"mode": "replace",
"what": " 3240 9600 65 1",
"by": " 6 9600 65 1",
"descr": "Set stepping (steps=6, dt=9600, fast=65)"
},{
"mode": "replace",
"what": " T 90 0 ",
"by": " T 1 0 ",
"descr": "Set dtout every steps"
}]
},
Available mode of patching are :
replace
insert-before
insert-after
insert-after-before
insert-at-begin
insert-at-end
Each patch must be filled with a dictionnary called rule with the following keys :
“mode” (str, required): One of listed modes above.
“what” (str, required): The line to match (after normalization) for certain operations.
“by” (str): Line used in ‘replace’ mode.
“insert” (str or list): Line(s) to insert in ‘insert-before’, ‘insert-after’, ‘insert-after-before’ modes.
“after” (str, required for ‘insert-after-before’): The line after which to insert.
“before” (str, required for ‘insert-after-before’): The line before which to insert.
“descr” (str, optional): Description of the rule for logging
16.1.4.6. Variables#
You might notice in the configuration file the use of some variables to be replaced at parsing time on the form of : {tuning.gnu} or {task}.
It avoids duplicating too many entries in the file.
You will mostly find them at two places :
On the cases to unpack king of template not to fully duplicate all cases for OpenMP and MPI.
To inject some values from host into the jobcomp-configure call.
To inject the mpi extra options in command_prefix.
16.1.5. Not implemented yet#
The script can not perform XIOS and coupled cases using OASIS yet.
Theses features are planned to be add in the future.
16.1.6. Frequently Asked Questions (FAQ)#
Below are some frequently asked questions about the CROCO test script.
16.1.6.1. Can I use intel compiler ?#
Yes you can, you will only have to check the tuning option for intel are given in host file and that the variant you used have the “tuning_familly”: “intel”. And of course intel compiler must be available in the host you used.
16.1.6.2. Does –cvtk option available with –debug and –restart ?#
Yes you can run tests with all these options at the same time. Therefore,
if the run failed due to non reproducibility or non restartability with
--cvtk option, the --continue or --report options will not work
and the script will exit at the first fail.
16.1.6.3. How can I run a simple analytical case ?#
One of the most simple use of the script is :
./bench-croco.py -c BASIN -v sequential
This command is running the BASIN case in sequential. If you encounter errors, check if the default host is corresponding to your machine. If not, you can configure your own host. See also : host
For a list of available analytical cases, you can search in config.d/cases the JSONC files for each configurations.
You can also search in the review of test cases.
16.1.6.4. How can I run a realist case ?#
Code is provide with realist configurations defined in files realist.jsonc and coastal.jsonc. To run them, you will need external datasets available at the url https://data-croco.ifremer.fr/ForCI-INRIA.tar.gz.
Once you have download the archive, extract it and put in
--data-root-path the path toward the directory
./bench-croco.py -c BENGUELA_LR*,VILAINE -v mpi-4 --data-root-path /my/data/path
See also : data_root_path
16.1.6.5. Can I define a new test based on an existing configuration ?#
Yes.
If your configuration fit with one already in cppdefs.h file, you will need to add your sub-test in the jsonc file corresponding to the configuration (example basin.jsonc or canyon.jsonc). Then you will have to complete the case config file.
For example, you can choose wich cppkeys you want to activate or deactivate
"cppkeys": {
"FRC_BRY": true,
"BULK_FLUX": true,
"GLS_MIXING": true,
"LMD_MIXING": false,
},
You can also specify another croco.in file if the default one for this configuration does not fit your needs.
"input_file": "TEST_CASES/croco.in.My_awesome_case",
If you need external datasets, do not forget to put the path in
"croco_files_path": "MY_PATH",
16.1.6.6. Can I define a new test NOT based on an existing configuration ?#
Yes.
If your test is a whole new configuration, you will have to :
update cppdefs.h file to add the configuration
add corresponding input file in TEST_CASES directory
add corresponding case config file in config.d/cases.
16.1.6.7. How can I get help ?#
For further assistance, refer to the documentation or contact the development team.
16.1.7. Conclusion#
This script provides full automation of CROCO tests, simplifying performance evaluation and simulation validation.
For any questions, post on the user forum or contact the development team.