Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,7 @@ machine_label.txt
*.exe
*.out
*.app

# generate default directories
/runs
/idefix
77 changes: 77 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,83 @@ These tests were written for Idefix v1.1
This directory contains scripts used to evaluate the single-process perfomance (frag_tests) as well as weak scaling tests for the lagrangian particles module.
These tests were written for Idefix v2.0 with Clément Robert's particle module (unreleased at the time of writing).

### scripts/bench.py

A complete python script to :

1. Build Idefix for the cluster (Sub-command **build**).
2. Generate the job files for the given cluster (Sub-command **gen**).
3. Launching the jobs (Sub-command **run**).
4. Extracting the scalability results and plotting (Sub-command **extract**).

In order to proceed, you need to first **download** and **compile** *Idefix*. You can either do it by hand or use the **build** sub-command of the script :

./scripts/bench.py --cluster <CLUSTER> --commit <GIT-COMMIT> build

Then you need to **generate** the job directory and files :

./scripts/bench.py --cluster <CLUSTER> --commit <GIT-COMMIT> \
--account <ACCOUNT> --max-cores <MAX-CORES> --cores-per-node <CORES-PER-NODE>\
--problem-size <PROBLEM-SIZE> \
gen

Note that when running on **GPUs**, the `core` options correspond to the number of GPUs and not cores.

Check that the jobs are created the right way, then launch them by calling :

./scripts/bench.py --cluster <CLUSTER> --commit <GIT-COMMIT> \
--account <ACCOUNT> --max-cores <MAX-CORES> --cores-per-node <CORES-PER-NODE> \
--problem-size <PROBLEM-SIZE> \
run

You can then finally parse the logs to extract the perf and get some report files as output :

./scripts/bench.py --cluster <CLUSTER> --commit <GIT-COMMIT> \
--account <ACCOUNT> --max-cores <MAX-CORES> --cores-per-node <CORES-PER-NODE> \
--problem-size <PROBLEM-SIZE> \
extract

It will produce :

- `runs/<COMMIT>/<CLUSTER>/<DATE>/<PROBLEM-SIZE>/summary.json`
- `runs/<COMMIT>/<CLUSTER>/<DATE>/<PROBLEM-SIZE>/summary.dat`
- `runs/<COMMIT>/<CLUSTER>/<DATE>/<PROBLEM-SIZE>/summary.gnuplot`
- `runs/<COMMIT>/<CLUSTER>/<DATE>/<PROBLEM-SIZE>/summary.pdf`

For a more concrete example, to use the **H100** of **Kraken**, you can use :

# compile
./scripts/bench.py --cluster kraken-gpu/h100 --commit master build

# generate the jobs
./scripts/bench.py --cluster kraken-gpu/h100 --commit master --account <ACCOUNT> \
--max-cores 8 --cores-per-node 2 --problem-size 256 \
gen

# submit the jobs to the cluster
./scripts/bench.py --cluster kraken-gpu/h100 --commit master --account <ACCOUNT> \
--max-cores 8 --cores-per-node 2 --problem-size 256 \
run

# extract the performance from the logs
./scripts/bench.py --cluster kraken-gpu/h100 --commit master --account <ACCOUNT> \
--max-cores 8 --cores-per-node 2 --problem-size 256 \
extract

In case you want to build Idefix yourself, use the case in **./OrszagTang3D/setup/** to prepare your build dir and then use the
`--build-directory` to configure in which directory your **idefix** binary and case files lands.

./scripts/bench.py --build-directory ./my_idefix/build/ --cluster kraken-gpu/h100 \
--commit master --account <ACCOUNT> --max-cores 8 --cores-per-node 2 \
--problem-size 256 \
gen

To extract the data from a past date, you can precise the date (following the format `year-month-day`) and force the plotting with gnuplot via `--plot`:

./scripts/bench.py --cluster kraken-gpu/h100 --commit master --account <ACCOUNT> \
--max-cores 8 --cores-per-node 2 --problem-size 256 \
gen --date 2026-06-15 --plot

### scripts/run-bench

A basic script to facilitate running the benchmark on a specific
Expand Down
5 changes: 5 additions & 0 deletions clusters/kraken/cpu/mpi/env.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
module purge

module load openmpi/5.0.8/gcc-14-2.0

export IDEFIX_FLAGS="-DIdefix_MPI=ON"
18 changes: 18 additions & 0 deletions clusters/kraken/cpu/mpi/script.oar
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/bin/bash
#OAR -l /nodes=@nodes@/core=@core@,walltime=00:10:00
#OAR --name @name@
#OAR --project @account@
#OAR --stdout idefix.out
#OAR --stderr idefix.err

# nettoyage des modules charges en interactif et herites par defaut
module purge

# chargement des modules
module load openmpi/5.0.8/gcc-14-2.0*

# echo des commandes lancees
set -x

# execution du code
mpirun -np $((@nodes@ * @core@)) ./idefix
7 changes: 7 additions & 0 deletions clusters/kraken/gpu/h100/env.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
module purge

module load cuda/12.9.41.patched
module load openmpi/5.0.8/cuda-12.9-gcc-14.2.0
module load ucx/1.19.0-cuda

export IDEFIX_FLAGS="-DKokkos_ENABLE_CUDA=ON -DKokkos_ARCH_HOPPER90=ON -DIdefix_MPI=ON"
27 changes: 27 additions & 0 deletions clusters/kraken/gpu/h100/script.oar
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/bin/bash
#OAR -l /nodes=@nodes@/gpu=@core@,walltime=00:10:00
#OAR --name @name@
#OAR --project @account@
#OAR --stdout idefix.out
#OAR --stderr idefix.err

# nettoyage des modules charges en interactif et herites par defaut
module purge

# chargement des modules
module load arch/h100
module load cmake/3.30.1
module load cuda/12.1.0
module load openmpi/4.1.5-cuda

# echo des commandes lancees
set -x
#export KOKKOS_PROFILE_LIBRARY=~/src/kokkos-tools/kp_nvprof_connector.so
#which nsys
#nsys -v
# execution du code
#export TMPDIR=$JOBSCRATCH
#ln -s $JOBSCRATCH /tmp/nvidia

# execution du code
./idefix
1 change: 1 addition & 0 deletions clusters/localhost/cpu/mpi/env.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export IDEFIX_FLAGS="-DIdefix_MPI=ON"
13 changes: 13 additions & 0 deletions clusters/localhost/cpu/mpi/script.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash

# echo des commandes lancees
set -x
#export KOKKOS_PROFILE_LIBRARY=~/src/kokkos-tools/kp_nvprof_connector.so
#which nsys
#nsys -v
# execution du code
#export TMPDIR=$JOBSCRATCH
#ln -s $JOBSCRATCH /tmp/nvidia

# execution du code
mpirun -np @core@ ./idefix 2>&1 | tee idefix.out
1 change: 1 addition & 0 deletions clusters/localhost/cpu/openmp/env.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export IDEFIX_FLAGS="-DKokkos_ENABLE_OPENMP=ON"
13 changes: 13 additions & 0 deletions clusters/localhost/cpu/openmp/script.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash

# echo des commandes lancees
set -x
#export KOKKOS_PROFILE_LIBRARY=~/src/kokkos-tools/kp_nvprof_connector.so
#which nsys
#nsys -v
# execution du code
#export TMPDIR=$JOBSCRATCH
#ln -s $JOBSCRATCH /tmp/nvidia

# execution du code
OMP_NUM_THREADS=@core@ OMP_PROC_BIND=spread ./idefix 2>&1 | tee idefix.out
Loading