Geant4

Running Geant4 simulations on the Remeis cluster

This page describes a practical workflow for building and running Geant4 Monte-Carlo simulations on the Remeis compute cluster, using SLURM for parameter scans and conda to provide the Geant4 toolkit. The examples are taken from Cu67 simulations, but the mechanics apply to any Geant4 application.

Overview

A Geant4 simulation on the cluster involves three steps, although the last step is optional:

• the compiled Geant4 application (a C++ executable, here cu67_sim);
• one or more macro files (.mac) that configure geometry, beam and run size;
• a Python wrapper that renders macros from a template, launches the executable, merges the output, and produces summary tables and plots.

For parameter scans the Python wrapper is submitted to a SLURM array job, so that each grid point runs as an independent task.

Geant4 via conda

Geant4 (including its data libraries) is available from conda-forge; there is no need to compile the toolkit by hand. Create the environment once:

module load conda
conda create -n geant4 -c conda-forge geant4 numpy scipy matplotlib

This might take a while. Afterwards, you can load it in every subsequent session (interactive shell or batch script) -- you might also place the following into you .bashrc /.cshrc:

module load conda
conda activate geant4

Activating the environment sets the Geant4 data-library variables (G4NEUTRONHPDATA, G4PARTICLEXSDATA, …), so — unlike a manual install — there is no geant4.sh to source. Version 11.4 is the version used so far.

Building the application

At this point you can the build your sepecific application and setup a file estructurer like this:

 +- CMakeLists.txt
   +- cu67.cc
   +- include/
   |  ... headers.hh ...
   +- src/
      ... sources.cc ...

CMake locates the conda-provided Geant4 through CMAKE_PREFIX_PATH, which is pointed at the active environment via $CONDA_PREFIX:´. And the application can be compiled via:

module load conda
conda activate geant4

mkdir -p build && cd build
cmake -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..
make -j$(nproc)

It can be usefull to package these, e.g. in a build.sh, scripts, which builds the simulation and copies the resulting cu67_sim executable to the project root - or use a makefile.

Macros and templates

What is simulated is set by macro files:

./cu67_sim run.mac

or interactively (with visualisation) by passing no argument.

Two points to keep in mind are:

• Commands that alter the geometry (e.g. /det/converter/halfThickness) must appear before /run/initialize, because the geometry is built when the run is initialised. Beam and run commands (/gun/…, /run/beamOn) come after.
• Multithreaded runs are requested with /run/numberOfThreads N. Note that high-precision (HP) physics data can cost ~1–2 GB per worker thread, which matters for the memory requested via SLURM.

For parameter scans the macro is kept as a template (run.mac.tpl) with Python str.format placeholders, which the driver substitutes at run time:

/det/converter/halfThickness {HALF_THICKNESS_MM} mm
/analysis/setFileName        {OUTPUT_BASE}

/run/initialize

/gun/particle  e-
/gun/energy    {BEAM_ENERGY_MEV} MeV
/gun/position  0 0 -5 cm
/gun/direction 0 0 1

/tracking/verbose 0
/run/beamOn {N_EVENTS}

Caveat: /analysis/setFileName treats text after the last dot as a file extension, so output base names must not contain dots (e.g. encode "1.0 mm" as a tag like t1000um).

Python Wrapper

Python can be useful to run everything together. To can e.g. several simulations on a grid of parameters. A script can be used that:

1. renders run.mac.tpl into a concrete macro with the point's parameters;
2. runs the executable and captures stdout;
3. merges the per-thread CSV output and moves it into the output directory;
4. parses the result from stdout with a regex.

At the end it writes a summary CSV and the plots.

Running scans with SLURM

For longer simulations it is best to use SLURM. Parameter scans are submitted as job arrays: the python script called by srun runs the simulation for one grid point per array task, indexed by $SLURM_ARRAY_TASK_ID, with each task using $SLURM_CPUS_PER_TASK Geant4 worker threads. The batch script. More CPUS per task can be used but as they all need to be available simultaneously, SLURM might delay the scheduling of the job:

#!/bin/bash
#SBATCH --job-name cu67_scan
#SBATCH --partition meggie
#SBATCH --account meggie
#SBATCH --time 12:00:00
#SBATCH --output log/scan_%A_%a.out
#SBATCH --error log/scan_%A_%a.err
#SBATCH --array 0-39
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 4
#SBATCH --mem-per-cpu 1G
#SBATCH --mail-type END,FAIL
#SBATCH --mail-user user@mail.de

module load conda
conda activate geant4

cd /path/to/project

srun /usr/bin/nice -n +19 \
    python3 scan_thickness_slurm.py \
        --task-id ${SLURM_ARRAY_TASK_ID} \
        --threads ${SLURM_CPUS_PER_TASK}

Submit it with:

 sbatch scan_thickness.slurm

A few points are relevant for running simulations on the cluster:

• The --array range must span the number of grid points (0–39 here = 40 points). Keep it in sync with the grid defined in the driver.
• Isolated working directories. Each task runs in its own work directory, so the flat per-thread CSVs of concurrently running tasks never collide.
• Rather than 40 tasks appending to one summary file, each task writes its single result to rows/<tag>.csv.
• nice -n +19 keeps the jobs from crowding out interactive use of the node.

Collecting and plotting

After the array finishes, a single collect step merges the per-task row files into the summary table and draws the plots (this is the step that needs matplotlib):

python3 scan_thickness_slurm.py --collect

It can also be chained as a dependent job so it runs automatically once the array completes:

sbatch --dependency=afterany:<jobid> --ntasks 1 \
       --wrap "python3 scan_thickness_slurm.py --collect"

If interest lies not in exploring the parameter but to maximize statistics for a single simulation, the parameters just need to be kept constant and the results of identical simulations merged in the end.

Summary

Step	Command
Load Geant4	module load conda && conda activate geant4
Build	cmake -DCMAKE_PREFIX_PATH=$CONDA_PREFIX .. && make -j
Single run	./cu67_sim run.mac
Submit scan	mkdir -p log && sbatch scan_thickness.slurm
Collect + plot	python3 scan_thickness_slurm.py --collect