Geant4 - Revision history

Thalhammer at 13:30, 24 July 2026

2026-07-24T13:30:01Z

Thalhammer at 07:03, 24 July 2026

2026-07-24T07:03:29Z

Thalhammer at 16:34, 23 July 2026

2026-07-23T16:34:07Z

Thalhammer at 16:32, 23 July 2026

2026-07-23T16:32:29Z

Thalhammer at 16:31, 23 July 2026

2026-07-23T16:31:15Z

Thalhammer: added conda-forge

2026-07-23T16:21:22Z

added conda-forge

Thalhammer: Create documentation for Geant4

2026-07-23T13:59:17Z

Create documentation for Geant4

New page

= Running Geant4 simulations on the Remeis cluster =

This page describes a practical workflow for building and running
[https://geant4.web.cern.ch/ Geant4] Monte-Carlo simulations on the Remeis
compute cluster, using SLURM for parameter scans and conda to provide the
Geant4 toolkit. The examples are taken from ''Cu67'' simulations,
but the mechanics apply to any Geant4 application.

== Overview ==

A Geant4 simulation on the cluster involves three steps, although the last step is optional:

* the compiled Geant4 application (a C++ executable, here <code>cu67_sim</code>);
* one or more macro files (<code>.mac</code>) that configure geometry, beam and run size;
* a Python wrapper that renders macros from a template, launches the executable, merges the output, and produces summary tables and plots.

For parameter scans the Python wrapper is submitted to a SLURM array job,
so that each grid point runs as an independent task.

== Geant4 via conda ==

Geant4 (including its data libraries) is available from
[https://conda-forge.org/ conda-forge]; there is no need to compile the toolkit
by hand. Create the environment once:

<syntaxhighlight lang="bash">
module load conda
conda create -n geant4 -c geant4 numpy scipy matplotlib
</syntaxhighlight>

and load it in every subsequent session (interactive shell or batch script):

<syntaxhighlight lang="bash">
module load conda
conda activate geant4
</syntaxhighlight>

Activating the environment sets the Geant4 data-library variables
(<code>G4NEUTRONHPDATA</code>, <code>G4PARTICLEXSDATA</code>, …), so — unlike a
manual install — there is no <code>geant4.sh</code> to source. Version 11.4 is the version used so far.

== Building the application ==

CMake locates the conda-provided Geant4 through <code>CMAKE_PREFIX_PATH</code>,
which is pointed at the active environment via <code>$CONDA_PREFIX</code>:

<syntaxhighlight lang="bash">
module load conda
conda activate geant4

mkdir -p build && cd build
cmake -DCMAKE_PREFIX_PATH=$CONDA_PREFIX ..
make -j$(nproc)
</syntaxhighlight>

It can be use full to contain this in e.g. a <code>build.sh</code>, scripts, which builds the simulation and copies the resulting <code>cu67_sim</code> executable to the
project root.

== Macros and templates ==

What is simulated is set by macro files:

<syntaxhighlight lang="bash">
./cu67_sim run.mac
</syntaxhighlight>

or interactively (with visualisation) by passing no argument.

Two points to keep in mind are:

* Commands that alter the geometry (e.g. <code>/det/converter/halfThickness</code>) must appear before <code>/run/initialize</code>, because the geometry is built when the run is initialised. Beam and run commands (<code>/gun/…</code>, <code>/run/beamOn</code>) come after.
* Multithreaded runs are requested with <code>/run/numberOfThreads N</code>. Note that high-precision (HP) physics data can cost ~1–2 GB per worker thread, which matters for the memory requested via SLURM.

For parameter scans the macro is kept as a template (<code>run.mac.tpl</code>)
with Python <code>str.format</code> placeholders, which the driver substitutes at
run time:

<syntaxhighlight lang="text">
/det/converter/halfThickness {HALF_THICKNESS_MM} mm
/analysis/setFileName {OUTPUT_BASE}

/run/initialize

/gun/particle e-
/gun/energy {BEAM_ENERGY_MEV} MeV
/gun/position 0 0 -5 cm
/gun/direction 0 0 1

/tracking/verbose 0
/run/beamOn {N_EVENTS}
</syntaxhighlight>

Caveat: <code>/analysis/setFileName</code> treats text after the last dot
as a file extension, so output base names must not contain dots (e.g. encode
"1.0 mm" as a tag like <code>t1000um</code>).

== Python drivers ==

Python can be used to run everything together. For a single-machine scan
(<code>scan_thickness.py</code>) each grid point:

# renders <code>run.mac.tpl</code> into a concrete macro with the point's parameters;
# runs the executable and captures stdout;
# merges the per-thread CSV output and moves it into the output directory;
# parses the result from stdout with a regex.

At the end it writes a summary CSV and the plots.

== Running scans with SLURM ==

Parameter scans are submitted as job arrays: the driver
(<code>scan_thickness_slurm.py</code>) runs one grid point per array task,
indexed by <code>$SLURM_ARRAY_TASK_ID</code>, with each task using
<code>$SLURM_CPUS_PER_TASK</code> Geant4 worker threads. The batch script:

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name cu67_scan
#SBATCH --partition meggie
#SBATCH --account meggie
#SBATCH --time 12:00:00
#SBATCH --output log/scan_%A_%a.out
#SBATCH --error log/scan_%A_%a.err
#SBATCH --array 0-39
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 4
#SBATCH --mem-per-cpu 1G
#SBATCH --mail-type END,FAIL
#SBATCH --mail-user user@mail.de

module load conda
conda activate geant4

cd /path/to/project

srun /usr/bin/nice -n +19 \
python3 scan_thickness_slurm.py \
--task-id ${SLURM_ARRAY_TASK_ID} \
--threads ${SLURM_CPUS_PER_TASK}
</syntaxhighlight>

Submit it (creating the log directory first) with:

<syntaxhighlight lang="bash">
mkdir -p log && sbatch scan_thickness.slurm
</syntaxhighlight>

A few points are relevant for running simulations on the cluster:

* The <code>--array</code> range must span the number of grid points (0–39 here = 40 points). Keep it in sync with the grid defined in the driver.
* Isolated working directories. Each task runs in its own work directory, so the flat per-thread CSVs of concurrently running tasks never collide.
* Rather than 40 tasks appending to one summary file, each task writes its single result row atomically to <code>rows/<tag>.csv</code>.
* A point whose row file already exists is skipped, so a partially failed array can simply be resubmitted.
* <code>nice -n +19</code> keeps the jobs from crowding out interactive use of the node.

== Collecting and plotting ==

After the array finishes, a single collect step merges the per-task row
files into the summary table and draws the plots (this is the step that needs
matplotlib):

<syntaxhighlight lang="bash">
python3 scan_thickness_slurm.py --collect
</syntaxhighlight>

It can also be chained as a dependent job so it runs automatically once the
array completes:

<syntaxhighlight lang="bash">
sbatch --dependency=afterany:<jobid> --ntasks 1 \
--wrap "python3 scan_thickness_slurm.py --collect"
</syntaxhighlight>

If interest lies not in exploring the parameter but to maximize statistics for a single simulation, the parameters just need to be kept constant and the results of identical simulations merged in the end.

== Summary ==

{| class="wikitable"
! Step !! Command
|-
| Load Geant4 || <code>module load conda && conda activate geant4</code>
|-
| Build || <code>cmake -DCMAKE_PREFIX_PATH=$CONDA_PREFIX .. && make -j</code>
|-
| Single run || <code>./cu67_sim run.mac</code>
|-
| Submit scan || <code>mkdir -p log && sbatch scan_thickness.slurm</code>
|-
| Collect + plot || <code>python3 scan_thickness_slurm.py --collect</code>
|}

[[Category:Software]]

@@ Line 107: / Line 107: @@
 "1.0 mm" as a tag like <code>t1000um</code>).
-== Python drivers ==
+== Python Wrapper ==
-Python can be used to run everything together. For a single-machine scan
+Python can be useful to run everything together. To can e.g. several simulations on a grid of parameters. A script can be used that:
 # renders <code>run.mac.tpl</code> into a concrete macro with the point's parameters;
@@ Line 121: / Line 120: @@
 == Running scans with SLURM ==
-Parameter scans are submitted as job arrays: the driver
+For longer simulations it is best to use SLURM. Parameter scans are submitted as job arrays: the python script called by srun runs the simulation for one grid point per array task,
 indexed by <code>$SLURM_ARRAY_TASK_ID</code>, with each task using
-<code>$SLURM_CPUS_PER_TASK</code> Geant4 worker threads. The batch script:
+<code>$SLURM_CPUS_PER_TASK</code> Geant4 worker threads. The batch script. More CPUS per task can be used but as they all need to be available simultaneously, SLURM might delay the scheduling of the job:
 <syntaxhighlight lang="bash">
@@ Line 152: / Line 150: @@
 </syntaxhighlight>
-Submit it (creating the log directory first) with:
+Submit it with:
 <syntaxhighlight lang="bash">
-mkdir -p log && sbatch scan_thickness.slurm
+ sbatch scan_thickness.slurm
 </syntaxhighlight>

@@ Line 41: / Line 41: @@
 == Building the application ==
 CMake locates the conda-provided Geant4 through <code>CMAKE_PREFIX_PATH</code>,
-which is pointed at the active environment via <code>$CONDA_PREFIX</code>:
+which is pointed at the active environment via <code>$CONDA_PREFIX</code>:´. And the application can be compiled via:
 <syntaxhighlight lang="bash">

@@ Line 54: / Line 54: @@
 </syntaxhighlight>
-It can be use full to contain this in e.g. a <code>build.sh</code>, scripts, which builds the simulation and copies the resulting <code>cu67_sim</code> executable to the
+It can be usefull to package these, e.g. in a <code>build.sh</code>, scripts, which builds the simulation and copies the resulting <code>cu67_sim</code> executable to the
-project root.
+project root - or use a makefile.
 == Macros and templates ==

@@ Line 150: / Line 150: @@
 * The <code>--array</code> range must span the number of grid points (0–39 here = 40 points). Keep it in sync with the grid defined in the driver.
 * Isolated working directories. Each task runs in its own work directory, so the flat per-thread CSVs of concurrently running tasks never collide.
-* Rather than 40 tasks appending to one summary file, each task writes its single result row atomically to <code>rows/&lt;tag&gt;.csv</code>.
+* Rather than 40 tasks appending to one summary file, each task writes its single result to <code>rows/&lt;tag&gt;.csv</code>.
 * <code>nice -n +19</code> keeps the jobs from crowding out interactive use of the node.

@@ Line 1: / Line 1: @@
-	= Running Geant4 simulations on the Remeis cluster =
+= Running Geant4 simulations on the Remeis cluster =
 This page describes a practical workflow for building and running
@@ Line 29: / Line 29: @@
 </syntaxhighlight>
-and load it in every subsequent session (interactive shell or batch script):
+This might take a while. Afterwards, you can load it in every subsequent session (interactive shell or batch script) -- you might also place the following into you .bashrc /.cshrc:
 <syntaxhighlight lang="bash">

@@ Line 26: / Line 26: @@
 <syntaxhighlight lang="bash">
 module load conda
-conda create -n geant4 -c geant4 numpy scipy matplotlib
+conda create -n geant4 -c conda-forge geant4 numpy scipy matplotlib
 </syntaxhighlight>