Fermi-LAT - Revision history

Gokus at 19:24, 17 March 2025

2025-03-17T19:24:03Z

Gokus at 21:32, 8 January 2025

2025-01-08T21:32:29Z

Show changes

Gokus at 09:38, 19 May 2022

2022-05-19T09:38:11Z

Gokus at 09:37, 19 May 2022

2022-05-19T09:37:13Z

Gokus at 16:52, 27 September 2021

2021-09-27T16:52:04Z

Gokus: /* Q&A */

2021-09-27T16:51:35Z

Q&A

Gokus: /* Troubleshooting */

2021-06-29T14:41:16Z

Troubleshooting

Gokus: /* Contact person */

2021-06-23T14:54:17Z

Contact person

Gokus at 14:53, 23 June 2021

2021-06-23T14:53:59Z

Gokus at 14:40, 23 June 2021

2021-06-23T14:40:01Z

@@ Line 7: / Line 7: @@
 To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment.
 At Remeis, analysis scripts written with ''fermipy'' are available to create either a spectrum or a light curve. These can be found at $FERMITOOLS, which links to /data/software/Science/satscripts/fermiscripts.<br />
-To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.
+To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is the recommended choice.
 === Option 1 (recommended) - Using the existing conda environment (Fermitools 2.2.0 & fermipy version 1.3.1) ===
@@ Line 22: / Line 22: @@
 <pre>
 #!/bin/bash
-source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.sh
+source /userdata/data/gokus/conda/miniconda3/etc/profile.d/conda.sh
 alias fermipy3 ='conda activate /userdata/data/gokus/conda/miniconda3/envs/fermipy_3'
 </pre>

@@ Line 35: / Line 35: @@
 However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.
-All photon files are linked at '''/home/X-ray/Fermi/archive/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/home/X-ray/Fermi/archive//spacecraft_ft2_files.fits'''.<br/><br/>
+All photon files are linked at '''/home/X-ray/Fermi/archive/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/home/X-ray/Fermi/archive/spacecraft_ft2_files.fits'''.<br/><br/>
 In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:

@@ Line 35: / Line 35: @@
 However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.
-All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/>
+All photon files are linked at '''/home/X-ray/Fermi/archive/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/home/X-ray/Fermi/archive//spacecraft_ft2_files.fits'''.<br/><br/>
 In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:
 <pre>
 #!/bin/tcsh
-setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs
+setenv FERMI_DIR /home/X-ray/Fermi/archive/catalogs
 setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources
 </pre>
@@ Line 46: / Line 46: @@
 <pre>
 #!/bin/bash
-export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/
+export FERMI_DIR=/home/X-ray/Fermi/archive/catalogs/
 export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources
 </pre>
@@ Line 52: / Line 52: @@
 In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/>
-In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.
+In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/home/X-ray/Fermi/archive/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.
 == Fermi-LAT analysis with the pipeline scripts ==

@@ Line 332: / Line 332: @@
 == Contact person ==
 The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/>
-Last status update on this wiki page: June 23, 2021
+Last status update on this wiki page: September 27, 2021
 [[Category:Data Extraction]]

@@ Line 239: / Line 239: @@
 ...
 </pre>
 == Troubleshooting ==

@@ Line 253: / Line 253: @@
          itimes = enumerate(zip(times[:-1], times[1:]))
-           &rArr; flux_const_for_calcTS_var = None</span>
+           &rArr; flux_const_for_calcTS_var = None
          for i, time in itimes:

← Older revision		Revision as of 14:53, 23 June 2021
Line 218:		Line 218:

	'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/>		'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/>
−	~~'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell.~~

		+	'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. I recommend to write a script to create the .slurm file that you want to submit to slurm. Here's an example how this array job can look like:
		+	<pre>
		+	#!/bin/bash
		+	#SBATCH --job-name LATspectra
		+	#SBATCH --output job.out-%a
		+	#SBATCH --error err.out-%a
		+	#SBATCH --time 48:00:00
		+	#SBATCH --partition=remeis
		+	#SBATCH --ntasks=1
		+	#SBATCH --hint=nomultithread
		+	#SBATCH --mem-per-cpu=4G
		+	#SBATCH --mail-type=END,FAIL,ARRAY_TASKS
		+	#SBATCH --mail-user=my@email.de
		+	#SBATCH --array 0-XXX%YY (XXX: number of tasks in array, YY: number of jobs running simultaneously

		+	COMMAND[0]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJXXX.X-XXXX mjd_start mjd_stop"
		+	COMMAND[1]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJYYY.Y-YYYY mjd_start mjd_stop"
		+	COMMAND[2]="..."
		+	...
		+	</pre>

	== Troubleshooting ==		== Troubleshooting ==

@@ Line 289: / Line 289: @@
 <br/>
 '''Running the original script to check what's going wrong''' <br/>
 <br/>