https://www.sternwarte.uni-erlangen.de/wiki/api.php?action=feedcontributions&feedformat=atom&user=GokusRemeis-Wiki - User contributions [en]2026-04-08T18:29:01ZUser contributionsMediaWiki 1.35.7https://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=3633Fermi-LAT2025-03-17T19:24:03Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. <br />
At Remeis, analysis scripts written with ''fermipy'' are available to create either a spectrum or a light curve. These can be found at $FERMITOOLS, which links to /data/software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is the recommended choice.<br />
<br />
=== Option 1 (recommended) - Using the existing conda environment (Fermitools 2.2.0 & fermipy version 1.3.1) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda3/etc/profile.d/conda.csh<br />
alias fermipy3 "conda activate fermipy_3" <br />
</pre><br />
<br />
'''If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing this conda installation by default!'''<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
source /userdata/data/gokus/conda/miniconda3/etc/profile.d/conda.sh<br />
alias fermipy3 ='conda activate /userdata/data/gokus/conda/miniconda3/envs/fermipy_3'<br />
</pre><br />
Using the command '''fermipy3''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
=== If you want to use the old conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
In an earlier version, the analysis pipeline ran on Python 2.7 and the most recent versions of the Fermitools and fermipy that were compatible with Python 2.7. It is recommended to use the newest version, however, the former scripts compatible with the old conda enviroment exist within a folder at $FERMITOOLS. If you want to use the old version, please reach out to andrea.gokus[at]fau.de.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/home/X-ray/Fermi/archive/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/home/X-ray/Fermi/archive/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /home/X-ray/Fermi/archive/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/home/X-ray/Fermi/archive/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/home/X-ray/Fermi/archive/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see Remeis Wiki page)]<br />
Note: Cannot be used to search for unknown source close to a known 4FGL source.<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? If set to NO and a 4FGL source is<br />
closer than 0.5 deg, the analysis will run on that close target! Default: YES<br />
<br />
--fix=yes/no Fixes parameters for all sources to 4FGL values. Only to be used to check for<br />
detection of new sources! Default: NO<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
<br />
--myscript=path Path to your own script if you do not want to use the standard Fermi scripts.<br />
Path needs to include the file name. Default: Using scripts in $FERMITOOLS<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see Remeis Wiki page)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for the analysis<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for the analysis<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--fix=yes/no Fixes parameters for all sources to 4FGL values. Only to be used to check for<br />
detection of new sources! Default: NO<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name, RAJ2000, and DEJ2000 for<br />
name and coordinates. If '--ra' and '--dec' are also given, the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
<br />
--myscript=path Path to your own script if you do not want to use the standard Fermi scripts.<br />
Path needs to include the file name. NOTE: Only implemented for spectra, not LC calculation!<br />
Default: Using scripts in $FERMITOOLS<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
<br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. I recommend to write a script to create the .slurm file that you want to submit to slurm. Here's an example how this array job can look like:<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --job-name LATspectra<br />
#SBATCH --output job.out-%a<br />
#SBATCH --error err.out-%a<br />
#SBATCH --time 48:00:00<br />
#SBATCH --partition=remeis<br />
#SBATCH --ntasks=1<br />
#SBATCH --hint=nomultithread<br />
#SBATCH --mem-per-cpu=4G<br />
#SBATCH --mail-type=END,FAIL,ARRAY_TASKS<br />
#SBATCH --mail-user=my@email.de<br />
#SBATCH --array 0-XXX%YY (XXX: number of tasks in array, YY: number of jobs running simultaneously<br />
<br />
COMMAND[0]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJXXX.X-XXXX mjd_start mjd_stop"<br />
COMMAND[1]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJYYY.Y-YYYY mjd_start mjd_stop"<br />
COMMAND[2]="..."<br />
...<br />
</pre><br />
<br />
'''Can the scripts find possibly unknown gamma-ray sources by itself?''' - Yes, when running the analysis script to create a spectrum (with make_spectrum.sl), there is a routine implemented (fermipy function ''find_sources''), which searches for excesses in the TS map with a significance of 5 sigma or higher. The coordinates of a found source aren saved in the folder in a file called ''new_source.dat'', in which the analysis is run, to allow further inspections and a follow-up analyis.<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code, my work-around for this issue is implemented in the wrapper scripts ''make_spectrum.sl'' and ''make_lightcurve.sl''. Deleting the subfolders is now handled in there and not via the routine in the source code. <br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. I have [https://github.com/fermiPy/fermipy/issues/405 | reported this issue to the developers], and it should be fixed now in the fermipy version 1.1.4 and up.<br />
If you're using an older version of fermipy, you can use the following hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is near the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None<br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
If you encounter a new error, you can also go ahead and start the python script for the analysis directly in the shell. Remember to load your conda environment first. You also need a ''config_SED.yaml'' or ''config_LC.yaml'' file in the folder where you are going to run the analysis, otherwise you won't get far. You can directly run the script, e.g. via<br />
<pre><br />
(fermipy)[user@computer]> $FERMITOOLS/SEDcreator_fermipy.py<br />
</pre><br />
<br />
<br/><br />
'''Light curve analysis fails because of too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: January 8, 2025<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=3549Fermi-LAT2025-01-08T21:32:29Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. <br />
At Remeis, analysis scripts written with ''fermipy'' are available to create either a spectrum or a light curve. These can be found at $FERMITOOLS, which links to /data/software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 (recommended) - Using the existing conda environment (Fermitools 2.2.0 & fermipy version 1.3.1) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda3/etc/profile.d/conda.csh<br />
alias fermipy3 "conda activate fermipy_3" <br />
</pre><br />
<br />
'''If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing this conda installation by default!'''<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.sh<br />
alias fermipy3 ='conda activate /userdata/data/gokus/conda/miniconda3/envs/fermipy_3'<br />
</pre><br />
Using the command '''fermipy3''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
=== If you want to use the old conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
In an earlier version, the analysis pipeline ran on Python 2.7 and the most recent versions of the Fermitools and fermipy that were compatible with Python 2.7. It is recommended to use the newest version, however, the former scripts compatible with the old conda enviroment exist within a folder at $FERMITOOLS. If you want to use the old version, please reach out to andrea.gokus[at]fau.de.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/home/X-ray/Fermi/archive/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/home/X-ray/Fermi/archive/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /home/X-ray/Fermi/archive/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/home/X-ray/Fermi/archive/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/home/X-ray/Fermi/archive/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see Remeis Wiki page)]<br />
Note: Cannot be used to search for unknown source close to a known 4FGL source.<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? If set to NO and a 4FGL source is<br />
closer than 0.5 deg, the analysis will run on that close target! Default: YES<br />
<br />
--fix=yes/no Fixes parameters for all sources to 4FGL values. Only to be used to check for<br />
detection of new sources! Default: NO<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
<br />
--myscript=path Path to your own script if you do not want to use the standard Fermi scripts.<br />
Path needs to include the file name. Default: Using scripts in $FERMITOOLS<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see Remeis Wiki page)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for the analysis<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for the analysis<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--fix=yes/no Fixes parameters for all sources to 4FGL values. Only to be used to check for<br />
detection of new sources! Default: NO<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name, RAJ2000, and DEJ2000 for<br />
name and coordinates. If '--ra' and '--dec' are also given, the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
<br />
--myscript=path Path to your own script if you do not want to use the standard Fermi scripts.<br />
Path needs to include the file name. NOTE: Only implemented for spectra, not LC calculation!<br />
Default: Using scripts in $FERMITOOLS<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
<br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. I recommend to write a script to create the .slurm file that you want to submit to slurm. Here's an example how this array job can look like:<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --job-name LATspectra<br />
#SBATCH --output job.out-%a<br />
#SBATCH --error err.out-%a<br />
#SBATCH --time 48:00:00<br />
#SBATCH --partition=remeis<br />
#SBATCH --ntasks=1<br />
#SBATCH --hint=nomultithread<br />
#SBATCH --mem-per-cpu=4G<br />
#SBATCH --mail-type=END,FAIL,ARRAY_TASKS<br />
#SBATCH --mail-user=my@email.de<br />
#SBATCH --array 0-XXX%YY (XXX: number of tasks in array, YY: number of jobs running simultaneously<br />
<br />
COMMAND[0]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJXXX.X-XXXX mjd_start mjd_stop"<br />
COMMAND[1]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJYYY.Y-YYYY mjd_start mjd_stop"<br />
COMMAND[2]="..."<br />
...<br />
</pre><br />
<br />
'''Can the scripts find possibly unknown gamma-ray sources by itself?''' - Yes, when running the analysis script to create a spectrum (with make_spectrum.sl), there is a routine implemented (fermipy function ''find_sources''), which searches for excesses in the TS map with a significance of 5 sigma or higher. The coordinates of a found source aren saved in the folder in a file called ''new_source.dat'', in which the analysis is run, to allow further inspections and a follow-up analyis.<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code, my work-around for this issue is implemented in the wrapper scripts ''make_spectrum.sl'' and ''make_lightcurve.sl''. Deleting the subfolders is now handled in there and not via the routine in the source code. <br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. I have [https://github.com/fermiPy/fermipy/issues/405 | reported this issue to the developers], and it should be fixed now in the fermipy version 1.1.4 and up.<br />
If you're using an older version of fermipy, you can use the following hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is near the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None<br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
If you encounter a new error, you can also go ahead and start the python script for the analysis directly in the shell. Remember to load your conda environment first. You also need a ''config_SED.yaml'' or ''config_LC.yaml'' file in the folder where you are going to run the analysis, otherwise you won't get far. You can directly run the script, e.g. via<br />
<pre><br />
(fermipy)[user@computer]> $FERMITOOLS/SEDcreator_fermipy.py<br />
</pre><br />
<br />
<br/><br />
'''Light curve analysis fails because of too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: January 8, 2025<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Skivergnuegen_2023&diff=2752Skivergnuegen 20232023-01-03T09:35:33Z<p>Gokus: /* Fahrer */</p>
<hr />
<div>'''Remeis Skifahren 2023''' The same procedure as every year ;-) <br />
Hier werden die wichtigsten organisatorischen Dinge festgehalten. Jeder darf hier gerne editieren; wer keinen Account für das Wiki hat, muss sich bitte an jemanden wenden, der Zugang hat.<br />
<br />
[[File:154854 silvretta-montafon-panorama.jpeg|right|700px]]<br />
<br />
== Allgemeine Infos ==<br />
<br />
*Ferienwohnung(en):<br />
**Wir kommen wieder unter im [https://www.grandau.at/montafon/zimmer/ferienhaus-enzian Haus Enzian] des [https://www.grandau.at/ Sporthotels Grandau].<br />
**Adresse des Sporthotel: Montafonerstraße 274a, 6791 St. Gallenkirch, Österreich. Unsere Unterkünfte liegen im Türkeiweg.<br />
<br />
*Skigebiet: Montafon (Vorarlberg)<br />
**Skigebiet Karte: https://winter.intermaps.com/montafon?lang=de<br />
**Skipass Preise: https://www.montafon.at/de/Service/Bergbahn-Preise-Tickets/Mehrtageskarte-Winter<br />
**Wer nicht alle 7 Tage Skifahren möchte: Es gibt Angebote wie z.B. 5 aus 7, mit einem solchen Skipass kann man innerhalb von 7 Tagen an 5 beliebigen Tagen skifahren.<br />
<br />
*Skiverleih: [http://www.sportharry.at/ Sport Harry], direkt an der Talstation.<br />
<br />
== Unterkunft ==<br />
=== Aufteilung ===<br />
<br />
Dieses Jahr gibt es nur die Hütte mit folgenden Zimmern (wir sind insgesamt 14, also bleiben keine Plätze frei).<br />
<br />
{| class="wikitable"<br />
! scope="col" | Zimmer<br />
! scope="col" | Insassen<br />
|-<br />
! scope="row" | 2-er<br />
| Amy, Aafia<br />
<br />
|-<br />
! scope="row" | 3-er<br />
| Jakob, Max, Katya<br />
|-<br />
! scope="row" | 4-er<br />
| Eva, Steffen, Flo, Christian<br />
|-<br />
! scope="row" | 5-er<br />
| Basti<br />
|}<br />
<br />
== Fahrer ==<br />
Fahrer können sich hier eintragen und Eckdaten angeben. Diejenige, die mitfahren möchten, sprechen sich mit den Fahrern ab und tragen sich ebenfalls ein. Klärt bitte auch die Gepäcklage, ggf. kann ein anderer Fahrer z.B. Skiausrüstung mitnehmen.<br />
<br />
'''Auf der Suche nach einer Mitfahrgelegenheit: '''<br />
<br />
{| class="wikitable"<br />
! scope="col" | Fahrer<br />
! scope="col" | # Plätze<br />
! scope="col" | Anreise<br />
! scope="col" | Mitfahrer Anreise<br />
! scope="col" | Abreise<br />
! scope="col" | Mitfahrer Abreise<br />
! scope="col" | Skimitnahme<br />
! scope="col" | Kommentar<br />
|-<br />
|Eva<br />
|0(3)<br />
|Samstag<br />
|Steffen+Christian<br />
|Samstag?<br />
|Steffen+Christian<br />
|eigene ja<br />
|Auto ist voll<br />
|-<br />
|Max<br />
|0(3)<br />
|Samstag<br />
|Jakob+Katya<br />
|Samstag<br />
|Jakob+Katya<br />
|eigene ja<br />
|Auto ist voll<br />
|-<br />
|Basti<br />
|3(3)<br />
|Sonntag ganz frueh<br />
|<br />
|Freitag Mittag<br />
|<br />
|Ja<br />
|<br />
|-<br />
<br />
|-<br />
<br />
|-<br />
<br />
|-<br />
|}<br />
<br />
== Schwarzes Brett ==<br />
<br />
=== Essen ===<br />
Hier gibt's den Essensplan, soweit übernommen von den letzten Jahren. Da wir sehr viele Leute sind, sollten wir wieder vorab grob planen, was wir kochen wollen. Wir können einige Dinge in Deutschland besorgen und mitnehmen, es gib aber auch einen nahe gelegen Supermarkt. Generelle Dinge, wie z.B. Gewürze oder Aufstriche für Frühstücksbrötchen könnte man auch mitbringen.<br />
<br />
Wer Kochvorschläge oder andere Ideen hat, gerne unten eintragen und kommentieren. Bedenkt, dass die Zubereitung (relativ) einfach sein sollte. Da sich die Liste in den letzten Jahren gut bewaehrt hat, behalten wir sie bei, aber die Reihenfolge/Tage kann man sicher noch hin und her bewegen. Einige (wenige) Sachen müssen wir dann manchmal trotzdem noch beim Spar kaufen, da ja auch der Kühlschrank nur begrenzt Platz hat.<br />
<br />
<br />
'''Essenplan:'''<br />
*Samstag: Spaghetti Bolognese + veg. Bolognese<br />
*Sonntag: Käse Spätzle<br />
*Montag: Burritos<br />
*Dienstag: Risotto<br />
*Mittwoch: Gulasch<br />
*Donnerstag: Kartoffeln mit Kräuter-Quark<br />
*Freitag: Dal mit Reis (od. Curry mit Suesskartoffeln )<br />
<br />
<br />
Einkaufsliste von 2022 (letztes Jahr): https://docs.google.com/document/d/166b_mIXxmoAgcv1upIGremd_36E1fmwwxsQADP2uEv8/<br />
<br />
@Christian: Vegane Option entwender selbst mitbringen oder hier mit reinschreiben (falls man die einfach bekommt)<br />
<br />
=== Alkohol/Party ===<br />
<br />
=== Spiele ===<br />
<br />
Immer lustig sind gemeinsame Spieleabende. Also, wer im Besitz von Gesellschaftsspielen ist, bringt diese gerne mit! Um eine Übersicht zu bekommen auch bitte hier eintragen:<br />
<br />
''* Spiele hier eintragen''<br />
<br />
=== Sonstiges ===<br />
<br />
=== Alternativprogramm ===<br />
* Rodeln (http://www.montafon.at/de/urlaubswelten/echte_naturliebhaber/rodeln)<br />
* Schneeschuhwandern (http://www.montafon.at/schneeschuhwanderungen)<br />
* Therme (http://www.montafon.at/schwimmen), z.B. http://www.aqua-dome.at/de (ca. 130km entfernt!)<br />
<br />
[[Category:Internal]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Skivergnuegen_2023&diff=2751Skivergnuegen 20232023-01-03T09:34:57Z<p>Gokus: /* Fahrer */</p>
<hr />
<div>'''Remeis Skifahren 2023''' The same procedure as every year ;-) <br />
Hier werden die wichtigsten organisatorischen Dinge festgehalten. Jeder darf hier gerne editieren; wer keinen Account für das Wiki hat, muss sich bitte an jemanden wenden, der Zugang hat.<br />
<br />
[[File:154854 silvretta-montafon-panorama.jpeg|right|700px]]<br />
<br />
== Allgemeine Infos ==<br />
<br />
*Ferienwohnung(en):<br />
**Wir kommen wieder unter im [https://www.grandau.at/montafon/zimmer/ferienhaus-enzian Haus Enzian] des [https://www.grandau.at/ Sporthotels Grandau].<br />
**Adresse des Sporthotel: Montafonerstraße 274a, 6791 St. Gallenkirch, Österreich. Unsere Unterkünfte liegen im Türkeiweg.<br />
<br />
*Skigebiet: Montafon (Vorarlberg)<br />
**Skigebiet Karte: https://winter.intermaps.com/montafon?lang=de<br />
**Skipass Preise: https://www.montafon.at/de/Service/Bergbahn-Preise-Tickets/Mehrtageskarte-Winter<br />
**Wer nicht alle 7 Tage Skifahren möchte: Es gibt Angebote wie z.B. 5 aus 7, mit einem solchen Skipass kann man innerhalb von 7 Tagen an 5 beliebigen Tagen skifahren.<br />
<br />
*Skiverleih: [http://www.sportharry.at/ Sport Harry], direkt an der Talstation.<br />
<br />
== Unterkunft ==<br />
=== Aufteilung ===<br />
<br />
Dieses Jahr gibt es nur die Hütte mit folgenden Zimmern (wir sind insgesamt 14, also bleiben keine Plätze frei).<br />
<br />
{| class="wikitable"<br />
! scope="col" | Zimmer<br />
! scope="col" | Insassen<br />
|-<br />
! scope="row" | 2-er<br />
| Amy, Aafia<br />
<br />
|-<br />
! scope="row" | 3-er<br />
| Jakob, Max, Katya<br />
|-<br />
! scope="row" | 4-er<br />
| Eva, Steffen, Flo, Christian<br />
|-<br />
! scope="row" | 5-er<br />
| Basti<br />
|}<br />
<br />
== Fahrer ==<br />
Fahrer können sich hier eintragen und Eckdaten angeben. Diejenige, die mitfahren möchten, sprechen sich mit den Fahrern ab und tragen sich ebenfalls ein. Klärt bitte auch die Gepäcklage, ggf. kann ein anderer Fahrer z.B. Skiausrüstung mitnehmen.<br />
<br />
'''Auf der Suche nach einer Mitfahrgelegenheit: '''<br />
<br />
{| class="wikitable"<br />
! scope="col" | Fahrer<br />
! scope="col" | # Plätze<br />
! scope="col" | Anreise<br />
! scope="col" | Mitfahrer Anreise<br />
! scope="col" | Abreise<br />
! scope="col" | Mitfahrer Abreise<br />
! scope="col" | Skimitnahme<br />
! scope="col" | Kommentar<br />
|-<br />
|Eva<br />
|0(3)<br />
|Samstag<br />
|Steffen+Christian<br />
|Samstag?<br />
|Steffen+Christian<br />
|eigene ja<br />
|Auto ist voll<br />
|-<br />
|Max<br />
|0(3)<br />
|Samstag<br />
|Jakob+Katya<br />
|Samstag<br />
|Jakob+Katya<br />
|eigene ja<br />
|Auto ist voll<br />
|-<br />
|Basti<br />
|3(3)<br />
|<br />
|Sonntag ganz frueh<br />
|<br />
|Freitag Mittag<br />
|Ja<br />
|<br />
|-<br />
<br />
|-<br />
<br />
|-<br />
<br />
|-<br />
|}<br />
<br />
== Schwarzes Brett ==<br />
<br />
=== Essen ===<br />
Hier gibt's den Essensplan, soweit übernommen von den letzten Jahren. Da wir sehr viele Leute sind, sollten wir wieder vorab grob planen, was wir kochen wollen. Wir können einige Dinge in Deutschland besorgen und mitnehmen, es gib aber auch einen nahe gelegen Supermarkt. Generelle Dinge, wie z.B. Gewürze oder Aufstriche für Frühstücksbrötchen könnte man auch mitbringen.<br />
<br />
Wer Kochvorschläge oder andere Ideen hat, gerne unten eintragen und kommentieren. Bedenkt, dass die Zubereitung (relativ) einfach sein sollte. Da sich die Liste in den letzten Jahren gut bewaehrt hat, behalten wir sie bei, aber die Reihenfolge/Tage kann man sicher noch hin und her bewegen. Einige (wenige) Sachen müssen wir dann manchmal trotzdem noch beim Spar kaufen, da ja auch der Kühlschrank nur begrenzt Platz hat.<br />
<br />
<br />
'''Essenplan:'''<br />
*Samstag: Spaghetti Bolognese + veg. Bolognese<br />
*Sonntag: Käse Spätzle<br />
*Montag: Burritos<br />
*Dienstag: Risotto<br />
*Mittwoch: Gulasch<br />
*Donnerstag: Kartoffeln mit Kräuter-Quark<br />
*Freitag: Dal mit Reis (od. Curry mit Suesskartoffeln )<br />
<br />
<br />
Einkaufsliste von 2022 (letztes Jahr): https://docs.google.com/document/d/166b_mIXxmoAgcv1upIGremd_36E1fmwwxsQADP2uEv8/<br />
<br />
@Christian: Vegane Option entwender selbst mitbringen oder hier mit reinschreiben (falls man die einfach bekommt)<br />
<br />
=== Alkohol/Party ===<br />
<br />
=== Spiele ===<br />
<br />
Immer lustig sind gemeinsame Spieleabende. Also, wer im Besitz von Gesellschaftsspielen ist, bringt diese gerne mit! Um eine Übersicht zu bekommen auch bitte hier eintragen:<br />
<br />
''* Spiele hier eintragen''<br />
<br />
=== Sonstiges ===<br />
<br />
=== Alternativprogramm ===<br />
* Rodeln (http://www.montafon.at/de/urlaubswelten/echte_naturliebhaber/rodeln)<br />
* Schneeschuhwandern (http://www.montafon.at/schneeschuhwanderungen)<br />
* Therme (http://www.montafon.at/schwimmen), z.B. http://www.aqua-dome.at/de (ca. 130km entfernt!)<br />
<br />
[[Category:Internal]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Skivergnuegen_2023&diff=2750Skivergnuegen 20232023-01-03T09:33:08Z<p>Gokus: /* Aufteilung */</p>
<hr />
<div>'''Remeis Skifahren 2023''' The same procedure as every year ;-) <br />
Hier werden die wichtigsten organisatorischen Dinge festgehalten. Jeder darf hier gerne editieren; wer keinen Account für das Wiki hat, muss sich bitte an jemanden wenden, der Zugang hat.<br />
<br />
[[File:154854 silvretta-montafon-panorama.jpeg|right|700px]]<br />
<br />
== Allgemeine Infos ==<br />
<br />
*Ferienwohnung(en):<br />
**Wir kommen wieder unter im [https://www.grandau.at/montafon/zimmer/ferienhaus-enzian Haus Enzian] des [https://www.grandau.at/ Sporthotels Grandau].<br />
**Adresse des Sporthotel: Montafonerstraße 274a, 6791 St. Gallenkirch, Österreich. Unsere Unterkünfte liegen im Türkeiweg.<br />
<br />
*Skigebiet: Montafon (Vorarlberg)<br />
**Skigebiet Karte: https://winter.intermaps.com/montafon?lang=de<br />
**Skipass Preise: https://www.montafon.at/de/Service/Bergbahn-Preise-Tickets/Mehrtageskarte-Winter<br />
**Wer nicht alle 7 Tage Skifahren möchte: Es gibt Angebote wie z.B. 5 aus 7, mit einem solchen Skipass kann man innerhalb von 7 Tagen an 5 beliebigen Tagen skifahren.<br />
<br />
*Skiverleih: [http://www.sportharry.at/ Sport Harry], direkt an der Talstation.<br />
<br />
== Unterkunft ==<br />
=== Aufteilung ===<br />
<br />
Dieses Jahr gibt es nur die Hütte mit folgenden Zimmern (wir sind insgesamt 14, also bleiben keine Plätze frei).<br />
<br />
{| class="wikitable"<br />
! scope="col" | Zimmer<br />
! scope="col" | Insassen<br />
|-<br />
! scope="row" | 2-er<br />
| Amy, Aafia<br />
<br />
|-<br />
! scope="row" | 3-er<br />
| Jakob, Max, Katya<br />
|-<br />
! scope="row" | 4-er<br />
| Eva, Steffen, Flo, Christian<br />
|-<br />
! scope="row" | 5-er<br />
| Basti<br />
|}<br />
<br />
== Fahrer ==<br />
Fahrer können sich hier eintragen und Eckdaten angeben. Diejenige, die mitfahren möchten, sprechen sich mit den Fahrern ab und tragen sich ebenfalls ein. Klärt bitte auch die Gepäcklage, ggf. kann ein anderer Fahrer z.B. Skiausrüstung mitnehmen.<br />
<br />
'''Auf der Suche nach einer Mitfahrgelegenheit: '''<br />
<br />
{| class="wikitable"<br />
! scope="col" | Fahrer<br />
! scope="col" | # Plätze<br />
! scope="col" | Anreise<br />
! scope="col" | Mitfahrer Anreise<br />
! scope="col" | Abreise<br />
! scope="col" | Mitfahrer Abreise<br />
! scope="col" | Skimitnahme<br />
! scope="col" | Kommentar<br />
|-<br />
|Eva<br />
|0(3)<br />
|Samstag<br />
|Steffen+Christian<br />
|Samstag?<br />
|Steffen+Christian<br />
|eigene ja<br />
|Auto ist voll<br />
|-<br />
|Max<br />
|0(3)<br />
|Samstag<br />
|Jakob+Katya<br />
|Samstag<br />
|Jakob+Katya<br />
|eigene ja<br />
|Auto ist voll<br />
|-<br />
<br />
|-<br />
<br />
|-<br />
<br />
|-<br />
<br />
|-<br />
|}<br />
<br />
== Schwarzes Brett ==<br />
<br />
=== Essen ===<br />
Hier gibt's den Essensplan, soweit übernommen von den letzten Jahren. Da wir sehr viele Leute sind, sollten wir wieder vorab grob planen, was wir kochen wollen. Wir können einige Dinge in Deutschland besorgen und mitnehmen, es gib aber auch einen nahe gelegen Supermarkt. Generelle Dinge, wie z.B. Gewürze oder Aufstriche für Frühstücksbrötchen könnte man auch mitbringen.<br />
<br />
Wer Kochvorschläge oder andere Ideen hat, gerne unten eintragen und kommentieren. Bedenkt, dass die Zubereitung (relativ) einfach sein sollte. Da sich die Liste in den letzten Jahren gut bewaehrt hat, behalten wir sie bei, aber die Reihenfolge/Tage kann man sicher noch hin und her bewegen. Einige (wenige) Sachen müssen wir dann manchmal trotzdem noch beim Spar kaufen, da ja auch der Kühlschrank nur begrenzt Platz hat.<br />
<br />
<br />
'''Essenplan:'''<br />
*Samstag: Spaghetti Bolognese + veg. Bolognese<br />
*Sonntag: Käse Spätzle<br />
*Montag: Burritos<br />
*Dienstag: Risotto<br />
*Mittwoch: Gulasch<br />
*Donnerstag: Kartoffeln mit Kräuter-Quark<br />
*Freitag: Dal mit Reis (od. Curry mit Suesskartoffeln )<br />
<br />
<br />
Einkaufsliste von 2022 (letztes Jahr): https://docs.google.com/document/d/166b_mIXxmoAgcv1upIGremd_36E1fmwwxsQADP2uEv8/<br />
<br />
@Christian: Vegane Option entwender selbst mitbringen oder hier mit reinschreiben (falls man die einfach bekommt)<br />
<br />
=== Alkohol/Party ===<br />
<br />
=== Spiele ===<br />
<br />
Immer lustig sind gemeinsame Spieleabende. Also, wer im Besitz von Gesellschaftsspielen ist, bringt diese gerne mit! Um eine Übersicht zu bekommen auch bitte hier eintragen:<br />
<br />
''* Spiele hier eintragen''<br />
<br />
=== Sonstiges ===<br />
<br />
=== Alternativprogramm ===<br />
* Rodeln (http://www.montafon.at/de/urlaubswelten/echte_naturliebhaber/rodeln)<br />
* Schneeschuhwandern (http://www.montafon.at/schneeschuhwanderungen)<br />
* Therme (http://www.montafon.at/schwimmen), z.B. http://www.aqua-dome.at/de (ca. 130km entfernt!)<br />
<br />
[[Category:Internal]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Install_ISIS_on_Linux&diff=2623Install ISIS on Linux2022-07-21T08:30:27Z<p>Gokus: </p>
<hr />
<div>These instructions were tested for Debian 10, Ubuntu 20.04, and openSUSE 15.3. They should help you to install both the X-ray isisscripts as well as the "stellar_isisscripts", but will focus on the latter. It also assumes that you use the bash shell. Using the csh requires minor changes.<br />
<br />
If you prefer an easy way to run the X-ray isisscripts (without the "stellar_isisscripts"), you can use the singularity container:<br />
<br />
https://www.sternwarte.uni-erlangen.de/wiki/index.php/Isis_tutorial_installing<br />
<br />
== 1. Install basic dependencies and general things ==<br />
<br />
If you don't have root access, you might have to build some things from source, e.g. pgplot, cfitsio, or libpng. <br />
However, it might be easier to rely on HEASOFT in this case. <br />
<br />
<pre><br />
sudo apt-get -y install libreadline-dev<br />
sudo apt-get -y install libcurl4<br />
sudo apt-get -y install libcurl4-gnutls-dev<br />
sudo apt-get -y install libncurses5-dev<br />
sudo apt-get -y install xorg-dev<br />
sudo apt-get -y install gcc g++ gfortran<br />
sudo apt-get -y install perl-modules-5*<br />
sudo apt-get -y install python3-dev<br />
sudo apt-get -y install fig2dev<br />
sudo apt-get -y install libpng-dev <br />
sudo apt-get -y install zlib1g-dev<br />
sudo apt-get -y install libpcre3-dev<br />
sudo apt-get -y install libonig-dev<br />
sudo apt-get -y install pgplot5<br />
sudo apt-get -y install wget<br />
sudo apt-get -y install git<br />
</pre> <br />
<br />
You have to define some things in your ~/.bashrc:<br />
<br />
<pre><br />
export CC=/usr/bin/gcc<br />
export CXX=/usr/bin/g++<br />
export FC=/usr/bin/gfortran<br />
export PERL=/usr/bin/perl<br />
export PYTHON=/usr/bin/python3<br />
</pre> <br />
<br />
Then do <br />
<br />
<pre><br />
source ~/.bashrc<br />
</pre><br />
<br />
== 2. Install HEASOFT ==<br />
<br />
This is ''not'' required for the "stellar_isisscripts" and can be skipped if you only want to install these. That is, unless you have no other way to install pgplot and cfitsio. HEASOFT seems to be required to run the X-ray isisscripts properly. To download HEASOFT, go to:<br />
<br />
https://heasarc.gsfc.nasa.gov/lheasoft/download.html<br />
<br />
unpack the download, and then:<br />
<br />
<pre><br />
cd heasoft-6.29/BUILD_DIR<br />
</pre><br />
<br />
Before I could run the configure script I had to make many files executable, <br />
so it seems easiest to make all of them executable (this is not good practice):<br />
<br />
<pre><br />
find .. -name '*' | xargs chmod +x<br />
</pre><br />
<br />
Building HEASOFT will take a while (~>30min). If you don't want to use root, set a different installation location by using ./configure with <tt>'''--prefix=/path/to/installation/folder/'''</tt>. <br />
<br />
<pre><br />
./configure<br />
make<br />
sudo make install<br />
find .. -name 'headas-init.sh' | xargs chmod +x<br />
</pre><br />
<br />
Finally, you need to define $HEASOFT. For me, this meant editing ~/.bashrc to <br />
include the two following lines. You need to adjust the path of course:<br />
<br />
<pre><br />
export HEADAS=/home/matti/programs/heasoft-6.29/x86_64-pc-linux-gnu-libc2.31<br />
. $HEADAS/headas-init.sh<br />
# or<br />
# . $HEADAS/headas-init.sh<br />
</pre><br />
<br />
for csh, use instead:<br />
<br />
<pre><br />
setenv HEADAS /home/volans/mdorsch/isis/heasoft/installation/x86_64-pc-linux-gnu-libc2.31<br />
. $HEADAS/headas-init.csh<br />
<br />
</pre><br />
<br />
Then, in the terminal:<br />
<br />
<pre><br />
source ~/.bashrc<br />
# or source ~/.cshrc<br />
</pre><br />
<br />
== 3. Install slang / slang modules / isis / isisscripts ==<br />
<br />
First get the latest versions of slang / jed/ slgsl / slxfig / slirp / isis / isisscripts. <br />
A scripted installation of slang and isis is possible, following the MIT guide:<br />
<br />
https://space.mit.edu/cxc/isis/install.html#XI<br />
<br />
However, I recommend using the git repositories. Go to the Software directory in your terminal and clone the repositories:<br />
<pre><br />
git clone https://github.com/houckj/isis<br />
git clone git://git.jedsoft.org/git/slang.git<br />
git clone git://git.jedsoft.org/git/slxfig.git<br />
git clone git://git.jedsoft.org/git/slgsl.git<br />
git clone git://git.jedsoft.org/git/jed.git<br />
git clone git://git.jedsoft.org/git/slirp.git<br />
git clone http://www.sternwarte.uni-erlangen.de/git.public/isisscripts <br />
</pre> <br />
<br />
The following will assume you want to use root access. If that is not the case, you can use use the option<br />
<br />
<pre><br />
--prefix=/path/to/installation/folder/<br />
</pre><br />
<br />
With the ./configure commands. It seems to work to install all of them to the same "--prefix". <br />
<br />
Install slang. Go to the slang directory.<br />
<pre><br />
sh ./configure<br />
make<br />
sudo make install<br />
</pre><br />
Make sure that all necessary standard modules are installed, including "png", which requires "libpng16" to be installed. <br />
If "libpng16" is not installed yet, you can easily build it from source:<br />
<br />
https://sourceforge.net/projects/libpng/files/<br />
<br />
and then add its installation folder to $PATH.<br />
<br />
Install isis. Go to the isis directory, here with HEADAS:<br />
<pre><br />
sh ./configure --with-headas=$HEADAS<br />
make<br />
sudo make install<br />
</pre><br />
If your aim is to install the "stellar_isisscripts", you don't need HEADAS, but may have to install fitsio:<br />
<br />
https://heasarc.gsfc.nasa.gov/fitsio/<br />
<br />
Then go to the isis directory and:<br />
<br />
<pre><br />
sh ./configure<br />
make<br />
sudo make install<br />
</pre><br />
<br />
Add <tt>'''--with-slang=/path/to/slang'''</tt> to the following configuration set ups in case you installed slang in a different path other than <tt>/usr/local/</tt><br />
<br />
Install jed. Go to the jed directory<br />
<pre><br />
sh ./configure<br />
make<br />
make xjed<br />
sudo make install<br />
</pre><br />
<br />
Install slgsl. Go to the slgsl directory<br />
<pre><br />
sh ./configure<br />
make<br />
sudo make install<br />
</pre><br />
<br />
Install slxfig. Go to the slxfig directory<br />
<pre><br />
sh ./configure<br />
make<br />
sudo make install<br />
</pre><br />
<br />
Install slirp. Go to the slirp directory<br />
<pre><br />
sh ./configure<br />
make<br />
sudo make install <br />
</pre><br />
<br />
Before building the isisscripts, install the "SLURP" perl module. <br />
<pre><br />
sudo perl -MCPAN -e shell<br />
install CPAN<br />
reload cpan<br />
install File::Slurp<br />
quit<br />
</pre><br />
<br />
Then go to the isisscripts directory and<br />
<pre><br />
make<br />
</pre><br />
<br />
Create / modify the [[ISIS auto start (.isisrc)|.isisrc]] file in your <tt>$HOME</tt> directory and add the following line<br />
<pre><br />
add_to_isis_load_path("/Path/To/isisscripts/share/");<br />
define ris() { require("isisscripts"); };<br />
</pre><br />
This allows the easy access to <tt>require("isisscripts")</tt> in isis.<br />
<br />
== 4. Install stellar_isisscripts ==<br />
<br />
Get the newest version of the stellar_isisscripts from git.<br />
If you have a Remeis/gitlab account and are at the Remeis observatory, do:<br />
<pre><br />
git clone git@serpens.sternwarte.uni-erlangen.de:irrgang/stellar.git<br />
mv stellar stellar_isisscripts<br />
</pre><br />
If you have a Remeis/gitlab account and are not at the Remeis observatory, do:<br />
<pre><br />
git clone http://www.sternwarte.uni-erlangen.de/gitlab/irrgang/stellar.git<br />
mv stellar stellar_isisscripts<br />
</pre><br />
Otherwise ask Matti Dorsch. Then:<br />
<pre><br />
cd stellar_isisscripts<br />
make <br />
</pre><br />
<br />
Now compile some necessary C functions:<br />
<pre><br />
cd stellar_isisscripts/slirp<br />
slirp -make -lm -lgsl -lgslcblas -lpthread c_functions.h c_functions.o<br />
sed -i -e 's/^CFLAGS[[:space:]]*= -g -O2/CFLAGS = -g -Ofast -Wall -Wextra/g' Makefile <br />
</pre><br />
<br />
Then:<br />
<pre><br />
make<br />
make test<br />
rm c_functions-test.sl ; rm c_functions.o ; rm c_functions_glue.o ; rm c_functions_glue.c ; rm Makefile ; rm *~ <br />
</pre><br />
<br />
Create modify the [[ISIS auto start (.isisrc)|.isisrc]] file in your <tt>$HOME</tt> directory and add the following line<br />
<pre><br />
add_to_isis_load_path("/Path/To/stellar_isisscripts/share/");<br />
define rmy() { require("stellar_isisscripts"); };<br />
</pre><br />
<br />
To use all functions, you need the stilts tool, which is related to TOPCAT. You can install it as a package:<br />
<pre><br />
sudo apt-get install stilts<br />
</pre><br />
<br />
Only if this does not work:<br />
<br />
<pre><br />
mkdir stilts; cd stilts<br />
wget http://www.star.bris.ac.uk/~mbt/stilts/stilts.jar<br />
wget 'http://www.star.bris.ac.uk/~mbt/stilts/stilts'<br />
chmod +x stilts<br />
sudo cp * /usr/bin/<br />
</pre><br />
<br />
Without root, you can add the folder that contains stilts to your $PATH in ~/.bashrc or ~/.cshrc.<br />
<br />
To test if the scripts work, try for example:<br />
<pre><br />
isis<br />
rmy;<br />
s = query_photometry ("HZ44");<br />
s.print();<br />
help query_photometry<br />
q<br />
exit;<br />
</pre></div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Install_ISIS_on_Linux&diff=2622Install ISIS on Linux2022-07-20T19:52:25Z<p>Gokus: </p>
<hr />
<div>These instructions were tested for Debian 10, Ubuntu 20.04, and openSUSE 15.3. They should help you to install both the X-ray isisscripts as well as the "stellar_isisscripts", but will focus on the latter. It also assumes that you use the bash shell. Using the csh requires minor changes.<br />
<br />
If you prefer an easy way to run the X-ray isisscripts (without the "stellar_isisscripts"), you can use the singularity container:<br />
<br />
https://www.sternwarte.uni-erlangen.de/wiki/index.php/Isis_tutorial_installing<br />
<br />
== 1. Install basic dependencies and general things ==<br />
<br />
If you don't have root access, you might have to build some things from source, e.g. pgplot, cfitsio, or libpng. <br />
However, it might be easier to rely on HEASOFT in this case. <br />
<br />
<pre><br />
sudo apt-get -y install libreadline-dev<br />
sudo apt-get -y install libcurl4<br />
sudo apt-get -y install libcurl4-gnutls-dev<br />
sudo apt-get -y install libncurses5-dev<br />
sudo apt-get -y install libgsl-dev (at least for Ubuntu 22.04)<br />
sudo apt-get -y install xorg-dev<br />
sudo apt-get -y install gcc g++ gfortran<br />
sudo apt-get -y install perl-modules-5*<br />
sudo apt-get -y install python3-dev<br />
sudo apt-get -y install fig2dev<br />
sudo apt-get -y install libpng-dev <br />
sudo apt-get -y install zlib1g-dev<br />
sudo apt-get -y install libpcre3-dev<br />
sudo apt-get -y install libonig-dev<br />
sudo apt-get -y install pgplot5<br />
sudo apt-get -y install wget<br />
sudo apt-get -y install git<br />
</pre> <br />
<br />
You have to define some things in your ~/.bashrc:<br />
<br />
<pre><br />
export CC=/usr/bin/gcc<br />
export CXX=/usr/bin/g++<br />
export FC=/usr/bin/gfortran<br />
export PERL=/usr/bin/perl<br />
export PYTHON=/usr/bin/python3<br />
</pre> <br />
<br />
Then do <br />
<br />
<pre><br />
source ~/.bashrc<br />
</pre><br />
<br />
== 2. Install HEASOFT ==<br />
<br />
This is ''not'' required for the "stellar_isisscripts" and can be skipped if you only want to install these. That is, unless you have no other way to install pgplot and cfitsio. HEASOFT seems to be required to run the X-ray isisscripts properly. To download HEASOFT, go to:<br />
<br />
https://heasarc.gsfc.nasa.gov/lheasoft/download.html<br />
<br />
unpack the download, and then:<br />
<br />
<pre><br />
cd heasoft-6.29/BUILD_DIR<br />
</pre><br />
<br />
Before I could run the configure script I had to make many files executable, <br />
so it seems easiest to make all of them executable (this is not good practice):<br />
<br />
<pre><br />
find .. -name '*' | xargs chmod +x<br />
</pre><br />
<br />
Building HEASOFT will take a while (~>30min). If you don't want to use root, set a different installation location by using ./configure with <tt>'''--prefix=/path/to/installation/folder/'''</tt>. <br />
<br />
<pre><br />
./configure<br />
make<br />
sudo make install<br />
find .. -name 'headas-init.sh' | xargs chmod +x<br />
</pre><br />
<br />
Finally, you need to define $HEASOFT. For me, this meant editing ~/.bashrc to <br />
include the two following lines. You need to adjust the path of course:<br />
<br />
<pre><br />
export HEADAS=/home/matti/programs/heasoft-6.29/x86_64-pc-linux-gnu-libc2.31<br />
. $HEADAS/headas-init.sh<br />
# or<br />
# . $HEADAS/headas-init.sh<br />
</pre><br />
<br />
for csh, use instead:<br />
<br />
<pre><br />
setenv HEADAS /home/volans/mdorsch/isis/heasoft/installation/x86_64-pc-linux-gnu-libc2.31<br />
. $HEADAS/headas-init.csh<br />
<br />
</pre><br />
<br />
Then, in the terminal:<br />
<br />
<pre><br />
source ~/.bashrc<br />
# or source ~/.cshrc<br />
</pre><br />
<br />
== 3. Install slang / slang modules / isis / isisscripts ==<br />
<br />
First get the latest versions of slang / jed/ slgsl / slxfig / slirp / isis / isisscripts. <br />
A scripted installation of slang and isis is possible, following the MIT guide:<br />
<br />
https://space.mit.edu/cxc/isis/install.html#XI<br />
<br />
However, I recommend using the git repositories. Go to the Software directory in your terminal and clone the repositories:<br />
<pre><br />
git clone https://github.com/houckj/isis<br />
git clone git://git.jedsoft.org/git/slang.git<br />
git clone git://git.jedsoft.org/git/slxfig.git<br />
git clone git://git.jedsoft.org/git/slgsl.git<br />
git clone git://git.jedsoft.org/git/jed.git<br />
git clone git://git.jedsoft.org/git/slirp.git<br />
git clone http://www.sternwarte.uni-erlangen.de/git.public/isisscripts <br />
</pre> <br />
<br />
The following will assume you want to use root access. If that is not the case, you can use use the option<br />
<br />
<pre><br />
--prefix=/path/to/installation/folder/<br />
</pre><br />
<br />
With the ./configure commands. It seems to work to install all of them to the same "--prefix". <br />
<br />
Install slang. Go to the slang directory.<br />
<pre><br />
sh ./configure<br />
make<br />
sudo make install<br />
</pre><br />
Make sure that all necessary standard modules are installed, including "png", which requires "libpng16" to be installed. <br />
If "libpng16" is not installed yet, you can easily build it from source:<br />
<br />
https://sourceforge.net/projects/libpng/files/<br />
<br />
and then add its installation folder to $PATH.<br />
<br />
Install isis. Go to the isis directory, here with HEADAS:<br />
<pre><br />
sh ./configure --with-headas=$HEADAS<br />
make<br />
sudo make install<br />
</pre><br />
If your aim is to install the "stellar_isisscripts", you don't need HEADAS, but may have to install fitsio:<br />
<br />
https://heasarc.gsfc.nasa.gov/fitsio/<br />
<br />
Then go to the isis directory and:<br />
<br />
<pre><br />
sh ./configure<br />
make<br />
sudo make install<br />
</pre><br />
<br />
Add <tt>'''--with-slang=/path/to/slang'''</tt> to the following configuration set ups in case you installed slang in a different path other than <tt>/usr/local/</tt><br />
<br />
Install jed. Go to the jed directory<br />
<pre><br />
sh ./configure<br />
make<br />
make xjed<br />
sudo make install<br />
</pre><br />
<br />
Install slgsl. Go to the slgsl directory<br />
<pre><br />
sh ./configure<br />
make<br />
sudo make install<br />
</pre><br />
<br />
Install slxfig. Go to the slxfig directory<br />
<pre><br />
sh ./configure<br />
make<br />
sudo make install<br />
</pre><br />
<br />
Install slirp. Go to the slirp directory<br />
<pre><br />
sh ./configure<br />
make<br />
sudo make install <br />
</pre><br />
<br />
Before building the isisscripts, install the "SLURP" perl module. <br />
<pre><br />
sudo perl -MCPAN -e shell<br />
install CPAN<br />
reload cpan<br />
install File::Slurp<br />
quit<br />
</pre><br />
<br />
Then go to the isisscripts directory and<br />
<pre><br />
make<br />
</pre><br />
<br />
Create / modify the [[ISIS auto start (.isisrc)|.isisrc]] file in your <tt>$HOME</tt> directory and add the following line<br />
<pre><br />
add_to_isis_load_path("/Path/To/isisscripts/share/");<br />
define ris() { require("isisscripts"); };<br />
</pre><br />
This allows the easy access to <tt>require("isisscripts")</tt> in isis.<br />
<br />
== 4. Install stellar_isisscripts ==<br />
<br />
Get the newest version of the stellar_isisscripts from git.<br />
If you have a Remeis/gitlab account and are at the Remeis observatory, do:<br />
<pre><br />
git clone git@serpens.sternwarte.uni-erlangen.de:irrgang/stellar.git<br />
mv stellar stellar_isisscripts<br />
</pre><br />
If you have a Remeis/gitlab account and are not at the Remeis observatory, do:<br />
<pre><br />
git clone http://www.sternwarte.uni-erlangen.de/gitlab/irrgang/stellar.git<br />
mv stellar stellar_isisscripts<br />
</pre><br />
Otherwise ask Matti Dorsch. Then:<br />
<pre><br />
cd stellar_isisscripts<br />
make <br />
</pre><br />
<br />
Now compile some necessary C functions:<br />
<pre><br />
cd stellar_isisscripts/slirp<br />
slirp -make -lm -lgsl -lgslcblas -lpthread c_functions.h c_functions.o<br />
sed -i -e 's/^CFLAGS[[:space:]]*= -g -O2/CFLAGS = -g -Ofast -Wall -Wextra/g' Makefile <br />
</pre><br />
<br />
Then:<br />
<pre><br />
make<br />
make test<br />
rm c_functions-test.sl ; rm c_functions.o ; rm c_functions_glue.o ; rm c_functions_glue.c ; rm Makefile ; rm *~ <br />
</pre><br />
<br />
Create modify the [[ISIS auto start (.isisrc)|.isisrc]] file in your <tt>$HOME</tt> directory and add the following line<br />
<pre><br />
add_to_isis_load_path("/Path/To/stellar_isisscripts/share/");<br />
define rmy() { require("stellar_isisscripts"); };<br />
</pre><br />
<br />
To use all functions, you need the stilts tool, which is related to TOPCAT. You can install it as a package:<br />
<pre><br />
sudo apt-get install stilts<br />
</pre><br />
<br />
Only if this does not work:<br />
<br />
<pre><br />
mkdir stilts; cd stilts<br />
wget http://www.star.bris.ac.uk/~mbt/stilts/stilts.jar<br />
wget 'http://www.star.bris.ac.uk/~mbt/stilts/stilts'<br />
chmod +x stilts<br />
sudo cp * /usr/bin/<br />
</pre><br />
<br />
Without root, you can add the folder that contains stilts to your $PATH in ~/.bashrc or ~/.cshrc.<br />
<br />
To test if the scripts work, try for example:<br />
<pre><br />
isis<br />
rmy;<br />
s = query_photometry ("HZ44");<br />
s.print();<br />
help query_photometry<br />
q<br />
exit;<br />
</pre></div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2509Fermi-LAT2022-05-19T09:38:11Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/home/X-ray/Fermi/archive/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/home/X-ray/Fermi/archive/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /home/X-ray/Fermi/archive/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/home/X-ray/Fermi/archive/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/home/X-ray/Fermi/archive/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
<br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. I recommend to write a script to create the .slurm file that you want to submit to slurm. Here's an example how this array job can look like:<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --job-name LATspectra<br />
#SBATCH --output job.out-%a<br />
#SBATCH --error err.out-%a<br />
#SBATCH --time 48:00:00<br />
#SBATCH --partition=remeis<br />
#SBATCH --ntasks=1<br />
#SBATCH --hint=nomultithread<br />
#SBATCH --mem-per-cpu=4G<br />
#SBATCH --mail-type=END,FAIL,ARRAY_TASKS<br />
#SBATCH --mail-user=my@email.de<br />
#SBATCH --array 0-XXX%YY (XXX: number of tasks in array, YY: number of jobs running simultaneously<br />
<br />
COMMAND[0]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJXXX.X-XXXX mjd_start mjd_stop"<br />
COMMAND[1]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJYYY.Y-YYYY mjd_start mjd_stop"<br />
COMMAND[2]="..."<br />
...<br />
</pre><br />
<br />
'''Can the scripts find possibly unknown gamma-ray sources by itself?''' - Yes, when running the analysis script to create a spectrum (with make_spectrum.sl), there is a routine implemented (fermipy function ''find_sources''), which searches for excesses in the TS map with a significance of 5 sigma or higher. The coordinates of this found source is then saved in the folder, in which the analysis is run, to allow further inspections and a follow-up analyis.<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code, my work-around for this issue is implemented in the wrapper scripts ''make_spectrum.sl'' and ''make_lightcurve.sl''. Deleting the subfolders is now handled in there and not via the routine in the source code. <br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None<br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
If you encounter a new error, you can also go ahead and start the python script for the analysis directly in the shell. Remember to load your conda environment first. You also need a ''config_SED.yaml'' or ''config_LC.yaml'' file in the folder where you are going to run the analysis, otherwise you won't get far. You can directly run the script, e.g. via<br />
<pre><br />
(fermipy)[user@computer]> $FERMITOOLS/SEDcreator_fermipy.py<br />
</pre><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: September 27, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2508Fermi-LAT2022-05-19T09:37:13Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/home/X-ray/Fermi/archive/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/home/X-ray/Fermi/archive//spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /home/X-ray/Fermi/archive/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/home/X-ray/Fermi/archive/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/home/X-ray/Fermi/archive/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
<br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. I recommend to write a script to create the .slurm file that you want to submit to slurm. Here's an example how this array job can look like:<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --job-name LATspectra<br />
#SBATCH --output job.out-%a<br />
#SBATCH --error err.out-%a<br />
#SBATCH --time 48:00:00<br />
#SBATCH --partition=remeis<br />
#SBATCH --ntasks=1<br />
#SBATCH --hint=nomultithread<br />
#SBATCH --mem-per-cpu=4G<br />
#SBATCH --mail-type=END,FAIL,ARRAY_TASKS<br />
#SBATCH --mail-user=my@email.de<br />
#SBATCH --array 0-XXX%YY (XXX: number of tasks in array, YY: number of jobs running simultaneously<br />
<br />
COMMAND[0]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJXXX.X-XXXX mjd_start mjd_stop"<br />
COMMAND[1]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJYYY.Y-YYYY mjd_start mjd_stop"<br />
COMMAND[2]="..."<br />
...<br />
</pre><br />
<br />
'''Can the scripts find possibly unknown gamma-ray sources by itself?''' - Yes, when running the analysis script to create a spectrum (with make_spectrum.sl), there is a routine implemented (fermipy function ''find_sources''), which searches for excesses in the TS map with a significance of 5 sigma or higher. The coordinates of this found source is then saved in the folder, in which the analysis is run, to allow further inspections and a follow-up analyis.<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code, my work-around for this issue is implemented in the wrapper scripts ''make_spectrum.sl'' and ''make_lightcurve.sl''. Deleting the subfolders is now handled in there and not via the routine in the source code. <br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None<br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
If you encounter a new error, you can also go ahead and start the python script for the analysis directly in the shell. Remember to load your conda environment first. You also need a ''config_SED.yaml'' or ''config_LC.yaml'' file in the folder where you are going to run the analysis, otherwise you won't get far. You can directly run the script, e.g. via<br />
<pre><br />
(fermipy)[user@computer]> $FERMITOOLS/SEDcreator_fermipy.py<br />
</pre><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: September 27, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2241Fermi-LAT2021-09-27T16:52:04Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
<br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. I recommend to write a script to create the .slurm file that you want to submit to slurm. Here's an example how this array job can look like:<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --job-name LATspectra<br />
#SBATCH --output job.out-%a<br />
#SBATCH --error err.out-%a<br />
#SBATCH --time 48:00:00<br />
#SBATCH --partition=remeis<br />
#SBATCH --ntasks=1<br />
#SBATCH --hint=nomultithread<br />
#SBATCH --mem-per-cpu=4G<br />
#SBATCH --mail-type=END,FAIL,ARRAY_TASKS<br />
#SBATCH --mail-user=my@email.de<br />
#SBATCH --array 0-XXX%YY (XXX: number of tasks in array, YY: number of jobs running simultaneously<br />
<br />
COMMAND[0]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJXXX.X-XXXX mjd_start mjd_stop"<br />
COMMAND[1]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJYYY.Y-YYYY mjd_start mjd_stop"<br />
COMMAND[2]="..."<br />
...<br />
</pre><br />
<br />
'''Can the scripts find possibly unknown gamma-ray sources by itself?''' - Yes, when running the analysis script to create a spectrum (with make_spectrum.sl), there is a routine implemented (fermipy function ''find_sources''), which searches for excesses in the TS map with a significance of 5 sigma or higher. The coordinates of this found source is then saved in the folder, in which the analysis is run, to allow further inspections and a follow-up analyis.<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code, my work-around for this issue is implemented in the wrapper scripts ''make_spectrum.sl'' and ''make_lightcurve.sl''. Deleting the subfolders is now handled in there and not via the routine in the source code. <br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None<br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
If you encounter a new error, you can also go ahead and start the python script for the analysis directly in the shell. Remember to load your conda environment first. You also need a ''config_SED.yaml'' or ''config_LC.yaml'' file in the folder where you are going to run the analysis, otherwise you won't get far. You can directly run the script, e.g. via<br />
<pre><br />
(fermipy)[user@computer]> $FERMITOOLS/SEDcreator_fermipy.py<br />
</pre><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: September 27, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2240Fermi-LAT2021-09-27T16:51:35Z<p>Gokus: /* Q&A */</p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
<br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. I recommend to write a script to create the .slurm file that you want to submit to slurm. Here's an example how this array job can look like:<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --job-name LATspectra<br />
#SBATCH --output job.out-%a<br />
#SBATCH --error err.out-%a<br />
#SBATCH --time 48:00:00<br />
#SBATCH --partition=remeis<br />
#SBATCH --ntasks=1<br />
#SBATCH --hint=nomultithread<br />
#SBATCH --mem-per-cpu=4G<br />
#SBATCH --mail-type=END,FAIL,ARRAY_TASKS<br />
#SBATCH --mail-user=my@email.de<br />
#SBATCH --array 0-XXX%YY (XXX: number of tasks in array, YY: number of jobs running simultaneously<br />
<br />
COMMAND[0]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJXXX.X-XXXX mjd_start mjd_stop"<br />
COMMAND[1]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJYYY.Y-YYYY mjd_start mjd_stop"<br />
COMMAND[2]="..."<br />
...<br />
</pre><br />
<br />
'''Can the scripts find possibly unknown gamma-ray sources by itself?''' - Yes, when running the analysis script to create a spectrum (with make_spectrum.sl), there is a routine implemented (fermipy function ''find_sources''), which searches for excesses in the TS map with a significance of 5 sigma or higher. The coordinates of this found source is then saved in the folder, in which the analysis is run, to allow further inspections and a follow-up analyis.<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code, my work-around for this issue is implemented in the wrapper scripts ''make_spectrum.sl'' and ''make_lightcurve.sl''. Deleting the subfolders is now handled in there and not via the routine in the source code. <br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None<br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
If you encounter a new error, you can also go ahead and start the python script for the analysis directly in the shell. Remember to load your conda environment first. You also need a ''config_SED.yaml'' or ''config_LC.yaml'' file in the folder where you are going to run the analysis, otherwise you won't get far. You can directly run the script, e.g. via<br />
<pre><br />
(fermipy)[user@computer]> $FERMITOOLS/SEDcreator_fermipy.py<br />
</pre><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: June 23, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2228Fermi-LAT2021-06-29T14:41:16Z<p>Gokus: /* Troubleshooting */</p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
<br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. I recommend to write a script to create the .slurm file that you want to submit to slurm. Here's an example how this array job can look like:<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --job-name LATspectra<br />
#SBATCH --output job.out-%a<br />
#SBATCH --error err.out-%a<br />
#SBATCH --time 48:00:00<br />
#SBATCH --partition=remeis<br />
#SBATCH --ntasks=1<br />
#SBATCH --hint=nomultithread<br />
#SBATCH --mem-per-cpu=4G<br />
#SBATCH --mail-type=END,FAIL,ARRAY_TASKS<br />
#SBATCH --mail-user=my@email.de<br />
#SBATCH --array 0-XXX%YY (XXX: number of tasks in array, YY: number of jobs running simultaneously<br />
<br />
COMMAND[0]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJXXX.X-XXXX mjd_start mjd_stop"<br />
COMMAND[1]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJYYY.Y-YYYY mjd_start mjd_stop"<br />
COMMAND[2]="..."<br />
...<br />
</pre><br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code, my work-around for this issue is implemented in the wrapper scripts ''make_spectrum.sl'' and ''make_lightcurve.sl''. Deleting the subfolders is now handled in there and not via the routine in the source code. <br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None<br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
If you encounter a new error, you can also go ahead and start the python script for the analysis directly in the shell. Remember to load your conda environment first. You also need a ''config_SED.yaml'' or ''config_LC.yaml'' file in the folder where you are going to run the analysis, otherwise you won't get far. You can directly run the script, e.g. via<br />
<pre><br />
(fermipy)[user@computer]> $FERMITOOLS/SEDcreator_fermipy.py<br />
</pre><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: June 23, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2225Fermi-LAT2021-06-23T14:54:17Z<p>Gokus: /* Contact person */</p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
<br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. I recommend to write a script to create the .slurm file that you want to submit to slurm. Here's an example how this array job can look like:<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --job-name LATspectra<br />
#SBATCH --output job.out-%a<br />
#SBATCH --error err.out-%a<br />
#SBATCH --time 48:00:00<br />
#SBATCH --partition=remeis<br />
#SBATCH --ntasks=1<br />
#SBATCH --hint=nomultithread<br />
#SBATCH --mem-per-cpu=4G<br />
#SBATCH --mail-type=END,FAIL,ARRAY_TASKS<br />
#SBATCH --mail-user=my@email.de<br />
#SBATCH --array 0-XXX%YY (XXX: number of tasks in array, YY: number of jobs running simultaneously<br />
<br />
COMMAND[0]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJXXX.X-XXXX mjd_start mjd_stop"<br />
COMMAND[1]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJYYY.Y-YYYY mjd_start mjd_stop"<br />
COMMAND[2]="..."<br />
...<br />
</pre><br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code, my work-around for this issue is implemented in the wrapper scripts ''make_spectrum.sl'' and ''make_lightcurve.sl''. Deleting the subfolders is now handled in there and not via the routine in the source code. <br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None</span><br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
If you encounter a new error, you can also go ahead and start the python script for the analysis directly in the shell. Remember to load your conda environment first. You also need a ''config_SED.yaml'' or ''config_LC.yaml'' file in the folder where you are going to run the analysis, otherwise you won't get far. You can directly run the script, e.g. via<br />
<pre><br />
(fermipy)[user@computer]> $FERMITOOLS/SEDcreator_fermipy.py<br />
</pre><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: June 23, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2224Fermi-LAT2021-06-23T14:53:59Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
<br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. I recommend to write a script to create the .slurm file that you want to submit to slurm. Here's an example how this array job can look like:<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --job-name LATspectra<br />
#SBATCH --output job.out-%a<br />
#SBATCH --error err.out-%a<br />
#SBATCH --time 48:00:00<br />
#SBATCH --partition=remeis<br />
#SBATCH --ntasks=1<br />
#SBATCH --hint=nomultithread<br />
#SBATCH --mem-per-cpu=4G<br />
#SBATCH --mail-type=END,FAIL,ARRAY_TASKS<br />
#SBATCH --mail-user=my@email.de<br />
#SBATCH --array 0-XXX%YY (XXX: number of tasks in array, YY: number of jobs running simultaneously<br />
<br />
COMMAND[0]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJXXX.X-XXXX mjd_start mjd_stop"<br />
COMMAND[1]="/software/Science/satscripts/fermiscripts/make_spectrum.sl --expath=/your_path/ 4FGLJYYY.Y-YYYY mjd_start mjd_stop"<br />
COMMAND[2]="..."<br />
...<br />
</pre><br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code, my work-around for this issue is implemented in the wrapper scripts ''make_spectrum.sl'' and ''make_lightcurve.sl''. Deleting the subfolders is now handled in there and not via the routine in the source code. <br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None</span><br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
If you encounter a new error, you can also go ahead and start the python script for the analysis directly in the shell. Remember to load your conda environment first. You also need a ''config_SED.yaml'' or ''config_LC.yaml'' file in the folder where you are going to run the analysis, otherwise you won't get far. You can directly run the script, e.g. via<br />
<pre><br />
(fermipy)[user@computer]> $FERMITOOLS/SEDcreator_fermipy.py<br />
</pre><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2223Fermi-LAT2021-06-23T14:40:01Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code, my work-around for this issue is implemented in the wrapper scripts ''make_spectrum.sl'' and ''make_lightcurve.sl''. Deleting the subfolders is now handled in there and not via the routine in the source code. <br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None</span><br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
If you encounter a new error, you can also go ahead and start the python script for the analysis directly in the shell. Remember to load your conda environment first. You also need a ''config_SED.yaml'' or ''config_LC.yaml'' file in the folder where you are going to run the analysis, otherwise you won't get far. You can directly run the script, e.g. via<br />
<pre><br />
(fermipy)[user@computer]> $FERMITOOLS/SEDcreator_fermipy.py<br />
</pre><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2222Fermi-LAT2021-06-23T14:15:10Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code, my work-around for this issue is implemented in the wrapper scripts ''make_spectrum.sl'' and ''make_lightcurve.sl''. Deleting the subfolders is now handled in there and not via the routine in the source code. <br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None</span><br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2221Fermi-LAT2021-06-23T14:12:57Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
<br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, that one of the LC bins folder was missing, and hence, the light curve could not be assembled into one fits file. Because I couldn't pinpoint the error in the source code<br />
<br/><br />
<br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None</span><br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2220Fermi-LAT2021-06-23T13:02:48Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
'''Error during deletion of LC subfolders''' <br/><br />
The internal sub-routine for creating the LCs throwed me an error a couple of times, <br />
<br/><br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None</span><br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2219Fermi-LAT2021-06-23T10:36:11Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
'''error during deletion of LC subfolders''' <br/><br />
<br />
<br/><br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix (additions marked with &rArr;):<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
&rArr; flux_const_for_calcTS_var = None</span><br />
for i, time in itimes:<br />
<br />
&rArr; if not 'fit_success' in mapo[i]:<br />
&rArr; self.logger.error(<br />
&rArr; 'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
&rArr; continue<br />
<br />
&rArr; [el]if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
&rArr; if 'flux_const' in mapo[i]:<br />
&rArr; flux_const_for_calcTS_var = mapo[i]['flux_const']<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
&rArr; flux_const=flux_const_for_calcTS_var,<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2218Fermi-LAT2021-06-23T10:26:34Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
'''error during deletion of LC subfolders''' <br/><br />
<br />
<br/><br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix:<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))</pre><br />
<span style="color:red;">flux_const_for_calcTS_var = None</span><br />
<pre>for i, time in itimes:<br />
<br />
'''if not 'fit_success' in mapo[i]:<br />
self.logger.error(<br />
'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
el'''if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
flux_const=mapo[0]['flux_const'],<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2217Fermi-LAT2021-06-23T10:18:55Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
'''error during deletion of LC subfolders''' <br/><br />
<br />
<br/><br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix:<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
<span style="color:red">flux_const_for_calcTS_var = None</span><br />
for i, time in itimes:<br />
<br />
<span style="color:red">if not 'fit_success' in mapo[i]:<br />
self.logger.error(<br />
'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
el</span>if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
flux_const=mapo[0]['flux_const'],<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
{{font color|red|This text is different}}<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2216Fermi-LAT2021-06-23T10:17:50Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
'''error during deletion of LC subfolders''' <br/><br />
<br />
<br/><br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success'', here is my hot fix:<br />
<pre><br />
# this snippet is nearly at the bottom of the file<br />
<br />
itimes = enumerate(zip(times[:-1], times[1:]))<br />
<span style="color:red">flux_const_for_calcTS_var = None</span><br />
for i, time in itimes:<br />
<br />
<span style="color:red">if not 'fit_success' in mapo[i]:<br />
self.logger.error(<br />
'fit_success not found in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
el</span>if not mapo[i]['fit_success']:<br />
self.logger.error(<br />
'Fit failed in bin %d in range %i %i.' % (i, time[0], time[1]))<br />
continue<br />
<br />
for k in o.keys():<br />
<br />
if k == 'config':<br />
continue<br />
if not k in mapo[i]:<br />
continue<br />
# if (isinstance(o[k], np.ndarray) and<br />
# o[k][i].shape != mapo[i][k].shape):<br />
# gta.logger.warning('Incompatible shape for column %s', k)<br />
# continue<br />
<br />
try:<br />
o[k][i] = mapo[i][k]<br />
except:<br />
pass<br />
<br />
systematic = kwargs.get('systematic', 0.02)<br />
<br />
o['ts_var'] = calcTS_var(loglike=o['loglike_fixed'],<br />
loglike_const=o['loglike_const'],<br />
flux_err=o['flux_err_fixed'],<br />
flux_const=mapo[0]['flux_const'],<br />
systematic=systematic,<br />
fit_success=o['fit_success_fixed'])<br />
<br />
return o<br />
<br />
</pre><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2215Fermi-LAT2021-06-23T10:05:24Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
'''error during deletion of LC subfolders''' <br/><br />
<br />
<br/><br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [[https://github.com/fermiPy/fermipy/issues/405|reported this issue to the developers]], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success''<br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2214Fermi-LAT2021-06-23T10:03:54Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
'''error during deletion of LC subfolders''' <br/><br />
<br />
<br/><br />
'''Individual bins crashed and then the light curve cannot be assembled''' <br/><br />
This is a problem that I fixed manually by editing the source code in the lightcurve.py file. While I have also [reported this issue to the developers|https://github.com/fermiPy/fermipy/issues/405], it has been currently not fixed yet (June 23 2021). In order to still manage creating the light curve, I implemented the following hot fix. In case you are setting up your own conda environment and you come across an error related to ''KeyError: fit_success''<br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2213Fermi-LAT2021-06-23T09:14:40Z<p>Gokus: /* Troubleshooting */</p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
'''error during deletion of LC subfolders''' <br/><br />
<br />
<br/><br />
'''problems when Fit of individual LC bins didn't even start''' <br/><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files. An error which can relate to that, looks, for example, like this:<br />
<pre><br />
GTBinnedAnalysis.run_gtapp(): Caught N3tip12TipExceptionE at the top level: setNumRecords could not insert rows in FITS table in extension "EVENTS" in file "/path_to_analysis/LC_analysis/ft1_00.fits" (CFITSIO ERROR 108: error reading from FITS file)<br />
</pre><br />
In case you need to compute a lot of spectra of light curves and want to do this on /karl, either limit the number of jobs running simultaneously to 10, or store the data on /userdata, where I have so far never encountered this problem.<br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2212Fermi-LAT2021-06-23T09:09:26Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br/><br />
'''How can I submit an array job?''' - In order to do that, you need to work from within a bash shell, not a (t)csh shell. <br />
<br />
<br />
<br />
== Troubleshooting ==<br />
During the Fermi data analysis, several errors can pop up and are not necessarily linked to the Fermi analysis itself. Here, I'll show those I encountered and what I have done to (hot)fix them.<br/><br />
'''error during deletion of LC subfolders''' <br/><br />
<br />
<br/><br />
'''problems when Fit of individual LC bins didn't even start''' <br/><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
While /karl has the benefit of storing large amounts of data, there is an issue with computing many Fermi spectra or light curve at the same time and storing that on /karl. The problems that arise because of this are not clearly visible from checking the log files, but a common error looks, for example like this:<br />
<pre><br />
dkdkdkdkd<br />
</pre><br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
Computing a long light curve with only a short binning (e.g. daily binning for a year-long light curve, or weekly binning for a light curve covering a few years), can result in a termination of the analysis process when the computer cannot handle the amount of open files anymore. I'm not entirely sure what goes wrong internally, however, my suggested hot fix for you is to split the computation of the light curve into manageable time intervals (e.g. chunks of ~100 bins). After all chunks have been computed, you can put them together using the command ''ftmerge'':<br />
<pre><br />
#!/bin/tcsh<br />
<br />
# Step 1: Create list of all light curve chunk files<br />
ls -1 /path_to_my_analysis_results/source_name/LC_analysis/*_lightcurve.fits > /path_to_my_analysis_results/source_name.list<br />
<br />
# /path_to_my_analysis_results/ should contain the folders of each chunk, e.g. named MJD_start_end<br />
<br />
# Step 2: Merge all light curve files into one fits file<br />
ftmerge @/path_to_my_analysis_results/source_name.list /path_to_my_analysis_results/lightcurve_source_name.fits clobber = yes<br />
</pre><br />
No worries about losing already computed bins in case your analysis crashes: those are not deleted when the script crashes mid-way, therefore already computed LC bins are available after the crash and it's not necessary to re-do the analysis for all of the light curve bins.<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2211Fermi-LAT2021-06-23T08:40:17Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br />
'''How can I submit an array job?''' - ...<br />
<br />
<br />
<br />
== Troubleshooting ==<br />
<br />
Errors I have encountered and (hot)fixed: <br/><br />
'''error during deletion of LC subfolders''' <br/><br />
<br />
<br/><br />
'''problems when Fit of individual LC bins didn't even start''' <br/><br />
<br />
<br/><br />
'''Using /karl'''<br/><br />
<br />
<br/><br />
'''Running the original script to check what's going wrong''' <br/><br />
<br />
<br/><br />
'''Light curve fails too many open files''' <br/><br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2208Fermi-LAT2021-05-15T21:20:11Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
While most options are the same as for the ''make_spectrum.sl'' and ''make_lightcurve.sl'' scripts, there are additional options regarding your setup on slurm. Especially the memory and walltime options need to be increased in case you are want to compute a lightcurve with more than 50 bins.<br />
<br />
== Q&A ==<br />
<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br />
'''How can I submit an array job?''' - ...<br />
<br />
<br />
<br />
== Troubleshooting ==<br />
<br />
Errors I have encountered and (hot)fixed: <br/><br />
- error during deletion of LC subfolders <br/><br />
- problems when Fit of individual LC bins didn't even start <br/><br />
- using /karl <br/><br />
- how to run the original script to check what's going wrong? <br/><br />
- light curve fails too many open files <br/><br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2207Fermi-LAT2021-05-15T21:16:33Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br/><br />
If you have comments or questions, please find the information on the Fermi contact person at the Remeis observatory at the bottom of the page.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you want to compute many bins for your light curve). Ideally you want to let the cluster perform your jobs for you and not run an analysis on your desktop machine, unless you want to test something. In order to make life easy for a job submission on slurm, you can use the script ''submit_fermijob.sl'', which is also available at the $FERMITOOLS location. The usage of ''submit_fermijob.sl'' should look like<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl ANALYSIS SOURCENAME TMIN TMAX<br />
</pre><br />
In addition to the name and the time ranges for the analysis, it is necessary to specify whether a spectral (SED) or light curve (LC) computation is requested.<br />
Calling the help on this script via '''$FERMITOOLS/submit_fermijob.sl --help''', gives a full overview of all options available here:<br />
<pre><br />
$FERMITOOLS/submit_fermijob.sl --help<br />
<br />
** Script to submit a Fermi Analysis to slurm **<br />
This script creates a file to submit a Fermi LC or SED analysis to the slurm queue<br />
[Make sure you submit the job when the Fermi Environment is already loaded!]<br />
<br />
<br />
Usage: submit_fermijob.sl [options] [analysis] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
analysis: specify whether it's a light curve [LC] or a SED analysis [SED]<br />
src: Name of source as string, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range the analysis<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known = yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, the coordinates<br />
from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number Number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin, only relevant for LC analysis! Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--submit=yes/no State whether you want to submit to slurm right away; Default: YES<br />
<br />
--walltime=hours Walltime in hours, note that the default walltime might not be enough time for a longer lightcurve.<br />
If you're running into a Slurm TIMEOUT error, set a longer walltime and resubmit. Previous steps of the analysis are saved<br />
and don't need to be repeated unless something in the configuration setup has to be changed; Default: 10 hours<br />
<br />
--mem=memory Memory per CPU, set only number in unit of Gigabyte; Default: 3GB<br />
<br />
--email=mailadress If you want to get informed about the slurm status per email, put email here.<br />
<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
<br />
== Q&A ==<br />
'''Is multi-threading available for the pipeline scripts?''' - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br/><br />
'''How can I submit an array job?''' - ...<br />
<br />
<br />
== Troubleshooting ==<br />
Errors I have encountered and (hot)fixed:<br />
- error during deletion of LC subfolders<br />
- problems when Fit of individual LC bins didn't even start<br />
- using /karl<br />
- how to run the original script to check what's going wrong?<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2206Fermi-LAT2021-05-15T21:07:19Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
The computation of LAT products can take several hours up to days (if you have <br />
<br />
== Q&A ==<br />
Is multi-threading available for the pipeline scripts? - No, this is not possible to use via slurm at the moment. If you're running an analysis on your machine with your own scripts, it is possible to activate multi-threading for the computation of a light curve. For more information on this, please check out https://fermipy.readthedocs.io/en/latest/advanced/lightcurve.html#optimizing-computation-speed.<br />
<br />
<br />
== Troubleshooting ==<br />
Errors<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2205Fermi-LAT2021-05-15T21:02:48Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs three arguments: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
The computation of a light curve is similarly called ''make_lightcurve.sl'' and is available via<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl SOURCENAME TMIN TMAX<br />
</pre><br />
The script needs at the same arguments that are needed for the spectral analysis: the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. Again, several options are provided for specific needs for the analysis, which you can call via '''$FERMITOOLS/make_lightcurve.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_lightcurve.sl --help<br />
<br />
** General script to compute Fermi lightcurves **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file)]<br />
<br />
Usage: make_lightcurve.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, end of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given, <br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
--timebin=number Length of one light curve bin. Default: 7 [days] (weekly binning)<br />
--savebins=True/False Save all the light curve bins? Careful, can take up a lot of space! Default: False<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
Please note that the default time binning for the light curve is one week (7 days).<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2204Fermi-LAT2021-05-15T20:49:55Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs at least three arguments, the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
</pre><br />
As you can see, it is assumed that you do the analysis on a known gamma-ray source, which is named in the 4FGL. If that is the case, it is easiest to give the 4FGL name (without blanks!) as first arguments to the script. If your source is not in the 4FGL, you need to add the coordinates via the --ra and --dec options.</br><br />
<br />
<br />
=== Creating a light curve ===<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2203Fermi-LAT2021-05-15T20:46:51Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
<br />
=== Creating a spectrum ===<br />
If you are interested in the gamma-ray spectrum of a source, the corresponding script is called ''make_spectrum.sl'' and can be called via<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl SOURCENAME TMIN TMAX<br />
</pre><br />
As you can see, the script needs at least three arguments, the name of the source, and both a start time and an end time, for which an average gamma-ray spectrum should be computed. There are several options you can choose from regarding the specifics of the analysis, which you can call via '''$FERMITOOLS/make_spectrum.sl --help'''.<br />
The full help output looks like this:<br />
<pre><br />
$FERMITOOLS/make_spectrum.sl --help<br />
** General script to compute Fermi spectra **<br />
[Make sure you have loaded the Fermitools and Fermipy (see README file/wiki entry)]<br />
<br />
Usage: make_spectrum.sl [options] [src] [tmin] [tmax]<br />
Mandatory inputs:<br />
src: Name of source without any blanks, either a 4FGL name or a different name if corresponding catalog is defined in options<br />
tmin: in MJD, start of time range for which SED will be computed<br />
tmax: in MJD, nd of time range<br />
<br />
Options for analysis:<br />
--known=yes/no Is source a known Fermi source (in the 4FGL)? Important to set correctly, otherwise<br />
the analysis will fail! Default: YES<br />
<br />
--ra=degrees RAJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--dec=degrees DEJ2000 position of source, only needed if source is NOT in 4FGL and no other<br />
alternative catalog is provided with the '--cat option'!<br />
<br />
--cat=catalog.fits Catalog to query for source coordinates, must have columns named Source_Name,<br />
RAJ2000, and DEJ2000 for name and coordinates. If '--ra' and '--dec' are also given,<br />
the coordinates from the catalog will be ignored. Default: 4FGL<br />
<br />
--roi=degrees Size of ROI in degrees. Default: 10 [deg]<br />
<br />
--emin=energy Minimum energy in MeV. Default: 100 [MeV]<br />
<br />
--emax=energy Maximum energy in MeV. Default: 300000 [MeV]<br />
<br />
--ebin=number number of energy bins per energy decade. Default: 10<br />
<br />
Options for setup:<br />
--expath=path Path where analysis should be saved. Default: location where script is started<br />
<br />
</pre><br />
<br />
=== Creating a light curve ===<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2202Fermi-LAT2021-05-15T20:41:38Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV. The scripts have only been tested for blazar analyses.<br/><br />
The scripts can <br />
<br />
=== Creating a spectrum ===<br />
<br />
<br />
=== Creating a light curve ===<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2201Fermi-LAT2021-05-15T20:40:03Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, and supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Fermi-LAT analysis with the pipeline scripts ==<br />
The scripts available at $FERMITOOLS are written to perform a standard analysis on a point source, i.e. a blazar. Please note that these analysis scripts might not provide correct results if you are analysing energies below 100 MeV<br />
<br />
=== Creating a spectrum ===<br />
<br />
<br />
=== Creating a light curve ===<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2200Fermi-LAT2021-05-15T20:36:28Z<p>Gokus: /* The LAT data */</p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, and supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''. Those are provided and regularly updated by the Fermi-LAT collaboration and help to improve your analysis results, as background, especially at lower energies (< 1 GeV), can have quite an impact on your results.<br />
<br />
== Creating a spectrum ==<br />
<br />
== Creating a light curve ==<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2199Fermi-LAT2021-05-15T20:34:22Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are mostly present in the galactic plane and the magellanic clouds (e.g. features of the LMC, the Centaurus A lobes, and supernova remnants), can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
<br />
In the $FERMI_DIR, you can find the current catalog as well as the templates for all extended sources.<br/><br/><br />
<br />
In addition, the LAT background models, needed to model the galactic and isotropic diffuse emission, can be found at '''/satdata/X-ray/Fermi/LAT_background_models'''.<br />
== Creating a spectrum ==<br />
<br />
== Creating a light curve ==<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2198Fermi-LAT2021-05-15T20:30:18Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh -f<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
In case you're using a bash shell, the addition in your .bashrc file should look something like this:<br />
<pre><br />
#!/bin/bash<br />
alias fermipy='conda activate /userdata/data/gokus/conda/miniconda2/envs/fermipy'<br />
</pre><br />
Using the command '''fermipy''', you can now change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br/><br/><br />
<br />
In the early steps of your analysis, a region of interest (ROI) around your desired source or source coordinates will be created, which contains all the sources that are included in the 10-year Fourth Point Source Catalog (4FGL). In addition, extended sources, which are present in the galactic plane and the magellanic clouds, can be part of that ROI, too. In order to run the pipeline scripts properly, two environment variables need to be added to your .cshrc file:<br />
<pre><br />
#!/bin/tcsh -f<br />
setenv FERMI_DIR /satdata/X-ray/Fermi/catalogs<br />
setenv LATEXTDIR $FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
Alternatively for the bash shell, add the following to your .bashrc file:<br />
<pre><br />
#!/bin/bash<br />
export FERMI_DIR=/satdata/X-ray/Fermi/catalogs/<br />
export LATEXTDIR=$FERMI_DIR/4FGL_extended_sources<br />
</pre><br />
== Creating a spectrum ==<br />
<br />
== Creating a light curve ==<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2197Fermi-LAT2021-05-15T20:20:37Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh -f<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
Using the command '''fermipy''', you can change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br/><br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset. If you are using the provided analysis scripts, it automatically looks through all the data that is available at the observatory.<br />
All photon files are linked at '''/satdata/X-ray/Fermi/photon_ft1_files.list''', while all of the spacecraft data is merged in '''/satdata/X-ray/Fermi/spacecraft_ft2_files.fits'''.<br />
== Creating a spectrum ==<br />
<br />
== Creating a light curve ==<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2196Fermi-LAT2021-05-15T20:15:59Z<p>Gokus: qui</p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh -f<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
Using the command '''fermipy''', you can change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files. One can acquire the data needed for a specific analysis via the online data query at https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi, or download weekly data products for the entire sky from https://fermi.gsfc.nasa.gov/ssc/data/access/.<br />
However, all data acquired by Fermi-LAT during its mission are also stored at the servers of the observatory, hence, it is not necessary to download a specific dataset.<br />
<br />
== Creating a spectrum ==<br />
<br />
== Creating a light curve ==<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2195Fermi-LAT2021-05-15T20:09:48Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh -f<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
Using the command '''fermipy''', you can change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). It is recommended to read the full installation instructions provided in the documentations.<br />
<br />
== The LAT data ==<br />
The data downlinked from the Fermi satellite contains information about the events/photons measured by the LAT, and information about status of the spacecraft itself at each time. Those information are needed for an analysis, as they are the main input. These files are called photon and spacecraft files.<br />
<br />
== Creating a spectrum ==<br />
<br />
== Creating a light curve ==<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2194Fermi-LAT2021-05-15T20:03:29Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. <br/><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh -f<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
Using the command '''fermipy''', you can change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure you are installing the right versions of each (as in between versions there was a change in the use of python2.7 and python3). You can follow the <br />
<br />
== Creating a spectrum ==<br />
<br />
== Creating a light curve ==<br />
<br />
== Performing Fermi-LAT analysis using slurm ==<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2193Fermi-LAT2021-05-15T20:00:48Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. </br><br />
Before you use the script, make sure you know what you're doing. If you are analysing gamma-ray data for the first time, please read the online documentation at https://fermi.gsfc.nasa.gov/ssc/data/analysis/LAT_essentials.html. Because the Fermi-LAT data can be accessed by everyone for free, there is an extensive documentation available online.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh -f<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
Using the command '''fermipy''', you can change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure to be sure<br />
<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2192Fermi-LAT2021-05-15T19:57:43Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. For more information on the spacecraft, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov. </br><br />
<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh -f<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
Using the command '''fermipy''', you can change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure to be sure<br />
<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
== Contact person ==<br />
The scripts and this wiki entry has been written by Andrea Gokus. She is also currently the maintainer of the LAT dataset on the Remeis server. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br/><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2191Fermi-LAT2021-05-15T19:51:37Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky. <br />
For more information, please check out the official Fermi webpage at https://fermi.gsfc.nasa.gov.<br />
<br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written with ''fermipy'', available to create either a spectrum or a light curve. Those are globally available at $FERMITOOLS, which links to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
The pipeline scripts for a standard Fermi-LAT analysis (spectrum or light curve) are optimized to run with the version of this conda environment. In order to envoke it, it is necessary to source the conda installation, where the environment for a Fermi-LAT analysis has been created, before activating the environment. In order to simplify this, you can copy the following in your .cshrc file.<br />
<br />
<pre><br />
#!/bin/tcsh -f<br />
source /userdata/data/gokus/conda/miniconda2/etc/profile.d/conda.csh<br />
alias fermipy "conda activate fermipy" <br />
</pre><br />
<br />
If you already have your own conda installation, which you regularly use, make sure that you create an alias instead of sourcing that conda installtion by default!<br />
Using the command '''fermipy''', you can change into the conda environment, in which the necessary tools for using the pipeline scripts are installed. Now you're ready to do some Science!<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure to be sure<br />
<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<br />
==== Contact person ====<br />
The scripts and this wiki entry has been written by Andrea Gokus. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.</br><br />
Last status update on this wiki page: Mai 16, 2021<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2190Fermi-LAT2021-05-15T19:31:01Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky.<br />
<br />
The following <br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written using fermipy, available to create either a spectrum or a light curve, which are available at $FERMITOOLS, linking to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: calling a currently existing conda environment, or creating the corresponding conda environment yourself. The first version is, at the current moment, the recommended choice, also because the pipeline scripts are written for python 2.7, while for the latter one needs to make sure the correct versions of the Fermitools and ''fermipy'' are installed.<br />
<br />
=== Option 1 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
<br />
=== Option 2 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
Please make sure to be sure<br />
<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<pre><br />
Programmiertext.<br />
</pre><br />
==== Contact person ====<br />
The scripts and this wiki entry has been written by Andrea Gokus. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2189Fermi-LAT2021-05-15T19:06:06Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky.<br />
<br />
The following <br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written using fermipy, available to create either a spectrum or a light curve, which are available at $FERMITOOLS, linking to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: creating the corresponding conda environment yourself, or calling a currently existing conda environment.<br />
<br />
=== Option 1 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at <br/><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda <br/><br />
''fermipy'': https://github.com/fermiPy/fermipy (Github), https://fermipy.readthedocs.io/en/latest/ (Documentation) <br/><br />
The newest version<br />
<br />
=== Option 2 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<pre><br />
Programmiertext.<br />
</pre><br />
==== Contact person ====<br />
The scripts and this wiki entry has been written by Andrea Gokus. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2188Fermi-LAT2021-05-15T19:04:40Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky.<br />
<br />
The following <br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written using fermipy, available to create either a spectrum or a light curve, which are available at $FERMITOOLS, linking to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: creating the corresponding conda environment yourself, or calling a currently existing conda environment.<br />
<br />
=== Option 1 - Installing and creating a conda environment yourself ===<br />
Both the Fermitools and ''fermipy'' are managed via GitHub and the full code, documentation and installation can be found at </br><br />
Fermitools: https://github.com/fermi-lat/Fermitools-conda </br><br />
''fermipy'': Github repository: https://github.com/fermiPy/fermipy Documentation: https://fermipy.readthedocs.io/en/latest/ </br><br />
The newest version<br />
<br />
=== Option 2 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<pre><br />
Programmiertext.<br />
</pre><br />
==== Contact person ====<br />
The scripts and this wiki entry has been written by Andrea Gokus. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2187Fermi-LAT2021-05-15T18:54:18Z<p>Gokus: </p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky.<br />
<br />
The following <br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written using fermipy, available to create either a spectrum or a light curve, which are available at $FERMITOOLS, linking to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: creating the corresponding conda environment yourself, or calling a currently existing conda environment.<br />
<br />
=== Option 1 - Installing and creating a conda environment yourself ===<br />
<br />
=== Option 2 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0) ===<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<pre><br />
Programmiertext.<br />
</pre><br />
==== Contact person ====<br />
The scripts and this wiki entry has been written by Andrea Gokus. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokushttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Fermi-LAT&diff=2186Fermi-LAT2021-05-15T18:53:53Z<p>Gokus: Created page with "The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray dat..."</p>
<hr />
<div>The Fermi Large Area Telescope (LAT) observes the gamma-ray sky from 20 MeV up to 1 TeV. It does so by scanning the entire sky three times a day, hence, there is gamma-ray data available for each day for every position on the sky.<br />
<br />
The following <br />
<br />
== Installation & preparation ==<br />
To analyse data from Fermi-LAT, the Fermitools are needed. Additionally, there is a python package, called ''fermipy'', such that one can do the data analysis from within a python environment. There are analysis scripts, written using fermipy, available to create either a spectrum or a light curve, which are available at $FERMITOOLS, linking to /software/Science/satscripts/fermiscripts.<br /><br />
To run those scripts, it is necessary to have both the Fermitools and ''fermipy'' installed, ideally in a conda environment. There are two options for the user to do so: creating the corresponding conda environment yourself, or calling a currently existing conda environment.<br />
<br />
'''Option 1 - Installing and creating a conda environment yourself'''<br />
<br />
===Option 2 - Using the existing conda environment (Fermitools 1.2.23 & fermipy version 0.20.0)===<br />
<br />
== Troubleshooting ==<br />
<br />
<br />
<pre><br />
Programmiertext.<br />
</pre><br />
==== Contact person ====<br />
The scripts and this wiki entry has been written by Andrea Gokus. In case of any further questions or problems, please contact her via andrea.gokus[at]fau.de.<br />
<br />
<br />
[[Category:Data Extraction]]</div>Gokus