Integral - Revision history

Thalhammer: /* Non-OSA INTEGRAL extraction */

2025-07-30T13:33:44Z

Non-OSA INTEGRAL extraction

Thalhammer at 13:18, 30 July 2025

2025-07-30T13:18:16Z

Thalhammer: Added PIF extraction

2024-04-24T20:07:54Z

Added PIF extraction

Thalhammer: /* Time selection */

2023-06-09T20:46:15Z

Time selection

Thalhammer at 19:38, 7 June 2023

2023-06-07T19:38:04Z

Thalhammer at 16:14, 6 June 2023

2023-06-06T16:14:20Z

Thalhammer: Added some details

2023-06-06T10:18:07Z

Added some details

Thalhammer: Added LC extraction and comment on non-remeis pipelines

2023-06-02T12:03:48Z

Added LC extraction and comment on non-remeis pipelines

Thalhammer: /* On catalogs */

2023-06-01T12:58:51Z

On catalogs

Thalhammer: Added remark on catalog file types

2023-05-26T09:42:51Z

Added remark on catalog file types

@@ Line 200: / Line 200: @@
 There are a few alternatives by now to using OSA+Remeis scripts:
 * To run IBIS and JEMX analysis online there is MMODA at https://www.astro.unige.ch/mmoda/help/mmoda/integral-mission
-* If you are interested in SPI data you can do this with the OSA installed here at Remeis, there are however much more reliable and stable pipelines available in form of either SPI-DAI or the SPI-pipeline at MPE organized by Tomas Siegert.
+* If you are interested in SPI data you can do this with the OSA installed here at Remeis, there are however much more reliable and stable pipelines available in form of either SPI-DAI or the SPI-pipeline at MPE organized by Thomas Siegert.
 [[Category:Data Extraction]]

@@ Line 200: / Line 200: @@
 There are a few alternatives by now to using OSA+Remeis scripts:
 * To run IBIS and JEMX analysis online there is MMODA at https://www.astro.unige.ch/mmoda/help/mmoda/integral-mission
-* If you are interested in SPI data you can do this with the OSA installed here at Remeis, there are however much more reliable and stable pipelines available in form of either SPI-DAI or the SPI-pipeline at MPE organized by Tomas Stiegert.
+* If you are interested in SPI data you can do this with the OSA installed here at Remeis, there are however much more reliable and stable pipelines available in form of either SPI-DAI or the SPI-pipeline at MPE organized by Tomas Siegert.
 [[Category:Data Extraction]]

@@ Line 63: / Line 63: @@
    ls obs/*/scw/*.001/*_Cyg_X-1_30-120.lc >> lcs.txt
    fmerge @lcs.txt Cyg_X1_ibis_lc.fits -
 === Typical JEMX spectral analysis ===
 Until revolution 976 (Oct. 10,2010) only one JEM-X unit was operating at any give time. During that time one should therefore run the analysis for both detectors.
-; 1) Find ScWs that cover your source. As JEMX has a smaller field of view, its max distance to the source should also be reduced to ~5 degreee:
+; 1) Find ScWs that cover your source. As JEMX has a smaller field of view, its max distance to the source should also be reduced to ~5 degree:
    objscw --scwlist "Cyg X-1" --maxdist=5 > scws_jmx.txt
 And again, you will probably further select in which ScWs you are interested.

← Older revision		Revision as of 20:46, 9 June 2023
Line 98:		Line 98:

	gti_user tstart=8170.5 tstop=8171.8 unit=sec begin=0.0 end=50 step=150 repeat=653 gti=user_gti.fits clobber=yes		gti_user tstart=8170.5 tstop=8171.8 unit=sec begin=0.0 end=50 step=150 repeat=653 gti=user_gti.fits clobber=yes
		+
		+	Or for a to calculate a reverse bary-corrected GTI file. Not that this requires the coordinated and the DOL file of a ScW or Observation Group.
		+
		+	gti_user tstart=8393.16 unit=sec begin=0.0 end=50 step=25 repeat=600 gti=user_gti.fits clobber=yes bary=y group=og_ibis.fits+1
		+
		+	Also note that the light-curves generated by imalc_sbs through master_sbs are already barycenter corrected.
		+

	More details can be found in the introduction to INTEGRAL analysis:		More details can be found in the introduction to INTEGRAL analysis:

@@ Line 25: / Line 25: @@
 === Typical IBIS image/mosaic ===
-) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:
+; 1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:
    objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt
-If you want to use scws_all.txt as input for hte next step remove the header of this file, i.e, the first 6 lines.
+If you want to use scws_all.txt as input for hte next step remove the header of this file, i.e, the first 6 lines. Importantly you will probalby want to select only a subset of the available observations and therefore perform some selection on the full list of science windows. As a simple example one might image to only use the latest 100 ScWs:
-) Start the ScW based analysis:
+  tail -n 100 scws_all.txt > scws_cut.txt
    master_sbs  --job=IMA --scwlist=scws_all.txt --energies="30,50,200" --verbose --clobber=yes  --path=$PWD
-) Generate the required input file for spe_pick
+; 3) Generate the required input file for spe_pick
    txt2idx element="scws_dols.txt" index="cygx1"
-) Run INT_mosa_ibis to generate the a combine mosaic
+; 4) Run INT_mosa_ibis to generate the a combine mosaic
    INT_mosa_ibis --obsgrp=$PWD --dol=scws_dols.txt --destin=$PWD/obs/mosa_cygx1
-) Check the images for artifacts and sort out affected ScWs through trial-and-error:
+; 5) Check the images for artifacts and sort out affected ScWs through trial-and-error:
    ds9 isgri_mosa_ima.fits[2] -region found.reg -cmap b -scale sqrt -scale limits 0 60 -zoom 2
@@ Line 45: / Line 46: @@
 === Typical IBIS spectral analysis ===
-) Again, find ScWs that cover your source:
+; 1) Again, find ScWs that cover your source and are of interest to you
-  objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt
+; 2) Start the ScW based analysis:
-) Start the ScW based analysis:
+   master_sbs  --job=SPE --scwlist=scws_cut.txt --source="Cyg X-1"  --verbose --clobber=yes  --path=$PWD --catalog=specat.fits
-   master_sbs  --job=SPE --scwlist=scws_all.txt --source="Cyg X-1"  --verbose --clobber=yes  --path=$PWD --catalog=specat.fits
+; 3) Generate the required input file for spe_pick. Check that that all objects in the DOL file  exist.
    txt2idx element="scws_dols.txt" index="cygx1"
-) Run spe_pick to generate the final spectra
+; 4) Run spe_pick to generate the final spectra
    spe_pick group="cygx1.fits" source="Cyg X-1" rootname="cygX1_all" instrument="ISGRI"
 This will result in background subtracted spectra for the given source. Remember that the source name has exactly the naming in the INTEGRAL reference catalog.
 === Typical IBIS light-curve generation ===
-) Quite similar to the spectral extraction. First ScWs that cover your source, or reuse the previous ScW list:
+; 1) Quite similar to the spectral extraction. First find ScWs that cover your source, or reuse the previous ScW list:
-  objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt
+; 2) Start the ScW based analysis, here for 100s time binning:
-) Start the ScW based analysis, here for 100s time binning:
+   master_sbs  --job=LC --scwlist=scws_cut.txt --source="Cyg X-1"  --verbose --clobber=yes  --path=$PWD --catalog=specat.fits --energies=30,120 --deltat=100
 This generates a separate light-curve for each ScW at, e.g., obs/250300170010/scw/250300170010.001/250300170010_Cyg_X-1_30-120.lc
-) These can then be merged together e.g. using fmerge or ISIS.
+; 3) These can then be merged together e.g. using fmerge or ISIS.
    ls obs/*/scw/*.001/*_Cyg_X-1_30-120.lc >> lcs.txt
    fmerge @lcs.txt Cyg_X1_ibis_lc.fits -
@@ Line 68: / Line 66: @@
 === Typical JEMX spectral analysis ===
 Until revolution 976 (Oct. 10,2010) only one JEM-X unit was operating at any give time. During that time one should therefore run the analysis for both detectors.
-) Find ScWs that cover your source. As JEMX has a smaller field of view, its max distance to the source should also be reduced to ~5 degreee:
+; 1) Find ScWs that cover your source. As JEMX has a smaller field of view, its max distance to the source should also be reduced to ~5 degreee:
    objscw --scwlist "Cyg X-1" --maxdist=5 > scws_jmx.txt
-) Generate a RMF with the required binning, e.g., 32 bins:
+And again, you will probably further select in which ScWs you are interested.
    j_rebin_rmf  binlist=STD_032
-The RMF does not change between observations might be reused as long as binning requirements do not change.
+The RMF does not change between observations and might be reused as long as binning requirements do not change.
 (Depending on your fitting software the resulting RMF might give you troubles and needs to be rewritten.)
-) Start the ScW based analysis with 32=2^5 logarithmic energy bins:
+; 3) Start the ScW based analysis with 32=2^5 logarithmic energy bins:
-   master_sbs  --path=$PWD --scwlist=scws_jemx.txt --job=SPE --source="Cyg X-1" --jemx=1 --clobber=yes --catalog=catalog_jemx.fits --response=jemx_rebinned_rmf.fits --nChanBins=-5
+   master_sbs  --path=$PWD --scwlist=scws_jemx_cut.txt --job=SPE --source="Cyg X-1" --jemx=1 --clobber=yes --catalog=catalog_jemx.fits --response=jemx_rebinned_rmf.fits --nChanBins=-5
-) Generate the required input file for spe_pick
+; 4) Generate the required input file for spe_pick
    txt2idx element="scws_jmx_dols.txt" index="cygx1"
-) Run spe_pick to generate the final spectra.
+; 5) Run spe_pick to generate the final spectra.
    spe_pick group=cygx1.fits source="Cyg X-1" rootname="cygx1" instrument="JMX1"
@@ Line 84: / Line 83: @@
 To generate JEMX images there are no dedicated scripts at Remeis, and you have to fall back to the OSA scripts, e.g.:
    og_create idxSwg=scw.lst ogid=oph2 baseDir="./" instrument=JMX1
-   jemx_science_analysis startLevel="IMA2" endLevel="IMA2"  jemxNum= COR_gainModel=2
+   jemx_science_analysis startLevel="IMA2" endLevel="IMA2" jemxNum=1 COR_gainModel=2
 === On catalogs ===
@@ Line 93: / Line 92: @@
 In practice one prepares the catalog used for spectral extraction from the results of the IMA step, taking only into account actually detected sources, e.g. by running the following on the results of INT_mosa_ibis:
   fcopy "isgri_srcl_res.fits[1][DETSIG>10]" specat.fits
 === Quick Look Analysis ===
@@ Line 168: / Line 177: @@
 === Non-OSA INTEGRAL extraction ===
 There are a few alternatives by now to using OSA+Remeis scripts:
 * To run IBIS and JEMX analysis online there is MMODA at https://www.astro.unige.ch/mmoda/help/mmoda/integral-mission

@@ Line 68: / Line 68: @@
 Adapted from the documentation:
 While for generally image generation a user generated catalog is not required, if supplied, it needs to keeps the same format of the general reference catalog from which it was built, i.e. it has the extension of the type ”GNRL-REFR-CAT”. This is the correct extension to be used as input for the Imaging step. In Section 7.2 of the OSA manual, you are been shown how to build a catalog for the lightcurve (LCR) and spectral (SPE) step. In that case the catalog, you obtained, specat.fits, was of the type of ”ISGR-SRCLRES” which is the correct format for the LCR and SPE steps. The two formats cannot be switched, meaning that you cannot use a ”GNRL-REFR-CAT” type catalog for SPE and LCR or an ”ISGR-SRCL-RES” one for IMA.
 === Quick Look Analysis ===

@@ Line 11: / Line 11: @@
 === General remarks on Integral data extraction at Remeis: ===
-Initially data is extracted in parallel on a ScWs basis and in a second step combined to the final products. The initial step is launch by the scripts master_sbs. The script can generate spectra and images for ISGRI and spectra for JEMX. No PICsIT, SPI or OMC analysis is run at Remeis.
+Initially data is extracted in parallel on a ScWs basis and in a second step combined to the final products. The initial step is launch by the scripts master_sbs. The script can generate spectra and images for ISGRI and spectra for JEMX. No PICsIT, SPI or OMC analysis is run at Remeis. The resulting spectra are then combined using spe_pick while the ISGRI/IBIS images are merged using INT_mosa_ibis. Generally it is a good idea, if you have not already to have a look at the OSA cookbook, e.g., to check how the paths in the data object locator (DOL) files is supposed to look like or how to prepare a suitable catalog for spectral extraction: https://www.isdc.unige.ch/integral/analysis
 Before any OSA analysis can be run the corresponding module has to be load with:
@@ Line 66: / Line 65: @@
    jemx_science_analysis startLevel="IMA2" endLevel="IMA2"  jemxNum= COR_gainModel=2
+=== On catalogs ===
 === Quick Look Analysis ===
-NRT data can also be analysed at Remeis with the ususal scipts, require, however, an additional --nrt parameter in most scripts.
+NRT data can also be analysed at Remeis with the usual scripts, require, however, an additional --nrt parameter in most scripts.
 Also have a look at http://isdc.unige.ch/heavens/
 === Documentation ===
-For common questions:
+Useful to know:
 http://www.isdc.unige.ch/integral/download/osa/doc/10.0/osa_um_ibis/node27.html
 And for the cook-book and general documentation. Reading the IBIS Analysis User Manual is recommend:
 https://www.isdc.unige.ch/integral/analysis
 === Abbreviations ===