Xmm - Revision history

Weber at 18:54, 19 May 2026

2026-05-19T18:54:59Z

← Older revision		Revision as of 20:54, 19 May 2026
Line 47:		Line 47:
	${XMMTOOLS}/xmmprepare --datadir=/eu/X-ray/XMM/data/2290/0679780201/odf/		${XMMTOOLS}/xmmprepare --datadir=/eu/X-ray/XMM/data/2290/0679780201/odf/
	--prepdir=./test --pn --analysisdate=2014-01-11T01:05:00.000		--prepdir=./test --pn --analysisdate=2014-01-11T01:05:00.000

			It is worth knowing that a certain degree of randomization is used in the extraction to avoid the effects of quantization on which the measurements are based. Performing the same extraction repeatedly will therefore not necessarily result in absolutely identical event lists, lightcurves, and spectra. In the case of high signal-to-noise ratio sources performing the same extraction multiple times will most likely lead to more or less the same scientific results. However, if a result has only tentative significance, such as a very low significance pulse period, because there is only a very limited number of counts available, this result might change on repeated extractions. Of course, when arguing on such a low level, many other systematics most likely play a role anyway and the result should at least be treated with the utmost care. A paper discussing such a situation is [https://ui.adsabs.harvard.edu/abs/2026arXiv260515137K/abstract Knight et al., 2026]. By default the random number generator of the SAS is initialized to the local time. This is explained in [https://xmm-tools.cosmos.esa.int/external/sas/current/doc/taskmain/node3.html the SAS documentation]. If you need to control at least this systematic, you can use the environment variable SAS_RAND_SEED to set the seed to the same value during repeated the extraction.

	== XMM OM Data Extraction ==		== XMM OM Data Extraction ==

Nekrasov: /* Data */

2025-08-12T10:06:15Z

Data

← Older revision		Revision as of 12:06, 12 August 2025
Line 2:		Line 2:

	Always use the remeis archives at /eu/X-ray/XMM/data/ instead of downloading observation files yourself. You can search for the data of an observation by typing something like '''find /eu/X-ray/xmm/data/????/obsID/odf -type d''' - replacing obsID with your observation ID.		Always use the remeis archives at /eu/X-ray/XMM/data/ instead of downloading observation files yourself. You can search for the data of an observation by typing something like '''find /eu/X-ray/xmm/data/????/obsID/odf -type d''' - replacing obsID with your observation ID.
	If the observation you want to analyze is not in the archives: contact the XMM admin (Outdated: ~~[[User:Nekrasov\|Nekrasov]] ([[User talk:Nekrasov\|talk]])~~Jonathan Knies~~[[User:Nekrasov\|Nekrasov]] ([[User talk:Nekrasov\|talk]])~~, ~~[[User:Nekrasov\|Nekrasov]] ([[User talk:Nekrasov\|talk]])~~Amy Joyce~~[[User:Nekrasov\|Nekrasov]] ([[User talk:Nekrasov\|talk]]))~~ ??Esin Gulbahar?? and the observation will be downloaded for you.		If the observation you want to analyze is not in the archives: contact the XMM admin (Outdated: ~~Jonathan Knies~~, ~~Amy Joyce~~ ??Esin Gulbahar?? and the observation will be downloaded for you.

	Note (as of 12.08.2025): since xmmsas version 22 we cannot run it natively on the cluster. Instead, we have a docker image for this. Otherwise, downgrade xmmsas to version 21 "module load xmmsas/21" if you want to use the algorithm below.		Note (as of 12.08.2025): since xmmsas version 22 we cannot run it natively on the cluster. Instead, we have a docker image for this. Otherwise, downgrade xmmsas to version 21 "module load xmmsas/21" if you want to use the algorithm below.

Nekrasov: /* Data */

2025-08-12T10:05:33Z

Data

← Older revision		Revision as of 12:05, 12 August 2025
Line 2:		Line 2:

	Always use the remeis archives at /eu/X-ray/XMM/data/ instead of downloading observation files yourself. You can search for the data of an observation by typing something like '''find /eu/X-ray/xmm/data/????/obsID/odf -type d''' - replacing obsID with your observation ID.		Always use the remeis archives at /eu/X-ray/XMM/data/ instead of downloading observation files yourself. You can search for the data of an observation by typing something like '''find /eu/X-ray/xmm/data/????/obsID/odf -type d''' - replacing obsID with your observation ID.
	If the observation you want to analyze is not in the archives: contact the XMM admin (Outdated: ~~{{strikethrough~~\|Jonathan Knies}}, ~~{{strikethrough~~\|Amy Joyce}}) ??Esin Gulbahar?? and the observation will be downloaded for you.		If the observation you want to analyze is not in the archives: contact the XMM admin (Outdated: [[User:Nekrasov\|Nekrasov]] ([[User talk:Nekrasov\|talk]])Jonathan Knies[[User:Nekrasov\|Nekrasov]] ([[User talk:Nekrasov\|talk]]), [[User:Nekrasov\|Nekrasov]] ([[User talk:Nekrasov\|talk]])Amy Joyce[[User:Nekrasov\|Nekrasov]] ([[User talk:Nekrasov\|talk]])) ??Esin Gulbahar?? and the observation will be downloaded for you.

	Note (as of 12.08.2025): since xmmsas version 22 we cannot run it natively on the cluster. Instead, we have a docker image for this. Otherwise, downgrade xmmsas to version 21 "module load xmmsas/21" if you want to use the algorithm below.		Note (as of 12.08.2025): since xmmsas version 22 we cannot run it natively on the cluster. Instead, we have a docker image for this. Otherwise, downgrade xmmsas to version 21 "module load xmmsas/21" if you want to use the algorithm below.

Nekrasov: /* Data */

2025-08-12T10:02:28Z

Data

← Older revision		Revision as of 12:02, 12 August 2025
Line 2:		Line 2:

	Always use the remeis archives at /eu/X-ray/XMM/data/ instead of downloading observation files yourself. You can search for the data of an observation by typing something like '''find /eu/X-ray/xmm/data/????/obsID/odf -type d''' - replacing obsID with your observation ID.		Always use the remeis archives at /eu/X-ray/XMM/data/ instead of downloading observation files yourself. You can search for the data of an observation by typing something like '''find /eu/X-ray/xmm/data/????/obsID/odf -type d''' - replacing obsID with your observation ID.
	If the observation you want to analyze is not in the archives: contact the XMM admin (Outdated: ~~Jonathan Knies~~, ~~Amy Joyce~~) ??Esin Gulbahar?? and the observation will be downloaded for you.		If the observation you want to analyze is not in the archives: contact the XMM admin (Outdated: {{strikethrough\|Jonathan Knies}}, {{strikethrough\|Amy Joyce}}) ??Esin Gulbahar?? and the observation will be downloaded for you.

	Note (as of 12.08.2025): since xmmsas version 22 we cannot run it natively on the cluster. Instead, we have a docker image for this. Otherwise, downgrade xmmsas to version 21 "module load xmmsas/21" if you want to use the algorithm below.		Note (as of 12.08.2025): since xmmsas version 22 we cannot run it natively on the cluster. Instead, we have a docker image for this. Otherwise, downgrade xmmsas to version 21 "module load xmmsas/21" if you want to use the algorithm below.

Nekrasov: /* Data */

2025-08-12T09:56:17Z

Data

← Older revision		Revision as of 11:56, 12 August 2025
Line 2:		Line 2:

	Always use the remeis archives at /eu/X-ray/XMM/data/ instead of downloading observation files yourself. You can search for the data of an observation by typing something like '''find /eu/X-ray/xmm/data/????/obsID/odf -type d''' - replacing obsID with your observation ID.		Always use the remeis archives at /eu/X-ray/XMM/data/ instead of downloading observation files yourself. You can search for the data of an observation by typing something like '''find /eu/X-ray/xmm/data/????/obsID/odf -type d''' - replacing obsID with your observation ID.
	If the observation you want to analyze is not in the archives: contact the XMM admin (Jonathan Knies ~~- !outdated person!~~) and the observation will be downloaded for you.		If the observation you want to analyze is not in the archives: contact the XMM admin (Outdated: ~~Jonathan Knies~~, ~~Amy Joyce~~) ??Esin Gulbahar?? and the observation will be downloaded for you.

			Note (as of 12.08.2025): since xmmsas version 22 we cannot run it natively on the cluster. Instead, we have a docker image for this. Otherwise, downgrade xmmsas to version 21 "module load xmmsas/21" if you want to use the algorithm below.

	== XMM EPIC Data Extraction ==		== XMM EPIC Data Extraction ==

Nekrasov: /* Data */

2025-08-11T13:11:08Z

Data

← Older revision		Revision as of 15:11, 11 August 2025
Line 2:		Line 2:

	Always use the remeis archives at /eu/X-ray/XMM/data/ instead of downloading observation files yourself. You can search for the data of an observation by typing something like '''find /eu/X-ray/xmm/data/????/obsID/odf -type d''' - replacing obsID with your observation ID.		Always use the remeis archives at /eu/X-ray/XMM/data/ instead of downloading observation files yourself. You can search for the data of an observation by typing something like '''find /eu/X-ray/xmm/data/????/obsID/odf -type d''' - replacing obsID with your observation ID.
	If the observation you want to analyze is not in the archives: contact the XMM admin (Jonathan Knies) and the observation will be downloaded for you.		If the observation you want to analyze is not in the archives: contact the XMM admin (Jonathan Knies - !outdated person!) and the observation will be downloaded for you.

	== XMM EPIC Data Extraction ==		== XMM EPIC Data Extraction ==

Wilms: /* Workaround for the xmmscripts */

2024-01-04T15:30:39Z

Workaround for the xmmscripts

← Older revision		Revision as of 17:30, 4 January 2024
Line 35:		Line 35:
	"xmmextract" is a very powerful script, use its "--help" argument to get a list of all arguments.		"xmmextract" is a very powerful script, use its "--help" argument to get a list of all arguments.

	~~==== Workaround for the xmmscripts ====~~		Further important points:
	~~There were some problems occuring with the XMM-data extraction. Joern has implemented a workaround in the xmmscripts that has to be called as shown in the example below.~~
	~~Here the issues:~~
	# If your observation includes sub-observations, choose one of them. Do this with "onlyexp=1" not "onlyexp=PNU001" for example.		# If your observation includes sub-observations, choose one of them. Do this with "onlyexp=1" not "onlyexp=PNU001" for example.
	# Since our Calibration database gets updated frequently, we ran into the problem that new calibration files only like the latest SAS. As it is very problematic to just install the new SAS, there is a workaround. You have to address the calibration files that are working with our SAS version until the new SAS is installed (we must be patient as this is not trivial). The following, however, works perfectly:		# Since our Calibration database gets updated frequently, we ran into the problem that new calibration files only like the latest SAS. As it is very problematic to just install the new SAS, there is a workaround. You have to address the calibration files that are working with our SAS version until the new SAS is installed (we must be patient as this is not trivial). The following, however, works perfectly:

Wilms: /* XMM EPIC Data Extraction */

2024-01-04T15:28:49Z

XMM EPIC Data Extraction

← Older revision		Revision as of 17:28, 4 January 2024
Line 6:		Line 6:
	== XMM EPIC Data Extraction ==		== XMM EPIC Data Extraction ==

	After the system ~~got~~ updated from Ubuntu 11 to Ubuntu 12 a problem occured with the function "dssetattr" inside the "epchain".		After the system was updated from Ubuntu 11 to Ubuntu 12 a problem occured with the function "dssetattr" inside the "epchain".
	This function can set/add an attribute to the HDU (Header/Data Unit).		This function can set/add an attribute to the HDU (Header/Data Unit).
	The solution is to call		The solution is to call
Line 13:		Line 13:

	A little example script:		A little example script:


	#!/bin/csh		#!/bin/csh
Line 26:		Line 25:
	${xmmscripts}/xmmextract --prepdir=$obsid --pn --srcreg=src.fits --bkgreg=bkg.fits # source and background extraction of the chip		${xmmscripts}/xmmextract --prepdir=$obsid --pn --srcreg=src.fits --bkgreg=bkg.fits # source and background extraction of the chip

	~~note~~ that it is usually better to have the data preparation and the extraction in separate scripts (since one usually prepares once and then tries several extractions with different settings).		Note that it is usually better to have the data preparation and the extraction in separate scripts (since one usually prepares once and then tries several extractions with different settings).

	The xmmscripts need the region files to be fits files. You can convert them via		The xmmscripts need the region files to be fits files. You can convert them via
	ds9tocxc outset=outfilename.fits < infilename.reg		ds9tocxc outset=outfilename.fits < infilename.reg

			For simple extractions of point sources creating region files is not necessary, you can just give the position of the source and background region as well as their extraction radii with the "--ra", "--dec" and "--bkgra", "--bkgdec" arguments.


			"xmmextract" is a very powerful script, use its "--help" argument to get a list of all arguments.

	==== Workaround for the xmmscripts ====		==== Workaround for the xmmscripts ====
Line 39:		Line 44:

	#!/bin/csh		#!/bin/csh
	~~setenv XMMTOOLS /home/wilms/xmmscripts~~
	~~source ~wilms/progmapper15/progmapper.csh~~
	${XMMTOOLS}/xmmprepare --datadir=/eu/X-ray/XMM/data/2290/0679780201/odf/		${XMMTOOLS}/xmmprepare --datadir=/eu/X-ray/XMM/data/2290/0679780201/odf/
	--prepdir=./test --pn --analysisdate=2014-01-11T01:05:00.000		--prepdir=./test --pn --analysisdate=2014-01-11T01:05:00.000


	== XMM OM Data Extraction ==		== XMM OM Data Extraction ==

Knies at 13:21, 1 August 2018

2018-08-01T13:21:18Z

@@ Line 1: / Line 1: @@
-====== XMM Data Extraction ======
+Always use the remeis archives at /eu/X-ray/XMM/data/ instead of downloading observation files yourself. You can search for the data of an observation by typing something like '''find /eu/X-ray/xmm/data/????/obsID/odf -type d''' - replacing obsID with your observation ID.
 After the system got updated from Ubuntu 11 to Ubuntu 12 a problem occured with the function "dssetattr" inside the "epchain".
@@ Line 41: / Line 45: @@
-====== XMM OM Data Extraction ======
+== XMM OM Data Extraction ==
 For extracting the data of the Optical Monitor (OM) onboard XMM, there exist three tools which do this automatically: ''omichain'', ''omfchain'' and ''omgchain''.
@@ Line 81: / Line 85: @@
-====== Choosing correct XMM-EPIC filter ======
+== Choosing correct XMM-EPIC filter ==
 Choosing the correct EPIC filter for your observation depends on the
@@ Line 132: / Line 136: @@
 from the center, so one cannot use the thin epic filter; one must use
 the MEDIUM filter for all three EPIC cameras.
 [[Category:Data Extraction]]

Ballhausen at 13:14, 11 April 2018

2018-04-11T13:14:43Z

← Older revision		Revision as of 15:14, 11 April 2018
Line 5:		Line 5:
	This function can set/add an attribute to the HDU (Header/Data Unit).		This function can set/add an attribute to the HDU (Header/Data Unit).
	The solution is to call		The solution is to call
	'source $SOFTDIR/sas_init.csh'		''''source $SOFTDIR/sas_init.csh''''
	at the beginning of each script in order to initialize SAS inside your script.		at the beginning of each script in order to initialize SAS inside your script.

Line 29:		Line 29:
	There were some problems occuring with the XMM-data extraction. Joern has implemented a workaround in the xmmscripts that has to be called as shown in the example below.		There were some problems occuring with the XMM-data extraction. Joern has implemented a workaround in the xmmscripts that has to be called as shown in the example below.
	Here the issues:		Here the issues:
	- If your observation includes sub-observations, choose one of them. Do this with "onlyexp=1" not "onlyexp=PNU001" for example.		# If your observation includes sub-observations, choose one of them. Do this with "onlyexp=1" not "onlyexp=PNU001" for example.
	- Since our Calibration database gets updated frequently, we ran into the problem that new calibration files only like the latest SAS. As it is very problematic to just install the new SAS, there is a workaround. You have to address the calibration files that are working with our SAS version until the new SAS is installed (we must be patient as this is not trivial). The following, however, works perfectly:		# Since our Calibration database gets updated frequently, we ran into the problem that new calibration files only like the latest SAS. As it is very problematic to just install the new SAS, there is a workaround. You have to address the calibration files that are working with our SAS version until the new SAS is installed (we must be patient as this is not trivial). The following, however, works perfectly:

	example:		example:
Line 44:		Line 44:

	For extracting the data of the Optical Monitor (OM) onboard XMM, there exist three tools which do this automatically: ''omichain'', ''omfchain'' and ''omgchain''.		For extracting the data of the Optical Monitor (OM) onboard XMM, there exist three tools which do this automatically: ''omichain'', ''omfchain'' and ''omgchain''.
	"i" stands for image mode, "f" for fast mode and "g" for grism mode. Which mode you have can be found on the XMM log browse site: [[http://xmm2.esac.esa.int/external/xmm_obs_info/obs_view_frame.shtml]].		"i" stands for image mode, "f" for fast mode and "g" for grism mode. Which mode you have can be found on the XMM log browse site: http://xmm2.esac.esa.int/external/xmm_obs_info/obs_view_frame.shtml.
	If you want to extract your images you first have to set some paths:		If you want to extract your images you first have to set some paths:

Line 58:		Line 58:
	$omichain #or omfchain, omgchain		$omichain #or omfchain, omgchain

	You can also do the extraction step-by-step. For a description see [[http://xmm.esac.esa.int/sas/current/documentation/threads/omi_stepbystep.shtml]].		You can also do the extraction step-by-step. For a description see http://xmm.esac.esa.int/sas/current/documentation/threads/omi_stepbystep.shtml.
	At the end you should have several images (''IMAGE''), region files (''REGION'') and source lists (''SWSRLI'') for the OM exposures and filters. The nomenclatur for the files are:		At the end you should have several images (<code>IMAGE</code>), region files (<code>REGION</code>) and source lists (<code>SWSRLI</code>) for the OM exposures and filters. The nomenclatur for the files are:

	* IOOOOOOOOOODDUEEETTTTTTSXXX.FFF or POOOOOOOOOODDUEEETTTTTTSXXX.FFF (for PPS product files)		* IOOOOOOOOOODDUEEETTTTTTSXXX.FFF or POOOOOOOOOODDUEEETTTTTTSXXX.FFF (for PPS product files)
	* OOOOOOOOOO = XMM-ObsID (10 characters)		* OOOOOOOOOO = XMM-ObsID (10 characters)
	* DD = Data source idetifier (here OM)		* DD = Data source idetifier (here OM)
	* U = exposure flag (S = sched, U = unsched, X = not aplicable)		* U = exposure flag (S = sched, U = unsched, X = not aplicable)
	* EEE = exposure number within the instrument observation (3 characters)		* EEE = exposure number within the instrument observation (3 characters)
	* TTTTTT = product type (6 characters)		* TTTTTT = product type (6 characters)
	* S = 0 or data subset number/character		* S = 0 or data subset number/character
	* XXX = source number or slew step number (3 characters)		* XXX = source number or slew step number (3 characters)
	* FFF = file format (3 characters)		* FFF = file format (3 characters)

	From all exposures for one filter a mosaiced sky image is produced which contains ''*SIMAG?.FIT''. The ? stands for the filter: L = UVW1, M = UVM2, S = UVW2, B = B, U = U, V = V.		From all exposures for one filter a mosaiced sky image is produced which contains *SIMAG?.FIT. The ? stands for the filter: L = UVW1, M = UVM2, S = UVW2, B = B, U = U, V = V.
	Open the image with ds9 and have a look if your source is seen and where it is. You can check this by loading a region file, produced e.g. by simbad2ds9, into the image.		Open the image with ds9 and have a look if your source is seen and where it is. You can check this by loading a region file, produced e.g. by simbad2ds9, into the image.

	Then check the detector images (''PIMAGE_000.FIT'') for your source (it can be, that some detector images don't contain your source because the window was off). Load the region file which belongs to the detector image (''REGION000.ASC'', e.g. ''P0090050801OMS006IMAGE_0000.FIT'' -> ''I0090050801OMS006REGION0000.ASC'') and identify the number of your source.		Then check the detector images (<code>PIMAGE_000.FIT</code>) for your source (it can be, that some detector images don't contain your source because the window was off). Load the region file which belongs to the detector image (<code>REGION000.ASC</code>, e.g. <code>P0090050801OMS006IMAGE_0000.FIT</code> -> ''I0090050801OMS006REGION0000.ASC</code>) and identify the number of your source.

	~~{{:satext~~:image_region.jpeg~~?200~~\|}}		[[File:image_region.jpeg\|thumb]]

	Look into the source list for your exposure (''PSWSRLI.FIT'') and note the SRC_ID in the last column. Open the combined source list (''COMBO'') and search for your SRC_ID. Here you can find all the magnitudes and fluxes of your source for the different filters.		Look into the source list for your exposure (<code>PSWSRLI.FIT</code>) and note the SRC_ID in the last column. Open the combined source list (<code>COMBO</code>) and search for your SRC_ID. Here you can find all the magnitudes and fluxes of your source for the different filters.


Line 119:		Line 119:
	by magnitude, e.g.,		by magnitude, e.g.,

	cat GS1826USNO \| grep -v '#' \| awk '{print $7 " " $12}' \| sort -g > starlist_sortB		cat GS1826USNO \| grep -v '#' \| awk '{print $7 " " $12}' \| sort -g > starlist_sortB
			cat GS1826USNO \| grep -v '#' \| awk '{print $8 " " $12}' \| sort -g > starlist_sortR
	cat GS1826USNO \| grep -v '#' \| awk '{print $8 " " $12}' \| sort -g > starlist_sortR

	since the B and R magnitudes are columns 7 and 8, and distance from the center		since the B and R magnitudes are columns 7 and 8, and distance from the center