How to use the Remeis cluster - Revision history

Weber at 14:17, 14 April 2025

2025-04-14T14:17:41Z

← Older revision		Revision as of 16:17, 14 April 2025
Line 5:		Line 5:
	=== Performing Computations ===		=== Performing Computations ===

	* ~~The~~ servers ~~''crux'', ''grus'',~~ and ~~''phoenix'' should~~ not ~~be used for~~ computations ~~at all~~.		* You should never log in on servers directly and especially not run any computations on servers without using slurm.
	* Computations and extractions on the Cluster should be submitted via '''slurm'''. Information on how to use it can be found [[Slurm\|here]].		* Computations and extractions on the Cluster should be submitted via '''slurm'''. Information on how to use it can be found [[Slurm\|here]].
	* Also interactive sessions can be started via '''slurm'''.		* Also interactive sessions can be started via '''slurm'''.
Line 13:		Line 13:
	* '''/userdata:''' Extracted data like images or spectra or other space consuming stuff or stuff which can re recreated/downloaded again should be put there. /userdata is a RAID6 system, i.e. two disks can fail simultaneously w/o data loss --> your data is safe on /userdata! However, there is NO backup of userdata, as /userdata is too big to be backed up simply. And there is no quota on /userdata.		* '''/userdata:''' Extracted data like images or spectra or other space consuming stuff or stuff which can re recreated/downloaded again should be put there. /userdata is a RAID6 system, i.e. two disks can fail simultaneously w/o data loss --> your data is safe on /userdata! However, there is NO backup of userdata, as /userdata is too big to be backed up simply. And there is no quota on /userdata.

	* '''/home:''' This is the place for self-developed software, scripts, documents, TeX files, diploma/phd/... theses, papers, spreadsheets, etc. /home is also a RAID system, i.e. more than one disk may fail w/o data loss. Furthermore, /home is backed up every night to TWO different disks (in two machines) AND once a week on an external disk that is stored outside the observatory for the worst case (e.g., fire). It therefore does not make sense to create your own private backups on local scratch disks etc. /home, however, is quite small (1 TB for all the institute); we therefore have a quota of about 10GB per user. It is no problem to get more space, if you can't move stuff around anymore, just drop an Email to the admins. On the other hand, you don't have to fill your quota completely! To check your quota you can use the ''quota'' command on ~~crux~~, like		* '''/home:''' This is the place for self-developed software, scripts, documents, TeX files, diploma/phd/... theses, papers, spreadsheets, etc. /home is also a RAID system, i.e. more than one disk may fail w/o data loss. Furthermore, /home is backed up every night to TWO different disks (in two machines) AND once a week on an external disk that is stored outside the observatory for the worst case (e.g., fire). It therefore does not make sense to create your own private backups on local scratch disks etc. /home, however, is quite small (5 TB for all the institute); we therefore have a quota of about 10GB per user. It is no problem to get more space, if you can't move stuff around anymore, just drop an Email to the admins. On the other hand, you don't have to fill your quota completely! To check your quota you can use the ''quota'' command on libra, like
	ssh ~~crux~~ quota -s -f /~~home~~		ssh libra quota -s -f /exports/disk1

	* '''/scratch:''' For temporary or otherwise not very important data. You should not store anything important on /scratch!! The scratch disks are NOT backedup and the scratch disks are not fail safe, i.e. if a scratch disk is broken (which happens regularly), your data is GONE.		* '''/scratch:''' For temporary or otherwise not very important data. You should not store anything important on /scratch!! The scratch disks are NOT backedup and the scratch disks are not fail safe, i.e. if a scratch disk is broken (which happens regularly), your data is GONE. scratch might be emptied at any time without warning.

	* '''Satellite Data:''' We already have archives for most satellite raw data (applies mostly to the X-ray people). So please do not download any raw data yourself, but talk to ~~the person in charge of a specific mission. You can find out who is responsible by executing <code>showSatelliteContacts</code> in the shell~~.		* '''Satellite Data:''' We already have archives for most satellite raw data (applies mostly to the X-ray people). So please do not download any raw data yourself, but talk to Ingo.

	=== Logging into Remeis ===		=== Logging into Remeis ===
Line 29:		Line 29:


	===== Using ~~the NX-Client~~ =====		===== Using x2go =====

	~~If you need~~ a ~~graphics~~ interface~~, it might be worth to~~ use ~~the <code>nxclient</code> to connect to the Remeis Cluster. The corresponding <code>nxserver</code> is installed on (almost) all machines. In order to use it you have to~~ [~~http://www.nomachine.com/download.php download~~] it and follow the Connection-Wizard to connect on a host. Always use the same machine, as only a limited number of total connections is allowed on each machine. This works for Linux, Mac OSX, and even Windows.		For remote login with a graphical interface you can use [[X2Go Remote Desktop]]

	===== Hosts =====		===== Hosts =====
	Host names are generally the constellation names, e.g., <code>norma</code> or <code>~~leo~~</code>~~. All available hosts can be found in the shell variable <code>$RemeisHosts</code>. The servers (<code>crux</code>, <code>grus</code>, and <code>phoenix</code>) should not be used~~.		Host names are generally the constellation names, e.g., <code>norma</code> or <code>mensa</code>.

	==== Connecting without password and short hostnames ====		==== Connecting without password and short hostnames ====
Line 51:		Line 51:
	In order to define a short-cut for a host, you have to add the following information to the ''~/.ssh/config''		In order to define a short-cut for a host, you have to add the following information to the ''~/.ssh/config''
	# the short-name you want to call the host (most useful is the general name of the machine		# the short-name you want to call the host (most useful is the general name of the machine
	Host=~~pulsar~~		Host=norma
	# now define user and host such that ''<User>@<Hostname>'' is the correct adress		# now define user and host such that ''<User>@<Hostname>'' is the correct adress
	Hostname=~~pulsar~~.sternwarte.uni-erlangen.de		Hostname=norma.sternwarte.uni-erlangen.de
	User=YourUsername		User=YourUsername

	===== ssh-keys =====		===== ssh-keys =====

	If you setup the ssh-keys properly, you can login via ssh without entering a password~~. This isn't only nice to have, but if~~ you ~~use [[torque:start\|Torque]], this is mandatory~~. A detailed description can be found [[SSH-Keys\|here]].		If you setup the ssh-keys properly, you can login via ssh without entering a password (although you should definitely protect your ssh key with a password). A detailed description can be found [[SSH-Keys\|here]].

	=== Setting the Default Options/Software ===		=== Setting the Default Options/Software ===
Line 68:		Line 68:

	# activate the X-ray software (HEADAS, satellite extraction scripts, ...)		# activate the X-ray software (HEADAS, satellite extraction scripts, ...)
	source ~~$SOFTDIR~~/~~softwarescript_Xray~~.~~csh~~		source /data/software/cshrc.in

Vashukla: /* The Data-Storage System */

2023-06-23T20:46:56Z

The Data-Storage System

← Older revision		Revision as of 22:46, 23 June 2023
Line 11:		Line 11:
	=== The Data-Storage System ===		=== The Data-Storage System ===

	* '''/userdata:''' Extracted data like images or spectra or other space consuming stuff or stuff which can re recreated/downloaded again should be put there. /userdata is a RAID6 system, i.e. two disks can fail simultaneously w/o data loss --> your data is safe on /userdata! However, there is NO backup of userdata, as /userdata is too big to be ~~backuped~~ simply. And there is no quota on /userdata.		* '''/userdata:''' Extracted data like images or spectra or other space consuming stuff or stuff which can re recreated/downloaded again should be put there. /userdata is a RAID6 system, i.e. two disks can fail simultaneously w/o data loss --> your data is safe on /userdata! However, there is NO backup of userdata, as /userdata is too big to be backed up simply. And there is no quota on /userdata.

	* '''/home:''' This is the place for self-~~developped~~ software, scripts, documents, TeX files, diploma/phd/... theses, papers, spreadsheets, etc. /home is also a RAID system, i.e. more than one disk may fail w/o data loss. Furthermore, /home is ~~backedup~~ every night to TWO different disks (in two machines) AND once a week on an external disk that is stored outside the observatory for the worst case (fire ~~etc~~). It therefore does not make sense to create your own private backups on local scratch disks etc. /home, however, is quite small (1 TB for all the institute); we therefore have a quota of about 10GB per user. It is no problem to get more space, if you can't move stuff around anymore, just drop an Email to the admins. On the other hand, you don't have to fill your quota completely! To check your quota you can use the ''quota'' command on crux, like		* '''/home:''' This is the place for self-developed software, scripts, documents, TeX files, diploma/phd/... theses, papers, spreadsheets, etc. /home is also a RAID system, i.e. more than one disk may fail w/o data loss. Furthermore, /home is backed up every night to TWO different disks (in two machines) AND once a week on an external disk that is stored outside the observatory for the worst case (e.g., fire). It therefore does not make sense to create your own private backups on local scratch disks etc. /home, however, is quite small (1 TB for all the institute); we therefore have a quota of about 10GB per user. It is no problem to get more space, if you can't move stuff around anymore, just drop an Email to the admins. On the other hand, you don't have to fill your quota completely! To check your quota you can use the ''quota'' command on crux, like
	ssh crux quota -s -f /home		ssh crux quota -s -f /home

Line 19:		Line 19:

	* '''Satellite Data:''' We already have archives for most satellite raw data (applies mostly to the X-ray people). So please do not download any raw data yourself, but talk to the person in charge of a specific mission. You can find out who is responsible by executing <code>showSatelliteContacts</code> in the shell.		* '''Satellite Data:''' We already have archives for most satellite raw data (applies mostly to the X-ray people). So please do not download any raw data yourself, but talk to the person in charge of a specific mission. You can find out who is responsible by executing <code>showSatelliteContacts</code> in the shell.




	=== Logging into Remeis ===		=== Logging into Remeis ===

Zainab: /* How To use the Remeis Cluster */

2021-09-16T12:59:57Z

How To use the Remeis Cluster

← Older revision		Revision as of 14:59, 16 September 2021
Line 1:		Line 1:
	== How To use the Remeis Cluster ==		== How To use the Remeis Cluster ==

	The Cluster of the Remeis ~~obsveratory~~ is designed to make all ~~ressources~~ available for easy usage. Nevertheless, there are a few important things one has to know in order to use the cluster properly. In the following the most important information will be listed. Other questions or problems with the system or software sould be directed to [mailto:admin@sternwarte.uni-erlangen.de admin@sternwarte.uni-erlangen.de].		The Cluster of the Remeis observatory is designed to make all resources available for easy usage. Nevertheless, there are a few important things one has to know in order to use the cluster properly. In the following the most important information will be listed. Other questions or problems with the system or software sould be directed to [mailto:admin@sternwarte.uni-erlangen.de admin@sternwarte.uni-erlangen.de].

	=== Performing Computations ===		=== Performing Computations ===

Dauser at 08:48, 20 February 2019

2019-02-20T08:48:33Z

← Older revision		Revision as of 10:48, 20 February 2019
Line 6:		Line 6:

	* The servers ''crux'', ''grus'', and ''phoenix'' should not be used for computations at all.		* The servers ''crux'', ''grus'', and ''phoenix'' should not be used for computations at all.
	* Computations and extractions on the Cluster should be submitted via '''~~torque~~'''. Information on how to use it can be found [[~~Torque~~\|here]].		* Computations and extractions on the Cluster should be submitted via '''slurm'''. Information on how to use it can be found [[Slurm\|here]].
	* Also ~~[[Torque#Interactive_Jobs\|~~interactive sessions]] can be started via '''~~torque~~'''.		* Also interactive sessions can be started via '''slurm'''.

	=== The Data-Storage System ===		=== The Data-Storage System ===

Obst at 08:22, 20 February 2019

2019-02-20T08:22:27Z

← Older revision		Revision as of 10:22, 20 February 2019
Line 76:		Line 76:
	More information on the TC shell and how to configure it can be found [[TcShell\|here]].		More information on the TC shell and how to configure it can be found [[TcShell\|here]].

	[[Category:Current Members]]		[[Category:Current Members]][[Category:Software]]

Dauser: /* How To use the Remeis Cluster */

2019-02-19T12:06:00Z

How To use the Remeis Cluster

← Older revision		Revision as of 14:06, 19 February 2019
Line 1:		Line 1:
	== How To use the Remeis Cluster ==		== How To use the Remeis Cluster ==

	The Cluster of the Remeis obsveratory is designed to make all ressources available for easy usage. Nevertheless, there are a few important things one has to know in order to use the cluster properly. In the following the most important information will be listed. Other questions or problems with the system or software sould be directed to [mailto:~~admins~~@sternwarte.uni-erlangen.de ~~admins~~@sternwarte.uni-erlangen.de].		The Cluster of the Remeis obsveratory is designed to make all ressources available for easy usage. Nevertheless, there are a few important things one has to know in order to use the cluster properly. In the following the most important information will be listed. Other questions or problems with the system or software sould be directed to [mailto:admin@sternwarte.uni-erlangen.de admin@sternwarte.uni-erlangen.de].

	=== Performing Computations ===		=== Performing Computations ===

Thalhammer: Fixed link

2018-06-21T11:01:55Z

Fixed link

← Older revision		Revision as of 13:01, 21 June 2018
Line 74:		Line 74:


	More information on the TC shell and how to configure it can be found [[~~csh:start~~\|here]].		More information on the TC shell and how to configure it can be found [[TcShell\|here]].

	[[Category:Current Members]]		[[Category:Current Members]]

Thalhammer: Fixed link

2018-06-21T11:00:11Z

Fixed link

← Older revision		Revision as of 13:00, 21 June 2018
Line 6:		Line 6:

	* The servers ''crux'', ''grus'', and ''phoenix'' should not be used for computations at all.		* The servers ''crux'', ''grus'', and ''phoenix'' should not be used for computations at all.
	* Computations and extractions on the Cluster should be submitted via '''torque'''. Information on how to use it can be found [[~~torque:start~~\|here]].		* Computations and extractions on the Cluster should be submitted via '''torque'''. Information on how to use it can be found [[Torque\|here]].
	* Also [[Torque#Interactive_Jobs\|interactive sessions]] can be started via '''torque'''.		* Also [[Torque#Interactive_Jobs\|interactive sessions]] can be started via '''torque'''.

Thalhammer: Fixed link

2018-06-21T10:42:04Z

Fixed link

← Older revision		Revision as of 12:42, 21 June 2018
Line 7:		Line 7:
	* The servers ''crux'', ''grus'', and ''phoenix'' should not be used for computations at all.		* The servers ''crux'', ''grus'', and ''phoenix'' should not be used for computations at all.
	* Computations and extractions on the Cluster should be submitted via '''torque'''. Information on how to use it can be found [[torque:start\|here]].		* Computations and extractions on the Cluster should be submitted via '''torque'''. Information on how to use it can be found [[torque:start\|here]].
	* Also [[~~torque:start~~#~~interactive_jobs~~\|interactive sessions]] can be started via '''torque'''.		* Also [[Torque#Interactive_Jobs\|interactive sessions]] can be started via '''torque'''.

	=== The Data-Storage System ===		=== The Data-Storage System ===

Lorenz: /* ssh-keys */

2018-04-12T10:01:58Z

ssh-keys

← Older revision		Revision as of 12:01, 12 April 2018
Line 61:		Line 61:
	===== ssh-keys =====		===== ssh-keys =====

	If you setup the ssh-keys properly, you can login via ssh without entering a password. This isn't only nice to have, but if you use [[torque:start\|Torque]], this is mandatory. A detailed description can be found [[~~ssh~~-~~keys:start~~\|here]]		If you setup the ssh-keys properly, you can login via ssh without entering a password. This isn't only nice to have, but if you use [[torque:start\|Torque]], this is mandatory. A detailed description can be found [[SSH-Keys\|here]].

	=== Setting the Default Options/Software ===		=== Setting the Default Options/Software ===